How To Access Code In Large w/ Vim

HOW TO ACCESS THE CODE IN LARGE CODEBASE WITH VIMEdgar Wang & Howard Chung & Ferry Chen Titan Camera Team, 2014-2015

MILLION LINES OF CODE!!

External camera ISP use non-standard solution provided by QCOM

non-streaming output for still image capturing

External ISP wrapped as YUV sensor

wrapped YUV sensor not in QCOM expectations

YUV sensor with META!!!

REALLY?


external ISP not provide CCI compatible commands

external ISP use SPI to download the firmware, SPI is not a part of camera V4L2 driver framework..

DOUBLE WHAMMY !!


many features requested need the changes cross all layers from APP to ISP, and the non-standard solution here make

THE WORST…

QCOM CANNOT support us with its PREPARATORY code…


HOW MUCH?


PATH LOCFRAMEWORKS/BASE/CORE/JAVA/ANDROID/HARDWARE/CAMERA.JAVA 1897

FRAMEWORK/BASE/CORE/JAVA/ANDROID/HARDWARE/CAMERA2/* 15099FRAMEWORKS/BASE/CORE/JNI/ANDROID_HARDWARE_CAMERA.CPP 1021FRAMEWORKS/BASE/CORE/JNI/ANDROID_HARDWARE_CAMERA2_* 2923

FRAMEWORKS/AV/SERVICES/CAMERA/LIBCAMERASERVICE/* 19940FRAMEWORKS/AV/CAMERA/* 5492

HARDWARE/QCOM/CAMERA/* 103386VENDOR/QCOM/PROPRIETARY/MM-CAMERA/* 816521

KERNEL/DRIVERS/MEDIA/PLATFORM/MSM/CAMERA_V2/* 51954KERNEL/ARCH/ARM64/BOOT/DTS/QCOM/MSM8994-CAMERA.DTSI 517

KERNEL/ARCH/ARM64/BOOT/DTS/QCOM/MSM8994-TITAN-CAMERA.DTSI 0SUM 1018750

1,000,000+ Lines of Code(exclude the comments)


PATH #OFCHGFRAMEWORKS/BASE/CORE/JAVA/ANDROID/HARDWARE/CAMERA.JAVA 114

FRAMEWORK/BASE/CORE/JAVA/ANDROID/HARDWARE/CAMERA2/* 1544FRAMEWORKS/BASE/CORE/JNI/ANDROID_HARDWARE_CAMERA.CPP 4FRAMEWORKS/BASE/CORE/JNI/ANDROID_HARDWARE_CAMERA2_* 153

FRAMEWORKS/AV/SERVICES/CAMERA/LIBCAMERASERVICE/* 329FRAMEWORKS/AV/CAMERA/* 0

HARDWARE/QCOM/CAMERA/* 11059VENDOR/QCOM/PROPRIETARY/MM-CAMERA/* 22120

KERNEL/DRIVERS/MEDIA/PLATFORM/MSM/CAMERA_V2/* 7271KERNEL/ARCH/ARM64/BOOT/DTS/QCOM/MSM8994-CAMERA.DTSI 4

KERNEL/ARCH/ARM64/BOOT/DTS/QCOM/MSM8994-TITAN-CAMERA.DTSI 110SUM 42708

4.2%,42078 line changed by Camera Team

CHALLENGES

Code Tracing

Features Implementation

Baseline Porting

Human Communication

Change submission

CHALLENGES - DETAILS

Call Path too long to trace

Feature’s implementation not only in 1 or 2 projects only

Porting - be the a real F! thing.

Hardly to talk with other guys in code level.

High missing rate with submission

Slow disk search

model complexities

etc……

DISLIKEDExhausted code searching, this take lots of time to do it and degrade our performance!

e.g. grep, find, egrep…etc

Unnecessary directly switchings, redundant works and easy to lost in codebase

e.g. cd ../../../../../ && cd <path_to_hal>

No links between caller/callee for constants and functions

This will be a nightmare when tracing the call with tens of hops

Dupe and repeated works

This real be a hated one when you do thousands of changes!!

Slow deployment process

Flashing system takes 6 mins; push tons of libraries also take tens of seconds and error-prone.

Code editor only support one PL well

We are in Java/C++/C mixed codebase.

WANTEDEasy, Simple & one hop switching between working folders & commands

Know the position of any symbol in a second

Trivial way to trace symbols with their caller and callee

Tabbed editor with FS navigator and list of symbols

The editor can support queries on symbols with tabs

The editor will be nice if it has bookmark abilities

THE START POINT…

Alfred Huang (ajch), early camera BSP Owner

Switching directories really a bullshit!

History commands not straightforward to use

Vi is good to edit, but not good to do code tracing

THE START POINT…

We must (by ajch)

more powerful cd command

reuse and recall command history with ease

enable Vi have full capability to do code tracing

ESSENTIAL COMMANDS

ecd - extended cd

the memo list of working directories

support quick switch between working directories

ease of list modification

sorting on memo list

ESSENTIAL COMMANDS

Usage

For any working directory, using following command to push it into list

Whenever your want to jump a recorded directory, get the idx for jumping first

And you can jump to the directory by

~ $ ecd .

~ $ ecd -l

~ $ ecd -<Number>

ESSENTIAL COMMANDS

~ $ ecd -l 0) /home/cm/workspaces/Edgar/V06_1260/LINUX/android 1) /home/cm/workspaces/Edgar/V06_1260/LINUX/android/kernel/drivers/media/platform/msm/camera_v2 2) /home/cm/workspaces/Edgar/V06_1260/LINUX/android/vendor/qcom/proprietary/mm-camera-titan 3) /home/cm/workspaces/Edgar/V06_1260/LINUX/android/hardware/qcom/camera_titan 4) /home/cm/workspaces/Edgar/V06_1260/LINUX/android/vendor/qcom/proprietary/mm-still 5) /home/cm/workspaces/Edgar/V06_1260_55A/LINUX/android 6) /home/cm/workspaces/Edgar/V06_1260_55A/LINUX/android/hardware/qcom/camera_titan 7) /home/cm/workspaces/Edgar/V06_1260_55A/LINUX/android/vendor/qcom/proprietary

~ $ ecd -7

ESSENTIAL COMMANDS

hist - sort out cached history in shell and .bash_history

provide a visible view with command history in both memory and .bash_history

not broken the bash’s command history but extend it

compatible to the exclamation mark (!) in bash

ESSENTIAL COMMANDS

Usage

For any working directory, using following command to push it into list

just pass the number after exclamation mark

~ $ hist|grep <keyword>

~ $ !<Number>

ESSENTIAL COMMANDS

~ $ hist|grep fastboot 220) adb shell logcat -v threadtime > ./log.txt 314) adb reboot bootloader && fastboot -i 0x27F1 flash boot /Volumes/Sanctuary/Ferry/V06_1260/LINUX/android/out/target/product/v06/boot.img && fastboot -i 0x27F1 flash system /Volumes/Sanctuary/Ferry/V06_1260/LINUX/android/out/target/product/v06/system.img && xfastflash reboot 317) adb shell setprop log.tag.ImageReaderTest VERBOSE 320) adb pull /storage/emulated/0/3840x2160_full_jpeg.jpg 321) adb pull /storage/emulated/0/3840x2160_full_yuv.jpg 368) adb shell logcat -v threadtime|grep "left ="

~ $ !314







GTAGS - GNU GLOBAL source code tagging system

support 6 PLs by default, 25 PLs with Exuberant Ctags plug-in parser. Java/C/C++ are default ones.

Support shell command line, Vi, Emacs

Performance!

support text pattern

…etc

https://www.gnu.org/software/global/ for details…

GET SYMBOLS IN A SECOND

https://www.gnu.org/software/global/


Preparation

Download

Install gtags

make and install

~ $ wget http://tamacom.com/global/global-6.5.1.tar.gz

~ $ tar zxvf global-6.5.1.tar.gz && cd global-6.5.1

~ $ ./configure && make && mkaeinstall

http://tamacom.com/global/global-6.5.1.tar.gz

GET SYMBOLS IN A SECONDUse gtags for Android codebase

switch to Android source tree

generate list of file to be indexed (optional)

android_src $ find . $ ! -path "./out*" -a ! -name ".git*" $ -a $ -name "*.cpp" -o -name "*.h" -o -name "*.c" -o -name "*.aidl" -o -name "*.java" -o -name "*.S" $ > somefile

~ user$ cd <Android_source_tree>


Index tags for Android codebase

Index the source tree

Update the source tree (when needed)

android_src $ gtags -f somefile

android_src $ global -uv


Use gtags for indexed Android codebase

query a symbol

android_src $ global PowerManagerService frameworks/base/services/core/java/com/android/server/power/PowerManagerService.java



query a symbol, have more info

android_src $ global -x PowerManagerService PowerManagerService 102 ../../../../../frameworks/base/services/core/java/com/android/server/power/PowerManagerService.java public final class PowerManagerService extends SystemService



query WHO use a symbol

android_src $ global -r PowerManagerService frameworks/base/services/core/java/com/android/server/power/PowerManagerService.java frameworks/base/services/core/java/com/android/server/power/ShutdownThread.java frameworks/base/services/java/com/android/server/SystemServer.java



query WHO use a symbol, with more info

android_src $ global -rx PowerManagerService PowerManagerService 2998 frameworks/base/services/core/java/com/android/server/power/PowerManagerService.java PowerManagerService.this.handleWakeLockDeath(this); PowerManagerService 53 frameworks/base/services/core/java/com/android/server/power/ShutdownThread.java import com.android.server.power.PowerManagerService; PowerManagerService 622 frameworks/base/services/core/java/com/android/server/power/ShutdownThread.java PowerManagerService.lowLevelReboot(reason); PowerManagerService 644 frameworks/base/services/core/java/com/android/server/power/ShutdownThread.java PowerManagerService.lowLevelShutdown(); …



query symbol support patterns (regular expression!)

android_src $ global PowerManager* cts/tests/tests/os/src/android/os/cts/PowerManagerTest.java cts/tests/tests/os/src/android/os/cts/PowerManager_WakeLockTest.java …

Caution! This takes minutes! But still much faster than search full codebase..







Vim/Vi - native supports on tabs

NerdTree - explore filesystem and to open files and directories for Vi/Vim

Tagbar - list symbols of current file for Vi/Vim

This depends on Exuberant ctags

TABBED EDITOR WITH FS NAVIGATOR AND LIST OF SYMBOLS

android_src $ sudo apt-get install exuberant-ctags

Linux

android_src $ brew install exuberant-ctags

Mac







GTAGS - GNU GLOBAL source code tagging system

Although gtags support much editors, but it is not good enough!

Only jump to symbol(callee) supported by gtags…

TRIVIAL WAY TO TRACE SYMBOLS


ajch’s gtags.vim - Alfred’s gtags plugin form Vi/Vim

we’re all vimers!!

support get the list callers of symbols!

the last pulse of code tracing!



Having list of caller and callee is not enough

We must have next page/prev page function too.

ajch’s mym.vim - stack behaviours of the gtags operations on Vi/Vim

Hooray! we’re all vimers!!

support next page/prev page operations

makes Vi/Vim be a browser like editor!

Alfred’s plugins are integrated to our VimSuite - later on..








QFEnter - A vim plugin for intuitive file opening from Quickfix window to tabs

THE EDITOR CAN SUPPORT QUERIES ON SYMBOLS WITH TABS

vim-bookmarks - A vim plugin for bookmark

support line based bookmark

support bookmark annotations

ease of use

integrated to quick fix window

…

THE EDITOR WILL BE NICE IF IT HAS BOOKMARK ABILITIES

EHFANDROIDVIMSUITE

F2 - open/close/update bookmark

F7

F8F5 - next tabF6 - prev tab

t

EHFANDROIDVIMSUITE

F3 - list of definesF4 - list of references







It is not enoughIt can be better

WANTED IIRandom access to codebase

Not only know the position of symbol, but put editor to the line directly

Dupe and Repeated works can be maximum eliminated

Deployment have quick, accurate, and easy solution

A secure and safe way to backup/save our works with trials

More & more…

xvim - put you to the symbol with Vi/Vim directly

based on gtags.

ease of use

list all candidates to choose first

great partner to debug with logs!!!!

RANDOM ACCESS TO CODEBASE

RANDOM ACCESS TO CODEBASE10-09 08:46:54.239 7427 7427 E mm-camera: mct: mct_pipeline_stop_session - E 10-09 08:46:54.240 7427 7427 E mm-camera: mct: mct_pipeline_process_set:command=800000b 10-09 08:46:54.240 7427 7427 E mm-camera: mct: mct_pipeline_process_set: Issuing DEL_STREAM on stream 0x1000f and stream type=10 10-09 08:46:54.240 7427 7427 I mm-camera-sensor: port_sensor_unlink_func:293ide 1000f 10-09 08:46:54.240 7427 7427 E mm-camera: port_iface_ext_unlink_func: E, identity = 0x1000f, port = 0xf4878e20, direction = 2 10-09 08:46:54.240 7427 7427 I mm-camera-sensor: port_sensor_caps_unreserve:204ide 1000f 10-09 08:46:54.240 7427 7427 E mm-camera: port_iface_check_caps_unreserve_func: E, identity = 0x1000f, port = 0xf4878e20, direction = 2 10-09 08:46:54.240 7427 7427 E mm-camera: iface_unreserve_sink_port: stream_id f mode 0 10-09 08:46:54.240 7427 7427 E mm-camera: port_iface_ext_unlink_func: E, identity = 0x1000f, port = 0xf4bbc160, direction = 1 10-09 08:46:54.240 7427 7427 E mm-camera: iface_ext_unlink_src_port: mct stream already removed from hw stream


java.util.ConcurrentModificationException at java.util.HashMap$HashIterator.nextEntry(HashMap.java:787) at java.util.HashMap$EntryIterator.next(HashMap.java:824) at java.util.HashMap$EntryIterator.next(HashMap.java:822) at com.android.internal.telephony.SubscriptionController.getPhoneId(SubscriptionController.java:1199) at com.android.phone.PhoneInterfaceManager.getPhone(PhoneInterfaceManager.java:789) at com.android.phone.PhoneInterfaceManager.getIccId(PhoneInterfaceManager.java:2161) at com.android.phone.PhoneInterfaceManager.getLine1NumberForDisplay(PhoneInterfaceManager.java:2205)

…

drop you to the symbol directly!


android_src $ xvim <symbol_name>

android_src $ xvim port_iface_check_caps_unreserve_func

Example

android_src $ xvim getPhoneId


android_src $ xvim getPhoneId1) frameworks/base/packages/SystemUI/src/com/android/systemui/statusbar/MSimSignalClusterView.java:378 2) frameworks/base/packages/SystemUI/src/com/android/systemui/statusbar/policy/MSimNetworkControllerImpl.java:280 3) frameworks/base/telephony/java/android/telephony/RadioAccessFamily.java:75 4) frameworks/base/telephony/java/android/telephony/SubscriptionManager.java:828 5) frameworks/opt/telephony/src/java/com/android/internal/telephony/Phone.java:1999 6) frameworks/opt/telephony/src/java/com/android/internal/telephony/PhoneBase.java:2203 7) frameworks/opt/telephony/src/java/com/android/internal/telephony/PhoneProxy.java:1470 8) frameworks/opt/telephony/src/java/com/android/internal/telephony/ServiceStateTracker.java:1101 9) frameworks/opt/telephony/src/java/com/android/internal/telephony/SubscriptionController.java:1169 … #?






More & more…

The problem…

comments for jira header/footer really a bullshit when code changes in 4+ files…

comments for jira header/footer, this is penalty to RD, it discourage RD to do changes to solve the problem!

DUPE AND REPEATED WORKS CAN BE MAXIMUM ELIMINATED

git xcommit - unlock you from jira comment blocks..

auto comment jira header/fooder surround changed code block

interactive way to input jira info for git-commits.

auto list changed files for git add and commit msg.

DUPE AND REPEATED WORKS CAN BE MAXIMUM ELIMINATED

PREFERRED IIRandom access to codebase





More & more…

The problem….

DEPLOYMENT HAVE QUICK, ACCURATE, AND EASY SOLUTION

project_XX $ mm

project_XX $ cd ../../../../../../out/target/prodcut/V99

V99 $ adb push system/…. /system/….

V99 $ cd -

project_XX $ mm

reinstallSomething.py

auto delivery .apk, .so, .xml, executable binaries to device with give name

support pattern search

accurate, reliable, and support both local and remote repositories



project_XX $ mm … Install: out/target/product/v06/system/lib/hw/camera.msm8994.so … Install: out/target/product/v06/system/lib/libmmcamera_interface.so … Install: out/target/product/v06/system/lib/libmm-qcamera.so ….. …..

project_XX $ adb root project_XX $ adb remount project_XX $ reinstallSomething.py -i camera.msm8994.so libmmcamera_interface.so libmm-qcamera.so project_XX$ adb reboot

pattern support

Support remote repository


project_XX $ reinstallSomething.py -i libmmcamera*

local_dir $ export OUT_OVERLAY=/Volumes/Sanctuary/Ferry/V06_1260/LINUX/android/out/target/product/v06

local_dir $ reinstallSomething.py -I libmmcamera* local_dir $ adb reboot






More & more…

Problem..

the patch is good… but sometimes not applicable

how to have full picture of changes

how to deliver the change to other guys with full contexts

…

A SECURE AND SAFE WAY TO BACKUP/SAVE OUR WORKS WITH TRIALS

git xcommit -backup

This command will backup modified things (note, new added one must be separately handled, doesn’t a matter)


folder_to_backup $ git xcommit -backup


PREFERRED IIRandom access to codebase





More & more…

xfastboot, xadb…

when multiple connected to host……

MORE & MORE…

user $ xadb shell more than 1 devices, choose 1 device: 1) V06001325 2) V06000136 3) V06001835 4) V06000137 #? 1 V06001325 root@v06:/ #

LINKS

https://github.com/ferrisky/EHFAndroidVimSuite

https://github.com/ferrisky/ecd

https://github.com/ferrisky/hist

https://github.com/xygod/Git_X_Commit

https://github.com/xygod/Gtags_X_Vim

https://github.com/xygod/reinstallSomething

https://github.com/ferrisky/EHFAndroidVimSuite

https://github.com/ferrisky/ecd

https://github.com/ferrisky/hist

https://github.com/xygod/Git_X_Commit

https://github.com/xygod/Gtags_X_Vim

https://github.com/xygod/reinstallSomething

WELCOME TO CONTRIBUTE CODES TO US!

Software

How To Access Code In Large w/ Vim