ADB: What Developers Need to Know

Many Android developers use adb on a daily basis, sometimes even without knowing. In this article, I will explain what adb is, what you can do with it and how it works. This article is based on my experience developing and roughly follows the talk on at . https://github.com/Malinskiy/adam How to work with adb Podlodka Droidcrew What exactly is adb? Many Android developers used ndroid ebug ridge (or adb as it’s more commonly known) in the following way: A D B However, it’s not just a terminal utility. adb is a contract between Android devices and developer tooling What are the jobs of adb? The general problem that adb is trying to solve is remote access to compute. To solve this problem, adb has first to manage devices, that is: Provide CRUD-like API for devices Abstract transport layer Version the contract between dev tooling and Android devices Besides managing devices, adb has to provide to remote devices as well as including files, networking and the device’s screen frame buffer. terminal access general-purpose input/output Managing devices CRUD-like API adb allows us to available devices, new ones and those no longer needed. While adb connects USB devices automatically, we might want to explicitly connect a remote Android device via TCP/IP (or simply pair over WiFi on Android 11+). list connect disconnect Besides listing devices once, we also might need to devices and receive notifications on status changes of devices (e.g. connected, disconnected). This simplifies the tooling code a lot since you receive real-time notifications instead of resorting to polling. continually monitor Abstract transport layer adb supports 2 transport layers: USB and TCP/IP. While USB is used for real devices, TCP/IP is used for emulators and can be used for connecting to real devices as well. This feature of adb cannot be underestimated: whatever works locally via a USB cable will also work via TCP/IP whether the device is in the same room as you or in a remote datacenter location thousands of kilometres away. The device in this case can be connected via WiFi or Ethernet for example. Contract versioning As with most of the contracts, there are hard requirements for communication and soft ones. With adb, the hard requirement is represented by a single integer that you can always request from the server. The soft or optional part of the contract is represented using . features Terminal access While it can seem trivial to provide terminal access to a remote device, the truth is it’s not a simple task. adb went through several iterations of terminal access. The very basic one only provides an interleaved stdout/stderr output without even notifying you of the process's exit code. Here is a simple demonstration of the problem: Suppose we have two devices: one with a relatively new version of the Android OS and one with an old version. Suppose we want to execute some script that returns an exit code (in this case, 3). While the new device behaves as expected, the old one doesn’t propagate the exit code because it doesn’t support an optional feature. Such behaviours are hidden from the user of adb but can become quite a pain if you’re writing automation around Android devices using adb. Let’s go over the different iterations of terminal access on Android. v1 This version is supported all the way back to the HTC Dream. This implementation is exhibiting some problems, namely: Interleaved stdout/stderr. Good luck trying to parse them into separate outputs Do you want to provide input to a remote terminal process? No luck The protocol doesn’t expose an exit code. Hence, the behaviour demonstrated above If the output of the remote process is binary (e.g. file), then have fun unmangling line feeds since they go through PTY v1.5 After some time, the was added that did not allocate a PTY device and attach directly to the stream. exec service This version supported the use-case where the remote Android process returned a file (e.g. as a screenshot using adb exec-out screencap -p > screen.png) and also provided a file using the stdin stream. This did come at a cost, though: you have to choose between providing the stdin or reading the stdout because, in this implementation, you can’t have both. Same as v1, you still don’t have the exit code of the remote process. v2 Addressing almost all the problems of the previous version, v2 provides separated stdout and stderr output streams. It also returns exit code value and supports real-time stdin stream. This version also had a cost: the adb code on the Android device had to be changed. Since it’s a soft requirement for a contract, this functionality is behind a feature and works roughly starting from Oct 2015. shell_v2 PTY's problem with mangling is still present in the v2 also. One particular use case with developer tooling in Android that deserves its own feature is communication with SystemServices, e.g. package management and activity management. An example of this interaction is listing installed packages with : adb shell pm list package Spawn a process with Pm.java CLI wrapper Inside the process call ServiceManager.getService("package") Pass the result to stdout This is a very slow way of communicating with a long-lived service and not very efficient in terms of coding. That’s why was created: instead of a separate entry point for system services, a generic CLI is able to talk to all of them. If a system service wants to expose itself, it implements the method where file descriptors for the process are being passed. This change saves the trouble for AOSP developers of writing yet another CLI for each system service. cmd handleShellCommand In order to solve the overhead of running such terminal commands and allocating device resources for spawning a new process for a small request, ndroid inder ridge . A B B was introduced Using abb, the same use-case for listing packages is achieved using . According to the commit introducing this feature, the latency was reduced six times. adb abb package list Emulator terminal Every official emulator exposes a TCP port that you can use to issue emulator-specific commands. As of today, this functionality can also be considered terminal access and is usually achieved using , e.g.: adb emu $CMD What’s possible using the emulator’s terminal? There is no fixed contract here, but you can usually find using : emu help Calls and SMS Screen rotation Fingerprint sensor input Physics input (gyro, accel, etc) Mobile network conditions simulation GPS This functionality is really powerful: for example, it allows you to test an application that behaves differently if you’re running or walking, e.g. fitness trackers, mock location for GoogleMaps fragment and much more. To be frank, this terminal access is unrelated to adb, but for ease of use, it’s there in the adb CLI. You can also use any TCP client, e.g. for executing commands. The port is the last part of the emulator serial, e.g. for the console port is 5554. telnet emulator-5554 Since this connection is not encrypted, the emulator asks for a secret key that is stored in the $HOME/.emulator_console_auth-token on the system that started the emulator. This key is asked whenever a new connection is established to the console port. Input/output File v1 Initial implementation of file transfer on android (sync service) models the files using: name mode (access permissions) size mtime (timestamp of last file content change) In terms of what you can do with files, there are four requests: List Stat Pull a single file Push a single file The directory sync is not a concern from the point of adb, so all those directory structures are the responsibility of the tooling that is built on top of adb. File v2 The newer implementation supports everything v1 does as well as: uid (owner user id) gid (owner group id) atime (last access timestamp) ctime (change timestamp) nlink (number of hard links) ino (inode) In terms of requests, this new implementation requires feature flags: List (feature ) ls_v2 Stat (feature ) stat_v2 Pull a single file (feature ) sendrecv_v2 Push a single file (feature ) sendrecv_v2 On top of additional properties, file v2 protocol supports on-the-fly compression and decompression using , and to speed up those chunky adb transfers, including application installation. No compression is also an option here. brotli lz4 zstd Network input/output adb allows you to do port forwarding (request to developer machine is forwarded to device) and reverse port forwarding (the other way). You can use a TCP or a Unix socket on the developer's machine. On the Android side, you can specify the TCP/Unix socket or PID of a thread for using a debugger or and more. JDWP device files In terms of requests, it’s rules, a rule, a rule and rules. list add remove remove all Screenbuffer adb support capturing the current frame of the Android’s screen. There are 3 versions that adb used historically: v16: uses format RGB565 (16 bits per pixel) v1: variable bits per pixel specified each time, but most commonly is ARGB8888 v2: also variable as v1, but usually ARGB8888 and supports non-sRGB color spaces such as DCI-P3 Each time you use this request, you must interpret the data differently, so adb doesn’t provide any common mapping layer. Other bits Besides the above functionality, adb supports switching the adbd daemon running on the Android device into different modes (e.g. USB transport switch to TCP and back), remounting Android partitions, graceful shutdown of adb server and sideloading commonly used to flash custom firmware. How does adb work? Now that you know what adb can do, let’s dive into the bit. how This diagram has three key components at play: adbd installed on real or emulated devices (left & right) adb clients that communicate with Android devices adb server in the centre being the middleman adb server is exposing a TCP port for client connections The telnet connection example above first selects any device for communication and then executes a shell command . echo HelloADB This example is trying to illustrate that the adb client -> server protocol is not encrypted and is somewhat human-readable. adb server adb server is the middleman between clients and Android devices. It is commonly a background process that acts as a set of for a single connection to the device (remember the serial part in the USB?). multiplexers By default, the adb server attaches to the loopback interface on port 5037 because the communication is not encrypted and using a public interface exposes you to security risks (although you can do that if you want to). adbd adbd is also a background process like an adb server, but this process is on the Android device. Usually found at , and is installed using APEX module on Android 11+. adbd is the one doing the process spawn and all the actual execution of the requests. /system/bin/adbd com.google.android.adbd adb client Anything can communicate with an adb server if it can open TCP connections. The most common adb client is the adb CLI that is bundled together with the adb server into a single binary (hopefully not just for lulz). Typical adb client to adb server communication Every interaction of the client with the server starts with opening a TCP socket. Verifying the hard requirements of adb protocol is done using the command, the response is ACKd with and then the version is returned. If this is a compatible version the client proceeds to select a device to communicate with, in this case, a device with a serial number using command . This selection is also ACKd with by the server. host:version OKAY 29 SERIAL host:transport:SERIAL OKAY After this part of the dance is out of the way, the client specifies the actual request that needs to be executed, the means that we want to execute a command using the v1 shell implementation. This request is ACKd with yet another and the stdout/stderr response is returned with a close of the underlying TCP. shell:echo Hello echo Hello OKAY Since this part of the communication is not encrypted, it’s easy to debug the flow of traffic using something like or : tcpdump Wireshark Transport target Since adb is essentially a huge set of multiplexers, the selection of a target device can be done in a number of ways: = any single device or the host of the adb server host = specific device by serial number host-serial = specific device by transport identifier (internal adb id for underlying connection to adbd) host-transport-id = any single device connected via USB host-usb = any single local(=emulator) device host-local Request steps Every interaction of the adb client with the server usually follows the following plan: Open TCP socket Check versions (hard version of the contract as well as features) Select transport target Execute the actual command Close TCP socket adb server -> adbd request flow Communication between the adb server and adbd is done on top of a secure transport layer, so debugging what’s going on here is a bit harder. The protocol itself is a bit more structured with clear : framing struct amessage { uint32_t command; /* command identifier constant */ uint32_t arg0; /* first argument */ uint32_t arg1; /* second argument */ uint32_t data_length; /* length of payload (0 is allowed) */ uint32_t data_check; /* checksum of data payload */ uint32_t magic; /* command ^ 0xffffffff */ }; struct apacket { using payload_type = Block; amessage msg; payload_type payload; }; Each packet consists of header that contains the command to execute, a pair of arguments for the command and the payload length with a checksum. The command is also verified using the checksum. amessage magic Each communication between adb server and adbd starts with establishing a secure transport, e.g. in case of pair over WiFi TLS is used. Any packets before this handshake are discarded. is the identifier of connect command that has protocol version and max payload size as arguments. During this connect command host is also sending the identifier for the adb server. CNXN adbd also establishes connection back to the server the same way indicating the metainformation about the device such as the serial number. Server to adbd packet types is the connection packet A_CNXN(version,maxdata, ) systemtype:serialno:banner establishes the secure connection using TLS A_STLS(type,version,) establishes the secure connection using asymmetric RSA encryption. A_AUTH(type,0,”data”) Establishing the connection involves generating a random secret and sending it to the adb server using . adb server then encrypts this random secret using a private RSA key and sends it over to the adbd using . adbd then uses locally stored keys and verifies if one of the public keys gives back the original random secret. If a match is found, the connection is established. If there is no match found, then this cycle repeats until a proper key is used by the adb server or the adb server responds with a packet containing the public RSA key of the adb server. This is when you see the trust connection dialogue on Android and have to accept or deny the adb connection. type=TOKEN type=SIGNATURE type=RSA opens a communication from some client interaction identified by local-id to some destination on the device, e.g. shell process A_OPEN(local-id,0,dst) is basically an ACK packet A_OKAY(local-id,remote-id,) is the meat of the communication sending the chunks of data from and to the device. Keep in mind that there is no read operation: A_OKAY on the A_WRITE == A_READ A_WRTE(local-id,remote-id,”data”) closes the communication. Usually, the connection is opened and closed both ways with confirmation using A_OKAY A_CLSE(local-id,remote-id,) As an example, let’s see the traffic flow when executing command using adb. ls First, secure transport has to be established, and connections should succeed both ways. Next, we need to open a destination with some local identifier for this interaction using . If that's easier for you, you can think of local-id and remote-id as ports in TCP connections. shell:ls id When the adb server opens the connection, it doesn’t yet know the remote-id, so it uses . When adbd confirms the connection using , it responds back with remote id . These identifiers are the key components to having parallel execution of multiple requests since, otherwise, we wouldn't be able to route the packets properly. 0 A_OKAY id2 Imagine the response of ls is so large (>256K) that the response is fragmented into two chunks. Upon receiving the first one, the adb server has to confirm the delivery. Only after the confirmation will the adbd send the second chunk and close this session. Remember that this interaction doesn’t close the underlying connection as opposed to the adb client -> adb server communication. Real-world examples In Java, there is a commonly used adb client called (maven ) ddmlib com.android.tools.ddms:ddmlib:$VERSION Let’s see some examples of its usage: IntelliJ IDEA When you press the green triangle to run your application using a device, you might see a selector for which device to use. : Here is the selection code Android Gradle Plugin When using task in Gradle for installing your application, the following code using adb is used: installXXX Spoon test runner When you execute your tests, the test runner has to where the output of the execution (e.g. screenshots/code coverage) is located: determine Appium Although a bit dated, appium during the execution using adb framebuffer request: takes a screenshot The good thing about ddmlib is that many developer tools already use it, and it’s a commonly shared dependency. On the negative side, the codebase is a bit old, utilises threads too much, doesn’t provide proper timeouts and has close to zero documentation. ddmlib doesn’t handle all the things possible with adb, e.g. sideload requests, but there is also stuff that ddmlib does that is not adb-related: JDWP support if you need to implement a debugger Logcat parsing (but only ) -v long Test runner support, i.e. command wrapper and parser of the results (both string and proto-based) am instrument adb cli The second most commonly used client is adb cli bundled with the adb server. Its advantage is speed: it can be multiple times faster than the ddmlib. For that, you trade off the Android-specific parts (adb cli doesn’t know the format of for example, so you’ll have to parse it yourself) pm list Apart from the numerous bash scripts, adb cli can be found in for and some testing tools, including . Genymotion/scrcpy port forwarding the connection KasperskyLab/Kaspresso Application usage of adb Although not part of the adb protocol, it facilitates a lot of developer usage of Android devices: Debugging code execution (JDWP) Application installation (push apk, execute install cmd, streaming/split/atomic installs) Running tests (execute command, return the output) Mock-server networking (port-forward the mock server) Record video (execute a command, pull the file) Logcat Explore device (e.g. getprop) Simulate events (emu commands on emulator, input on all Android devices) I’d like to highlight the difference between using native adb functions and using adb for using Android device functions with the following example of taking a screenshot: the request is made using adb only adb framebuffer: is a terminal command on an Android device that is triggered via adb. If the command fails, it’s not adb’s fault adb shell screencap -p is a framework method that doesn’t use adb at all androidx.test.uiautomator.UiDevice#takeScreenshot adb caveats While device state means the Android device is online, adb doesn’t have any idea if the device is booted or not. Usually, devices will setup property to when init finishes device sys.boot_completed 1 While the adb server usually forks to the background, it might be desirable to start in the foreground to control the lifecycle properly. To do that, you need to kill the existing server and then start a new one adb nodaemon server Working with legacy devices using shell v1, you can still get the value of the exit code if you change the command a bit . When parsing the result, you will have to strip the suffix and extract the value of the exit code cmd; echo x$? If emulator functionality for mocking calls and fingerprints is something you desire, try using the for doing the same thing but with a contract and autogenerated client libs emulator_controller.proto Most physical devices have a stable serial number. Emulators, on the other hand, do not and the same emulator booted twice might have a different serial Thanks for checking this article out! All this knowledge came from developing an alternative adb client written in Kotlin called . If you work with adb server using Kotlin — try it. adm feel free to Also published . here