How to Run 5000+ Tests on Mobile Devices Everyday; From inDrive's Playbook (Part 1)

Hi all! My name is Taras Egorov; I'm an engineer at inDrive. I'll show you how we set up an infrastructure capable of running more than 5,000 tests per day on iOS and Android devices combined. The secret is simple: we used Selenoid.

Preface

Last year, our colleagues conducted a study of auto testing, and we took part in a survey as part of the study.

We were pleased with the survey's findings, so we decided to write an article to share our experience with you and get some advice in return. We thought it a good idea to divide the material into two parts: the first one focusing on Android and the second on iOS.

Let's begin with Android.

Running the tests on Android

Selenoid is a popular tool for running and managing browsers and Android emulators in a Docker container. You can read more about this in the documentation.

To write Appium tests, we use:

• Kotlin;
• JUnit 5;
• Maven.

The first run. Setting up Selenoid

1. Create a browsers.json config file:

{

  "android": {

    "default": "10.0",

    "versions": {

      "10.0": {

        "image": "browsers/android:10.0",

        "port": "4444",

        "path": "/wd/hub"

      }

    }

  }

}

The emulator image is specified in image. The guys from aerokube have prepared ready-made images of Android emulators. You can check them out here or here. The images don't differ from each other in any way.

Let’s take the image browsers/android:10.0 as an example. The image must be downloaded beforehand: docker pull browsers/android:10.0, otherwise the tests will not run:

Original error: create container: Error response from daemon: No such image: browsers/android:10.0

2. The next step is to run Selenoid. We do that directly via docker, or there’s the option of using Configuration Manager.

docker run -d \

           -v /var/run/docker.sock:/var/run/docker.sock \

           -v "$(pwd)/selenoid/config/":/etc/selenoid/:ro \

           -p 4444:4444 \

           --name selenoid \

           aerokube/selenoid:1.10.7

You can check to see if Selenoid is working properly by following the link http://localhost:4444 in your browser:

You are using Selenoid 1.10.7!

3. Specify the Selenoid address in the Appium tests in the driver:

   
   

...
val driver = AndroidDriver(URL("http://localhost:4444/wd/hub"), capabilities)
...

4. Next, specify the link to the build in capabilities:

... 
capabilities.setCapability("appium:app", "https://storage.example.com/builds/app.apk")
...

If unable to provide a link, you can specify the path to the build:**

...
capabilities.setCapability("appium:app", "/builds/app.apk")
...

Where /builds/app.apk is the path inside the container where the emulator is being run. For this option to work properly, be sure to specify the volumes in browsers.json:

{

  "android": {

    "default": "10.0",

    "versions": {

      "10.0": {

        ...

        "volumes": [

          "/home/username/app.apk:/builds/app.apk:ro"

        ]

        ...

      }

    }

  }

}

Where /home/username/app.apk is the path to the build on the host platform.

That's it, we've almost set up Selenoid, and now we can try running the tests:

./mvnw test

But, unfortunately, the tests will not be able to run. Let's look into this and see what's wrong.

The second run. Taking a look at the logs and video

The first thing to do after a failed startup is to check out the Selenoid logs:

docker logs selenoid

[INIT] [Loading configuration files...]
[INIT] [Loaded configuration from /etc/selenoid/browsers.json]
[INIT] [Video Dir: /opt/selenoid/video]
[INIT] [Your Docker API version is 1.41]
[INIT] [Timezone: UTC]
[INIT] [Listening on :4444]
[NEW_REQUEST] [unknown] [172.17.0.1]
[NEW_REQUEST_ACCEPTED] [unknown] [172.17.0.1]
[LOCATING_SERVICE] [android] [10.0]
[USING_DOCKER] [android] [10.0]
[CREATING_CONTAINER] [selenoid/android:10.0]
[STARTING_CONTAINER] [selenoid/android:10.0] [75e454341da7fc4b58ba104a5180813bac6cd7c124037a759b6c976e65b168fa]
[CONTAINER_STARTED] [selenoid/android:10.0] [75e454341da7fc4b58ba104a5180813bac6cd7c124037a759b6c976e65b168fa] [0.40s]
[0] [REMOVING_CONTAINER] [75e454341da7fc4b58ba104a5180813bac6cd7c124037a759b6c976e65b168fa]
[0] [CONTAINER_REMOVED] [75e454341da7fc4b58ba104a5180813bac6cd7c124037a759b6c976e65b168fa]
[0] [SERVICE_STARTUP_FAILED] [http://172.17.0.3:4444/wd/hub does not respond in 30s]

We see that the status is SERVICE_STARTUP_FAILED. Go to the documentation and look at the status value:

SERVICE_STARTUP_FAILED - Failed to start Docker container or driver binary

The error doesn’t tell you much, and more information is required. It would be good to take a look at the container logs. Let's do that by enabling logging:

docker run -d \

           -p 4444:4444 \

           -v /var/run/docker.sock:/var/run/docker.sock \

           -v "$(pwd)/selenoid/config/":/etc/selenoid/:ro \

           -v  "$(pwd)/selenoid/logs/":/opt/selenoid/logs/ \

           aerokube/selenoid:1.10.7 \

           -log-output-dir /opt/selenoid/logs

We also enable logs in the Capabilities section:

...
capabilities.setCapability("enableLog", true)
...

Run the tests and review the logs using the browser http://localhost:4444/logs/:

2023-04-16T13:44:43.909768530Z Waiting X server...
2023-04-16T13:44:44.009494775Z Logging to: /tmp/fluxbox.log
2023-04-16T13:44:44.047587277Z Waiting X server...
2023-04-16T13:44:44.151933325Z Waiting X server...
2023-04-16T13:44:44.262850410Z * daemon not running; starting now at tcp:5037
2023-04-16T13:44:44.457972956Z * daemon started successfully
2023-04-16T13:44:44.458249266Z adb: no devices/emulators found
2023-04-16T13:44:45.463480812Z adb: no devices/emulators found
2023-04-16T13:44:46.471547723Z adb: no devices/emulators found
2023-04-16T13:44:47.476093515Z adb: no devices/emulators found
2023-04-16T13:44:48.481987351Z adb: no devices/emulators found
2023-04-16T13:44:49.486503149Z adb: no devices/emulators found
2023-04-16T13:44:50.492757801Z adb: no devices/emulators found
2023-04-16T13:44:51.499094108Z adb: no devices/emulators found
2023-04-16T13:44:52.505862428Z adb: no devices/emulators found
2023-04-16T13:44:53.513276412Z adb: no devices/emulators found
2023-04-16T13:44:54.520642210Z adb: no devices/emulators found
2023-04-16T13:44:55.527420189Z adb: no devices/emulators found
2023-04-16T13:44:56.534631013Z adb: no devices/emulators found
2023-04-16T13:44:57.316094939Z WARNING. Using fallback path for the emulator registration directory.
2023-04-16T13:44:57.335415397Z checkValid: hw configs not eq
2023-04-16T13:44:57.541959741Z adb: device offline
2023-04-16T13:44:58.547907700Z adb: device offline
2023-04-16T13:44:58.565504866Z emulator: WARNING: System image is writable
2023-04-16T13:44:58.565528396Z emulator: Cold boot: different AVD configuration
2023-04-16T13:44:58.565532576Z Your emulator is out of date, please update by launching Android Studio:
2023-04-16T13:44:58.565536346Z  - Start Android Studio
2023-04-16T13:44:58.565539506Z  - Select menu "Tools > Android > SDK Manager"
2023-04-16T13:44:58.565543076Z  - Click "SDK Tools" tab
2023-04-16T13:44:58.565546216Z  - Check "Android Emulator" checkbox
2023-04-16T13:44:58.565549216Z  - Click "OK"
2023-04-16T13:44:58.565552146Z 
2023-04-16T13:44:59.554451514Z adb: device offline
2023-04-16T13:45:00.560926060Z adb: device offline
2023-04-16T13:45:01.568777440Z adb: device offline
2023-04-16T13:45:12.124226047Z emulator: INFO: boot completed
2023-04-16T13:45:12.124251007Z emulator: INFO: boot time 27848 ms
2023-04-16T13:45:12.124255077Z emulator: Increasing screen off timeout, logcat buffer size to 2M.
2023-04-16T13:45:12.152557294Z emulator: Revoking microphone permissions for Google App.

The container logs are not much help here either, because you can't see the Appium logs. Now let's try to enable them. To do this, let's look at the script entry point.sh:

...
if [ -z "$VERBOSE" ]; then
    APPIUM_ARGS="$APPIUM_ARGS --log-level error"
else
    EMULATOR_ARGS="$EMULATOR_ARGS -verbose"
fi
...

To enable Appium logs, the parameters VERBOSE=true and APPIUM_ARGS=--log-level debug: must be passed to the container:

{

  "android": {

    "default": "10.0",

    "versions": {

      "10.0": {

        ...

        "env": [

          "VERBOSE=true",

          "APPIUM_ARGS=--log-level debug"

        ]

        ...

      }

    }

  }

}

To enable Appium to debug logs, you need to pass VERBOSE; in this case, the emulator logs turn on and start filling the "ether"” But we fixed that for future images =)

Now it is enough to pass to APPIUM_ARGS=-log-level debug.

...
[HTTP] --> POST /wd/hub/session/c89fa9c2-ca2b-49cd-ab31-590eeccf77d1/element
[HTTP] {"using":"id","value":"authorization_edittext_phone"}
[debug] [W3C (c89fa9c2)] Calling AppiumDriver.findElement() with args: ["id","authorization_edittext_phone","
c89fa9c2-ca2b-49cd-ab31-590eeccf77d1"]
[debug] [BaseDriver] Valid locator strategies for this request: xpath, id, class name, accessibility id, -and
roid uiautomator
[debug] [BaseDriver] Waiting up to 0 ms for condition
[debug] [WD Proxy] Matched '/element' to command name 'findElement'
[debug] [WD Proxy] Proxying [POST /element] to [POST http://127.0.0.1:8200/wd/hub/session/65943f03-3b35-4d3eb221-d6dc7988f935/element] with body: {"strategy":"id","selector":
"authorization_edittext_phone","context":"","multiple":false}
[WD Proxy] Got response with status 404: {"sessionId":"65943f03-3b35-4d3e-b221-d6dc7988f935","value":{"error"
:"no such element","message":"An element could not be located on the page using the given search parameters","stacktrace":"io.appium.uiautomator2.common.exceptions.El
ementNotFoundException: An element could not be located on the page using the given search parameters\n\tat io.appium.uiautomator2.handler.FindElement.safeHandle(Find
Element.java:73)\n\tat io.appium.uiautomator2.handler.request.SafeRequestHandler.handle(SafeRequestHandler.java:41)\n\tat io.appium.uiautomator2.server.AppiumServlet.
handleRequest(AppiumServlet.java:253)\n\tat io.appium.uiautomator2.server.AppiumServlet.handleHttpRequest(AppiumServlet.java:247)\n\tat io.appium.uiautomator2.http.Se
rverHandler.channelRead(ServerHandler.java:68)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:366)\n\tat io
.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:352)\n\tat io.netty.chann...
[debug] [W3C] Matched W3C error code 'no such element' to NoSuchElementError
[debug] [W3C (c89fa9c2)] Encountered internal error running command: NoSuchElementError: An element could not
be located on the page using the given search parameters.
[debug] [W3C (c89fa9c2)]     at AndroidUiautomator2Driver.findElOrEls (/opt/node_modules/appium/node_modules/appium-android-driver/lib/commands/find.js:75:11)
[debug] [W3C (c89fa9c2)]     at process._tickCallback (internal/process/next_tick.js:68:7)
[HTTP] <-- POST /wd/hub/session/c89fa9c2-ca2b-49cd-ab31-590eeccf77d1/element 404 23 ms - 444
...

As seen from the logs, Appium cannot find our element. Let's see what is happening on the emulator screen. To do this, we have to run the Selenoid UI:

docker run -d \

           --name selenoid-ui \

           -p 8080:8080 \

           --link selenoid:selenoid \

           aerokube/selenoid-ui:1.10.4 \

           --selenoid-uri "http://selenoid:4444"

Go to http://0.0.0.0:8080 and open the Selenoid UI:

Be sure to enable VNC and video recording in the tests:

...

capabilities.setCapability("enableVNC", true)

capabilities.setCapability("enableVideo", true)

...

The Selenoid startup command ends up looking like this:

docker run -d \

-v /var/run/docker.sock:/var/run/docker.sock \

-v "(pwd)/selenoid/logs/":/opt/selenoid/logs/ \

-v /opt/selenoid/video/:/opt/selenoid/video/ \

-e OVERRIDE_VIDEO_OUTPUT_DIR="/opt/selenoid/video/" \

-p 4444:4444 \

-name selenoid \

aerokube/selenoid:1.10.7 \

-log-output-dir /opt/selenoid/logs

Open the Selenoid UI once the tests are up and running:

And here's a video of it.

We found the cause of the startup error. Great! Let’s move on.

The third run. Building an emulator image

As it turns out, the images of Selenoid emulators won’t work without Google Play services. To remedy this situation, you must build an emulator image yourself. The guys from aerokube have assembled everything you need for this: a repository with photos and documentation.

1. Downloading the repository.
2. Go to the selenium folder.
3. Run the script ./automate_android.sh and answer the questions. This is what it looks like in our case:

Specify Appium version: [1.18.1]

>> 1.18.1

Specify Android image type (possible values: "default", "google_apis", "google_apis_playstore", "android-tv", "android-wear"): [default]

>> google_apis

Specify Application Binary Interface (possible values: "armeabi-v7a", "arm64-v8a", "x86", "x86_64"): [x86]

>> x86

Specify Android version: [8.1]

>> 10.0

Specify device preset name if needed (e.g. "Nexus 4"):

>>

Specify SD card size, Mb: [500]

>>

Specify userdata.img size, Mb: [500]

>>

Are you building a Chrome Mobile image (for mobile web testing): [n]

>> y

Specify Chromedriver version if needed (required for Chrome Mobile):

>> 74.0.3729.6

Specify image tag: [selenoid/chrome-mobile:74.0]

>> android-emulator:10.0

Add Android quick boot snapshot? [y]

>> n

When I saw the question Add Android quick boot snapshot?, I thought it was an emulator snapshot. But if you look at the code, it calls a script, which is needed to install APK applications. Basically, it doesn't give us any gain.

We already use snapshot emulators to speed up container launches, but we'll talk about that in other articles.

4. Once the image is built, we will be prompted to push it into the registry. Since we don't need that yet, we will kindly decline the offer:

   
   

Push?  >> n

We have put together a build with Google Play services. Remember to change the image parameter in browsers.json, and restart Selenoid.

Now let's try rerunning the tests:

[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------

And here’s a video of the test run.

To sum up

What we have done:

• Configured Selenoid to run Android tests.
• Learned how to check out the logs and videos to troubleshoot.
• Built our own emulator image with Google Play services.

Something else I would like to tell you about:

• Selenoid timeouts. If your app is large, you might run into problems with timeouts.
• How we tried making the container run faster. The video shows that the run takes about a minute or so.
• How we tried running native Android tests written in Espresso on Selenoid. Spoiler alert: this works!

Meanwhile, in the next part, we'll tell you how we scaled up the infrastructure and ran tests on iOS.