CI/CD for Android Guide: Bitbucket Pipelines and Gradle Play Publisher

Motivation

There are many ways to build, test, deploy and publish an Android app.

You can do it manually.

It’s not hard or time-consuming to create a signed APK or Bundle in Android Studio, upload it to Google Play and promote it through the different testing tracks up to production. I did it for many years. I thought the time automation saves won’t compensate for the effort to set up a working pipeline. I was wrong.

Automation doesn’t just save time, it also makes the process more reliable, less error-prone (to human error) and encourages to deploy/publish more often. In general, the development cycle is sped up, not just when it comes to bug fixes but also feature releases. Why wait to bundle a big release when features can be pushed out to the customer by a simple pr/merge?

You can use Fastlane.

I did that for most of my apps. It’s convenient to type fastlane build_deploy and have the app built, signed, and published to Google Play automatically. It’s probably good enough if you’re the only developer on the team. For teams there are some major issues though:

1. Fastlane needs to be installed on each build machine. With multiple developers or if you use multiple computers to develop, you need to setup and maintain it multiple times. Fastlane’s documentation isn’t great imo and especially the setup on Linux is not straight-forward (time consuming). Having multiple build machines also impacts the reproducibility of the build, differences in the Fastlane dependencies (Ruby libs) can result in different results.
2. Signing and deploying requires the signing/upload key and also the Google Play API key. As few people as possible should have access to those keys. It’s possible to create new keys every time an employee with access to the keys leaves the company (provided upload keys are used to sign the app) but it’s certainly not an ideal process.
3. One of my requirements was to start the build process upon merge. Fastlane isn’t the tool for that unless it’s combined with other tools which leads us to…

You can use Jenkins.

Jenkins is a great tool. It’s very flexible and we used it (in combination with Rundeck) successfully to build and deploy a microservice backend to an AWS hosted Docker/Kubernetes environment. We also had a dedicated DevOps team to maintain it. If you don’t want to host your own CI/CD infrastructure and don’t have experts taking care of the installation and configuration, then Jenkins is probably not the right tool.

You can use <insert your SAAS CI/CD solution here>.

There are many cloud based CI/CD solutions like CircleCI, Travis CI or TeamCity to name just a few. I’m sure they are all great and integrate with your preferred Git provider but it’s another tool to integrate (while Bitbucket Pipeline is obviously tightly integrated already) and they seem to be sledge hammers for the requirements I had which are:

Requirements

• Trigger upon merge into a specific branch (master in my case)
• Run unit tests (I have ui tests as well but they aren’t part of the pipeline yet)
• Run lint and fail the build if errors are detected
• Build a free and a paid version of the app as bundles
• Sign the app
• Automatically increment the build number
• Upload app metadata (release notes, description, screen shots) if needed
• Deploy the app to the internal testing track
• Manual promotion from the internal testing track to the production track by app flavor (free/paid in my case)
• Secrets (signing key, Google Play API key, passwords) can’t be retrieved by developers with access to the repo

It turns out Bitbucket and Gradle Play Publisher are the tools I needed to make this happen. Let’s see how.

Overview

There are four steps to setup the pipeline:

1. We need programmatic access to Google Play to publish and promote apps (plus manage the meta data) -> we need an API key for the Google Play Developer API.
2. We need to configure the Gradle build to use a signing configuration that reads the signing information from environment variables.
3. We need to configure the Gradle Play Publisher plugin to interact with Google Play (upload, publish apps and manage meta data).We need to configure Bitbucket Pipeline to tie everything together.
4. The pipeline configuration defines the trigger (upon merge), the build steps (Gradle), the deploy steps (Gradle Play Publisher) and injects the environment variables (secrets, build number).

Google Play Developer API

The official documentation explains all steps in detail: https://developers.google.com/android-publisher/getting_started.

TL;TLTR (too long; too lazy to read):

• Go to your Google Play Developer Console as owner of the account and open the API access page:
  

• Accept the Terms of Service (if not done yet)

• Create a new Google Cloud project if you haven’t created one already (otherwise link an existing one).

• Under Service Accounts click on “Create new service account” and open the link that leads to the Google Cloud Platform:
  

• In Google Cloud Platform click on “CREATE SERVICE ACCOUNT”:
  

• Pick a meaningful name and description before hitting the “CREATE” button:
  

• The account needs the role “Service Account User”:

• You don’t need to grant user access to the new service account, Google Cloud adds the required users automatically with the correct permissions (a Google Play service and your own user) so just hit “DONE”:
  

• Next, you need to create an API key for the account.
  Open the actions menu (the three dots) and select “Manage keys”:
  

• Under “ADD Key” select “Create new key”:
  

• Create a JSON key:
  

• After hitting the “CREATE” button, the key file will be downloaded to your computer. I recommend to rename the file to make its purpose more obvious:
  

• Now you’re done in Google Cloud Platform and you can go back to the Google Play Console (to the API access screen). The newly created account should appear under “Service accounts” (hit the “Refresh service account” button). Click on “Grant access”:

• Click on “Add app” and select all apps you want to manage with this service account:
  

• The Account permissions are already set correctly so that the service can manage all release related activities (create releases including publication to production, management of meta data etc.).

• Click on “Invite user” and you’re done.
  We will use the Gradle Play Publisher plugin to validate API key setup later on.

Gradle Build

The Gradle build needs to be configured to include a signing configuration that reads the secrets from environment variables (or the gradle.properties file in your ~/.gradle folder).
If you already have one then you can skip this chapter.

The assumption is that you have a published app in Google Play and that you have access to the keystore and a signing key (or upload key) including the passwords.

For a local build the location of the keystore, the keystore password, the key alias and the key password will be configured in your ~/.gradle/gradle.properties file.

If you don’t have a ~/.gradle/gradle.properties file, please create one and add these four parameters (the bold part needs to be configured to fit your setup):

KEYSTORE_FILE=/path to the keystore file/playstore.keystore
KEYSTORE_PASSWORD=keystore password
KEYSTORE_KEY_ALIAS=key alias
KEYSTORE_KEY_PASSWORD=key password

Note: don’t use ~ for your home directory but use absolute paths. ~ works in a shell context but not with Gradle, Gradle Play Publisher and Bitbucket Pipeline.

Create a signing config in your app’s gradle.build file:

signingConfigs {
    release {
        storeFile file(KEYSTORE_FILE)
        storePassword KEYSTORE_PASSWORD
        keyAlias KEYSTORE_KEY_ALIAS
        keyPassword KEYSTORE_KEY_PASSWORD
    }
}

Add the signing config to the build type:

buildTypes {
   debug {
        // debug build type configuration ...
   }

   release {
        // release build type configuration ...
        signingConfig signingConfigs.release
   }
}

If the signing configuration is correct then the following command should run and create one or more aab files in your build/outputs/bundle folder:

./gradlew bundleRelease

Build Number

One of the requirements is the auto-increment of build numbers / versionCode. We will use Bitbucket’s $BITBUCKET_BUILD_NUMBER to set an environment variable that defines the versionCode. In order to process this environment variable, change your build.gradle file from:

versionCode 124

to:

versionCode project.hasProperty('BUILD_NUMBER') ? project['BUILD_NUMBER'].toInteger() : 124

Last but not least we need to set the initial value for $BITBUCKET_BUILD_NUMBER as it needs to be higher than the last used versionCode. Please follow this article to do so: https://support.atlassian.com/bitbucket-cloud/docs/set-a-new-value-for-the-pipelines-build-number/.

Gradle Play Publisher

While we are now able to build the app and create a signed bundle (or apk), we still need to configure Gradle Play Publisher to publish the signed app to Google Play (plus manage the meta data like screen shots, description etc.).

We could also use Fastlane for this but I don’t recommend to go down that path (been there, done it). Just trust me ;-)

Setting up the Gradle Play Publisher plugin is easy (see also https://github.com/Triple-T/gradle-play-publisher):

• Add the plugin to the app’s build.gradle file:

plugins {
    id 'com.android.application'
    id 'com.github.triplet.play' version '3.3.0'
    // other plugins...
}

• Add a configuration block to the app’s build.gradle file (after the android block):

android { ... }
play {
    serviceAccountCredentials = file(GOOGLE_PLAY_API_KEY)
}

• You’ll notice the GOOGLE_PLAY_API_KEY parameter. It’s a reference to the api key file we got when setting up the key for the service account -> google-play-api-key.json. The parameter needs to be defined in the ~/.gradle/gradle.properties file (analogous the signing config parameters):

GOOGLE_PLAY_API_KEY=/path to the api key file/google-play-api-key.json

• If everything was setup properly, the following command when run from the root directory of your app will download the app’s meta data:

./gradlew bootstrap

Bitbucket Pipeline

To get started with Bitbucket Pipeline and to create your first pipeline, please follow this excellent article: https://www.rockandnull.com/android-continuous-integration-bitbucket/

After going through that article you should now have a bitbucket-pipelines.yml file in your app’s root directory.

What we want now is to create this specific pipeline:

It consists of five steps (each step runs a separate Docker container):

1. Create secret files (keystore and API key file) and make then available to the pipeline
2. Run unit tests
3. Build the app bundle(s) and deploy them to the internal testing track
4. Promote the free version of the app to the production track (manual step)
5. Promote the pro/paid version of the app to the production track (manual step)

1. Create secret files

Neither the signing key (keystore + key) nor the Google Play API key should be added to the repo, otherwise every developer with repo access would be able to read them.

Bitbucket has the ability to define repository variables. By default they are encrypted and can’t be read by regular users but only by scripts. We will use this feature to define the five arguments our build/publish scripts need:

• KEYSTORE_FILE
• KEYSTORE_PASSWORD
• KEYSTORE_KEY_ALIAS
• KEYSTORE_KEY_PASSWORD
• GOOGLE_PLAY_API_KEY

It’s easy to define the three values for KEYSTORE_PASSWORD, KEYSTORE_KEY_ALIAS and KEYSTORE_KEY_PASSWORD since they are just text values. To do so go to the “Repository settings” and scroll down to “Repository variables”. Enter all three variables with the correct values:

To store the KEYSTORE_FILE and the GOOGLE_PLAY_API_KEY in a repository variable we encode the files with base64. The build pipeline will decode the text and recreate the original files.

Run the following commands to encode the two files:

base64 google-play-api-key.json
base64 playstore.keystore

Copy the base64 strings and create repository variables in Bitbucket. The strings should look somewhat like this (much longer though): YmFzZTY0IGdvb2dsZS1wbGF5LWFwaS1rZXkuanNvbg==

I also created two variables KEYSTORE_FILE and GOOGLE_PLAY_API_KEY to define the files names used for the decoded secrets:

Now we’re ready to define the first step of the actual pipeline in the bitbucket-pipelines.yml file.

image: androidsdk/android-30

pipelines:
  branches:
    master:
      - step:
          name: Create keystore and API key
          script:
            # create the keystore file and the google play api key file
            - mkdir keys
            - echo $KEYSTORE_FILE_BASE64 | base64 --decode > keys/$KEYSTORE_FILE
            - echo $GOOGLE_PLAY_API_KEY_BASE64 | base64 --decode > keys/$GOOGLE_PLAY_API_KEY
          artifacts:
            - keys/**

We use androidsdk/android-30 as the Docker image. That image has all the tools to build apps up to API 30 so no “manual” installation of build tools and writing code to accept the licenses.

In our case we want to build the master branch upon commit, hence the:

branches:
    master:

This is how to extract the key store file and the api key file from the repository variables:

• Create a new directory:

mkdir keys

• Decode the keystore file and write it to ./keys/$KEYSTORE_FILE ($KEYSTORE_FILE_BASE64 being the base64 encoded keystore file, $KEYSTORE_FILE being the name of the keystore file to create in the Docker container):

echo $KEYSTORE_FILE_BASE64 | base64 --decode > keys/$KEYSTORE_FILE

• Decode the api key file and write it to ./keys/$GOOGLE_PLAY_API_KEY ($GOOGLE_PLAY_API_KEY_BASE64 being the base64 encoded api key file, $GOOGLE_PLAY_API_KEY being the name of the api key file to create in the Docker container):

echo $GOOGLE_PLAY_API_KEY_BASE64 | base64 --decode > keys/$GOOGLE_PLAY_API_KEY

The artifacts tag defines which files are kept and are accessible by the subsequent pipeline steps. In this case we want to keep the two key files:

artifacts:
            - keys/**

2. Run unit tests

When run locally, Gradle reads the build arguments from the ~/.gradle/gradle.properties file. When run in the build pipeline we need to pass in the parameters as environment variables like so:

./gradlew -PKEYSTORE_FILE=../keys/$KEYSTORE_FILE
          -PKEYSTORE_PASSWORD=$KEYSTORE_PASSWORD
          -PKEY_ALIAS=$KEY_ALIAS
          -PKEY_PASSWORD=$KEY_PASSWORD
          -PGOOGLE_PLAY_API_KEY=../keys/$GOOGLE_PLAY_API_KEY
          test

PKEYSTORE_FILE creates an argument for Gradle with the name KEYSTORE_FILE and the value ../keys/$KEYSTORE_FILE with $KEYSTORE_FILE referencing the repository variable we defined earlier (translates to ../keys/playstore.keystore).

Putting everything together we get this step:

step:
    name: Run unit tests
    caches:
      - gradle
    script:
      - export GRADLE_OPTS='-XX:+UseG1GC -XX:MaxGCPauseMillis=1000 -Dorg.gradle.jvmargs="-Xmx2048m -XX:MaxPermSize=1024m -XX:ReservedCodeCacheSize=440m -XX:+UseCompressedOops -XX:+HeapDumpOnOutOfMemoryError -Dfile.encoding=UTF-8" -Dorg.gradle.parallel=false -Dorg.gradle.daemon=false -Dorg.gradle.configureondemand=true'
      - "./gradlew -PKEYSTORE_FILE=../keys/$KEYSTORE_FILE
                   -PKEYSTORE_PASSWORD=$KEYSTORE_PASSWORD
                   -PKEY_ALIAS=$KEY_ALIAS
                   -PKEY_PASSWORD=$KEY_PASSWORD
                   -PGOOGLE_PLAY_API_KEY=../keys/$GOOGLE_PLAY_API_KEY
                   test"
    artifacts:
      - app/build/outputs/**
      - app/build/reports/**

The Gradle options are optional. Of interest is mostly the org.gradle.daemon option. It prevents the script from failing if more than one Gradle task is run (e.g. by doing ./gradlew … clean test). For some reason the Gradle daemon is killed after the first task is done and the second one fails with:

Gradle build daemon disappeared unexpectedly (it may have been killed or may have crashed)

I’m sure there’s a better solution for this but for now the org.gradle.daemon is good enough for me.

3. Build and deploy the bundle

Building the app and deploying it to Google Play is simple with the Gradle Play Publisher plugin properly configured. The tasks publishFreeReleaseBundle and publishProReleaseBundle (with a Free and a Pro flavor of the app) will do all the heavy lifting. The pipeline step is:

- step:
    name: Build & deploy
    caches:
      - gradle
    script:
      - export GRADLE_OPTS='-XX:+UseG1GC -XX:MaxGCPauseMillis=1000 -Dorg.gradle.jvmargs="-Xmx2048m -XX:MaxPermSize=1024m -XX:ReservedCodeCacheSize=440m -XX:+UseCompressedOops -XX:+HeapDumpOnOutOfMemoryError -Dfile.encoding=UTF-8" -Dorg.gradle.parallel=false -Dorg.gradle.daemon=false -Dorg.gradle.configureondemand=true'
      - "./gradlew -PKEYSTORE_FILE=../keys/$KEYSTORE_FILE
                   -PKEYSTORE_PASSWORD=$KEYSTORE_PASSWORD
                   -PKEY_ALIAS=$KEY_ALIAS
                   -PKEY_PASSWORD=$KEY_PASSWORD
                   -PGOOGLE_PLAY_API_KEY=../keys/$GOOGLE_PLAY_API_KEY
                   -PBUILD_NUMBER=$BITBUCKET_BUILD_NUMBER
                   clean :app:publishFreeReleaseBundle :app:publishProReleaseBundle"
    artifacts:
      - app/build/outputs/

• We pass in -PBUILD_NUMBER=$BITBUCKET_BUILD_NUMBER to set the versionCode of the app

• The plugin publishes to the internal testing track by default, use the --track argument to publish to a different track (internal, alpha, beta, production)

4. Promote to production

These last step(s) are equally simple:

- parallel:
    - step:
        name: Promote free version
        caches:
          - gradle
        trigger: manual
        script:
          - export GRADLE_OPTS='-XX:+UseG1GC -XX:MaxGCPauseMillis=1000 -Dorg.gradle.jvmargs="-Xmx2048m -XX:MaxPermSize=1024m -XX:ReservedCodeCacheSize=440m -XX:+UseCompressedOops -XX:+HeapDumpOnOutOfMemoryError -Dfile.encoding=UTF-8" -Dorg.gradle.parallel=false -Dorg.gradle.daemon=false -Dorg.gradle.configureondemand=true'
          - "./gradlew -PKEYSTORE_FILE=../keys/$KEYSTORE_FILE
                   -PKEYSTORE_PASSWORD=$KEYSTORE_PASSWORD
                   -PKEY_ALIAS=$KEY_ALIAS
                   -PKEY_PASSWORD=$KEY_PASSWORD
                   -PGOOGLE_PLAY_API_KEY=../keys/$GOOGLE_PLAY_API_KEY
                   -PBUILD_NUMBER=$BITBUCKET_BUILD_NUMBER
                        promoteFreeReleaseArtifact --from-track internal --promote-track production --release-status completed"
    - step:
        name: Promote pro version
        caches:
          - gradle
        trigger: manual
        script:
          - export GRADLE_OPTS='-XX:+UseG1GC -XX:MaxGCPauseMillis=1000 -Dorg.gradle.jvmargs="-Xmx2048m -XX:MaxPermSize=1024m -XX:ReservedCodeCacheSize=440m -XX:+UseCompressedOops -XX:+HeapDumpOnOutOfMemoryError -Dfile.encoding=UTF-8" -Dorg.gradle.parallel=false -Dorg.gradle.daemon=false -Dorg.gradle.configureondemand=true'
          - "./gradlew -PKEYSTORE_FILE=../keys/$KEYSTORE_FILE
                   -PKEYSTORE_PASSWORD=$KEYSTORE_PASSWORD
                   -PKEY_ALIAS=$KEY_ALIAS
                   -PKEY_PASSWORD=$KEY_PASSWORD
                   -PGOOGLE_PLAY_API_KEY=../keys/$GOOGLE_PLAY_API_KEY
                   -PBUILD_NUMBER=$BITBUCKET_BUILD_NUMBER
                        promoteProReleaseArtifact --from-track internal --promote-track production --release-status inProgress --user-fraction .5"

This promotes the free version from the internal testing track to the production track for 100% of all users:

promoteFreeReleaseArtifact --from-track internal --promote-track production --release-status completed

This promotes the pro version from the internal testing track to the production track for 50% of all users (staged rollout):

promoteProReleaseArtifact --from-track internal --promote-track production --release-status inProgress --user-fraction .5

We define a manual trigger

trigger: manual

and thus the pipeline needs human intervention to run these steps. If you want automatic deployment to the production track, just remove that manual trigger. I prefer to do at least a quick smoke test before hitting the publish button.

For reference here’s the complete bitbucket-pipelines.yml file: https://gist.github.com/1gravity/d5a160094e5408fbff8f54c27b6c9e5c.

Happy coding!

Previously published behind a paywall: https://medium.com/nerd-for-tech/ci-cd-for-android-using-bitbucket-pipelines-and-gradle-play-publisher-f00d6047ecb5