CI/CD for Android Guide: Bitbucket Pipelines and Gradle Play Publisher

Motivation There are many ways to build, test, deploy and publish an Android app. You can do it . manually It’s not hard or time-consuming to create a signed APK or Bundle in Android Studio, upload it to Google Play and promote it through the different testing tracks up to production. I did it for many years. I thought the time automation saves won’t compensate for the effort to set up a working pipeline. I was wrong. Automation doesn’t just save time, it also makes the process more reliable, less error-prone (to human error) and encourages to deploy/publish more often. In general, the development cycle is sped up, not just when it comes to bug fixes but also feature releases. Why wait to bundle a big release when features can be pushed out to the customer by a simple pr/merge? You can use . Fastlane I did that for most of my apps. It’s convenient to type fastlane build_deploy and have the app built, signed, and published to Google Play automatically. It’s probably good enough if you’re the only developer on the team. For teams there are some major issues though: Fastlane needs to be installed on each build machine. With multiple developers or if you use multiple computers to develop, you need to setup and maintain it multiple times. Fastlane’s documentation isn’t great imo and especially the setup on Linux is not straight-forward (time consuming). Having multiple build machines also impacts the reproducibility of the build, differences in the Fastlane dependencies (Ruby libs) can result in different results. Signing and deploying requires the signing/upload key and also the Google Play API key. As few people as possible should have access to those keys. It’s possible to create new keys every time an employee with access to the keys leaves the company (provided upload keys are used to sign the app) but it’s certainly not an ideal process. One of my requirements was to start the build process upon merge. Fastlane isn’t the tool for that unless it’s combined with other tools which leads us to… You can use . Jenkins Jenkins is a great tool. It’s very flexible and we used it (in combination with Rundeck) successfully to build and deploy a microservice backend to an AWS hosted Docker/Kubernetes environment. We also had a dedicated DevOps team to maintain it. If you don’t want to host your own CI/CD infrastructure and don’t have experts taking care of the installation and configuration, then Jenkins is probably not the right tool. You can use . There are many cloud based CI/CD solutions like , or to name just a few. I’m sure they are all great and integrate with your preferred Git provider but it’s another tool to integrate (while Bitbucket Pipeline is obviously tightly integrated already) and they seem to be sledge hammers for the requirements I had which are: CircleCI Travis CI TeamCity Requirements upon merge into a specific branch (master in my case) Trigger Run (I have ui tests as well but they aren’t part of the pipeline yet) unit tests Run and fail the build if errors are detected lint Build a of the app as bundles free and a paid version the app Sign Automatically increment the build number Upload app (release notes, description, screen shots) if needed metadata the app to the Deploy internal testing track from the internal testing track to the production track by app flavor (free/paid in my case) Manual promotion (signing key, Google Play API key, passwords) can’t be retrieved by developers with access to the repo Secrets It turns out and are the tools I needed to make this happen. Let’s see how. Bitbucket Gradle Play Publisher Overview There are four steps to setup the pipeline: We need programmatic access to Google Play to publish and promote apps (plus manage the meta data) -> we need an API key for the . Google Play Developer API We need to configure the to use a signing configuration that reads the signing information from environment variables. Gradle build We need to configure the plugin to interact with Google Play (upload, publish apps and manage meta data).We need to configure to tie everything together. Gradle Play Publisher Bitbucket Pipeline The pipeline configuration defines the trigger (upon merge), the build steps (Gradle), the deploy steps (Gradle Play Publisher) and injects the environment variables (secrets, build number). Google Play Developer API The official documentation explains all steps in detail: . https://developers.google.com/android-publisher/getting_started TL;TLTR (too long; too lazy to read): Go to your as owner of the account and open the page: Google Play Developer Console API access Accept the Terms of Service (if not done yet) Create a new Google Cloud project if you haven’t created one already (otherwise link an existing one). Under click on and open the link that leads to the Google Cloud Platform: Service Accounts “Create new service account” In Google Cloud Platform click on : “CREATE SERVICE ACCOUNT” Pick a meaningful name and description before hitting the button: “CREATE” The account needs the role : “Service Account User” You don’t need to grant user access to the new service account, Google Cloud adds the required users automatically with the correct permissions (a Google Play service and your own user) so just hit : “DONE” Next, you need to create an API key for the account. Open the actions menu (the three dots) and select : “Manage keys” Under select : “ADD Key” “Create new key” Create a JSON key: After hitting the button, the key file will be downloaded to your computer. I recommend to rename the file to make its purpose more obvious: “CREATE” Now you’re done in Google Cloud Platform and you can go back to the Google Play Console (to the API access screen). The newly created account should appear under (hit the button). Click on : “Service accounts” “Refresh service account” “Grant access” Click on and select all apps you want to manage with this service account: “Add app” The Account permissions are already set correctly so that the service can manage all release related activities (create releases including publication to production, management of meta data etc.). Click on and you’re done. We will use the Gradle Play Publisher plugin to validate API key setup later on. “Invite user” Gradle Build The Gradle build needs to be configured to include a that reads the secrets from environment variables (or the gradle.properties file in your ). If you already have one then you can skip this chapter. signing configuration ~/.gradle folder The assumption is that you have a published app in Google Play and that you have access to the keystore and a signing key (or upload key) including the passwords. For a local build the location of the keystore, the keystore password, the key alias and the key password will be configured in your file. ~/.gradle/gradle.properties If you don’t have a file, please create one and add these four parameters (the bold part needs to be configured to fit your setup): ~/.gradle/gradle.properties = = = = KEYSTORE_FILE /path to the keystore file/playstore.keystore KEYSTORE_PASSWORD keystore password KEYSTORE_KEY_ALIAS key alias KEYSTORE_KEY_PASSWORD key password : don’t use ~ for your home directory but use absolute paths. ~ works in a shell context but not with Gradle, Gradle Play Publisher and Bitbucket Pipeline. Note Create a signing config in your app’s gradle.build file: signingConfigs { release { storeFile file(KEYSTORE_FILE) storePassword KEYSTORE_PASSWORD keyAlias KEYSTORE_KEY_ALIAS keyPassword KEYSTORE_KEY_PASSWORD } } Add the signing config to the build type: buildTypes { debug { } release { signingConfig signingConfigs.release } } // debug build type configuration ... // release build type configuration ... If the signing configuration is correct then the following command should run and create one or more aab files in your folder: build/outputs/bundle bundleRelease ./gradlew Build Number One of the requirements is the auto-increment of build numbers / versionCode. We will use Bitbucket’s to set an environment variable that defines the versionCode. In order to process this environment variable, change your file from: $ BITBUCKET_BUILD_NUMBER build.gradle versionCode 124 to: versionCode project.hasProperty( ) ? project[ ].toInteger() : 'BUILD_NUMBER' 'BUILD_NUMBER' 124 Last but not least we need to set the initial value for as it needs to be higher than the last used versionCode. Please follow this article to do so: . $ BITBUCKET_BUILD_NUMBER https://support.atlassian.com/bitbucket-cloud/docs/set-a-new-value-for-the-pipelines-build-number/ Gradle Play Publisher While we are now able to build the app and create a signed bundle (or apk), we still need to configure to publish the signed app to Google Play (plus manage the meta data like screen shots, description etc.). Gradle Play Publisher We could also use Fastlane for this but I don’t recommend to go down that path (been there, done it). Just trust me ;-) Setting up the Gradle Play Publisher plugin is easy (see also ): https://github.com/Triple-T/gradle-play-publisher Add the plugin to the app’s file: build.gradle plugins { id id version } 'com.android.application' 'com.github.triplet.play' '3.3.0' // other plugins... Add a configuration block to the app’s file (after the android block): build.gradle android { ... } play { serviceAccountCredentials = file(GOOGLE_PLAY_API_KEY) } You’ll notice the parameter. It’s a reference to the api key file we got when setting up the key for the service account -> . The parameter needs to be defined in the file (analogous the signing config parameters): GOOGLE_PLAY_API_KEY google-play-api-key.json ~/.gradle/gradle.properties = GOOGLE_PLAY_API_KEY /path to the api key file/google-play-api-key.json If everything was setup properly, the following command when run from the root directory of your app will download the app’s meta data: bootstrap ./gradlew Bitbucket Pipeline To get started with Bitbucket Pipeline and to create your first pipeline, please follow this excellent article: https://www.rockandnull.com/android-continuous-integration-bitbucket/ After going through that article you should now have a file in your app’s root directory. bitbucket-pipelines.yml What we want now is to create this specific pipeline: It consists of five steps (each step runs a separate Docker container): Create secret files (keystore and API key file) and make then available to the pipeline Run unit tests Build the app bundle(s) and deploy them to the internal testing track Promote the free version of the app to the production track (manual step) Promote the pro/paid version of the app to the production track (manual step) 1. Create secret files Neither the signing key (keystore + key) nor the Google Play API key should be added to the repo, otherwise every developer with repo access would be able to read them. Bitbucket has the ability to define repository variables. By default they are encrypted and can’t be read by regular users but only by scripts. We will use this feature to define the five arguments our build/publish scripts need: KEYSTORE_FILE KEYSTORE_PASSWORD KEYSTORE_KEY_ALIAS KEYSTORE_KEY_PASSWORD GOOGLE_PLAY_API_KEY It’s easy to define the three values for and since they are just text values. To do so go to the and scroll down to . Enter all three variables with the correct values: KEYSTORE_PASSWORD, KEYSTORE_KEY_ALIAS KEYSTORE_KEY_PASSWORD “Repository settings” “Repository variables” To store the in a repository variable we encode the files with base64. The build pipeline will decode the text and recreate the original files. KEYSTORE_FILE and the GOOGLE_PLAY_API_KEY Run the following commands to encode the two files: google-play-api- .json playstore.keystore base64 key base64 Copy the base64 strings and create repository variables in Bitbucket. The strings should look somewhat like this (much longer though): YmFzZTY0IGdvb2dsZS1wbGF5LWFwaS1rZXkuanNvbg== I also created two variables and to define the files names used for the decoded secrets: KEYSTORE_FILE GOOGLE_PLAY_API_KEY Now we’re ready to define the first step of the actual pipeline in the file. bitbucket-pipelines.yml image: androidsdk/android-30 pipelines: branches: master: - step: name: Create keystore and API key script: # create the keystore file and the google play api key file - mkdir keys - echo $KEYSTORE_FILE_BASE64 | base64 --decode > keys/$KEYSTORE_FILE - echo $GOOGLE_PLAY_API_KEY_BASE64 | base64 --decode > keys/$GOOGLE_PLAY_API_KEY artifacts: - keys/** We use as the Docker image. That image has all the tools to build apps up to API 30 so no “manual” installation of build tools and writing code to accept the licenses. androidsdk/android-30 In our case we want to build the master branch upon commit, hence the: branches: master: This is how to extract the key store file and the api key file from the repository variables: Create a new directory: mkdir keys Decode the keystore file and write it to ( being the base64 encoded keystore file, being the name of the keystore file to create in the Docker container): ./keys/$KEYSTORE_FILE $KEYSTORE_FILE_BASE64 $KEYSTORE_FILE echo $KEYSTORE_FILE_BASE64 | base64 --decode > keys/$KEYSTORE_FILE Decode the api key file and write it to ( GOOGLE_PLAY_API_KEY_BASE64 being the base64 encoded api key file, GOOGLE_PLAY_API_KEY being the name of the api key file to create in the Docker container): ./keys/$GOOGLE_PLAY_API_KEY $ $ echo $GOOGLE_PLAY_API_KEY_BASE64 | base64 --decode > keys/$GOOGLE_PLAY_API_KEY The artifacts tag defines which files are kept and are accessible by the subsequent pipeline steps. In this case we want to keep the two key files: artifacts: - keys /** 2. Run unit tests When run locally, Gradle reads the build arguments from the file. When run in the build pipeline we need to pass in the parameters as environment variables like so: ~/.gradle/gradle.properties ./gradlew -PKEYSTORE_FILE=../keys/$KEYSTORE_FILE - - - - test PKEYSTORE_PASSWORD=$KEYSTORE_PASSWORD PKEY_ALIAS=$KEY_ALIAS PKEY_PASSWORD=$KEY_PASSWORD PGOOGLE_PLAY_API_KEY=../keys/$GOOGLE_PLAY_API_KEY creates an argument for Gradle with the name and the value with $ referencing the repository variable we defined earlier (translates to . PKEYSTORE_FILE KEYSTORE_FILE ../keys/$KEYSTORE_FILE KEYSTORE_FILE ../keys/playstore.keystore) Putting everything together we get this step: step: name: Run unit tests caches: - gradle script: - export GRADLE_OPTS='-XX:+UseG1GC -XX:MaxGCPauseMillis=1000 -Dorg.gradle.jvmargs="-Xmx2048m -XX:MaxPermSize=1024m -XX:ReservedCodeCacheSize=440m -XX:+UseCompressedOops -XX:+HeapDumpOnOutOfMemoryError -Dfile.encoding=UTF-8" -Dorg.gradle.parallel=false -Dorg.gradle.daemon=false -Dorg.gradle.configureondemand=true' - "./gradlew -PKEYSTORE_FILE=../keys/$KEYSTORE_FILE -PKEYSTORE_PASSWORD=$KEYSTORE_PASSWORD -PKEY_ALIAS=$KEY_ALIAS -PKEY_PASSWORD=$KEY_PASSWORD -PGOOGLE_PLAY_API_KEY=../keys/$GOOGLE_PLAY_API_KEY test" artifacts: - app/build/outputs/** - app/build/reports/** The Gradle options are optional. Of interest is mostly the org.gradle.daemon option. It prevents the script from failing if more than one Gradle task is run (e.g. by doing ./gradlew … clean test). For some reason the Gradle daemon is killed after the first task is done and the second one fails with: Gradle build daemon disappeared unexpectedly (it may have been killed or may have crashed) I’m sure there’s a better solution for this but for now the org.gradle.daemon is good enough for me. 3. Build and deploy the bundle Building the app and deploying it to Google Play is simple with the Gradle Play Publisher plugin properly configured. The tasks and (with a Free and a Pro flavor of the app) will do all the heavy lifting. The pipeline step is: publishFreeReleaseBundle publishProReleaseBundle - step: name: Build & deploy caches: - gradle script: - export GRADLE_OPTS='-XX:+UseG1GC -XX:MaxGCPauseMillis=1000 -Dorg.gradle.jvmargs="-Xmx2048m -XX:MaxPermSize=1024m -XX:ReservedCodeCacheSize=440m -XX:+UseCompressedOops -XX:+HeapDumpOnOutOfMemoryError -Dfile.encoding=UTF-8" -Dorg.gradle.parallel=false -Dorg.gradle.daemon=false -Dorg.gradle.configureondemand=true' - "./gradlew -PKEYSTORE_FILE=../keys/$KEYSTORE_FILE -PKEYSTORE_PASSWORD=$KEYSTORE_PASSWORD -PKEY_ALIAS=$KEY_ALIAS -PKEY_PASSWORD=$KEY_PASSWORD -PGOOGLE_PLAY_API_KEY=../keys/$GOOGLE_PLAY_API_KEY -PBUILD_NUMBER=$BITBUCKET_BUILD_NUMBER clean :app:publishFreeReleaseBundle :app:publishProReleaseBundle" artifacts: - app/build/outputs/ We pass in to set the of the app -PBUILD_NUMBER=$BITBUCKET_BUILD_NUMBER versionCode The plugin publishes to the internal testing track by default, use the -- argument to publish to a different track (internal, alpha, beta, production) track 4. Promote to production These last step(s) are equally simple: - parallel: - step: name: Promote free version caches: - gradle trigger: manual script: - export GRADLE_OPTS='-XX:+UseG1GC -XX:MaxGCPauseMillis=1000 -Dorg.gradle.jvmargs="-Xmx2048m -XX:MaxPermSize=1024m -XX:ReservedCodeCacheSize=440m -XX:+UseCompressedOops -XX:+HeapDumpOnOutOfMemoryError -Dfile.encoding=UTF-8" -Dorg.gradle.parallel=false -Dorg.gradle.daemon=false -Dorg.gradle.configureondemand=true' - "./gradlew -PKEYSTORE_FILE=../keys/$KEYSTORE_FILE -PKEYSTORE_PASSWORD=$KEYSTORE_PASSWORD -PKEY_ALIAS=$KEY_ALIAS -PKEY_PASSWORD=$KEY_PASSWORD -PGOOGLE_PLAY_API_KEY=../keys/$GOOGLE_PLAY_API_KEY -PBUILD_NUMBER=$BITBUCKET_BUILD_NUMBER promoteFreeReleaseArtifact --from-track internal --promote-track production --release-status completed" - step: name: Promote pro version caches: - gradle trigger: manual script: - export GRADLE_OPTS='-XX:+UseG1GC -XX:MaxGCPauseMillis=1000 -Dorg.gradle.jvmargs="-Xmx2048m -XX:MaxPermSize=1024m -XX:ReservedCodeCacheSize=440m -XX:+UseCompressedOops -XX:+HeapDumpOnOutOfMemoryError -Dfile.encoding=UTF-8" -Dorg.gradle.parallel=false -Dorg.gradle.daemon=false -Dorg.gradle.configureondemand=true' - "./gradlew -PKEYSTORE_FILE=../keys/$KEYSTORE_FILE -PKEYSTORE_PASSWORD=$KEYSTORE_PASSWORD -PKEY_ALIAS=$KEY_ALIAS -PKEY_PASSWORD=$KEY_PASSWORD -PGOOGLE_PLAY_API_KEY=../keys/$GOOGLE_PLAY_API_KEY -PBUILD_NUMBER=$BITBUCKET_BUILD_NUMBER promoteProReleaseArtifact --from-track internal --promote-track production --release-status inProgress --user-fraction .5" This promotes the free version from the internal testing track to the production track for 100% of all users: promoteFreeReleaseArtifact -- -track internal --promote-track production --release-status completed from This promotes the pro version from the internal testing track to the production track for 50% of all users (staged rollout): promoteProReleaseArtifact -- -track internal --promote-track production --release-status inProgress --user-fraction from .5 We define a manual trigger trigger: manual and thus the pipeline needs human intervention to run these steps. If you want automatic deployment to the production track, just remove that manual trigger. I prefer to do at least a quick smoke test before hitting the publish button. For reference here’s the complete file: . bitbucket-pipelines.yml https://gist.github.com/1gravity/d5a160094e5408fbff8f54c27b6c9e5c Happy coding! https://medium.com/nerd-for-tech/ci-cd-for-android-using-bitbucket-pipelines-and-gradle-play-publisher-f00d6047ecb5 Previously published behind a paywall: