Alexey Alter-Pesotskiy

4 minute read

Optimizing Android Test Runs: Parsing Test Names and Implementing Orchestration

Intro

Today I’d like to talk about test orchestration on Android. And about parsing test names in particular, as one of the main pitfalls along the way.

Imagine you have hundreds or thousands of tests that take more than an hour in a raw. You better start thinking how to speed them up or run them in parallel.

There are multiple options to parallel the tests:

  • Parallelization at the CI level: when you split the tests and run them on different jobs (e.g. via GitHub Actions matrix)
  • Parallelization at the test runner level: when you split the tests and run them on different devices as part of a single CI job (some test runners are smart and offer this out of the box, some are dumb and should be executed in different threads)
  • Parallelization at the test framework level: really depends on the framework, but usually this means splitting the tests into suites
  • <…>

Thus, most options require a list of test names or test classes in order to run them in parallel. But how can we parse the tests, get their names, and not miss a single one? On Android, Dex Test Parser comes to the rescue.

All the code examples below are presented using fastlane lanes for the simplicity. Feel free to extract the main idea and translate it to your favourite language/DSL if required.

Precondition

I assume you already have an apk with your tests and an apk with your test target. If not, you can do something like this:

./gradlew assembleDebugAndroidTest assembleDebug

or using fastlane:

lane :build do
  gradle(tasks: ['assembleDebugAndroidTest', 'assembleDebug'])
end

Install test parser

First of all, let’s install dex-test-parser. According to the docs, this can be done using Maven, Gradle or manually. I choose the latter option:

wget -O test-parser.jar 'https://linkedin.jfrog.io/artifactory/open-source/com/linkedin/dextestparser/parser/2.2.1/parser-2.2.1-all.jar'

or using fastlane:

lane :install_test_parser do
  version = '2.2.1'
  output = 'test-parser.jar'
  url = "https://linkedin.jfrog.io/artifactory/open-source/com/linkedin/dextestparser/parser/#{version}/parser-#{version}-all.jar"
  sh("wget -O #{output} '#{url}' 2>/dev/null") unless File.exist?(output)
end

Parse the tests

Now let’s parse the tests using dex-test-parser:

java -jar test-parser.jar "${path_to_test_apk_path}" ./

This produces an AllTests.txt file that contains all absolute test paths.

Batch the tests

To batch the tests we need to know the total amount of batches and what is the current batch number:

lane :batch_tests do |options|
  current_batch = []
  if options[:batch_number] && options[:batch_count]
    sh("java -jar test-parser.jar #{options[:test_apk_path]} ./")
    test_names = File.read('AllTests.txt').split
    current_batch = test_names.each_slice((test_names.size.to_f / options[:batch_count].to_i).ceil).to_a[options[:batch_number].to_i]
    UI.success("Contents of the current batch: #{current_batch}")
  end
  current_batch
end

For instance, we have 3 batches in total and we want to get the tests for the first batch:

bundle exec fastlane batch_tests batch_count:3 batch_number:0 test_apk_path:"path/to/test.apk"

Keep in mind that the batches will always be identical and equally divided. If new tests are added, they will be distributed proportionally between the batches at the upcoming run.

Service the tests

To be able to run the tests with pre-built apks we have to install test services:

lane :install_test_services do
  device_api_level = sh('adb shell getprop ro.build.version.sdk').strip.to_i
  force_queryable = device_api_level >= 30 ? '--force-queryable' : ''
  version = '1.5.0'

  output = 'test-services.apk'
  url = "https://dl.google.com/dl/android/maven2/androidx/test/services/test-services/#{version}/test-services-#{version}.apk"
  sh("wget -O #{output} '#{url}' 2>/dev/null") unless File.exist?(output)
  sh("adb install #{force_queryable} -r #{output}")

  output = 'orchestrator.apk'
  url = "https://dl.google.com/dl/android/maven2/androidx/test/services/orchestrator/#{version}/orchestrator-#{version}.apk"
  sh("wget -O #{output} '#{url}' 2>/dev/null") unless File.exist?(output)
  sh("adb install #{force_queryable} -r #{output}")
end

Run the tests

Let’s sum it up and add the cherry on top:

lane :test do
  install_test_services
  install_test_parser

  apk_path = "path/to/app.apk"
  test_apk_path = "path/to/test.apk"

  sh("adb install -r #{apk_path}")
  sh("adb install -r #{test_apk_path}")

  current_test_batch = batch_tests(
    batch_number: options[:batch_number],
    batch_count: options[:batch_count],
    test_apk_path: test_apk_path
  )

  app_package_name = 'com.example'
  test_package_name = "#{app_package_name}.test"
  test_runner_package_name = 'androidx.test.runner.AndroidJUnitRunner'
  test_orchestrator_package_name = 'androidx.test.orchestrator/.AndroidTestOrchestrator'
  androidx_test_services_path = sh('adb shell pm path androidx.test.services').strip

  result = sh(
    "adb shell 'CLASSPATH=#{androidx_test_services_path}' " \
    'app_process / androidx.test.services.shellexecutor.ShellMain am instrument -w '\
    '-e clearPackageData true ' \
    "-e targetInstrumentation #{test_package_name}/#{test_runner_package_name} " \
    "-e class #{current_test_batch.join(',')} #{test_orchestrator_package_name}"
  )
  UI.user_error!('Tests have failed!') if result.include?('Failures')
end

Showcase

On GitHub Actions it would look like this:

name: CI

on:
  push:

jobs:
  build:
    name: Build
    runs-on: ubuntu-24.04
    steps:
      - uses: actions/[email protected]
      - uses: ruby/setup-ruby@v1
        with:
          ruby-version: 3.1
          bundler-cache: true
      - uses: actions/setup-java@v4
        with:
          distribution: adopt
          java-version: 17
      - name: Enable KVM group permissions
        run: |
          echo 'KERNEL=="kvm", GROUP="kvm", MODE="0666", OPTIONS+="static_node=kvm"' | sudo tee /etc/udev/rules.d/99-kvm4all.rules
          sudo udevadm control --reload-rules
          sudo udevadm trigger --name-match=kvm
      - run: bundle exec fastlane build
      - uses: actions/[email protected]
        with:
          name: apks
          path: |
            ./**/path/to/app.apk
            ./**/path/to/test.apk

  test:
    name: Test
    runs-on: ubuntu-24.04
    needs: build
    strategy:
      matrix:
        include:
          - batch_number: 0
          - batch_number: 1
          - batch_number: 2
    steps:
      - uses: actions/[email protected]
      - uses: ruby/setup-ruby@v1
        with:
          ruby-version: 3.1
          bundler-cache: true
      - uses: actions/setup-java@v4
        with:
          distribution: adopt
          java-version: 17
      - uses: actions/[email protected]
        with:
          name: apks
      - name: Enable KVM group permissions
        run: |
          echo 'KERNEL=="kvm", GROUP="kvm", MODE="0666", OPTIONS+="static_node=kvm"' | sudo tee /etc/udev/rules.d/99-kvm4all.rules
          sudo udevadm control --reload-rules
          sudo udevadm trigger --name-match=kvm
      - uses: reactivecircus/android-emulator-runner@v2
        with:
          api-level: 33
          disable-animations: true
          profile: pixel
          arch : x86_64
          emulator-options: -no-snapshot-save -no-window -no-audio -no-boot-anim -gpu swiftshader_indirect -camera-back none -camera-front none
          script: bundle exec fastlane test batch_count:$ batch_number:$
      - uses: actions/[email protected]
        if: failure()
        with:
          name: report
          path: ./**/build/reports/androidTests/*

If you want to randomize things a bit, consider:

  • moving the tests’ parsing step to the Build job
  • shuffling test names in AllTests.txt file
  • uploading it as an artifact alongside the apks
  • downloading it as part of the matrix step

After that, it should be pretty much the same as the initial plan.

Resources

Updated: