Alexey Alter-Pesotskiy

2 minute read

Running tests on multiple iOS versions on GitHub Actions

As with everything, there are at least two ways to run tests in parallel on multiple iOS versions:

  • Parallelization at the CI level
  • Parallelization at the test framework level

In our case, the CI will be GitHub Actions and the test framework will be Fastlane - the wrapper on top of XCTest. Just to be clear, we are going to use GitHub Actions as well as Fastlane in both scenarios, the only difference is where exactly we will parallelize the tests.

Parallelization at the CI level

Let’s create a GitHub Actions workflow, that will:

  • be executed by cron
  • use a matrix with different iOS versions, devices and macOS versions if needed
  • install the required iOS Simulator
  • pass a device name and an iOS version to Fastlane
name: Cron

on:
  schedule:
    # Running the pipeline "At 03:00 every night"
    - cron: '0 3 * * *'

jobs:
  test:
    name: Run tests on multiple iOS versions
    strategy:
      matrix:
        ios: [12.4, 13.7, 14.5, latest]
        device: ["iPhone 8"]
        macos: ["macos-12"]
      fail-fast: false
    runs-on: ${{ matrix.macos }}
    steps:
    - uses: actions/checkout@v2

    # Grabbing the full name of a macOS image
    - run: echo "IMAGE=${ImageOS}-${ImageVersion}" >> $GITHUB_ENV

    # Caching ruby gems
    - uses: actions/cache@v2
      with:
        path: vendor/bundle
        key: ${{ env.IMAGE }}-gem-${{ hashFiles('**/Gemfile.lock') }}
        restore-keys: ${{ env.IMAGE }}-gem-

    # Installing ruby gems
    - run: bundle install

    # Installing iOS Simulator if required
    - name: Install iOS Simulator ${{ matrix.ios }}
      if: ${{ matrix.ios != 'latest' }}
      run: xcversion simulators --install='iOS ${{ matrix.ios }}' || true

    # Here we go
    - name: Run Tests
      run: bundle exec fastlane cron_checks device:"${{ matrix.device }}" ios:"${{ matrix.ios }}"

Then let’s create a Fastlane lane, that will:

  • take in a device name and an iOS version
  • run tests on the provided device
lane :cron_checks do |options|
  device_name = options[:ios] == 'latest' ? options[:device] : "#{options[:device]} (#{ios})"

  scan(
    project: 'SampleApp.xcodeproj',
    scheme: 'SampleApp',
    testplan: 'SampleAppTestPlan',
    configuration: 'Debug',
    devices: [device_name]
  )
end

This way, the tests are run on multiple iOS versions in parallel in different matrix jobs on GitHub Actions.

Parallelization at the test framework level

Let’s create a GitHub Actions workflow, that will:

  • be executed by cron
  • pass a device name to Fastlane
name: Cron

on:
  schedule:
    # Running the pipeline "At 03:00 every night"
    - cron: '0 3 * * *'

jobs:
  test:
    name: Run tests on multiple iOS versions
    runs-on: macos-12
    steps:
    - uses: actions/checkout@v2

    # Grabbing the full name of a macOS image
    - run: echo "IMAGE=${ImageOS}-${ImageVersion}" >> $GITHUB_ENV

    # Caching ruby gems
    - uses: actions/cache@v2
      with:
        path: vendor/bundle
        key: ${{ env.IMAGE }}-gem-${{ hashFiles('**/Gemfile.lock') }}
        restore-keys: ${{ env.IMAGE }}-gem-

    # Installing ruby gems
    - run: bundle install

    # Here we go
    - name: Run Tests
      run: bundle exec fastlane cron_checks device:"iPhone 8"

Then let’s create a Fastlane lane, that will:

  • take in a device name
  • install the required iOS Simulators
  • run tests on multiple iOS versions
lane :cron_checks do |options|
  ios = ['12.4', '13.7', '14.5', 'latest']

  # Installing iOS Simulators if required
  ios.each { |version| sh("xcversion simulators --install='iOS #{version}' || true") if version != ios.last }

  # Setting up device names
  devices = ios.reverse.drop(1).reverse.map { |v| "#{options[:device]} (#{v})" }.append(options[:device])

  scan(
    project: 'SampleApp.xcodeproj',
    scheme: 'SampleApp',
    testplan: 'SampleAppTestPlan',
    configuration: 'Debug',
    devices: devices
  )
end

Thus, the tests are run on multiple iOS versions in parallel in the same job on GitHub Actions.

Updated: