Video recording of failed tests on iOS

November 12, 2022 2 minute read

It’s no secret that tests can fail with or without a reason, especially those flaky UI ones on CI. Would be great to know exactly why they failed, wouldn’t it?

XCTest runs in the sandbox, which makes it hard to get anything out of it, but let’s hack it and record the failed tests!

Precondition

The easiest way to record a video is to run simctl command from the Terminal:

xcrun simctl io booted recordVideo "filename"

But how to run Shell commands from XCTest? Any web server will do the trick. Let’s use my favourite one:

gem install sinatra

Code outside the sandbox

Create a server.rb file
Fill it with the code below

require 'sinatra'
require 'fileutils'

post '/record_video/:udid/:test_name' do
  recordings_dir = 'recordings'
  video_base_name = "#{recordings_dir}/#{params['test_name']}"
  recordings = (0..Dir["#{recordings_dir}/*"].length + 1).to_a
  body = JSON.parse(request.body.read)
  FileUtils.mkdir_p(recordings_dir)

  video_file = ''
  if body['delete']
    recordings.reverse_each do |i|
      video_file = "#{video_base_name}_#{i}.mp4"
      break if File.exist?(video_file)
    end
  else
    recordings.each do |i|
      video_file = "#{video_base_name}_#{i}.mp4"
      break unless File.exist?(video_file)
    end
  end

  if body['stop']
    simctl_processes = `pgrep simctl`.strip.split("\n")
    simctl_processes.each { |pid| `kill -s SIGINT #{pid}` }
    File.delete(video_file) if body['delete'] && File.exist?(video_file)
  else
    puts `xcrun simctl io #{params['udid']} recordVideo --codec h264 --force #{video_file} &`
  end
end

Comments

Web server has only one endpoint: /record_video/:udid/:test_name
Simulator UDID and test name are passed as URL parameters
- :udid is used to run simctl command
- :test_name is used to name the video file
This endpoint gets POST requests with JSON:
- stop: Bool - should recording be stopped (will kill the recording process)
- delete: Bool - should recording be deleted (will delete the recording if test passed)
Web server saves the video in the recordings directory
- It’s smart enough to find the next available file name if test is run multiple times (e.g. retry strategy)

Code in the sandbox

Open an Xcode project
Create a test file
Fill it with the code below

import XCTest

final class SampleTest: XCTestCase {

  override func setUpWithError() throws {
    try super.setUpWithError()
    recordVideo()
    XCUIApplication().launch()
  }

  override func tearDownWithError() throws {
      recordVideo(stop: true)
      XCUIApplication().terminate()
      try super.tearDownWithError()
  }

  func testPass() {
    XCTAssertTrue(true)
  }

  func testFail() {
    XCTAssertTrue(false)
  }

  func recordVideo(stop: Bool = false) {
    let testName = String(name.split(separator: " ")[1].dropLast())
    let json: [String: Any] = ["delete": !isTestFailed(), "stop": stop]
    let udid = ProcessInfo.processInfo.environment["SIMULATOR_UDID"] ?? ""
    let urlString = "http://localhost:4567/record_video/\(udid)/\(testName)"
    guard let url = URL(string: urlString) else { return }

    var request = URLRequest(url: url)
    request.httpMethod = "POST"
    request.httpBody = try? JSONSerialization.data(withJSONObject: json, options: [])
    URLSession.shared.dataTask(with: request).resume()
  }

  func isTestFailed() -> Bool {
    if let testRun = testRun {
      let failureCount = testRun.failureCount + testRun.unexpectedExceptionCount
      return failureCount > 0
    }
    return false
  }
}

Comments

There are two tests: testPass and testFail, so we can make sure that the recording is saved only if the test failed. I suppose there is no need to keep recording if the test passed
setUpWithError and tearDownWithError are the best places to start and stop the recording
recordVideo method sends the request to the web server with all the required details
- host: localhost
- port: 4567 (default for Sinatra)
- endpoint: /record_video/:udid/:test_name
isTestFailed method is used to get a test result
Dictionary extension with jsonToString method is used to convert JSON body to String

Result

Run the web server:
```
 ruby server.rb
```
Run the tests
- from Xcode or from the Terminal, whatever you prefer
- on any iOS Simulator
Check out the recordings folder, you should only find the video from the failed test there