3 minute read

Preview

It’s no secret that tests can fail with or without a reason, especially those flaky UI ones on CI. Would be great to know exactly why they failed, wouldn’t it?

XCTest runs in the sandbox, which makes it hard to get anything out of it, but let’s hack it and record the failed tests!

Precondition

The easiest way to record a video is to run simctl command from the Terminal:

xcrun simctl io booted recordVideo "filename"

But how to run Shell commands from XCTest? Any web server will do the trick. Let’s use my favorite one:

gem install sinatra

Code outside the sandbox

  1. Create a server.rb file
  2. Fill it with the code below
require 'sinatra'
require 'fileutils'

post '/record_video/:udid/:test_name' do
  recordings_dir = 'recordings'
  video_base_name = "#{recordings_dir}/#{params['test_name']}"
  recordings = (0..Dir["#{recordings_dir}/*"].length + 1).to_a
  body = JSON.parse(request.body.read)
  FileUtils.mkdir_p(recordings_dir)

  video_file = ''
  if body['delete']
    recordings.reverse_each do |i|
      video_file = "#{video_base_name}_#{i}.mp4"
      break if File.exist?(video_file)
    end
  else
    recordings.each do |i|
      video_file = "#{video_base_name}_#{i}.mp4"
      break unless File.exist?(video_file)
    end
  end

  if body['stop']
    simctl_processes = `pgrep simctl`.strip.split("\n")
    simctl_processes.each { |pid| `kill -s SIGINT #{pid}` }
    File.delete(video_file) if body['delete'] && File.exist?(video_file)
  else
    puts `xcrun simctl io #{params['udid']} recordVideo --codec h264 --force #{video_file} &`
  end
end

Comments

  1. Web server has only one endpoint: /record_video/:udid/:test_name
  2. Simulator UDID and test name are passed as URL parameters
    • :udid is used to run simctl command
    • :test_name is used to name the video file
  3. This endpoint gets POST requests with JSON:
    • stop: Bool - should recording be stopped (will kill the recording process)
    • delete: Bool - should recording be deleted (will delete the recording if test passed)
  4. Web server saves the video in the recordings directory
    • It’s smart enough to find the next available file name if test is run multiple times (e.g. retry strategy)

Code in the sandbox

  1. Open an Xcode project
  2. Create a test file
  3. Fill it with the code below
import XCTest

final class SampleTest: XCTestCase {

  override func setUpWithError() throws {
    try super.setUpWithError()
    recordVideo()
    XCUIApplication().launch()
  }

  override func tearDownWithError() throws {
      recordVideo(stop: true)
      XCUIApplication().terminate()
      try super.tearDownWithError()
  }

  func testPass() {
    XCTAssertTrue(true)
  }

  func testFail() {
    XCTAssertTrue(false)
  }

  func recordVideo(stop: Bool = false) {
    let testName = String(name.split(separator: " ")[1].dropLast())
    let json: [String: Any] = ["delete": !isTestFailed(), "stop": stop]
    let udid = ProcessInfo.processInfo.environment["SIMULATOR_UDID"] ?? ""
    let urlString = "http://localhost:4567/record_video/\(udid)/\(testName)"
    guard let url = URL(string: urlString) else { return }

    var request = URLRequest(url: url)
    request.httpMethod = "POST"
    request.httpBody = json.jsonToString().data(using: .utf8)
    URLSession.shared.dataTask(with: request).resume()
  }

  func isTestFailed() -> Bool {
    if let testRun = testRun {
      let failureCount = testRun.failureCount + testRun.unexpectedExceptionCount
      return failureCount > 0
    }
    return false
  }
}

extension Dictionary {

  func jsonToString(prettyPrinted: Bool = false) -> String {
    var options: JSONSerialization.WritingOptions = []
    if prettyPrinted {
      options = JSONSerialization.WritingOptions.prettyPrinted
    }

    do {
      let data = try JSONSerialization.data(withJSONObject: self, options: options)
      if let string = String(data: data, encoding: String.Encoding.utf8) {
        return string
      }
    } catch {
      print(error)
    }

    return ""
  }
}

Comments

  1. There are two tests: testPass and testFail, so we can make sure that recording is saved only if the test failed. I suppose there is no need to keep recording if the test passed
  2. setUpWithError and tearDownWithError are the best places to start and stop the recording
  3. recordVideo method sends the request to the web server with all the required details
    • host: localhost
    • port: 4567 (default for Sinatra)
    • endpoint: /record_video/:udid/:test_name
  4. isTestFailed method is used to get test result
  5. Dictionary extension with jsonToString method is used to convert JSON body to String

Result

  1. Run the web server:

     ruby server.rb
    
  2. Run the tests
    • from Xcode or from the Terminal, whatever you prefer
    • on any iOS Simulator
  3. Check out the recordings folder, you should only find the video from the failed test there

Feel free to adjust the scripts to your needs and don’t hesitate to reach out if you have any questions. See ya!

Updated: