Beautifying XCTest results

February 6, 2023 5 minute read

Sponsored

XCTest is an excellent tool for testing your apps on iOS. It’s fast, reliable, and easy to use. But the default test reports aren’t as pretty as they could be, especially for UI tests. In this post, I’ll show you how to make them look much better.

Quick intro

Usually, tests’ result representation has two «heads»:

trace in logs — basically test names with ✅ or ❌ prefixes
json/xml/xcresult reports — more detailed info that is packed into a file

Although the first option does show which tests are failing, and sometimes it even gives us an idea why, in some cases, it’s not enough because it lacks some important details such as screenshots, logs, videos, etc, which can lead to a lot of time spent on debugging.

The second one can give us a lot more if configured correctly but in a computer-readable format.

Let’s take as much as possible out of these two options and convert their output into a human-being-readable format.

Sample project

I created a sample project to show you how to beautify XCTest results from both logs’ and reports’ points of view. So if you’re in a hurry, just jump on this repo and copy-paste schtuff from there.

Beautifying the logs

There is one rock star tool that has been around for ages and is able to hide the useless parts of xcodebuild output for us so we can focus only on the important things. It’s called xcpretty.

gem install xcpretty

Just try to build or test any .xcodeproj using the naked xcodebuild command with and w/o xcpretty and you’ll see the difference.

Without xcpretty:

xcodebuild test -scheme XCTestReport -destination 'platform=iOS Simulator,name=iPhone 14,OS=16.2'

🙈

With xcpretty:

xcodebuild test -scheme XCTestReport -destination 'platform=iOS Simulator,name=iPhone 14,OS=16.2' | xcpretty

Beautiful, innit?

I won’t go into details about xcpretty here because its pros (and cons?) have been covered in many articles already. It’s a great tool, and I highly recommend using it, so you don’t strain your eyes.

Fun fact, if you’re using fastlane, it has xcpretty built-in. I bet you already know that, so let’s move on.

Beautifying the reports

Reporting is a bit more complicated thingy. There are a couple of popular tools for doing this job, for instance:

XCTestHTMLReport
Allure Report (XCTest integration tutorial)

Allure is cross-platform, XCTestHTMLReport is kind of «native», both are really great with their own advantages and disadvantages and both able to do what we’re up to. I’ll show you how to use both of them, in parallel, so you can choose the one you like more.

Installing XCTestHTMLReport is as easy as:

brew install xctesthtmlreport

Installing Allure Report is a bit more complicated, but still not rocket science:

brew install allure
DOWNLOAD_URL="https://github.com/eroshenkoam/xcresults/releases/download/1.13.1/xcresults"
curl -sL "${DOWNLOAD_URL}" -o ./xcresults
chmod +x ./xcresults

Let’s run the command we already know to execute the tests and generate .xcresult:
```
 xcodebuild test -scheme XCTestReport -destination 'platform=iOS Simulator,name=iPhone 14,OS=16.2' -resultBundlePath TestResult | xcpretty
```
- resultBundlePath argument takes the path where to save the tests’ result
- so passing -resultBundlePath TestResult to xcodebuild test command will create a TestResult.xcresult in the root folder
Now we can generate a report from the .xcresult file:

2.1. For XCTestHTMLReport:
```
 xchtmlreport TestResult.xcresult
```
- this command will create an index.html in the root folder
2.2. For Allure Report:
```
 ./xcresults export TestResult.xcresult allure-results
 allure generate allure-results
```
- these commands will create two folders in the root folder: allure-results and allure-report
Let’s open the report:

3.1. For XCTestHTMLReport:
```
 open index.html
```
Beautiful, innit?

3.2. For Allure Report:
```
 allure open allure-report
```
Gorgeous, innit?

All above on CI

Alrighty, let’s try to replicate all that on CI. I’ll use GitHub Actions for this purpose, but guess what, you can use almost any other CI service you want.

Let's start with the XCTestHTMLReport:

First of all, we need to create a workflow file. I’ll name it test.yml and put it into .github/workflows folder:

 name: Beautifying XCTest Results

 on:
   push:
     branches:
       - main

   workflow_dispatch:

 jobs:
   xctest_html_report:
     name: XCTestHTMLReport
     runs-on: macos-12
     steps:
       - uses: actions/[email protected]

       - run: brew install xctesthtmlreport

       - run: xcodebuild test -scheme XCTestReport -destination 'platform=iOS Simulator,name=iPhone 14,OS=16.2' -resultBundlePath TestResult | xcpretty

       - run: xchtmlreport TestResult.xcresult

       - name: Deploy to GitHub Pages
         uses: peaceiris/[email protected]
         with:
           github_token: ${{ secrets.GITHUB_TOKEN }}
           publish_dir: ./

as you may have noticed, we’re installing XCTestHTMLReport as it’s not pre-installed unlike xcpretty, and then running the same commands we used locally
and then the third-party GitHub action peaceiris/actions-gh-pages is used to publish the report to GitHub Pages
see sample project for more details

PS: you might want to keep a history of test runs, so you can use the destination_dir: ${{ github.run_id }} option to publish the report to a separate folder for each run. Keep in mind that XCTestHTMLReport depends on *.xcresult which is super heavy and this will increase the size of the branch/repo, so you may need to clean up the old reports from time to time:

 - uses: peaceiris/[email protected]
   with:
     github_token: ${{ secrets.GITHUB_TOKEN }}
     publish_dir: ./
     destination_dir: ${{ github.run_id }}
     exclude_assets: '.github,XCTestReport.xcodeproj,XCTestReport,XCTestReportUITests'

Then let’s allow GitHub to publish the test reports for us. For this, we need to:
- create a branch called gh-pages:
- enable GitHub Pages for our project, Settings -> Pages:
- enable required workflow permissions, Settings -> Actions -> General -> Workflow permissions:
That’s basically it. Every time we run the tests on GitHub Actions, it will automatically update gh-pages branch and publish the test reports for us on GitHub Pages:

Example:
- https://testableapple.github.io/beautifying-xctest-results/4098130575/
- https://testableapple.github.io/beautifying-xctest-results/4098181045/

Let's implement Allure Report:

First of all, we need to add a new job to the existing workflow file .github/workflows/test.yml:

 allure:
   name: Allure Report
   runs-on: macos-12
   timeout-minutes: 20
   steps:
     - uses: actions/[email protected]

     - run: |
         brew install allure
         DOWNLOAD_URL="https://github.com/eroshenkoam/xcresults/releases/download/1.13.1/xcresults"
         curl -sL "${DOWNLOAD_URL}" -o ./xcresults
         chmod +x ./xcresults

     - run: xcodebuild test -scheme XCTestReport -destination 'platform=iOS Simulator,name=iPhone 14,OS=16.2' -resultBundlePath TestResult | xcpretty

     - run: |
         ./xcresults export TestResult.xcresult allure-results
         allure generate allure-results

     - uses: actions/upload-artifact@v3
       with:
         name: Allure Report
         path: allure-report/

as you may have noticed, we’re installing allure and xcresults as they are not pre-installed unlike xcpretty, and then running the same commands we used locally
and then the native GitHub action actions/upload-artifact is used to upload the report to GitHub Actions artifacts
see sample project for more details

PS: we don’t publish the test report to GitHub Pages in this example. Consider using a third-party GitHub action simple-elf/allure-report-action if you’d like to achieve this.

That’s basically it. Every time we run the tests on GitHub Actions, it will automatically attach the test report to the action’s artifacts. Thus, we can download the report from there and open it locally as described above.

Example:
- https://github.com/testableapple/beautifying-xctest-results/actions/runs/4113524123

Quick outro

Setting up all the nice little things needed to make test results look gorgeous takes some time and effort, but it’s a one-time thingy and it’s worth it. This is a great way to make life easier for you and your team and make test results more readable and understandable.

Hope you enjoyed this post. See you in the next one! 🤠