How do we overcome Xcode’s tendency to give slow test feedback? Let’s work on a brand-new project to reduce the test feedback time from 47 seconds down to 1.6 seconds.
[This post is part of the TDD with SwiftUI series.]
Why Shorter Test Time Matters
In Xcode, you run all tests in a test plan by pressing ⌘U. Or you might run a suite of tests by clicking the test diamond at the top of a test class. And sometimes you may want to run a single test by clicking the test diamond next to a test function. Whatever you do, how long does it take before you get the test results?
Fast feedback doesn’t mean you’ll run the tests more often. But it does mean you can if you want to. And in test-driven development (TDD), I want to. The TDD ideal is to take a tiny step and validate it. Then another. And then another.
But slow feedback makes this much harder. If it takes too long, we batch smaller changes into big steps before we dare to run the tests. This defeats continuous validation, the heartbeat of TDD.
And even if you never TDD, slow feedback is no fun.
So it’s important to tackle feedback time and to be quite determined about it. Over time, even tiny gains can make a difference for your entire team. I am scraping for every second, even every tenth of a second, that I can find.
Let’s dive in, shall we?
Start With a New Project
I’m starting a new project called GitInbox, an app for reading GitHub notifications. I want it to be a place to explore TDD for SwiftUI. But that’s down the road. For now, let’s create a new project, see how Xcode’s default settings feel, and then improve them.
I encourage you to follow along by making your own new project. Measure how it goes on your computer.
Launch Xcode and create a new project. I’ll be using the name GitInbox.
Choose SwiftUI as the interface. This sets the language to Swift.
Set Storage to “None” to start with, because YAGNI (you aren't gonna need it). YAGNI means we don't include anything until we actually need it. What will storage be like for this project? We don’t know. So let’s defer any storage decisions until later.
And make sure “Include Tests” is checked.
Even in this New Project dialog, Xcode is working against the fast feedback I crave. It used to offer separate checkboxes for “Include Unit Tests” and “Include UI Tests.” (Was this back in Xcode 12?) Now there is no choice—we have to “Include Tests,” which is all or nothing.
Now the new project is in place with Xcode’s default settings. Before we change those settings, let’s measure the feedback time we’re starting with.
Measure Initial Test Time
Set Xcode’s destination to an iOS simulator. For our first trial, make sure the simulator isn’t already running. I want to try a cold start. Nothing built, no simulator running.
Find the Clock app on your phone, and go to the Stopwatch tab. Why use a stopwatch instead of checking what Xcode reports? Because I want to measure the start-to-finish human experience, not the machine’s experience. (Checking Xcode’s reports may become useful later as the time gets shorter.)
With one finger on the Start button, get ready to use your other hand to press ⌘U to run tests. (If you can’t do this, recruit a helper.) And when the tests stop, press the Stop button. Ready? Go.
My first run took around 2 minutes 50 seconds. And I’m on a very capable computer with an M1 and loads of RAM. Sitting around for nearly 3 minutes is not the wonderful experience of fast feedback I want. But let’s keep in mind that this is the first launch. Building takes time. Launching the simulator takes time. But then it seemed to sit there… and sit there… and sit there, before installing the app and starting to run the first tests.
Let’s try a second run with everything warmed up. My second time took 47 seconds. That’s still quite long, especially when you remember that we haven’t yet written a single test of our own. What's going on?
Stop Running the UI Tests
For TDD, we want only microtests, which are unit tests that are fast, specific, and consistent. UI tests fall outside these bounds. So let's turn off the UI tests.
Xcode now keeps test settings in a “test plan.” In the Xcode menu, select Product > Test Plan > Edit Test Plan or press ⌘>.
This opens a tab showing the test plan. The tab says “autocreated” because these are default settings that haven’t been saved. And now we can see that the project has two sets of tests, starting with the project name. My project is named GitInbox, so GitInboxTests are the unit tests, and GitInboxUITests are the UI tests.
Select the UI tests, then delete them from the plan by clicking the minus (–) button in the lower-left, or pressing the Delete key. Since this test plan has never been saved before, Xcode asks if we want to save it.
Select Save. It now prompts for the name and location of the test plan. To make a clear association between the test plan and the test target, I like to use the same name for both. So let's name it GitInboxTests, and save it in the project root.
Why delete the UI tests instead of just deselecting the “Enabled” checkbox? Because I never want to mix fast tests with slow tests. For TDD, the test plan should contain only microtests: unit tests that are fast, specific, and consistent.
Let's ⌘U again to check the time. With everything still warmed up from before, this run took 4 seconds. Now we’re really starting to get somewhere. But let’s not stop yet.
Stop Building the UI Tests, Just Delete Them
If you already have UI tests, make a separate test plan to run them. And make a separate scheme that uses that test plan. Keep them out of the fast tests.
But what if you haven't written any UI tests? Then delete the ones that are there. Xcode created a target and a source folder with files. Let's start by deleting the target.
If Xcode isn't already showing the Project Navigator in the left pane, select View > Navigators > Project from the menu, or press ⌘1. Then select the project at the very top (not the source folder inside). This will bring up the project settings. Select the General tab if you’re not already there. You should see a list of targets.
Select the UITests target. Click the – button in the lower-left (or press the Delete key) and confirm the deletion.
That deletes the target. Finally, delete the UI test source files. Select the folder in the Project Navigator, press the Delete key, and select “Move to Trash” in the confirmation dialog.
Now we won’t spend any time building tests we don’t need. Goodbye, and good riddance.
Turn Off Parallel Tests for XCTest
Another unfortunate change Xcode made is to run tests in parallel by default. You can spot this by watching the simulator during a test run. When tests are run in parallel, Xcode spins up multiple clones of the simulator. These simulator windows are named “Clone 1” and so on.
This makes sense for slow-running UI tests. You pay a cost to set up the clones, but that cost is more than made up for because the tests are so slow.
But microtests are fast-running. For most microtests, it takes longer for Xcode to set up the simulator clones than is worth it. And even if you reach a massive test suite, we’re going to want fast feedback when running a subset of them locally. So let’s not run them in parallel from our main test plan.
Go to the test plan, either by selecting it in the Project Navigator on the left, or selecting Product > Test Plan > Edit Test Plan (⌘>) from the menu. Notice that it says “Execute in parallel” under the tests. This is what we want to change.
Click the “Options…” button and click on the “Parallelization” popup menu.
Change it from “Enabled (If Possible)” to “Swift Testing Only”.
Try running the tests again. That has shaved off another second, bringing my time down to 3 seconds.
Turn Off the Test Debugger
The feedback time is getting faster, but we can still do better. When we run tests, Xcode goes through the following steps:
- Build the project
- Build the tests
- Launch the app in the simulator
- Inject the tests into the running app
- Attach the debugger
- Run the tests
- Quit the app
There are ways to avoid launching an app that we may want to look into later. Right now, let’s get rid of that “attach the debugger” step. To do this, edit the scheme by selecting Product > Scheme > Edit Scheme… from the menu, or press ⌘<.
See where the Debugger checkbox is? Click to deselect it. Run tests to see how it reduces the time before tests start running.
The checkbox at the bottom shows that this is a shared scheme. This means that everyone on your team who uses this same scheme will not have the debugger during test runs unless they turn it back on. This may cause surprises, so discuss this as a team before making a change to a shared scheme. You may want to duplicate the scheme so that you have two, one for fast feedback and one for debugging.
Not taking time to attach the debugger saves another second, bringing my total time for a rerun to 2 seconds.
Remove Test Template Cruft
When you make a new project (or a new test target), it comes with some template test code. If this is an XCTest template, it comes with a performance test. Like UI tests, these don’t belong with microtests. We shouldn’t even run them in Debug builds. And the work a performance test does to set up, record, and report timing statistics costs time.
I see many, many projects with these templates still in place. I’ve written about them before in Simplify Your iOS Test Writing: Introducing a Streamlined File Template. They’re doing nothing but cluttering your code, as well as taking up some time. Delete them.
Note that XCTest will be unhappy if we try to run tests but have none. So while we work on the initial project setup, let’s create a do-nothing placeholder test.
This carves off nearly half a second. My rerun time now comes to just 1.6 seconds.
Check Existing Projects for These Time Wasters
That takes care of changes to new project settings. However, your existing projects may have other settings that impact your test feedback time. Here are two to check for.
Custom Build Scripts
Does your project have Build Phases that run custom shell scripts? For example, many projects run their source files through SwiftFormat.
If you’re not careful, this could cause your project to rebuild everything, every time.
Back in the old days, developers managed “makefiles” to specify their source file dependencies. If this file changes, these other files need to be rebuilt. But Xcode manages these source file dependencies for us.
Except when it can’t.
Inside a Run Script build phase, Xcode has no idea what’s going on. What needs to be rebuilt after the script runs? Since Xcode can’t guess, it throws up its hands and says, “Well, rebuild everything then.”
But like makefiles of old, we can solve this by defining the dependencies ourselves. A Run Script build phase has a way to specify the inputs and outputs. To learn more, read Specify the input and output files for your script in Apple’s article “Running custom scripts during a build.”
dSYM Generation
Xcode Release builds leave out information to stay lean. Debugging symbols go into a separate file called a dSYM. But a Debug build already has this information inside, so there’s no need to generate a dSYM. Any time spent doing so is waste.
To check this, select your project in the Project Navigator (⌘1), select your project in the Project list, and open the Build Settings tab. Filter by “debug information format.” What you want to see is a setup as shown below, where the Release build makes a dSYM file, but the Debug build doesn’t.
Kudos to Arek Holko for his article Speeding up Development Build Times With Conditional dSYM Generation.
I’ve seen this trick save 0.1 seconds. No, that’s not much, but when you examine the low cost of making this change against the benefit, it’s still worth it. Remember, I want to shave every little bit because every developer on the team will experience the test feedback cycle many times each day.
Conclusion
Let’s look at the difference our improvements made, both from a cold start and also with everything warmed up.
These changes will help your developer experience even if no one on your team practices TDD. But they are essential for TDD because fast test feedback lets us run tests more frequently. This acts as an incentive to move in small, verified steps, and to keep aiming for smaller steps.
Do you have any additional speed tips for Xcode?
[This post is part of the TDD with SwiftUI series.]
Another trick I’ve learned, is have the non-frontend tests run on the Mac instead of the simulator.
in my (this) case it only shaved off a second, but in bigger apps this is noticable.
Axel, do you put your non-frontend code into a separate module of some kind, with its own test target?
Yes, that’s the plan. Separate layers, like network, models etc. can go in a separate module which can be tested faster on macos. Coordinators can be tested as well if the interface is protocolised. But that’s probably in your book (which I have, but is still on the bucket list)
But this was in pre-M1 times, so perhaps the speedup is less prominent now)