Since writing about dealing with continuous integration and Apple Silicon as well as retrying failed tests, I’ve been thinking some more about iOS and Xcode test automation and would like to share some useful tips and thoughts on that front. Hopefully these pointers will help you out with something with which you might be dealing or that you might see in the future in your iOS, iPadOS, or other apps.
Automate, Automate, Automate
Even if you’re a developer working on your own code (and possibly especially in that case), automating your UI (and other) tests so that they can be run asynchronously from the rest of your work flow is critical to improving productivity. While unit and even integration tests tend to run in seconds and very quickly, it’s common for UI tests to take as long as the user would (and sometimes longer given app spin up times, etc.). This can tie up your machine and work flow if you are running them interactively through Xcode itself. Watching console messages fly by can be addicting but isn’t what I’d prefer to do all day. 
Consider using a tool such as Xcode Cloud (when it comes out beta or the wait list opens up) or Jenkins (even if installed locally on your laptop) to make your processes simpler. You could also look at third party services such as Firebase Test Lab or AWS Device Farm as well – although be aware of free vs. paid tier thresholds.
Being able to kick off a test suite in the background and headless on check in or with just a mouse click can free your time to focus on other more interesting tasks. There is certainly effort, planning, and monitoring that is necessary with setting up automation; but in my experience the effort has paid off in the long run. Give it a shot if you haven’t! I think you’ll find it very cool to get working and rewarding as well.
Consider Third Party Libraries
Yes, it’s always possible to write every last piece of code yourself and fully control all of your source. This need or desire has to be balanced, though, against real life and getting work product out the door. If you find yourself writing a ton of repetitive or hard to manage test code, consider testing and/or matching frameworks like Quick, Nimble, or Google’s EarlGrey 2 to simplify some of your UI test logic. For purely test scenarios, third party integrations present somewhat less risk than frameworks that ship with your app in production binaries.
For example, here’s a simple case of an XCTAssert statement converted to use the Nimble expect function.
// with XCTAssert
XCTAssertEqual(grass.color, .green)
// with Nimble
expect(grass.color).to(equal(.green))
As you can see, readability and intention is more clear with the Nimble test assertion. Where Nimble shines as well is providing useful syntax for UI tests that have to wait and block with toEventually. In the example below, we’re waiting for that grass to green up for three seconds and checking every 250 milliseconds.
expect(grass.color).toEventually(equal(.green), timeout: .seconds(3), pollInterval: .milliseconds(250))
You will want to keep in mind that adopting one of these frameworks involves writing tests in a different way than you might have done in the past. Converting back to standard XCTAssert type tests may be difficult and time intensive if you change your mind. As with any third party library, try it first in a limited capacity and be sure you want to stick with it before adopting wholesale.
isHittable versus exists
When getting started with Xcode UI testing, it’s pretty common to be tossing around isHittable and exists interchangeably (or possibly only use exists). As the Apple developer documentation points out though about exists:
“The fact that an element exists does not imply that it is hittable. Elements can exist offscreen, or exist onscreen but be hidden by another element…”
I’ve been burned a few times using exists with buttons and labels hidden behind alerts, offscreen in a table view, or displaying only partially through the safe area. They technically exist but cannot be tapped by the user and may not be visible. isHittable is a better check for element accessibility than exists if any of these cases are a possibility.
// Using `isHittable` to verify our label both exists and can be seen/tapped
XCTAssertTrue(app.staticTexts["Hello, world"].isHittable)
Note though that isHittable is not a perfect visibility check as it will still return true for views that are onscreen but hidden by other views in the same hierarchy (e.g. a Text SwiftUI view lower and completely covered in the ZStack by another view). isHittable is more about your views being covered by form sheets, modals and alerts, or being scrolled offscreen.
Accessibility Identifiers Are Your Friend
It’s pretty common when spinning up UI tests, either manually or from UI recording, to just use the UILabel text or the UINavigationBar title as the identifier for the element you are addressing. However Xcode provides the very useful accessibilityIdentifier property (accessibilityIdentifier(_:) in SwiftUI) to let you uniquely identify every element in your UI. It is meant for UI automation and does not impact Voice Over or actual app interaction.
(Note that Apple has deprecated the SwiftUI accessibility(identifier:) method in favor of the one above as of iOS 14.)
Using the accessibility identifier can help insulate your UI tests from having to be updated when the text on a view changes. For tappable elements like buttons, this can be very useful when you just need to tap a button but don’t care what it says. Note that using an accessibility identifier does not preclude you from still checking the correctness of the text or value of an element through the label or value properties.
// Using the text from a label as the identifier. If the
// text ever changes, this will fail.
XCTAssertTrue(app.staticTexts["Hello, world"].isHittable)
// Setting the accessibility identifier in SwiftUI
Text("Hello World")
.accessibilityIdentifier("Home_View_Label")
// Label using accessibilityIdentifier will have different identifier and label properties
// StaticText, 0x132053b20, {{0.0, 0.0}, {88.0, 20.5}}, identifier: 'Home_View_Label', label: 'Hello World'
// Using the accessibility label from a label as the identifier
XCTAssertTrue(app.staticTexts["Home_View_Label"].isHittable)
Be Careful of Testing Negative cases
When writing UI tests and using XCTAssertFalse, be aware that a query that returns zero elements can return false as well. If we consider the case below, we would expect the first cases where we are testing for an invalid identifier (‘XYZ’) to fail the test at run time with an ‘element not found’ type error. Instead the all of these tests will pass because a query with no elements is considered unhittable.
// All of these statements pass regardless of whether an actual element exists or not
// Element was never defined
XCTAssertFalse(app.staticTexts["XYZ"].firstMatch.isHittable)
XCTAssertFalse(app.staticTexts["XYZ"].isHittable)
// Element was defined but is hidden by a form sheet
XCTAssertFalse(app.staticTexts["Home_Hidden_Label"].firstMatch.isHittable)
If you aren’t careful, you might think you have tested that a label got covered up by a form sheet; but because of a typo in the identifier are really not testing anything at all.
Be Prepared for Instability and Unexpected Behavior
When writing unit tests against iOS, iPadOS, or really any platform, dependencies are mocked out, there are fewer moving parts, and tests run reliably – over and over and over. Once you start introducing dependencies and live data, it’s possible that some level of instability may creep in. If you are coding and running integration tests, for example, against a live API, success will depend very much on the stability and availability of that API.
With UI tests, in addition to the dependency on live API, the simulator and its host operating system can become an additional source of problems and failed tests. In my experience, there have been quite a few scenarios running iOS UI tests where the simulator has crashed, stalled, or done something unexpected. The most common scenarios are system popups, system overlays, app rating dialogs, and other events that pop user interface elements on top of your app in an unexpected way. Most of these can be dealt with in code one way or the other but you need to be prepared.
But I also have found UI testing through xcodebuild to occasionally crash or hang the simulator. In some cases the causes of the events have been code-related but not always. Resetting simulator state, uninstalling the app prior to testing, and similar remedies can help if you are running into those kinds of problems.
Enjoy Testing!
Hopefully some of this will help you as you automate your application and improve your apps. Please let me know if you have any thoughts in the comments and feel free to like, follow, share, etc.
