Hello, everyone! I am Ihor Tkach, B2C iOS Developer at the UK-based fintech company Wirex with an extensive R&D centre in Kyiv. As experience shows, an end-to-end test is one of the most effective tools in identifying critical bugs in the UI functionality of an app. However, if you decide to start writing UI Tests in your project, you could face issues linked to using CI machines in the tests and other problems related to identifying the object on your screen.
In this story, I’ll share tips from the Wirex R&D team on how to troubleshoot these issues that we explored whilst working with our product, endeavouring to make all currencies equal, combining traditional money and crypto into one app.
Running UI Tests on Simulators vs Real device problems
As we know, tests on a real device run smoothly without flickering because it takes most of our memory to run an app. Not only this, but we don’t have as many background processes running at the same time. As a result, the screen will only usually flicker because of memory overloading by simulating the iPhone and processes like Safari, Xcode, and others.
At Wirex, we use real devices to test our apps. However, when we wrote our first E2E test on the simulator and pushed it to our CI, the test failed, and we couldn’t get our TextField to focus. Rerunning the test led to the same problem, and after checking our CI machine's logs, we realized that the software keyboard didn’t appear after tapping on our TextFields or TextViews, so we couldn’t type any text into our fields.
The problem was in the default connection of our hardware keyboard to the simulator, so we had to turn it off while running the test. To do that we needed to go to our Simulator -> I/O -> Keyboard and make sure that “Connected Hardware Keyboard” wasn’t marked.
But what about CI; how can we handle it there? For this purpose, we can add this code to the AppDelegate.swift. That makes your keyboard always use the software keyboard for your simulator environment. application(_:didFinishLaunchingWithOptions:)
#if targetEnvironment(simulator)
// Disable hardware keyboards.
let setHardwareLayout = NSSelectorFromString("setHardwareLayout:")
UITextInputMode.activeInputModes
// Filter `UIKeyboardInputMode`s.
.filter({ $0.responds(to: setHardwareLayout) })
.forEach { $0.perform(setHardwareLayout, with: nil) }
#endif
Running E2E Tests on your Local machine != CI machine
I don’t care if it works on your machine! We are not shipping your machine! - Vidiu Platon
Performance of our Local machine and CI:
You can see that our local machine performs better than the СI machine. The thing is that we didn’t take that into account and had multiple errors on the CI. That’s why our screens have displayed perfectly on the local machine, but CI didn’t load all the elements on the screen and marked the test as failed. We tried to rerun the test, but there were still problems.
To deal with this issue, we need to use the waitForExistence(:) method during the waiting time of making all elements visible on the screen while the view hierarchy is loading them.
Most errors you get in the future will look like this:
We don’t wait for all pages to show up with waitForExistence(:), but only the heaviest pages that have a lot of elements in the view hierarchy.
For the first time, to load all elements we’ve set our timeout for a random 3–10 sec.
func testXAccountCreate() throws {
app.launch()
let dashboardPage = DashboardPage(app: app)
guard dashboardPage.view.waitForExistence(timeout: Constants.defaultTimeout) else { return XCTFail("Dashboard Scene must have been shown") }
let xAccountsTabPage = dashboardPage.tapXAccountTabButton()
guard xAccountsTabPage.view.waitForExistence(timeout: Constants.defaultTimeout) else { return XCTFail("X-AccountsTabPage must have been shown") }
let xAccountListScene = xAccountsTabPage
.closeFeatureExplanationPageIfNeeded()
.tapCreateXAccountButton()
guard xAccountListScene.view.waitForExistence(timeout: Constants.defaultTimeout) else { return XCTFail("X-AccountsTabPage must have been shown") }
}
When we’ve written about 15 UI Tests, we changed our waiting time to a “defaultTimeout = 20 sec” for all our E2E Tests.
Why 20 sec, you ask? We ran a small investigation:
According to these numbers, 20 sec for loading the elements on the screen provided us with a satisfactory result. Therefore, we didn’t increase the execution time for our UI Tests, since 8/10 is a great result for E2E Tests.
The most valuable lesson you need to keep in mind while writing UI Tests is the difference in the performance between Local and CI machines. This is because the performance of the simulator directly depends on the performance of the machine which runs the testing environment and other processes at the same time.
Elements Tree — Accessibility identifier — Recording
There is no right way to find your element, you need to try them all and choose what works the best for a particular case.
Elements Tree
Xcode gives you a visual tree of elements from your current screen. To launch it, you need to set a breakpoint inside your test in a specific view you want to debug and write the command “po app” into the console. As a result, you’ll get a visual elements tree like this:
On the screenshots above we can see the elements tree for the current Add Funds view, so we can grab these objects to use further in our code.
For example, if we want to tap on EUR cell, we search the image of the desired object in the elements tree.
Image, 0x6000036710a0, {{20.0, 198.0}, {32.0, 32.3}}, identifier: ‘eur’
And in the Page Object it looks like this:
private let eurWallet: XCUIElement
eurWallet = app.images[“eur"]
eurWallet.tap()
But there could be a situation when we can’t get the identifier for the element we wanted to tap. In this case, the accessibility identifier comes into play.
Accessibility identifier
Accessibility in UIKit is not something new. It helps users with disabilities to interact with apps in many ways. Every UIView conforms to a protocol named UIAccessibility that allows iOS features such as VoiceOver to identify the different UI elements on the screen.
The thing is, if we can’t get an element from an elements tree by identifier and can’t match the object because it’s void, the Accessibility in the UIKit could be an effective instrument to get all the other elements required.
There are two ways to set an identifier for an element in the Accessibility identifier:
- By code imageView.accessibilityLabel = “mainImage”.
- Go to the Identity Inspector pane in our storyboards or xibs.
On the Page Object it looks like this:
private let mainImage: XCUIElement
mainImage = app.images[“mainImage"]
mainImage.tap()
The issue is, the two problems could lead to potential memory overloads and performance spikes on your machine:
-
The query looks for all images on the screen, no matter their position in the view hierarchy.
-
The query does a full scan on the elements tree even after it has already found an element named “mainImage”.
To avoid these issues, we need to be more precise about the position of our element in the view hierarchy:
mainImage = app.cells.images[“mainImage”].firstMatch
So we made two changes to avoid performance spikes:
- Now we only search elements in cells of our view while eliminating the rest of the tree.
- Likewise, we add the property “firstMatch” to return the first element we found in our cell view tree. If you have only one element on the screen, such as there only being one TextField on the screen, you can avoid an identifier and get the first matched TextField element like this.
private let nameTextField: XCUIElement
nameTextField = app.textFields.firstMatch
nameTextField.tap()
Recording
Finding the object in the elements tree with the help of an accessibility identifier is not the only way to find an element on your screen. There was a different way introduced by Apple starting from the iOS 9️ release that let us record our actions on the simulator and convert them to a script of our XCTestCase.
To start recording you need to locate your cursor inside a UI Test method and start the app. When the app is being launched you need to hit the record button at the bottom of the Xcode window and start tapping on the screen:
This generated code is the result of tapping on the TextField and writing the name “Mark” into it. To stop the recording, you need to click the record button again.
Notice that generated code by tapping on the screen is not readable. It’s better and cleaner to write it yourself instead of using this method.
private let nameTextField: XCUIElement
nameTextField = app.textFields.firstMatch
nameTextField.tap()
nameTextField.typeText(“Mark")
This looks a lot better!
But recording saves us a lot of time in finding the specific element.
private let firstExchangeButton: XCUIElement
firstExchangeButton = app.tabBars.buttons.element(boundBy: 2)
firstExchangeButton.tap()
It’s not a perfect solution, but it works.
There are screens on which recording doesn’t work at all, and everything just crashes. In a situation like this, you might want to use other methods that we’ve considered today. However, it should be noted that from version to version, the Xcode recording instrument may not work properly, which is why it’s not a tool that you can rely on 100% of the time.
Recording only helps when other tools like an elements tree and accessibility identifier do not work, and it really comes into play for situations like swiping, scrolling, or dragging the elements on the screen.
UI Tests with CI/CD Retry and Parallel Testing
The biggest pain is UI Tests failing on CI again and again. All Teams Globally
UI Tests need to be a part of CI/CD like Unit Tests. They should be run on CI builds, otherwise we won’t know if we’ve broken something. It’s crucial to ensure that your UI Tests are stable and fast enough to run on CI builds without making that process troublesome for contributors.
We decided to write our first UI Test about six months ago, and we didn’t know that it would be so painful for us because the tests on CI had been failing while the tests on the local machine passed successfully. Every time we had to rerun our tests on the CI machine manually, but now, we just use an automated tool for this.
Use a Fastlane plugin “fastlane-plugin-test_center“ to make your life easier when running your UI Tests on CI!
To install:
- Go to your Fastlane folder and open Pluginfile
- Add the new plugin to it gem ‘fastlane-plugin-test_center’
- Open the Fastlane file
- Instead of scan use multi_scan
- Choose what parameter you need to add:
multi_scan(
clean: true,
scheme: $scheme_name,
try_count: 3, #retry _failing_ test up to 3 times
parallel_testrun_count: 2,
device: "iPhone 12 mini”
)
- Update your Fastlane file with command-line “bundle exec fastlane update_plugins“
Remember to be careful with the “parallel_testrun_count” if your machine has less than 6 cores CPU and 16GB of RAM. It may affect your performance 🐢.
One of the problems with adding “parallel_testrun_count = 2” was that it increased our build time for about 2–3 minutes after each run, so the normal build time increased from 17 minutes up to 38–45 minutes.
It took us some time to investigate what had gone wrong. The thing that we didn’t take into account was that running two simulators at once gives a lot of drops in performance on CI. So we needed to delete paralel_testrun from Fastlane and add it later after getting our new M1 machine for CI.
More simulators for UI Tests are not always a good choice:
We decided to pick up with 3 reruns because our build time didn’t increase as much, and we had the same success / running attempts number, so we stuck with 4️ reruns.
It’s always helpful to take advantage of new plugins that make our contribution process easier and save us a lot of time.
Thank you for reading!