Test Automation Tools: Detox
At spriteCloud, we work a lot with eCommerce clients and one of the most important tests we conduct for them are end-to-end tests. End-to-end tests are at the tip of the testing pyramid and are used to test whether the flow of an application is performing as expected from beginning to end. This testing involves ensuring that integrated components of the application work. For example, think of the process of going to Amazon, searching for an item, adding it to the cart, checking out, completing payment, and having the application properly start the fulfillment process.
Working with many different eCommerce clients that use different tech stacks and tools, we have the benefit of becoming familiar with a wide variety of test automation tools and frameworks.
In this particular case we asked our Test Automation Engineer, Rajeswari Narayanareddy, to give us a review of her experience working with Detox as an automated end-to-end testing framework for mobile automation.
End-to-end testing in mobile applications
End-to-end (or E2E) testing in mobile applications is a process that is similar to that of E2E testing with web applications.
The end-to-end testing process entails;
- Setting up the environment; create an instance of an emulator for Android or IOS device.
- Installing the application on the device/emulator.
- Running the application on the device/emulator.
- Executing actions; depending on the type of framework used, the actions such as click, type, hover, etc. are performed on the elements by using the references such as button, view, and screenwrapper.
- Expecting outcomes; assert/expect functions can be used to test the outcomes by performing the same functions as described in the step above.
While end-to-end tests are important as it tests the entire application in an environment that exactly imitates real-world users, if is often prone to flakiness. Due to this unreliability, the maintainability of end-to-end automation test suites are highly difficult as compared to other frameworks, even on different platforms.
- Failures can occur for several reasons;
- Tests might fail for no reason, even without making any changes to the code.
- Tests are non-deterministic. There are different internals within the app which may finish executing in a different order on different runs.
- We can’t be sure when the application goes idle, because it is not clear when the app has finished handling user interactions.
- Users working on the end-to-end testing frameworks have to use multiple sleeps in order to make the tests work as the user doesn’t know what happens internally with the app (i.e., handling the synchronization manually).
Many of the tools available in the market for mobile app testing are based on “Black Box” testing. Testing without knowing the internal workings of the Application Under Test (AUT).
Mobile black Box E2E frameworks go over the GUI looking for an element (if not found, it sleeps and loops until a certain timeout), then interacts with that particular element. One of the main causes for the unreliability of such black box end-to-end tests is that inability to understand what is happening inside the AUT. This problem gets compounded when end-to-end testing React Native apps.
A lot happens under the hood with React Native apps that can make testing more flaky. To address the flakiness problem, Detox’s creator Wix explains some Troubleshooting tips for Flakiness.
Appium, the industry standard
Appium is one of the most used and often considered a standard in the industry for mobile test automation. Appium is based on the black box testing model which replicates the external methods of interacting with the web app, like how an actual user would.
In fact we have a series of articles explaining how to use Appium for mobile testing automation in our automation guide.
- Pioneering Appium with Ruby for Mobile Test Automation, Part 1: Required Software
- Pioneering Appium with Ruby for Mobile Test Automation, Part 2: OSx software
- Pioneering Appium with Ruby for Mobile Test Automation, Part 3: Building the Environment
Tests with Appium are fairly unreliable. It often yields different results on different devices and there are frequent continuous integration failures. A solution to these failures (adding more sleeps) causes slower tests.
What is Detox?
Detox is a Javascript-based end-to-end automation testing framework for mobile apps originally developed by Wix, one of the top contributors from React Native community.
I was using Detox to write E2E automation tests for mobile apps both on Android and iOS. Though, initially, the framework setup was a little complex, it was quite an nice experience working on Detox to setup tests using Cucumber. Detox can also be used with Jest and Mocha test runners and can also be used with Continuous Integration (CI) pipelines using Travis and TeamCity without any hassle.
Setting up tests with Detox and Cucumber helped to easily integrate the test results to spriteCloud’s Calliope Pro dashboard for both Android and iOS devices. Calliope Pro is a test results monitoring and reporting dashboard created by spriteCloud so that development teams can share, compare, and monitor quality assurance testing results in one location. Give Calliope Pro a try by signing up for a free starter account today.
Some advantages of using Detox
Detox is built from the ground up for native mobile apps and has very good support for React Native apps running on the mobile devices. Detox evaluates expectations natively directly in the tested app running on a simulator or device, instead of the test script running on node.js.
Grey Box
Unlike most E2E tools like Appium, Detox is based on a “Grey Box” testing model. Grey Box testing means that the tester has some knowledge of the internal workings of the application and this allows the framework to monitor the application from within and synchronize with it. Not only is the Grey Box testing more stable than Black Box testing, but also much faster.
No Webdriver
Since we are not on the web, Detox does not use a Webdriver. Pretty obvious, but worth mentioning. Detox is built to interact with the mobile’s native layers directly. Detox interacts with the native drivers Espresso and EarlGrey using the JSON based reflection mechanism. This then invokes the native methods on the device directly.
Espresso and EarlGrey
The native Grey Box drivers, EarlGrey for iOS and Espresso for Android, are developed by Google. EarlGrey and Espresso help us to write clear tests and give us the ability to synchronize with the app and interact with it when the app is idle. This increases the reliability of the tests and also eliminates the need for adding sleeps in the tests.
Simple Javascript-based API
Detox as a simple API written in Javascript with a very small configuration process. Check out Wix’s GitHub repo for more about the configuration of the API.
How Detox works
The sequence diagram below shows the general communication scheme between the components in Detox. For a more detailed explanation of how Detox works, head to the documentation for Detox.
Summary
Though tools like Appium have wider capabilities and are more commonly used in the industry for mobile app testing, Detox has the added advantages of being less flaky (though not completely flake-free) and it works well with Javascript-based applications (especially React Native apps.) It is always good to try out different frameworks to see what are its capabilities and pick the right one for the Application Under Test.
In my role as a Test Automation Engineer that works with multiple clients a year on a variety of products, I need to keep myself abreast of the new tools and frameworks available. It is also likely that I will find myself working on more mobile apps in the future.
So I recommend trying Detox and taking the time to learn how it works. It is interesting to any tester who wants to automate mobile apps and particularly those working on with React Native. A small downside of the E2E framework is that the tests can still be a little less reliable due to network and server issues, but overall Detox is a less flaky method for end-to-end testing mobile apps.
Ready to unlock the true potential of Detox for your test automation efforts?
Contact projects@spritecloud.com for a free consultation and discover how we can transform your testing process!