What is Test Automation?

Test automation means using automation tools to execute pre-scripted tests. In other words, automated software is used to test predetermined functions of the System Under Test (SUT) to check if an application performs correctly after changes are implemented.

Test automation is very useful because it can automate some necessary but repetitive tasks of testing processes that have already been established. They are also helpful in performing additional testing that would be hard or time-consuming to do manually.

Test automation is typically implemented after initial manual testing has been completed. A test script (usually written in a programming or scripting language) is used to navigate through the SUT and to compare pre-defined expected outcomes with actual outcomes. These results are recorded to a test log, and the scripts can be changed or reused to run additional scenarios throughout the system.

In layman’s terms, manual testing identifies the method/process of how a function is to be tested, and then a script is written to replicate that process to make it possible to be run automatically. Scale this up to dozens or hundreds of tests scripts, and you have test automation suite that allows you to monitor your software development effectively with little human intervention (for a time anyway). The differences between manual and test automation testing will be discussed later.

Automation is now a very crucial part of the software development lifecycle as it allows for frequent and efficient testing of business-critical functions that befit the speed necessary in modern software development projects. Automated testing is meant to make software testing quicker and more efficient in repetitive and difficult areas, not to replace manual testing entirely.

Now that we understand what test automation is, it is important to know why, with all the potential for cost overruns and difficulties, test automation is important in software development projects.

‍

Why Implement Test Automation?

The goal of a software development team is to deliver software that works as expected. Still, due to the rapid pace of development, large teams of developers, and cost constraints, defects will invariably slip through. When a defect slips through to the production environment, it can result in:

• Loss of money
• Loss of time
• Loss of business reputation
• Injury or death

The costs associated with fixing software defects (read bugs) often depends on which stage of the development process the defects were discovered. As shown in the image above, most defects are introduced during the coding phase. As indicated by the red line, the later defects are found, the more they cost to resolve. Ideally, testers want to find as many defects as possible during unit testing when they are still relatively cheap to fix. This is where the concepts of automation and specifically test driven development come into play.

‍

Manual testing may be the most thorough process for catching defects, but it is incredibly time-consuming for a tester to manually test and record outcomes at the scale and scope of modern software development projects.

‍

Automating tests is, therefore, the best way to increase the speed, efficiency, effectiveness, and coverage of your software testing efforts. Test automation allows your team to perform more tests in less time, free up human testers for more high-level testing, and increases coverage. Automated tests can also be run outside of working hours to minimise the impact on other project stakeholders.

‍

Due to these reasons, test automation is considered critical for big software development organisations. As the test automation market diversifies, costs are coming down, making it more affordable for smaller organisations. Test automation still has its barriers preventing organisations from adopting it, however, software testing service providers like spriteCloud can help development teams with test automation in multiple ways. spriteCloud can work in a consultancy capacity by helping to set up your automation suite and train your staff, with the idea to hand it off to your team late. Alternatively, spriteCloud can take over your test automation processes entirely allowing your team to focus on development; we often refer to this as Test Automation as a Service (TAaaS).

‍

No and low-code solutions are also becoming increasingly available, though they are often limited to specific ecosystems.

‍

The Differences Between Manual Testing and Test Automation

While test automation clearly has its benefits, it is not a replacement for manual testing. This idea of automated testing taking over entirely from manual testing is a foolish idea. Test automation is a process and toolset for supporting and increasing the efficiencies of testing as a whole. Test automation is designed to take easily repeatable tests and perform them much faster for the tester.

In doing this, a testers time is freed up to increase the coverage of the tests via exploratory testing. A (manual) tester’s time is then typically split between developing and maintaining test scripts (maintenance), 80%, and performing manual (exploratory) testing, 20%. With modern Agile and DevOps methodologies, test automation is a more important tool in the quality assurance toolbox, but it still requires an operator.

Curious to understand the differences between manual testing and automated testing? Read more about the differences between manual testing and test automation in our in-depth article.

The difference between manual testing and test automation

While the differences between these two forms of testing is pretty obvious by their names, manual and automated. While it seems clear, it is important to give a fuller explanation


‍

The Benefits and Drawbacks of Test Automation

Like all processes, test automation possesses a duality.

When done correctly, automation can significantly increase your testing team’s performance. On the other hand, test automation can also be challenging to implement and keep costs under control, especially with an inexperienced team and over-ambitious project managers. Below are some of the more prominent pros and cons of automation, for a more in-depth analysis of the pros and cons of test automation, continue reading our post on the topic.

The benefits of test automation

• Saves time and money
• Increases test coverage
• Improves accuracy
• Allows increased scale of testing
• Helps both developers and testers
• Improves morale

The drawbacks of test automation

• Misunderstanding the role of TA
• Quality issues
• Maintenance costs
• Proficiency in script writing



‍

Test Automation Best Practices

Sharing best practices can help shorten the learning curve by setting up automation in your development process. Here you can find some bullet points to consider when planning your test automation strategy. You should continue reading about test automation best practices and considering hiring an experienced organisation to help your team start correctly. Here are some best practice to consider.

• Hire or create a dedicated team
• Create tests with reliability and maintainability in mind
• Develop Independent and modular test cases
• Make everyone accountable for test automation
• Evaluate the tools well before implementing (read more in our tools and framework reviews section)
• Base test scripts on good manual tests
• Use alternatives to UI tests when possible (as they are costly and flaky)
• Test early and test often, within reason

If you are reading this because you are unsure of how to start implementing automation, spriteCloud can help you set up a test automation suite and train your testers with our test automation consultancy services.



‍

Test Automation Strategy

Like all processes, test automation is a coin with two sides to it. When done correctly, automation can significantly increase your testing team’s performance. On the other hand, automated testing can also be challenging to implement and keep costs under control, especially with an inexperienced team and over-ambitious project managers.

‍

Your strategy should focus on the prevention of defects being introduced into the application rather than the mere detection of these defects. This is because a defect discovered in the earlier Unit Test stages of testing is much easier to correct than when it is found later, like during UI testing, for instance. Following this quality over quantity thinking, any newly developed feature in an application should not introduce any regression defects. In other words, a release with one feature that is fully-functioning and defect-free is much better than a release with many flaky features.

‍

Keeping tests robust and flexible (by making sure they are more modular in nature and less prone to breaking as the complexity of the code increases) is a process that should be built into your automation strategy from the beginning. In our post about working with complex workflows in Cucumber, we give some tips on writing test cases that are easy to maintain and re-use.

‍

As the project grows and the system under test (SUT) changes, the test cases and test scripts will begin to deviate and cause more false failures as they are testing the system less accurately. Testing time and costs accumulate as the tests required to ensure a properly functioning system are continually being added to. While developers are working on developing a new feature (a small portion of the code), testers must re-test the entire code base for defects caused by this new code. Every release ensures that the code base grows larger and larger while getting more difficult to test.

A good automation strategy, therefore, should take into consideration this understanding that test cases will accumulate and that testing will create a bottleneck, if not properly planned and anticipated.

‍

Test automation’s goal is not to replace manual testing but to automate repetitive tasks, thereby making this increasingly growing need to more tests manageable. Automating 80% of testing and manually testing 20% should be a good rule of thumb. As testing grow ins in complexity as the development project progresses, making good decisions about automation, early on is important. It is for this reason that hiring experts or a consultancy organisation to help set up test suites is actually very cost-effective.

‍

The Test Automation Pyramid

The test automation pyramid is a useful guide for your strategy. This pyramid used in combination with the image shown in Section 2 helps give a great understanding of where to focus a project’s testing resources for maximum return on investment. As you move up from the base of the pyramid to the top, tests become more time-consuming, more expensive, and more prone to failure. Unit tests should represent the majority of the tests conducted while UI tests are conducted less frequently.

Unit tests are an important part of writing quality code, and this type of test is typically done by developers. Using the test-driven development (TDD) strategy, test cases are written before code, making this the solid foundation for your automation strategy. This is the quickest way for a developer to find out that new code broke existing code. Due to the quick feedback and simple nature of the unit test, these should be run as often as it makes sense to do so.

‍

API or End-to-End tests come next in terms of frequency and time commitment. These automated tests layer is used to test the business logic without involving the user interface (UI), which can cause more complications. Most of your code should be able to be tested by the previous layers in the pyramid.

‍

User Interface (UI) tests are more prone to false failures and are slower to perform. Therefore, a solid foundation created by the other layers in the test automation pyramid is crucial and will save time, effort, and headache. Looking for some tips on debugging your web driver test automation scripts, do it via the command line



‍

Test Automation for Mobile Applications

The chances that your product or service will also need to be used via a mobile phone application of some sort, is quite high and only gets higher in the future. Test automation for mobile applications is very similar to conventional test automation with the exception that there are different approaches adopted by the application, and the tools differ for testing those applications.

‍

Mobile apps are typically served in three ways:

• Native: integrated into the devices, and content stored on the device. It’s fast but harder to develop and test.
• Web: designed to look like a native app but limited to serving content via a web browser. It is slower to serve content because of this.
• Hybrid: the best of both worlds, though still dependent on browser speeds/internet connection.

When looking at testing tools for mobile applications, you will typically come across two flavours;

• Open-source tools, like Appium and Robotium.
• Enterprise-level tools, like SeeTest and Eggplant.

The principles behind automation are similar but require different tools for mobile applications. For a more in-depth look at tools for mobile applications automated testing, be sure to read our post about Mobile Automation Testing Tools



‍

Automated Testing for Mobile Applications with Appium

When it comes to the testing of mobile applications we prefer to use Appium, unless of course our client’s require another tool to be used. Our preferred test stack at spriteCloud is Cucumber/Gherkin, Ruby and (in the case of mobile app automated testing) Appium.

Appium is great for mobile apps because both native and hybrid apps can be tested for both Android and iOS. This means that testers can save some time by reusing code for testing on both operating systems. We love Ruby and Appium so much, we’ve written a three-part series to guide you through set up and use of the stack.

Pioneering Appium with Ruby for Mobile Test Automation, Part 1: Required Software

Pioneering Appium with Ruby for Mobile Test Automation, Part 2: OSx software

Pioneering Appium with Ruby for Mobile Test Automation, Part 3: Building the Environment

‍

Test Automation in CI/CD

Being competitive in a lucrative software market these days means building new products and services faster than other organisation’s. Software is also getting more complex as a greater level of integrations and functionalities are expected. This need for a more competitive edge has lead to the increased adoption of methodologies to speed up development, like Agile and DevOps. Unfortunately, as seen by some high-profile software failures, the need for faster development cycles often leads to increases in defects.

When software development speeds up, so must software testing, and test automation is key to meeting the demands of faster development cycles through the use of Continuous Integrations and Continous Delivery (CI/CD) pipelines. Adopting test automation for CI/CD pipelines allows development teams to test fast and test early.

Take a moment to read our lengthly article about why test automation is essential in a CI/CD pipeline and learn some tips that can help in setting up automation.

‍

Test Automation Tools and Frameworks

On one side, automated testing is a powerful process that can significantly increase your development team’s performance and test coverage. On the other side, automation can be difficult to implement, maintain, and stay within budget, especially with inexperienced teams and over-ambitious project managers. Even given the challenges of automation, when it is done right, the benefits of test automation can hugely outweigh the risks of getting it wrong.

You might be reading this as an experienced automation engineer researching the many automation tools and frameworks, or as a newcomer to test automation, hoping to make sense of it all. Whatever your experience level, we hope that this collection of reviews and demos created by our team of automation engineers will be useful to you. At spriteCloud, we love open-source tools which is why we dedicate much of our time working with these where appropriate. Our favourite open-source test stack is using Cucumber and Gherkin. We’ve written a lot about Cucumber and Gherkin, which we’ve turned into a guide for you to use.

At spriteCloud, we deliver quality assurance consulting, software testing, and cybersecurity services to clients ranging in size from small start-ups/scale-ups to multinational enterprises. With one of our core disciplines being in test automation, we find it important to maintain and develop our in-house knowledge of automation. We encourage our test engineers to continually expand on their knowledge of tools, languages, and processes.

spriteCloud’s automation engineers have the opportunity to work with many companies in many different industries, resulting in very diverse experiences. We are also a tools agnostic organisation which means that when we go to a client we use whatever tooling they are working with. This means that our engineers need to continuously learn new tools and frameworks.

If you would like to request a review of a test automation tool or framework, please use the form on the Contact Us page to do so.

Below you can find our reviews of test automation tools and frameworks.


Cypress.io

Test Automation Tools: Cypress.io


TestCafe

Detox

Test Automation Tools: Detox


Xamarin

Test Automation Tools: Xamarin


Calliope Pro

While on the topic of test automation tools, our DevOps tool for test results monitoring is the perfect compliment for your TA practices. Its tool and framework agnostic meaning that it supports most major test frameworks. Calliope is created by testers, for testers so you can help your test results tell as story and shorten feedback cycles.

Calliope Pro is the one location for your development team to share, compare, and monitor test results to quickly and clearly understand the health of the code and identify regressions. Give Calliope Pro a try, for free, today and discover the power of collaboration.

Learn more about Calliope Pro

‍

Testing Titans Interview Series

Sometimes it feels like testing never gets any love and that all the glory goes to the developers. Our Testing Titans interviews series tries to correct this by lifting the veil on the software testing world and shining a light on the concepts, tools, and people that are shaping the industry.

Software Testing and quality assurance is a crucial aspect of software development and should be treated as such.

In our interview series, you can look forward to interviews from highly experienced and influential experts on the subjects of DevOps, test automation, cybersecurity, and software development – all from the perspective of software testing. Have a lot at some of our current interviews below.

‍