Case Study: Ben NL

Ben NL, or Ben, is a Dutch virtual mobile network operated by T-Mobile Netherlands with roughly 600,000 users. Its users are mainly students and people within the 30-40-year-old age group. Ben offers its voice, text, and data services at a highly discounted rate and this is achieved by offering relatively limited customer service.

‍

With this high level of self-service, the mobile app plays a central role within the Ben ecosystem, as around 60-70% of the users are using it. The most common actions a user performs in the app are checking their calling and mobile data limits, with the option of buying extra data (as MBs) and also modifying subscription options and perks from month to month. Through the app customers also have access to invoices and account information.

‍

“Together with spriteCloud we were able to professionalise the testing procedure of the Ben app with test automation.”

Stefanie Uijttenhout

Product Owner

Ben Mobile App

‍

In September 2019, spriteCloud was contacted by Stefanie Uijttenhout (Product Owner) in regards to a quality assurance request for Ben (T-Mobile). Stefanie had heard of our successes when we had worked with T-Mobile previously and wanted to ask if we could bring our mobile testing expertise to the Ben team as well.  spriteCloud completed a successful transition to Ben/T-Mobile’s development partner, Oberon, and set up mobile test automation, including the test results dashboard tool Calliope Pro.

‍

In this case study, we will present how we worked together with both Ben and Oberon to provide our quality assurance and test automation expertise.

‍

As Product Owner of the Ben mobile application, Stefanie asked spriteCloud’s assistance.

‍

The team working on the Ben mobile app was relatively small, consisting of two developers and the Product Owner (PO) for the app, Stefanie. Both developers on the project were from Ben/T-Mobile’s development partner, Oberon, which made the team quite distributed. The development schedule of the Ben mobile app was in sprints of two weeks with a standard release every 2-3 sprints.

‍

Before spriteCloud joined the team, the Ben mobile app teams shared a manual tester with another Ben development team. When that tester left, the Scrum Master took over this role as the manual tester of the Ben mobile app, alongside his scrum master work. We see this situation occurring on many occasions, testing is devalued to a part-time/dual-role after which the quality of the application drops significantly.

‍

The need for faster and safer development

‍

Initially, Ben was looking for a single manual tester to join the mobile app team, however, according to the PO, the number of “ app users and the incoming errors increased so [much], that we realised manual testing [alone] was not sufficient” and hiring an external software testing provider would be beneficial. Following this event, the decision was made to source external specialised/dedicated testers to focus on both manual testing of the mobile application and to set up, expand and maintain mobile test automation on the native mobile application.

‍

‍

spriteCloud’s involvement at Ben

‍

In the next section, we will explain spriteCloud’s involvement in helping the Ben mobile app team resolve their problem. First, we will explain the team structure, the processes used on this project, a word on collaboration, and finally discuss the successes that spriteCloud and Ben working together were able to create.

‍

QA team structure

‍

spriteCloud’s first man on the ground was Mobile Test Engineer Aron Febles. Aron was brought on to fulfil the role of the previous tester that had left the organisation. Aron was initially asked to join to manually test the mobile application of Ben, for the ‘in-development’ functionalities. Aron was also simultaneously creating documentation and testing cycles for the project. A very important milestone during this initial phase was to get a complete overview of the functionalities by creating a regression test document. This would be used to validate the existing functionality of all future releases.

‍

A few months later, as the app increased in functionality, the number of users and the number of incoming errors, the team decided that they needed more efficiency in their approach to testing. spriteCloud was asked to present a business case in regards to the setup of mobile test automation. A presentation was created by spriteCloud founder and Test Automation Architect Mark Barzilay.

‍

Two approaches were presented; spriteCloud would set up the mobile test automation suite and continue to expand and maintain this framework or spriteCloud would solely focus on the setup, after which the framework would be handed over to the Ben team. A hybrid version was chosen where both options were combined; spriteCloud would set up the framework with a specialist, after which our tester within the Ben team would be responsible for maintaining and expanding the test suite.  After a period, the framework would then be handed over to the Ben project team internal stakeholders.

‍

Ben was fortunate with the timing as our Senior Test Automation Specialist, Rajeswari, was available to provide her hands-on experience with Mobile Test Automation, which helped the setup phase tremendously. The setup period took 6 weeks from start to finish.

‍

Tools and processes brought in

‍

Rajeswari and the team decided to go with the Detox framework for automation testing for two reasons.

First, the Ben app was implemented using React Native technology making the Detox framework very compatible and suitable. It was way more comfortable for the developers to work with since the development technology was the same for them and thus easy to implement. The alternative that was discussed, Appium, was dropped due to the previously mentioned arguments. Second, another important consideration for choosing Detox was that, generally, tests are less flaky when written for Detox.

Ease and reliability were important factors for the relatively small mobile app team.

‍

The developers were able to have a basic setup to write the regression tests in Detox. Rajeswari worked with the team on a regular basis, helped improve the framework, and assisted in implementing it with Cucumber and Detox. This made it easy for everyone to understand what’s going on within the tests as the scenarios were written in plain text using Cucumber’s Gherkin syntax.

‍

The automation tests were implemented for all the major functionalities of the app such as in the subscription dashboard, purchasing additional benefits (like increasing data by buying more MBs), upgrading/downgrading subscriptions, and including user locale changes, amongst other features.

‍

Together with the developers, the spriteCloud team had implemented the setup of running tests in TeamCity (as a Continuous Integration tool). spriteCloud testers also integrated the test automation suite with spriteCloud’s own automated test results reporting and monitoring dashboard Calliope Pro.

‍

Calliope Pro provided the entire development team with better insight into the stability of the system-under-test (SUT), increasing collaboration between team members and visibility on issues. As the team was fairly decentralised, Calliope provided a reliable central source of truth for testers, developers, and the product owner to be able to collaborate on issues more effectively.

Smooth integration and collaboration

‍

The integration of spriteCloud’s manual tester, Aron, in the initial phases of the project went very smoothly and he fit in with the team pretty quickly, according to the Product Owner, Stefanie. After test automation was set up for a few weeks by Rajeswari, the handover of the test automation suite was initiated. The handover also went very smoothly as the entire process was very clear for both the developers and the tester, who was able to implement it further on for the most important customer journeys in the Ben mobile app.

‍

Due to the high maintainability and clean setup, the test suite was easy to hand over to new testers. The test suite was eventually successfully handed over to Ben as spriteCloud left the project. This handover also went without any problems as the internal stakeholders had experience with it and good documentation to support them.

‍

Success Story at Ben

‍

Based on the continuous improvements and the excellent integration of the testing procedures in the development cycle, for a six month period following spriteCloud’s involvement most of the releases for the app went very well with no or minimal issues.

‍

T-Mobile was able to significantly improve the testing procedures for the Ben mobile app. This has the benefit that transitioning new testers to the project has become very efficient.

‍

“ The manual testing and documentation of regression testing are much better. It helps the team to find more bugs earlier in the process before releasing it […]”

– Stefanie Uijttenhout, Product Owner of Ben mobile app

‍

‍

Also, based on the successful development, and the ensuing ease of use and stability of the app, it was decided by Ben that the mobile app development team should work on developing a way to transfer part of the app to the web interface of the Ben user’s accounts. spriteCloud’s testers and automation specialists played an important role in supporting the development team’s success which led to their code application being used elsewhere.

‍

After the experience of spriteCloud lending its expertise to Ben,  the product owner, Stefanie can confidently say that “the importance of proper and robust testing of the app has proven itself at Ben over the last two years.” Not only was the stability and user experience of the Ben mobile app increased, but the developers themselves were empowered to create better code by writing the test scripts themselves.  

‍

Ben was able to meet their business goals with the app development project and spriteCloud was able to support the development and train them to eventually take over the automated testing.

‍

‍

“The importance of proper and robust testing of the app has proven itself at Ben over the last two years.”

‍

Ready to revolutionize your testing process?
Reach out to projects@spritecloud.com for a free consultation and discover more!