blog

Let's talk about testing

‘Our biggest competitor is the option to not test at all’

This week, our Business Development Manager talks about the benefits of spriteCloud’s approach to software testing.

14 months ago, I joined spriteCloud as a ‘rookie’ in the software testing world. After being rapidly pumped full of information and presentations, the first customer visits presented themselves. After my first meetings with the responsible managers, it was clear that spriteCloud was offering something special. Even this non-technical business guy now knows the added value of external testing.

spriteCloud is a great company to work for in general, with short lines and creative business ideas that can be implemented a month after being brought up. It is an excellent environment for somebody new in the business. After my first few months as the sales guy for spriteCloud, it hit me that most of our clients found us through word-of-mouth, so our service was selling itself…

We need to conquer the world with spriteCloud

Nonetheless, like every company, we want to expand, or as our management team puts it: we need to conquer the world with spriteCloud! Therefore, we needed to listen more to the companies that we were serving, plus we needed to convince other companies why testing is so important for them. We needed to spread the spriteCloud message: Test your software, not your reputation! And so we did. Soon we found out that our service was very specialized and our market segment in the digital world was already strongly represented: the foundation was there, ready to expand! Testing for creative agencies on a project basis – a popular service we provide – is still pretty unique, especially with the flexibility that we offer as a service provider. We prefer software testing on-demand, without any tremendously long discussions about the scope of the project or pricing models, providing ad-hoc solutions by working together with the client.

Hire an expert to do the same job better

What we found was that a lot of companies, especially in the creative tech industry, respond to our testing requests in the same way. A large number of companies that we reached out to mentioned that they test internally. Sometimes that means a receptionist tests, the whole development team drops everything and tests, or other stakeholders test. Another thing, which is interesting for us, was that all of those stakeholders did not like to test, and were not good at testing. Lets be honest, everyone wants to work on something they are good at. Developers are great at writing code, project managers are great at organizing, and testers are great at testing. Furthermore, developers testing on a project budget is not free either, is it? So you might as well hire an expert to do the same job better!

A lot of companies I speak to tell me the same: with small budgets, already thinly spread across the project, the first thing cut out is usually testing and I get that. Convincing clients that we are able to think with them instead of think for them is always the hard part. But when they are convinced, they will most likely understand the added value of having a testing partner who can staff a 3 day project for the next week, can build their test automation suite from scratch, and can execute performance tests in-house. All the testing you desire underneath the same roof.

A small suggestion: do not put your (web-)application live without us looking at it. Because you should test your software, not your reputation.

Testnet Spring Event 2017

Testnet – the largest professional organization for testers in the Netherlands – hosts yearly a large number of events. This year, the Testnet Spring Event 2017 was organized on 15th May. The theme was ‘Widen your base: new skills for testers’, with a variety of workshops and presentations.

One of these workshops was ‘Storytelling for testers’ hosted by René Tuinhout and Marinus Stam. This workshop gave a short introduction to storytelling with practical examples and Do’s and Don’ts. The workshop focused on how to build up a story, and how to tell the story. How to write the story is a logical next step, but not covered in this course.

So, why Storytelling?

Storytelling in software testing is an important aspect for testers, because we do it all the time. The trick with storytelling is to make the story powerful. Telling and writing stories can be used when creating test reports. A test report is a description, explanation or justification of the status of a test project, and is set up professionally and with care to serve the clients. A report is not just a summary of facts; it is a story about the facts.

Different levels of stories for reports include:

  • Level 1: A story about the status of the product
  • Level 2: A story about how you tested
  • Level 3: A story about the value of the tests
  • Level 3+: A story about the value of your story
  • Another way to use storytelling in testing is for test automation. Within the software tool Cucumber, Gherkin is a programming language used to define use cases. These use cases are like small stories (scenarios) written in structured plain-text English. Using the steps ‘given’, ‘when’ and ‘then’, the tester creates a story that can be run by the test tool. For example:

    Given Testnet organized the Spring Event 2017
    And I attended the event
    When Participate in the workshop storytelling
    Then I learn something about storytelling
    And I can share this information

    What did we learn in this workshop?

    The workshop was intended to give an introduction to storytelling and the practical use of it. In this course, it was not important what type of story was used; whether you want to tell a fairy tale or write a testing report, the form in general is the same. The program was divided into four basic steps:

    1. The way you order the story
    2. The use of your voice
    3. The use of emotions
    4. The delivery of the story (in front of an audience)

    Every story is constructed from a set of ingredients and is ordered from beginning to end. Logically you start with the main subject(s), introduce a problem or a challenge, create a solution and eventually a wrap-up of the story. This ensures that the story is smooth and easy to understand.

    The steps about the use of your voice and emotions are specifically for the telling of the story, and not so much for writing the stories when using them in reports or in test automation.

    Finally, the delivery of the story depends on the purpose of it.

    When creating a story, there are some general Do’s and Don’ts as a guideline:

    Do’s:

  • Use everything you introduce (e.g. subjects or problems) in your story.
  • Keep the amount of problems or challenges limited to keep the story as short as needed.
  • Know what the purpose of the story is before telling or writing the story.
  • Don’ts:

  • Don’t use a ‘deus ex machina’ in the story. Deus ex machina is latin for ‘god from the machine’. The term has evolved into a plot device whereby a problem is suddenly resolved by some new event or character.
  • The workshop

    This workshop gave me a nice introduction to storytelling and information I can build from. Yes, this helps when creating a kid’s bedtime-story – but for a tester it is really helpful too.

    Sources:
    Testnet website: https://www.testnet.org/
    Testnet downloads: Storytelling for testers

    Lapis Lazuli: Watir, Selenium and Cucumber on steroids

    This entry is part 10 of 10 in the series Test Automation

    What does Lapis Lazuli add to Watir, Selenium and Cucumber?

    Hello, I am Gijs, one of the developers of Lapis Lazuli. I often get the question why we use Lapis Lazuli in addition to Watir. In this post I will explain each of the systems unique abilities and then I will list the advantages of using Lapis Lazuli.

    Cucumber, Selenium and Watir

    Below I will describe in a short summary the main functions of these solutions.

    Cucumber

    This is what makes Ruby code usable for Continous Integration. It helps you turn code into readable text. Most commonly used for Gherkin style output.


    Selenium

    The core purpose of Selenium is to convert browser functionality to coded functions, so a developer can communicate with a browser. Every browser has their own so called ‘driver’ and each driver has its own functions and methods to communicate with it. Selenium Webdriver unifies these differences and translates it to your favourite programming language. In our case: Ruby.

    Watir

    Watir makes the usage of Selenium much simpler. Where Selenium is still quite complicated and “codey” to use, Watir jumps in and simplifies it for you. For example browser.div selects the first div element on a page and browser.divs selects all div elements on a page. Thanks to Watir, test automation becomes easier and more readable.

    Lapis Lazuli

    With the combination of the 3 solutions, you can build a fully functional test automation suite, which can run your code automatically and gets your results in an orderly fashion. This is what Lapis Lazuli does. When installing Lapis Lazuli, you can use a command lapis_lazuli create [project-name] and it will generate a basic project in which you can instantly start automation.

    In the past, we combined the 3 systems manually here at spriteCloud, but we noticed that often the test automation engineer was dropping the ball on writing good code. We started seeing too many false-positives or false-negatives in our code. Mainly false-positives were a problem, because it doesn’t show issues.

    Enforcing best practises

    We’re still completely dependent on Watir’s way of interacting with elements, like clicking, hovering or setting values of input fields. But we have re-written a major part of how to select elements. Lapis Lazuli enforces the best practises in the following ways:

    • Automatically throw an informative error when an element is not found, with an option to disable this.
    • Discourage using regular expressions to find an element by providing a better alternative.
    • Finding an element is just as easy as waiting for an element, preventing timeout errors.

    Finally, in combination with Cucumber, screenshots are automatically taken when a scenario fails. The screenshot is given a unique name that consists of a timestamp and the step in which it failed. This enables the possibility to link failed steps to the screenshot and display it.

    Final thoughts

    Selenium, Watir and Cucumber are legendary solutions that bring great possibilities for TA-engineers in the Ruby world. Lapis Lazuli simply makes use of this to combine it into 1 well oiled TA-package.

    Gijs Paulides – spriteCloud – test automation engineer

    Mobile app testing: An introduction

    This entry is part 1 of 1 in the series Mobile app testing

    In the world we are living in, cell phones have become a big part of our lives. We are spending more time on mobile devices than with the people present around us. We want to do every possible thing on our cell phones. Big companies like Amazon are creating their own line of apps to make it easier for their customers to access their services. Now with the kind of priority a mobile app has, we need to make sure that the app works as intended without any issues.

    As a testing company we see that most of our clients are concentrating on making apps for iOS and Android. A few clients ask us to test their Windows OS client too. So how are we going to test the different apps? Are we going to test on a real device or are we going to use an emulator?

    Each technique has its own advantages and disadvantages, I will explain the pro’s and con’s of each technique below.

    Pros of testing on real devices:

  • We can know whether the app is responsive enough or not.
  • We can know how the user feels about the navigation of the app. When we use an emulator we use the mouse to click on the options and we do not know how this feels for the user. As an example, on iOS we tend to keep the navigation icons at the bottom as it will be more handy for navigation rather than if the icons are at the top. We do not experience the real feel of an app if we use an emulator where we can easily click those buttons.
  • We are able to test on different networks and data speeds. So we can test how the app works at different network speeds like 2G, 3G and 4G.
  • We can check how the app works when we receive a call or notifications.
  • Cons of testing on real devices:

  • It is expensive to test on real devices as you need to buy and maintain most used devices in the current market – and there are a lot of them.
  • Pros of testing on an emulator:

  • Free and open source.
  • Can be connected to an IDE for early testing during development.
  • Cons of testing on an emulator:

  • Emulators can be a bit slow.
  • Emulators may be incompatible with the app or app elements, meaning that you will need to create patches here and there to be able to use the emulator.
  • Now we have a basic idea of which testing procedure we have to follow based on our app requirements. As we move further testing an app, we see that we will come across bugs which are reproducible and bugs which are not.

    To hunt reproducible bugs, we try to gather as much information as possible about the bug and we get the crash log for the crashes that occur randomly without persistent steps. The crash logs are recorded on the application we use to distribute the apps like HockeyApp or Crashlytics.

    For bugs that are not reproducible, or ‘ghost bugs’ as my client calls them, we follow a specific procedure for tracking purposes. When we come across these ghost bugs, we try to remember the last steps we have done and on which functional areas we saw the issue. We communicate with the other testers to keep an eye on that area by sharing the relative steps. When my fellow testers test the same functionality, it can happen that they come across the same bug but with one or two additional steps or even with lesser steps. We combine all the steps and figure out the best way to reproduce the issue. By working together, the probability of finding the exact steps for reproducing the issue gets higher and we are able to track out most of the issues.

    Testing is not only about testing the functionality of the app. We also need to make sure that the app is secure. We will get into this more about it in my next post regarding the security testing of mobile applications.

    Game testing – An Introduction

    This entry is part 3 of 3 in the series Game Testing

    “A delayed game is eventually good, but a rushed game is forever bad.”
    – Shigeru Miyamoto

    Game testing is a specific type of software testing focused on video games, that can be extended and used in the ever growing market of entertainment and educational software (apps, simulators, video games, presentations etc.)

    As a part of software testing, most of the testing methodologies, practices and techniques will work and are used within game testing. Proper understanding of what is a defect (*bug), how can you find it and proper reporting and triaging (sorting of issues based on their level of priority) are the ‘bread and butter’ work of any tester involved in such projects.

    Also, game testing will show its own set of particularities and aspects that will have a bigger impact on the testing process compared to for example a web testing project.

    In my opinion, it all starts with the product and its intended audience. Video games tend to be very complex pieces of software that, besides a complex and stable technology (the game engine) also have different elements and mechanics (gameplay) that are created to trigger overlapping responses in and from the user (emotions, mechanical proficiency, intellectual challenges, self fulfilment etc.). As such, a game tester must always have in mind all these specific aspects when verifying and reporting on the project he is assigned to.

    The very simple answer to that is to ‘always think like an end-user’. What am I supposed to see, do, understand, feel and react as a user? Is this an intended experience, is this an intended behavior or response that the ‘game’ is requiring of me? Am I having fun? If not, why? All this layer of thinking gets over the ‘normal’ scrutiny that a tester has when observing and using the software.

    The product (type, platform, hardware) will determine the expected responses, while the end-users thinking will define how good the ‘entertainment’ value is. This approach will also generate a huge amount of reports and issues, for example technical reports, gameplay reports and logical reports. As such, the triaging part of the testing process becomes critical in managing and eventually getting ready to release and deliver the product.

    Entertainment software development tends to be a complex affair and most of the time on a ‘not enough time’ schedule. Close to the release date, the pressure is quite high and the whole team gets to feel the unglamorous effects of development crunch. This is a ‘normal day at the office’ part of the game testers life. How can a tester manage this and how do you plan to test such complex software? We’ll talk about this.. soon™!!.

    Which Android versions should I test on?

    This entry is part 5 of 5 in the series Android

    Android statistics chart

    It has been an interesting couple of months for Android enthusiasts. The highlight probably has been the release of the Samsung Galaxy S8 and S8 Plus. Next to releases of new devices, Android Nougat is being released by manufacturers to more devices. We can see this in the Android statistics for the last three months. Android Nougat has gained a market share of 7.1% compared to the 1.2% it had in February. Together with the growth of Nougat, we can see the stabilization of Android Marshmallow and Lollipop. Kitkat still holds a marketshare of 18.8%, losing only 3.1% over the last three months.


    Fig.1 Overview by Android codename (click to enlarge)

    Looking at specific version numbers we can see the same trend. Android 7.0.x and 7.1.x are gaining marketshare while Android 5.0.x, 5.1.x and 6.0.x are stabilizing. All older versions are slowly but steadily losing marketshare.


    Fig.2 Overview by version (click to enlarge)

    Based on these statistics, we can conclude that Android Nougat is really taking of now. For testing, it should be one of your priorities. Based on the trends we have seen in the past, Android Nougat will quickly gain more marketshare in the coming months. We expect it to be the largest Android version in about a year.

    Source: https://developer.android.com/about/dashboards/index.html

    Successful test automation with Cucumber in the Cloud

    How can you make Test Automation succeed?

    The biggest challenge to making a success out of test automation (TA) is maintainability. Maintainability of both your TA test suite as well as the TA infrastructure you are using is critical. I will explain both and describe how to get a successful TA setup on the API and UI level, using open source frameworks and the test automation platform Calliope.pro that can run your Cucumber TA suite in the cloud.

    Maintainable TA suite

    Things happen… A tester or developer goes on vacation or leaves your development team; changes in the tested application are made and break the test scenarios. When the TA suite is complex and hard to understand, in time and with a changing environment, it will soon require too much effort by the team to keep the suite up and running; it is doomed to fail. Using a scripting language such as Ruby, as opposed to a programming language, will keep the test suite easy to understand for non-developers. Using Cucumber (Gherkin language) as the top layer of your TA suite will make the test scenarios understandable for the business. On www.testautomation.info you can read about creating a stable TA suite from scratch using open source software. You can also find an Ecommere TA Suite template on github.

    Maintainable TA infrastructure

    Things still happen… Regular updates on web browsers and (test automation) frameworks get in the way and break your TA setup. The complete test automation infrastructure requires the ongoing attention of system engineers; but your system engineers are already busy maintaining the production environment… These factors affect whether you succeed or fail with your TA setup. Online TA platforms maintain the infrastructure for you, allowing you to put your effort elsewhere. spriteCloud Calliope.pro is such an online TA platform. It enables you to have a very cost effective test automation setup running all your Cucumber/Ruby TA suites.

    Want to see for yourself how successful test automation in the Cloud works? Calliope.pro offers a demo functionality where you can see online test automation in action. Enjoy and don’t hesitate to contact us if you have any questions about test automation!

    The testing recruitment Challenge

    Monday, starting the week with another day in recruitment, where you hope that your inbox will be filled with good CV’s of software testers. Not only non-EU candidates but also Dutch speaking testers with test automation skills. Unfortunately for me, the market has flipped totally from a client driven market to a candidate driven market. And my inbox is almost empty.

    Instead, the good software testers have an inbox full of incredible job opportunities. They get to choose between different companies to work for. Most testers nowadays get a headache of recruiters and are annoyed by constantly being phoned at work during the day.

    As the talent acquisition specialist of spriteCloud in this market, recruitment is a time-consuming and grafting exercise. To find the testers I want to hire (YOU!) I need to fish and source extensively.

    But after the fishing, sourcing and interviewing you every so often find someone who actually is a good fit, and has the skills we are looking for. Then you have to get them in. The negotiation starts and the offering of the right competitive contract is key with nice employee benefits. In this market it is not only about the money. Remuneration is one thing and every company has a cap. And lets be honest; beyond a certain amount of money, people are not motivated to work harder or perform better.

    People are mostly motivated by secondary benefits like:

    • Flexible/part-time working hours
    • Training and personal development
    • The latest, most powerful hardware
    • Newest technical environments
    • Time off work to attend conferences
    • Etc. etc.

    Fortunately spriteCloud’s secondary benefits have a competitive edge. And that makes my job rewarding instead of frustrating. I can tell our candidates that we have great clients: Heineken, adidas and G-Star RAW. That we do game testing and VR testing. That the opportunity to cross train in load testing or pentesting is possible. That we have our own open source test automation platform and our own test lab with a broad range of different devices. All assets to show why it is so much fun to work at spriteCloud.

    So, if you are interested to know more and want to be part of the test community spriteCloud contact me, and I will tell you much more about our great company.

    Patricia van Boxtel
    Talent acquisition
    0627107660
    patricia.van.boxtel@spritecloud.com

    Oedipus and the testing Sphinx

    A long time ago, in ancient Greek, Oedipus was travelling to Thebe. On his way, he crossed the path of a Sphinx. The Sphinx stopped all travellers on the road to Thebe to ask them a riddle. If they were not able to answer the riddle correctly, they would be killed and eaten. If the travellers were able to answer the riddle correctly, they could continue their journey. The Sphinx asked Oedipus this riddle: ‘What walks on four feet in the morning, two in the afternoon and three at night?’ Oedipus took a short moment to think and answered: ‘Man: as an infant, he crawls on all fours; as an adult, he walks on two legs and in old age, he uses a ‘walking’ stick’. The Sphinx, puzzled by the fact that somebody actually was able to answer the question, allowed Oedipus to continue his journey to Thebe.

    Now the testing Sphinx asks you: ‘What walks on four feet in the morning, two in the afternoon and three at night?’ You give him the same answer as Oedipus did. But she is not puzzled by the fact you are able to answer the question. Instead, she challenges you to give the smallest amount of test cases to test the theorem. If you are able to give the Sphinx the correct test cases, she will let you continue your journey. If not … you have to do overtime forever.

    As a seasoned tester, you know what to do: use boundary analysis to identify the test cases. So first you check what the age groups are. The answer is 0-17, 18-64 and 65 and up. You define the minimum required test cases as 17, 18, 64 and 65. You choose these test cases, because you know that to test the boundaries, you will need to test the smallest possible step from the boundary, in this case 1 (year).

    The Sphinx is puzzled by your testing knowledge and accepts your answer. You are allowed to continue your work without having to do overtime. But beware! The Sphinx may come back with another riddle, even more difficult. If you cannot answer the riddle … it is overtime forever for you.

    Mobile website test automation with cucumber but without the hassle

    This entry is part 9 of 10 in the series Test Automation

    Introduction

    In the test automation world the client often requests to run the tests on mobile devices, next to doing the testing on regular desktop. When running your automation in Cucumber, you have multiple possibilities to solve this. In this article I will talk about these possibilities and a new (simple) way to do it with Lapis Lazuli, a Ruby Gem that cooperates with Selenium Webdriver.

    I will talk about the following solutions:

    • Lapis Lazuli device simulation
    • Using Appium
    • Using Browserstack
    • Using a phone via USB.

    Continue Reading »

    Reputation. Meet spriteCloud

    Find out today why startups, SMBs, enterprises, brands, digital agencies, e-commerce, and mobile clients turn to spriteCloud to help improve their customer experiences. And their reputation. With complete range of QA services, we provide a full service that includes test planning, functional testing, test automation, performance testing, consultancy, mobile testing, and security testing. We even have a test lab — open to all our clients to use — with a full range of devices and platforms.

    Discover how our process can boost your reputation.