1. Test Management tool selection process
Although this guide is focused on XRay, it is recommended to begin by following these steps to ensure you make the right choice. With numerous test management tools on the market offering similar functionalities, it is important to make the right decision by identifying the most important areas.
We often receive questions about whether test management can be done in Excel, such as within Sharepoint. It is doable for a small project with approximately 50 test cases and one tester, but when you are working with multiple testers and a high number of test cases in different functional areas, it will be challenging to maintain. A Test Management tool is therefore recommended.
Test Management tooling selection sheet
We recommend the following criteria to make the selection of the Test Management tool. Take into consideration your project specifics, but in general, this will help to make a good choice.
To ensure you select the most suitable option for your requirements, please complete our short Tool Selection Form: https://forms.gle/QKR8db5hejRzht4n7
For Xray, questions 4, 5, and 9 are essential. Xray is a tool that works as a fully integrated add-on for Jira, when Jira is not part of the project issue tracking tooling, it does not make sense to acquire Xray.
Question 9 is related to the budget. The pricing of Xray is about €10 per Jira user (independent of using the tool). If there is no budget, an option would be a tool such as Testlink. Although it is not a fully Jira-integrated solution, it can still help with Test Management and linking to bugs.
When Xray is selected, I recommend starting with the free 30-day trial to set everything up in the right manner.
2. Why Xray?
Based on best practices and working with tools such as TestRail, Zephyr, and TestLink, certain aspects stand out, indicating that Xray is an excellent option for Test Management.
1. Integration with Jira: Xray is a tool that only works with Jira, it is the best Jira integrated test management tool on the market currently. This is also directly a limitation. Without Jira, the tool does not work.
2. Improved Traceability: Traceability is very helpful for Test Management reporting. It helps to confirm User Stories are covered and bugs are connected to the right test cases. As all is done within Jira, this works very effectively in Xray.
3. Support for Different Test Types: Any test type can be tracked in Xray. From Unit Tests, Functional Tests, and Integration tests to User Acceptance Tests. The benefit is that you can also easily reuse test cases from different test types.
4. Reporting and Analytics: You have multiple Xray widgets for reporting in Jira. It makes it easy to create comprehensive reports and dashboards. Also, Xray provides multiple options to export to Excel for additional analytics where needed.
5. Automatic generation of multiple test executions based upon Data: Xray can make a combination of potential test executions based upon different data sets.
6. User-friendly Interface: For everyone already acquainted with Jira, the interface of Xray is user-friendly. It’s suggested for additional instructions to be provided to users who are not familiar with Jira.
Multiple help and support: Xray has very good instruction pages, and you can find a lot of information on both Atlassian and Xray websites and blogs.
3. Installation and configuration
Installation
To install Xray, you do not need to go outside Jira. Follow these steps:
- Log in to your Jira instance and click on Apps.
- Search for Xray
- Click on Xray
- On the Xray page click on the yellow button “Try for free”
- This installs the plugin
- When completed you get the message Xray Test Management for JIRA installed successfully
The next step is the configuration of a project.
Configuration of a Project:
The X-ray project configuration is an important step. While you can update the configuration later in the project, it's advisable to take the time to set it up correctly from the start.
- Go to JIRA -> Apps -> XRay
- Click on Configure Project -> Configure
- As you see, the Issue Types are not yet configured. You first need to create the Xray issue types in Jira project settings.
- Add Issue Types: Test, Test Set, Test Plan, and Test Execution.
* You can also add Pre-Condition, but I recommend to add pre-conditions to the description of a Test Case
- Create Test Coverage: You can create coverage from Story Issue Type and others where needed for tests. This makes test coverage reporting easy.
- Set up defect mapping. By doing this, when you execute a test and create a defect, this automatically opens the correct issue type. In this case, a bug.
- All other settings can be left as default. You can update what is needed, for example creating Test Environments.
4. Best Practices using Xray
Create and maintain test cases in Excel
It might seem unconventional, but it's advisable due to the limitation of test case exports from Xray, which exclude test steps, and the quicker process compared to creating test cases directly within Xray. Once the test cases are crafted in the template, transferring them to Xray becomes straightforward.
Remember:
- Create test cases in Excel template (see template)
- Save as CSV
- Upload to Xray (See chapter upload CSV)
- Export test cases and add the Jira IDs to the test cases in Excel
- To update test cases, make the adjustments in the Excel file, then proceed with the upload process accordingly.
Upload Test Cases with the Xray importer
- Access the Importer:some text
- Log in to your Jira instance with an account that has Jira administrator permissions.
- Navigate to "System administration" > "Import & Export" > "External System Import."
- Select "Xray Test Case Importer" from the available options.
- Upload and Map Fields:some text
- Click "Import CSV".
- Choose the CSV file containing your test cases.
- Xray will display a preview of your data and a mapping section.
- In the mapping section, match the data headers in your CSV file with the corresponding Xray Test Issue fields. Common fields include "Issue Type," "Summary," "Description," "Steps," and "Expected Result."
- Import:some text
- Once you've mapped the fields and configured any settings, click "Begin Import" to initiate the upload process.
- Xray will import your test cases and provide a summary report detailing any errors or warnings encountered during the process.
Nomenclature and tags for test cases
The following tips help to ensure test cases can be identified easily and organized.
- Ensure test case titles are clear and contain some keywords.
- Do not use underscoressome text
- Instead of writing: Address_Change_for_existing_Employee_in_DepartmentX1 try Address Change for existing Employee in Department X1
- Write a clear description and pre-condition
- Use tags that help find test cases in Xray.some text
- Examples: Negative, Login, Hiring
- Use priorities (High, Medium, Low)
- Add a custom field if needed, this helps later in the dashboard functionalitysome text
- For example: Country, Test Type
Status of a Test Case
The status of an X-ray test case is not an indication of the execution. It is advisable to include a "Review" and a "Ready for Execution" status in the Jira workflow for this issue type and remove the "In Progress" status.
Meaning of the status for Test Cases:
- To do: Test case created, needs to be reviewed
- Review: The test case is being reviewed
- Ready for Execution: Test Case can be used in Test Execution
- Done: Test Case can not be used anymore for new test executions
Export test cases' basic functionality is limited
There is no standard function to export all test cases to Excel format (including the test steps). There are options to do it using additional JIRA export add-ons like Xporter. To prevent having issues when you need to do updates of test cases, it is recommended to do updates in the Excel file.
Test sets, test plan, and test execution
Xray offers a valuable feature allowing users to generate Test Sets, Test Plans, and Test Executions. What do these issue types entail, and how can they be optimally utilized?
Test Sets help to group tests. You can organize your test cases logically based on functionality, feature area, or any other relevant criteria.
Tips:
- Do not make test sets too long (maximum 15 test cases).
- Ensure all test cases in a test set are related
- Create reusable test sets for common functionalities or modules that can be included in various test plans
Test Plans help to define the testing scope for a specific release or iteration by outlining the test sets, testers assigned, and estimated execution time. Schedule test plans for specific dates or project milestones.
Tips:
- Make sure test plans have very clear title
- Link test plans to requirements to demonstrate coverage and ensure all functionalities are tested
- Add Test Sets to Test Plans and not individual test cases
Test Execution helps to run individual test cases or entire test sets or test plans, recording pass/fail results, execution time, and any additional details like screenshots or logs.
Tips:
- Create test executions per tester instead of multiple testers within a test execution
- Try to use Test Sets and not individual test cases
- You can create test executions per device or browser
- Keep test executions limited to a maximum of 50 test casessome text
- It is better to have 5 test executions than 1 of 250 test cases
5. Dashboards and reporting
Utilize Jira's standard dashboards along with its built-in Pie Charts gadget for bug status, and incorporate Xray gadgets such as Overall Test Results, Test Runs Summary, and Test Executions List. Stakeholders can be directed to the dashboard for an overview, and for deeper insights, they can directly access various filters by clicking on them.
Tips for dashboards:
- It's advisable to keep it simple
- 2 pie charts with bugs should be sufficient (Open bugs per status and all bugs per priority)
- Display overall test results using the Xray Gadgets: some text
- Overall test coverage: See the percentage of requirements covered by tests.
- Test execution status: Track the progress of test runs, showing how many tests have passed, failed, or are still in progress.
How to use X-ray gadgets?
- Create a Jira dashboard or use an existing one.
- Click "Add Gadget" and select an X-ray gadget from the available options.
- Configure the gadget by specifying a project or filter to determine which data it should display.
By combining Xray gadgets with other Jira gadgets, you can create a comprehensive dashboard that gives you a clear picture of your testing process and progress.
6. Where do you go when you an issue with Xray?
There is good support when using Xray and you should be able to find quick answers on the following websites.
Official Xray Documentation:
The primary resource for X-ray questions should be the official X-ray documentation provided by the makers. This documentation usually covers installation, configuration, using features, and troubleshooting. You can find it here: https://docs.getxray.app/display/XRAY/Getting+started
Atlassian Jira Support:
Since Xray is an add-on for Jira, the Atlassian Jira support website might also have helpful resources or troubleshooting guides related to using Xray within Jira. Check the Jira support section for any Xray-specific content: https://confluence.atlassian.com/jira
Community Forums:
Xray might have its own user forums or communities where users can ask questions and get help from other users or the Xray team. You can search online to see if there's an official Xray forum.
General Jira and Test Management Forums:
Even if there isn't a dedicated Xray forum, there are many online communities focused on Jira and test management in general. These communities might have discussions or solutions related to Xray functionalities within Jira. Search for forums related to "Jira" and "Test Management".
For example: https://stackoverflow.com/
spriteCloud
Our team possesses unparalleled experience in implementing and configuring Xray Test Management. Through countless successful international projects, we've honed our skills to become Xray implementation ninjas. We don't just set it up; we transform Xray into a strategic asset that supercharges your testing process.
Here's what sets us apart
- Deep Xray Expertise: Our team has a proven track record of successfully implementing Xray across diverse project scopes and industries. We understand the nuances of X-ray and how to tailor them to your unique testing needs.
- Streamlined Configuration: We don't believe in one-size-fits-all approaches. We meticulously configure Xray to seamlessly integrate with your existing Jira workflows, ensuring a smooth and efficient testing experience.
- International Project Acumen: Having tackled projects of various scales and complexities around the world, we possess the global perspective to navigate challenges and deliver optimal solutions.
- Customized Test Management Solutions: We don't just implement Xray; we help you leverage it to its full potential. We work closely with you to design a customized test management strategy that maximizes efficiency and delivers clear visibility into your testing process.
Why Choose Us?
- Reduced Testing Costs: Our optimized Xray configuration streamlines workflows, saving you time and resources.
- Enhanced Test Coverage: Achieve comprehensive test coverage with our strategic X-ray implementation plan.
- Improved Quality and Faster Releases: Gain immediate visibility into testing progress, leading to faster and bug-free releases.
- Increased Team Collaboration: Foster seamless collaboration between developers and testers with a unified Xray platform.
Conclusion
In conclusion, implementing a test management tool like Xray is essential for efficiently handling medium to large-scale projects. This guide has explored the advantages of Xray, a powerful JIRA add-on that integrates seamlessly with JIRA to streamline test planning, execution, and reporting. By following best practices for selecting and configuring test management tools, you can enhance traceability, support various test types, and utilize robust reporting and analytics. With Xray's user-friendly interface and extensive support resources, teams can maintain high standards of software quality, improve collaboration, and achieve faster, more reliable releases. Leveraging tools like Xray transforms your testing process into a strategic asset, ensuring comprehensive test coverage and optimized workflows.
Ready to unlock the true potential of Xray Test Management?
Contact projects@spritecloud.com for a free consultation and discover how we can transform your testing process!