As famous quality management expert W. Edwards Deming pointed out, the quality of a product is not detected, the quality is already there. Therefore, software quality cannot be improved by strengthening testing assurance, quality needs to be built into software products.
" Inspection does not improve the quality, nor guarantee quality. Inspection is too late. The quality, good or bad, is already in the product. Quality cannot be inspected into a product or service; it must be built into it. "
W. Edwards Deming
The timeline is crucial in this regard: the later the software defects are exposed, the higher the repair cost. Fewer defect prevention in the early stages, also means that more defects will be discovered later. Consequently, we need to prevent defects in advance, to avoid finding issues after the development is completed. This is the quality built-in.
Quality built-in is a core principle of the agile mindset. To foster built-in quality we use bug analysis as a way to prevent bugs. Our objective is to discover the problems in technology, personnel, process, project and organisation – so that we can promote continuous improvement.
Why Do We Need Bug Analysis?
First things first, why is it crucial not to skip bug analysis? A good way to prevent bugs is to learn from our existing experience. By analysing existing bugs, and finding root causes and problems in the process, we can prevent future bugs. Thinking about ways to optimize and improve we can control the quality risk. As we all know, a zero-defect product is not realistic. Since it is impossible to completely prevent the defects, we need to consider how to ensure that the defects that have occurred are effectively dealt with, and how we can use the existing defects to guide the built-in quality process. That is the focus of defect management.
The Bug Life Cycle or Why Timing Matters
Finding bugs is central to improving the quality of our software product. That being said, not every bug can be processed in the same way: timing matters. Each bug has a life cycle and for the bugs found in sprint and production, the processing is a bit different.
When the Bug Is Found in the Sprint
There are some rules we need to follow when it comes to the timing for fixing defects in the sprint:
- If it is a defect that blocks the development or testing progress, it should be fixed immediately.
- If it is a function-related defect that must be delivered in this iteration, and the cost is high or the defect has a large impact, it should be fixed as soon as possible.
- If it is a function-related defect that must be delivered in this iteration, but the fix cost is not very high or the defect has a small impact scope, it could be fixed in the later stage of the iteration.
When the Bug Is Found in Production
When the defect is found in production, we need to follow these guidelines:
- We need to determine its priority. Does the defect need to be fixed as soon as possible? If it needs to be fixed immediately, the release happens after regression testing.
- If its priority is low, the defect will be moved into the backlog. The processing flow is the same as when the defect is found in a sprint.
- After the defect is fixed, we need to check if we are missing some testing. What tests should be added?
- The defects found in production need to be managed for review.
- In retrospect, we need to review the production defects and make an improvement plan.
Create a Bug Report
Now that we know how to process the bugs found in different phases of the development process let’s have a look at the tools that can help you streamline your bug analysis.
How to Select a Management Tool
We need a tool to manage bugs. This can be a management software or you can simply record bugs in documents such as excel. Collecting this data in a centralised place is a must for bug analysis. The more information you have, the more benefit you will get from your data analysis.
When selecting a management tool, consider these points:
- Does it support collaboration?
- Is it easy to use and easy to learn?
- Is it easy to track the bug status and does it support clear bug reports?
- Can you use the tool to analyse your data?
One of the advantages of using a tool, such as Jira, that integrates requirements tracking to record defects is that defect records and story cards are in a unified system and panel. This is convenient as it allows all team members to view and maintain defects together, involving everyone in the process, not only quality assurance.
How to Write a Clear Bug Report
Once you selected the tool to manage your bugs, let’s look into how bug reports should be written. You need to follow at least two rules to file a bug report:
- Make a clear and easy-to-read description.
- Standardise your reports to allow for easy data collection.
As your bug report will be read by team members, making a clean and complete description is essential. The analysis is based on standardised data structures. When you file a bug report, you need to think about what you will analyze and standardise your input of a bug.
It is common to use a bug report template, all the team members can use the same template to file bug reports. Discussing how the template of the bug report should look, is an important step. Generally, a template includes the following elements: bug type, priority, severity, functional modules, root cause etc.
How Should Bugs be Analysed
Analysing each bug individually is not realistic and probably not the most efficient way to go about this. Except for production defects, each one of those deserves your attention. Of course, if you have enough time, you can make defect analysis a daily job.
That being said, the number of defects in an iteration is relatively large, and most of them are caused by development logic errors. Generally speaking, the analysis value is not large. If each iteration runs smoothly and the number of defects is stable, it is recommended not to analyze them specifically. After all, analyzing defects is time-consuming. If it is obvious that the number of defects in an iteration is increasing, the defects in this iteration can be analyzed in a targeted manner.
The common point of view to analyse bugs goes as follows: each point can be divided into multiple different attribute values, and the focus of different projects will be different. Generally, bugs can be analysed from different angles, as described below:
Analyse the Bugs From their Root Cause to Improve Quality
Through root cause analysis, meaning finding the root cause of bugs, you can make early adjustments and introduce control measures. Root causes can be classified as missing or unclear requirements, technical design problems, code logic errors, test coverage, dirty data, and environment-related problems (hardware, software, third parties, etc.).
These root causes can be targeted to improve quality. By optimising existing methods and processes to prevent and control the spread of problems you can effectively avoid new issues from arising.
Back to Timing
When is the earliest moment a bug can be found? For the defects in the iteration, the discovery phase can be classified into the story acceptance phase, functional testing phase and regression testing phase. We know that the earlier the defect is found, the lower the repair cost. If the analysis shows that there are more defects in a certain stage, we should reflect on which link is wrong and whether we can expose the defect earlier.
Analyse Defects According to Functional Modules
The functional modules depend on the system. This type of analysis helps us think about whether the defects are mostly due to technical debt or insufficient test coverage, which can help us improve our testing strategy.
Focussing on the Type of Bug
Defect types can be categorized as functional defects, performance defects, security defects etc. Quality assurance professionals working on agile projects need to pay attention to the quality of all aspects of the product, and clearly divide the types of defects, which can help identify weaknesses.
According to the General Trend
By analysing the overall bug growth and reduction trends, we can get more insights into testing efficiency, efficiency of developing and fixing bugs, testing bottlenecks, reasons for testing delays, and the testing life cycle in general. Trend analysis includes daily additions, daily closures, cumulative activities, cumulative closures, the total number of bugs, the proportion of serious bugs, the proportion of repeatedly opened bugs, etc.
Trends are more important than absolute values. Compared with the absolute value, the changing trend can reflect the dynamic process of the statistical data better.
According to the Efficiency
Through the duration distribution of the bug state, we can analyse the timeliness of the bug state. So we can find out how long it takes to fix the bug. We can also compare the team efficiency based on some matrix, like how many bugs there are in this sprint? How many bugs there were in the last sprint. How many bugs were found in the work of the other team? We can compare this data to see which team can take care of more new features or which team needs to improve these skills.
How to Analyse Your Data
When the data is ready, you can move on to the next step: the analysis. Depending on the capabilities of the bug management tool you chose the analysis will be done either in the same tool or in a different one. In the end, the data should be presented in a visually appealing way that can be easily shared with the team.
Jira, Tableau and Power Point are examples of tools you can use to analyse your data.
Jira
A lot of teams use Jira as a project management tool. Jira can automatically generate charts based on cards, such as pie charts and trend charts, making it very convenient to record and analyse them together.
Jira offers many options to analyse and visualise your data (Source: Medium).
Tableau
Tableau is a commercial data analysis tool that can draw various graphs and is also very convenient for displaying trend graphs. However, the cost of getting started is not low, and its main function is to perform data analysis. Drawing is only a drop in the bucket. Using it for defect analysis is a bit overkill.
Power Point Presentation
If your management system doesn’t support generating charts, you can also export your data to Power Point Presentation. It is an easy way to help you visualize the data trend.
Plugins
You can also ask developers to help develop some plugins based on your management system to help you collect data and generate charts. It is also a common solution.
Bug Analysis: Your Best Strategy to Improve Built-In Quality and Reduce Costs
In conclusion, to improve the built-in quality of software products, a core principle of the agile mindset, you need to make bug analysis a part of your software development process.There are many benefits of embedding bug analysis in your way of working and therefore promoting continuous improvement in your team, your processes and your organisation. By analysing current bugs you can not only improve the quality of the product but also prevent future bugs and ultimately achieve higher built-in quality. As we know from the testing pyramid this has the benefit of lowering costs, as bugs detected earlier in the process are also cheaper to fix.
Written by: Jessie Li