Introduction
This is one of a number of articles I am writing to help identify common problems related to Quality Assurance (QA) the impacts the release of products to market. While I have been a product manager for most of my career I have helped develop QA test plans and spent 6 years developing a QA test management tool. The guidelines I outline here are based on the QA best practices that we helped our customers implement in the field.
When possible I will provide real world examples from QA experts to illustrate the root cause of the challenges, and the real world techniques that are used that can help you avoid these pitfalls in your day to day efforts. Hopefully if you can employ these suggestions you’ll reduce the churn between teams and demonstrate a more collaborative path forward is possible with the right implementation.
The Challenge: Reducing Duplicate Bugs
Nothing frustrates an engineering team more than seeing the same issues over and over again, across builds, sprints and releases. Looking at the cause can uncover some understandable origins.
The Problems
Lack of Centralized Bug Reporting and Tracking
The first challenge may be simply that multiple testers are reporting bugs separately and do not know when a bug has already been filed. Some teams even lack access to a central defect tracking tool such as JIRA, Bugzilla, FogBugz, etc. In this case testers sometimes have to resort to reporting bugs via excel spreadsheets or google docs, which is in turn given to engineering later. Sometimes the QA lead will collate the separate bug reports and remove redundant bugs; however, duplicate bugs still slip through. This is surprisingly common, particularly with external QA teams.
Solution: Most modern test management tools can integrate with your bug tracking tool. By integrating your bug tracking and testing tools you can enforce a workflow with proper permissions and controls. By forcing QA to link a bug directly from a test suite / test scenario you can reduce duplicate bugs since users can see the existing bugs that are already associated with the test scenario before filing a new one.
Large and / or Distributed Teams
Even with a centralized bug tracking tool duplicate bugs are still far too common in large and distributed teams. This is often because multiple testers are often required to test the same or similar functions. A test case is often run through many different test scenarios, and against many different hardware and or software configurations. For instance, an online retailer will want to test the checkout process with many different shopping cart statuses, payment methods, and user types, and against multiple different browsers. Add mobile checkout to the list and you have many permutations of the same basic test. It is often very difficult for one person to do all the testing, but if two people are testing the same function then how do they know when a bug has already been filed?
The most straightforward solution is use an agreed upon naming and tagging process to categorize the bugs and then have everyone search for the existing bugs. This sounds good in theory, but in practice is very difficult. Not only is there a productivity loss as users search for bugs, but it is still easy for things to slip through.
Solution: Use a tool, or workflow that shows bugs that have already been linked to a test case. QMetry and qTest both provide this UX. Whether you are testing a test case in one or more different user scenario you will be able to see if there are any bugs related to the test case. This allows android testers to associate UI bugs that were already found on iPhone. A browser-based bug that was previously reported on Firefox could be updated when it is reproduced on Chrome, IE or Safari. In many organizations these would be considered separate bugs; however, the fix is often the same.
Bug Count as a Measure of Productivity and Value
Many QA teams measure productivity in terms of issues found (or bug count), so individual team members feel compelled to file as many issues as they can in a week to show they’re working hard and making progress throughout their testing. Even if this doesn’t describe your group’s mentality, it’s human nature to want to feel like you’ve accomplished something new each day even if what you’re seeing has been found in other environments or across other builds. If this is a driving force behind our assessment of personal value than we should take stock and consider the impact on our counterparts who, for most engineers, would prefer to be working on new areas of the product, coding new features and functions, not drowning in a sea of duplicate issues. Don’t get me wrong – unique issues are measurements of progress and value for any QA team, but a daily bug count, no matter the quality does more harm than it does good.
In my over ten years of working in and around QA with a wide range of projects I’ve seen some example of Quality Management making the decision to write up everything you see as a new issue and let Engineering and Product Management deal with the mess. You can think of this as the “Bug everything and let God sort ’em out” methodology. Clearly, this can come across as a very hostile and counterproductive stance to take when each group’s objectives can be accomplished without compromising your team’s integrity and their underlying value to the company. Of course, each QA group has their own reporting rules, but if the issue is not unique to what’s already in the system, linking it to the existing issue and maybe providing an update with the platform information and a log file for your configuration is typically good enough to help communicate to Engineering that the issue is reproducible across environments or releases and still needs to be addressed without adding to the ever growing list of issues to triage and fix.
Though easier said than done, you can overcome this trap by emphasizing quality over quantity. As we mentioned, test engineers want to show they have accomplished something. Testers don’t get credit for not filing a bug- no matter how high quality their work is.
Solution: Give incentives to the QA group for linking existing bugs to new test runs. This will save a ton of time, won’t result in duplicate bugs that distract and upset developers, and provides more details on what environments a bug fails in. If the bug already exists the user can quickly link it to the new test run, and if needed add additional details to the existing bug. The added details should actually help development resolve the issue faster rather than distract them. In this case both the developer and tester saves time and productivity increases.
Search and Destroy
Encourage your team to practice better search and discovery efforts before filing new issues. If you are working on a small team and have the luxury of working closely with your fellow test engineers, take the time to discuss an issue with the original reporter to make sure you’ve got something unique on your hands. Maybe their issue just needs a few more details and you found an identical bug. Many test management tools can help by proactively listing out similar issues or issues found in the same test section or test module, which can significantly cut down on the dreaded “dupes.”
Putting it All Together
As a product manager I want my team moving fast and not wasting time on redundant documentation, infighting, or wading through a long list of bugs. In addition to bothering developers, duplicate bugs often distract me and the management team too. Looking through progress reports can be frustrating when time and again the bug count doesn’t seem to match up with reality.
Thanks for reading and let me know what you think.