Guide to Product Planning: The Four Feature Buckets

I have personally never had the good fortune to have less features than my development team could build, or my sales team “needed” to win a sale, or management wanted to expand the market.  I always faced tough choices and needed to find the right balance of features to satisfy all the stakeholders and result in a successful release.

Most of the time you will not be able to satisfy all stakeholders; however, the following guidelines have helped me develop successful roadmaps that my team can get behind and that satisfies the needs of sales and marketing.

The key for me was to  have a blend of features that satisfies the needs of different stakeholders within the organization.  To do this I always identify features as part of one of four buckets:

  1. Market Movers
  2. Customer Feedback
  3. User Experience (UX)
  4. Internal Issues

Don’t get me wrong – there are some features that can fall in more than one bucket.  For instance customers of the checkin app Burbn identified pictures, and the filters for those pictures, as their favorite aspect of the app.  The customer feedback highlighted not only a major feature that needed attention, but a market mover. The solution was a market moving change based on a the photo filters that the users liked. Eventually it was identified that Burbn had a lot of UX challenges too, and probably some internal issues.  The solution was pivoting from the Burbn app to a more simplified app called Instagram.

You are not likely going to find a lot of features like the above that fill all the buckets.  In most cases you will find that  categorizing features into these buckets forces product teams to be intellectually honest with why they are implementing a certain feature.  Is it because we want to differentiate ourselves from the competition? Do customers want it?  Is it just cool? Do we need to make a change to avoid a bigger problem?

Lets talk about the 4 buckets in more detail.

1. Market Movers 

These features are intended to improve your market position and increase product metrics (Engagement / Growth / Revenue). Typically few features are truly market movers.

Identify market movers and build a solid business case around them- traditionally this is often done with a Market Requirement Doc (MRD).  Even if you are not creating a formal MRD it is important to spend more time on analyzing Market Movers.  Often these features can take your product in new directions.

Not all growth is good.  Will the new users compliment your existing users? Does adding an older demographic of user help or hurt your business?

Engagement sounds great, but you need to plan it. Can your organization handle the feedback?  Are your servers ready for the additional volume?

Not all revenue is worth seeking. If the revenue coming in from in app video commercials is leading to a heavy churn you may want to reconsider your strategy. A large Non-Recurring Engineering (NRE) fee can definitely benefit a company, but can also side track development and cost your market lead.

Since the success (or failure) of the product will be based on the evaluation of the improved product metrics do your best to build support for the market movers from all stakeholders.

2. Customer Feedback

These are feature requests (and bug fixes) that customers are actively requesting.  This may sound obvious, but listen to your customers. You may not want to implement every suggestion, but keep an open mind.  A a good product manager will know which features the customers are asking for the most.

At a minimum write up the user story for each feature request.  I have found great success getting support to write up the initial user story and then massaging it from there.

Note: If your product is for businesses it can also be helpful to add a business case.  While the user story focus on what the user expects the product to do, the business case identifies why the feature will benefit the customer.

Always try to include some obvious customer features and bugs with each release.   Nothing irritates a customer more that to see new features released that exclude features they have actively petitioned for.

Don’t forget the bugs. Customers like new features, but even small bugs can be irritating. A bug fix won’t help the sales team, and you can’t crow too loudly to the retail channel about fixing a bug; however, a bug left untreated will irritate a customer to no end.  There is a reason why we call issues “bugs”- they bug people.

3. User Experience (UX)

UX features are those that have not been asked for, but delight customers when they see them.

No matter how much you reach out to customers they will rarely let you know what they like or don’t like. Numbers for user feedback are hard to come by, but consider the 90-9-1 Rule used for Social Media participation. Basically 90% of the users contribute nothing, 9% occasionally contribute, and 1% are heavy contributors.  This also holds true for online product reviews on Amazon in which on average 10% of buyers leave reviews.  The lack of user feedback means that UX most often requires progressive action to improve.

User on boarding is a classic example of a UX problem that you need to focus on.  Users that try your product, but don’t like it rarely give you feedback. In fact, most users only provide feedback if they really like or dislike a product.

Analytics can also be used to discover issues in the current UX that need to be relieved.  Analytics can help spot bottlenecks, unused features that should be removed, and identify high use features and workflows that should be refined.

A simple analytic is your Retention Rate (# of people who use your app during a set time frame  / # of people who used your app within a previously time frame).  By analyzing the retention rate you can see if you have a high churn rate and need to consider improve your user on boarding. Since the problem may not be the on boarding, but the value the user finds with the app you should also look at the Daily Active Users (DAU).

A good product manager will always try to go beyond a customer’s feature request(s), or a bug(s) fix and often solve it in an unexpected way.   Whether you are using focus groups, analytics, or intuition the best UX solutions take the feedback of multiple users and create an innovative design that leverages technology to come up with an unexpected solution to a customer pain point.

4. Internal Issues

These are features that no user wants, but need to be done anyone.  Whether the feature is a database redesign, the addition of analytics, or an upgrade to the middleware an internal issue often only has one or two stakeholder who are pushing for it and little to no value to the customer.

Internal issues can be the most contentious features because of the difficulty in getting all stakeholders to agree.  Often internal features are not considered urgent because customers are not asking for them.  Too often internal issues are considered not important for the same reason.  This can be a big mistake.  A good tool I use for evaluating the importance of internal issues is to figure out where they fit on the 4 quadrants scale (read  7 Habits of Highly Effective People, Stephen Covey for a full description).

Urgent Not Urgent
Important  1  2
Not Important  3  4

The above table shows the correct order in which you should handle tasks.  The biggest challenge is the Not Urgent, but important issues.  This is often the area where internal issues fall.  Everyone recognizes that the feature is important, but it gets ignored while everyone fights the urgent fires.

I have also found that development can get stuck on Not Urgent and Not Important issues- features / fixes that just doesn’t sit well with them.  I have been guilty of this plenty of times too.  When I took over QMetry Test Manager (QTM) I was very irritated that many of the text was hard coded.  This did cause a number of bugs that needed to be fixed, but because of my previous experience with localized application I inflated this issue’s importance and urgency. The reality was that we had very little market demand  for a localized product. Hard coded text is a bad practice, and did cause some minor support issues, but after evaluating the overall need of the product I couldn’t argue that it was not in quadrant 4.

Putting it All Together

For optimal success with your releases it is important to consider features from all 4 buckets.  Your market movers ensure that the your strategic vision continues to provide the resources to invest in future iterations. Customer features ensure that your customers see that their invested time in your product is rewarded by a company who listens and delivers.    Improved UX helps to keep the customer delighted and highlight your ability to keep up and innovate with the marketplace.  And the internal fixes prevent bumps in the tracks from derailing your development train.

Conversely, if your releases lack features from one of these buckets represented, it likely means that you are excluding parts of your organization or your customer.  Eventually this will significantly impact your long term success.

Most development teams don’t ship monolithic feature redesign.  Most likely you are using some form of Agile with small iterations.  While many organizations are shooting for 2 week sprint releases, most companies are using a hybrid approach in which a series of sprints are combined over the course of a three month quarterly release. Releases are often either “Minor” (1.1) or “Major” (2.0).  In the case of Minor releases you will likely focus more on customer and internal issues, while Major releases will likely contain more Market Movers and UX issues.

No matter what type of release you are working on take the time to find a set of well rounded features that will keep your customers and internal stakeholders happy.

 

Product Manager’s Framework for Developing Successful User Experiences

As a long time product manager I have had to develop many new applications and features. Over the years I have learned many lessons. The below guidelines will help you design a good user interface (UI) by focusing on developing a successful user experience (UX).

This article will not focus on specific points of style. Whether you are building a consumer or enterprise product style is very important, but that I’ll let the UX designers cover that topic in more detail.

These 10 guidelines should apply to all features, large and small.  Whether the feature is simple or complex a product manager should think through the guidelines I have listed. I have learned the hard way that even small tweaks to an application can benefit from these guidelines.  Something as simple as whether to use “Close” or “Cancel” can make a big difference.  (see a Cautionary Tale: Cancel is not Close).

#1 Understand the Original Pain Point

It is important to understand why this UI/UX feature was requested (e.g. problem is it trying to solve). If it was a customer request then discover what the original reported problem was (note: this is not always what has been escalated).

Once you know the original pain point ask yourself “Are there any other related problems that can be solved?” Often one thing leads to another.  While you want to be careful of mission (and feature) creep you also don’t want to stop short of thinking through an issue. For instance when I was asked to add Active Directory (AD) keywords to the user profiles of my last product I could have easily added the keywords to help admin’s sort their users, but stop short of solving the real pain point of  providing AD based permissions .

#2 Get To Know Your Users

Understanding your users helps and what they are looking for help you solve a user’s problem in a way that they would expect them to be solved.

Don’t stop with just the persona that is requesting the feature, where a persona is a fictional stakeholder (e.g. user). Ask yourself if there are any persona’s that might object or be confused by the new feature.

Note: A persona  may include a name, picture; characteristics, behaviours, attitudes, and a goal which the product should help them achieve.

Find out as much as you can about all the personas that use your product before you start.

#3 Define the User Story

Do not assume that others on your team have your same understanding.  Ambiguity can easily slow development, irritate the team, and produce lots of bugs.

Whether you are using Agile or not you should always have a user story for every features and thus every UI element.  A user story goes like this:

“As a <role>, I want <goal/desire> so that <benefit>.”

See other alternatives to this format here.

Whether it is a simple button that needs to be added, or a complex user experience you need to be clear on the user story before you begin the design.

If you are working with a UI/UX designer review the user stories with them.   Let them ask questions, and give them answers.  Debate those answers.  Often this debate will result in a better understanding of the pain points and the persona’s involved.  This will also test a user story’s merits, if you can not convince the designer that the user story makes sense you may need to reprioritize.

#4 Develop An Initial Wireframe Workflow

So, now that we know who our personas are and what problems they are experiencing, we can start creating a wireframe, right?

Wrong.

One of the common mistakes I see in many applications and websites is poor workflow.  A feature is technically available, but it is hard to find, or it is found in one location, but not another.  It is always useful to walk through the user stories and see if there are any alternative workflows that make sense.  For instance:

  • Are all your users going to start from the same place?
  • Is the feature available if a user is not logged in?
  • Is the feature different when a user is not logged in?
  • Is the UI different for different users types (admin, user, guest, etc.)?
  • Does (or should) this feature require permissions?
  • Is this current UI the real problem?
  • Should this new feature be applied to similar functions?

Really think about the last two question. It’s tempting to want to not over complicate things (e.g. use the KISS principle); however, sometimes fixing a bug or adding a feature is just scratching an itch and the real problem won’t go away.  Adding something will inevitably lead to new user expectations. For more read a Cautionary Tale:   Unexpected Consequences and  Permissions Permissions .

#5 Develop A Successful User Interface

It is time to start thinking about how to simplify the user experience so that users know how to find and take advantage of the feature. It is very important to communicate effectively to the user.  I have been shocked how many times a simple UI mistake has ruined a new feature.

  • What language should you use?
  • Is there a need for an icon?
  • Does user need confirmation?
  • Would a pop up / guide message help new users?
  • Should you have a link to more information?

Word choice can make a big difference.  A clear and short title helps users understand features. Icons can help users find features, and often can assist in understanding.

Complex features may have different options that require a bit of explanation.  Sometimes that can be handled inline, but too many options or text might overwhelm or confuse the user.  Offer too many options and the user will probably do nothing. So choose wisely. Be sure that everything you add has a valid reason for being there. If it doesn’t solve a problem remove it.

Read  Attention To Detail for an example.

#6 Review Usability

Now that you have solved the user story, and started to develop a solid UI it is time to analyze your UI for usability. You have hopefully already looked a It is your job to ensure that your users not only understand the feature, but that i want in the quickest, most effective way possible. Problems might include:

  • Not optimized for mobile
  • UI too busy or font and icons too small
  • Not accessible to screen readers or color blind people.
  • Load time is too long
  • Too many steps in sign up flow
  • Is there help?
  • Do you have examples?

If you are developing a product that is web based you need to think about mobile. Even if your target market is not mobiel you need to consider that  54.34% of web browsing is now done via mobile devices (source: W3 May 2017). With the majority of users on mobile even if your niche of users is not mobile heavy there is likely going to be some crossover.

Another point to consider is that if your product is not mobile, maybe you should add a mobile app to compliment it.

Whether you are using mobile or not you need to think about how visually friendly the UI is for the average users.  Icons are great, but are they big enough to see clearly (or too big)?  Are there too many icons? 1 Are you including too much text? Are the icons obvious to the color blind?

When a user opens your UI they should immediately see how it can benefit them.  If they do not then they will likely leave and won’t come back anytime soon. To help with this you can add some guides that explain the user interface.  Having user guides is extremely important if you are:

  1. Introducing a new UX
  2. Using a UX that might be new to your audience

When “swipe right” was first introduced it needed to be explained.  Once it had been in use for a few years with trend setting apps like Tinder many people thought that swiping was obvious to everyone and started to add it to all sorts of apps including enterprise applications. The problem was that many enterprise users were not Tinder users. Not only did users find the application difficult to use, but enterprise users were not happy to find out that they were not hip enough to use the new UX.

Bottom line: A good UI does not need to be explained.  This was pointed out to me a few years ago by a customer.  I had redeveloped a complex UI and had added a whole bunch of tool tips and popup guides. I thought they were necessary due to the complexity of the UI; however, how could I argue with the statement “A good UI should be obvious”.  I had a lot of room for improvement.

1 Many years agoMicrosoft Office switched from having multiple independent tool bars to a single ribbon menu scheme. I believe the idea was to avoid the problems with multiple toolbars taking up so much space. The new ribbon also allowed them to increased the size of some of the more important icons. Both problems were important to fix; however, I was irritated myself since I was comfortable with the old system and had created custom toolbars that were no longer available. Worse, I noticed that most of my coworkers were confused by the changes. My coworkers were simply overwhelmed with the choices and often could not find the features they were looking for.  The ribbon was a new UX concept and took awhile for users to adjust.

#7 Find Ways To Encourage Users

One of the big ways that you can improve many apps is providing some sample data. Design a journey that seems natural.  Encourage users to follow an obvious path.
When a user makes a mistake find a friendly way to inform them and offer help in figuring out what to do to fix the problem.  Too often users are informed they made an error and left hanging to figure out the solution on their own.

This often leads to frustration, and for the user to blame you and to instantly lose trust.

#8 Add Rewards and/or Gamification

Gamification does not apply to every feature, but give it some thought.  How can you reward your users for using the application, using the feature?  Is there a reward that you can offer? Can you at least congratulate them in a human tone?

Reward your users and they will feel compelled to use your product. Make them feel loved. In turn, they will love your product.

To be clear “Gamification” does not mean that you turn your application into a game. It simply means that you are trying to use the principles of gaming to encourage the continued use of your application.

#9 Consider Analytics

Whenever possible add analytics so that you can determine in the future if you have made the right choices.  You are not always going to make that right choice.  Worse, users are not always going to tell you when you got it wrong.  In fact, most of the time users will never tell you they have a problem.  They just won’t buy, or will stop using your product.

You want to be able to analyze what features, and options, are being used.  The more feedback you can gather and analyze the better you will be able to refine your application.

If you have already added analytics to your product then you need to review what the analytics is telling you as far as the importance of the feature.  I’m sure analytics was considered as part of step 1, but now that you have moved forward it is a good idea to check again.  Based on the current analytics is the feature still worth developing? What are you hoping to gain?

#10 Define the Scope and Roadmap

Now that you have developed the user story(s) and workflow(s) and considered the UI you need to determine if the feature should be developed as a whole or in increments.

After fully analyzing a feature request, developing the user stories, and considering all of the unintended consequences you may discover that you have an Epic and not a feature.  You then need to consider if you have the bandwidth to handle the whole feature set in a sprint or two.  You might even need to span a feature across releases like I had to when I implemented AD based permissions.

If you have slid down a slippery slope and can’t figure out how you are going to climb out of the hole you dug, don’t worry. Start reviewing the stories and placing those that are needed first (the predicates) at the top.  Sometimes this means that stories that solve high priority pain point will be lower on overall priority because they are dependent on the other predicates stories.  Make sure that you link the issues together. It is important to note that as you build a ladder of features that some of the features at the bottom rungs  won’t be obvious advantages.  Often they are backend changes that development needs to make before the real pain points are made.

It is important to be as transparent as possible and communicate to everyone what is going on. Let development know the end goals, let sales know where you are headed,  and when appropriate let customers know that back end changes are happening.  Sometimes you won’t be able to show immediate progress, but as long as you keep communication going you should be good.

Bringing it all together

Whether you are fixing a UI bug, adding a small feature, or creating a brand new UI take the time to pause and think about the long term effects and goals of your product roadmap. In many cases you will be able to run through the above steps quickly- after all not ever problem is complex. When you do encounter a pothole in the road then take the time to explore fixing it. While often you will chose to delay filling the pothole, your analysis will help avoid passing by potholes that grow to upset the customer experience or those that grow into sinkholes that might swallow your forward progress.


Cautionary Tales

Cancel is not Close

When redesigning QMetry Test Manager we decided focus on providing a multi-tasking UI/UX. Users had been consistent in complaining that the old UI did not allow them to view multiple asset at the same time. To solve this problem we developed a desktop like UX that allowed users to open windows. 1 The standard window template had an OK and Cancel button to allow users to close the window. This template worked well in almost all cases; however, we had a subset of windows that allow users to take multiple actions. Each action had a separate confirmation so those windows had the OK button removed; however, the Cancel button remained. This consistency was correct, right?

Wrong.

Because we chose to use “Cancel” users felt that it should cancel the previous actions. Many users were confused and afraid of clicking on the cancel button. When the user base was small the problem and was easily explained away and was unfortunately ignored. Eventually a large enterprise customer’s not only complained about the UX, but submitted a “bug” that the Cancel button needed to be fixed– they were requesting the Cancel button to Cancel all previous actions. 2 Changing Cancel to Close was no longer an easy text fix, and the relationship with our largest enterprise account was strained.

1 Many criticized the use of windows UI framework for the new UI as “old technology”. Some of the fiercest critiques came from UX specialists who wanted to use the latest technology and make the UI “mobile friendly”. The reason we chose a windows framework was A) it solved the main UX pain point of multi-tasking B) All our potential users were very comfortable with it which improved user on boarding (another pain point) C) after multiple trials with a separate mobile application we had concluded that our user base was not interested in using our service on mobile phones (Note: Our windows based web interface did work with mobile browsers if needed).
2 The request for Cancel to Cancel all previous actions was actually greeted by development as “a feature request not a bug”. The feature vs bug debate is common between development and product managers and is worthy of its own article. For now I’ll just say that I agreed with development. Not only was this a feature, but it was actually a very complicated feature that involved a few technical challenges. To the user the problem was small and simple; however, the solution would be difficult, take a long time to develop, and would not produce a value add.

Providing AD Based Permissions

One of my largest customers requested us to add Active Directory (AD) keywords to our application. They were using AD to keywords to organize their thousands of users and wanted to use the same keywords to organize the users in our application. Since we already supported AD the import of the keywords was actually very simple to add. We could also easily add a column to the user list for the keywords to allow the admin to sort and filter by keyword. Unfortunately, the real pain point was not sorting or filtering. What the admin really wanted to do is assign permissions to our application based on the AD keyword.
This was a problem for many reasons too technical to go into here, but suffice it to say that had we moved forward with only the original reported pain point we would have not pleased the customer. As it was we had to create an Agile Epic issue which we broke down into separate features that we implemented incrementally.

Unexpected Consequences

When I took over product management for cloud based test management software I discovered that the the development team had added the ability to delete requirements.  This was based on a customer request; however, the customer had only asked for requirements to be deleted and not to delete the core data type used: test cases. The button was placed in an unusual location which made it hard to find.

After the initial release few customers were pleased and more customers were actually upset with the limitations. Instead of the new feature showing our users that we were listening to them, they took the new feature for granted and wanted more. Due to the complex nature of our SaaS application and the database schema we could not add the ability to delete test cases without considering a host of complex use cases- some of which were technically unfeasible.  Too late, we were headed down a slippery slope…

Moral: If you are going to do something, then do it right.

Permissions Permissions

A second problem that was created when delete was added to the application was that the original customer just wanted the delete feature.  They had a small flat organization and didn’t care who had the ability to delete an asset.  This did not work for our other customers who either wanted no one to be able to delete items, or wanted the feature limited to specific users.  Adding a user permission for delete had to be considered a requirement for the delete- and many other features.

Attention To Detail

One of the last bugs I solved with QMetry Test Manager was to unify the dialog boxes so that the escape button would always close the dialog box.  Believe it or not this was a big problem for us.  Why?  We had tens of thousands of core users that preferred navigating with the keyboard for speed.  We added keyboard hotkeys to improve the UI for them; however, this caused a bigger problem.  We started to field complaints from our enterprise users- the management team – because users were complaining of inconsistent behavior.    The original spec had defined the dialog boxes to have a close button; however, after v1.0 the original spec was rarely referenced and many new team members had never seen it.  Since a dialog box was so simple it was rarely mocked up which led to many new dialog boxes were created without a close button.  So, when we added hotkeys many dialog boxes would not close with the escape key and our power users complained.

Why was this a bigger problem?  Before the users were making a feature request- hotkeys.  Now they were reporting bugs and telling their management about an inconsistent UI.  To make matters worse other inconsistencies started to be highlighted!  The fix was simple, but finding all of the dialog boxes was a chore for development and QA.

Note: In case any of you think that development should have kept following the original spec.  Yes, in a perfect world that is true; however, if you expect a perfect world, and don’t want to take responsibility for the imperfections then don’t become of a product manager.

Preventing Major Bugs Late in Development

Introduction

This is one of a number of articles I am writing to help identify common problems related to Quality Assurance (QA) the impacts the release of  products to market.  While I have been a product manager for most of my career I have helped develop QA test plans and spent 6 years developing a QA test management tool.  The guidelines I outline here are based on the QA best practices that we helped our customers implement in the field.

When possible I will provide real world examples from QA experts to illustrate the root cause of the challenges, and the real world techniques that are used that can help you avoid these pitfalls in your day to day efforts. Hopefully if you can employ these suggestions you’ll reduce the churn between teams and demonstrate a more collaborative path forward is possible with the right implementation.

The Challenge: Preventing Major Bugs Late in Development

Moving away from Waterfall to Agile

A majority of testing organizations have traditionally followed the Waterfall method, thus allowing engineering to bring the project to a “testable state” at which point all available test resources would move from test authoring and other activities to performing a full pass or regression effort against a very complex set of features and changes that were un-tested (by QA) for weeks or possibly months. While most development team have transitioned to a more Agile methodology, many test teams are still using Waterfall type processes.

We don’t believe this is due to a lack of skill or experience from QA professionals, as there are many valid arguments that can be made in support of validation type testing.  This is especially true for third party testing when products are ready to be “certified” for submission into a program or marketplace. Another reasons for waterfall persisiting in QA is that many QA teams are outsourced.  Outsourced QA usually follows a Scope of Work (SOW) with defined requirements and milestones.  It is hard to teach Agile when the team is told to wait for, and then work towards, a SOW.

No matter the reason, testing efforts that are being undertaken as part of an ongoing development effort can receive many benefits in becoming more agile themselves.  Whenever possible they should work in parallel with engineering at as early a point in time as possible.

Striking a Balance Between Documentation, Coverage and Agility

When should your team begin testing a work-in-progress development effort? It’s not an easy decision for any QA Stakeholder to make. If your team has been provided with limited documentation on a new product or new feature set that you will be testing in the coming weeks the old mentality from Waterfall-style testing was to spend weeks or months writing test cases in a vacuum. Sure, you can share your progress with Product Management or Engineering to ensure coverage is adequate once you get your hands on a build to begin testing, but you’re not actively testing anything. Before we go any further, let’s be clear about our intentions – documentation is critical.

User Stories as a Framework for Testing

If your engineering team is working off of User Stories to define new features and there is adequate documentation for engineering to begin their work, then it’s likely that there will soon be an opportunity for your test engineers to begin testing early code integrations in support of the new stories. In fact, writing test cases in an agile environment reduces the common problem where QA has to start testing new features in “ad hoc” mode with little to no documentation. The stories themselves should provide enough information as a jumping off point and short, collaborative contact with engineering can help flesh out the test engineer’s authoring process so writing a valid test of a new feature doesn’t take weeks to complete.

We’re not drawing a line in the sand stating that spending time on writing formal test cases has no value, far from it. Consider the longer your resources are not engaged in providing incremental test support for engineering then there’s a good chance that major bugs are being written into the product’s systems and engineering is building on top of an unstable foundation. Depending on what flavor of Agile you’re working in you may even have assigned a 1:1 resource for QA and Engineering, in which case this engagement can be started earlier in the sprint than you might think.

Shift Left and Why It Matters

If QA is working in parallel with development than they should be able to test earlier in the cycle than waiting for a miracle build and then starting their regression testing. The Shift Left movement is helping Quality Assurance transition to Quality Engineering and allowing test efforts to begin much earlier in development, which should help to alleviate many of these last minute roadblocks. Many organizations are moving aggressively to automation testing with the goals of enhancing efficiency and expanding scope, but many are being driven by basic market forces. Their competition is working faster and the market disruptors can turn out more innovation through nimble product release schedules across many device platforms, most of which would not be possible and still carry a reasonable level of quality and functionality without the support of automation tools and new processes. For a different perspective, look to your own past of product testing and you must have experienced a disastrous show stopping issue that impacted the business and your team that might have been avoided if found earlier in the cycle. I certainly have and surely if you haven’t yet stay vigilant and keep improving your skills and maybe you’ll be able to avoid the worst case scenarios.

Thanks for reading and keep an eye out for more best practices.

Reducing Duplicate Issues

Introduction

This is one of a number of articles I am writing to help identify common problems related to Quality Assurance (QA) the impacts the release of  products to market.  While I have been a product manager for most of my career I have helped develop QA test plans and spent 6 years developing a QA test management tool.  The guidelines I outline here are based on the QA best practices that we helped our customers implement in the field.

When possible I will provide real world examples from QA experts to illustrate the root cause of the challenges, and the real world techniques that are used that can help you avoid these pitfalls in your day to day efforts. Hopefully if you can employ these suggestions you’ll reduce the churn between teams and demonstrate a more collaborative path forward is possible with the right implementation.

The Challenge: Reducing Duplicate Bugs

Nothing frustrates an engineering team more than seeing the same issues over and over again, across builds, sprints and releases. Looking at the cause can uncover some understandable origins.

The Problems

Lack of Centralized Bug Reporting and Tracking

The first challenge may be simply that multiple testers are reporting bugs separately and do not know when a bug has already been filed.  Some teams even lack access to a central defect tracking tool such as JIRA, Bugzilla, FogBugz, etc.  In this case testers sometimes have to resort to reporting bugs via excel spreadsheets or google docs, which is in turn given to engineering later.  Sometimes the QA lead will collate the separate bug reports and remove redundant bugs; however, duplicate bugs still slip through. This is surprisingly common, particularly with external QA teams.

Solution: Most modern test management tools can integrate with your bug tracking tool. By integrating your bug tracking and testing tools you can enforce a workflow with proper permissions and controls.  By forcing QA to link a bug directly from a test suite / test scenario you can reduce duplicate bugs since users can see the existing bugs that are already associated with the test scenario before filing a new one. 

Large and / or Distributed Teams

Even with a centralized bug tracking tool duplicate bugs are still far too common in large and distributed teams. This is often because multiple testers are often required to test the same or similar functions.  A test case is often run through many different test scenarios, and against many different hardware and or software configurations.  For instance, an online retailer will want to test the checkout process with many different shopping cart statuses, payment methods, and user types, and against multiple different browsers.  Add mobile checkout to the list and you have many permutations of the same basic test. It is often very difficult for one person to do all the testing, but if two people are testing the same function then how do they know when a bug has already been filed?

The most straightforward solution is use an agreed upon naming and tagging process to categorize the bugs and then have everyone search for the existing bugs.  This sounds good in theory, but in practice is very difficult.  Not only is there a productivity loss as users search for bugs, but it is still easy for things to slip through.

Solution: Use a tool, or workflow that shows bugs that have already been linked to a test case.  QMetry and qTest both provide this UX.  Whether you are testing a test case in one or more different user scenario you will be able to see if there are any bugs related to the test case.  This allows android testers to associate UI bugs that were already found on iPhone. A browser-based bug that was previously reported on Firefox could be updated when it is reproduced on Chrome, IE or Safari. In many organizations these would be considered separate bugs; however, the fix is often the same. 

Bug Count as a Measure of Productivity and Value

Many QA teams measure productivity in terms of issues found (or bug count), so individual team members feel compelled to file as many issues as they can in a week to show they’re working hard and making progress throughout their testing. Even if this doesn’t describe your group’s mentality, it’s human nature to want to feel like you’ve accomplished something new each day even if what you’re seeing has been found in other environments or across other builds. If this is a driving force behind our assessment of personal value than we should take stock and consider the impact on our counterparts who, for most engineers, would prefer to be working on new areas of the product, coding new features and functions, not drowning in a sea of duplicate issues. Don’t get me wrong – unique issues are measurements of progress and value for any QA team, but a daily bug count, no matter the quality does more harm than it does good.

In my over ten years of working in and around QA with a wide range of projects I’ve seen some example of Quality Management making the decision to write up everything you see as a new issue and let Engineering and Product Management deal with the mess. You can think of this as the “Bug everything and let God sort ’em out” methodology. Clearly, this can come across as a very hostile and counterproductive stance to take when each group’s objectives can be accomplished without compromising your team’s integrity and their underlying value to the company. Of course, each QA group has their own reporting rules, but if the issue is not unique to what’s already in the system, linking it to the existing issue and maybe providing an update with the platform information and a log file for your configuration is typically good enough to help communicate to Engineering that the issue is reproducible across environments or releases and still needs to be addressed without adding to the ever growing list of issues to triage and fix.

Though easier said than done, you can overcome this trap by emphasizing quality over quantity. As we mentioned, test engineers want to show they have accomplished something.  Testers don’t get credit for not filing a bug- no matter how high quality their work is.

Solution: Give incentives to the QA group for linking existing bugs to new test runs.  This will save a ton of time, won’t result in duplicate bugs that distract and upset developers, and provides more details on what environments a bug fails in.  If the bug already exists the user can quickly link it to the new test run, and if needed add additional details to the existing bug.  The added details should actually help development resolve the issue faster rather than distract them.  In this case both the developer and tester saves time and productivity increases.

Search and Destroy

Encourage your team to practice better search and discovery efforts before filing new issues. If you are working on a small team and have the luxury of working closely with your fellow test engineers, take the time to discuss an issue with the original reporter to make sure you’ve got something unique on your hands.  Maybe their issue just needs a few more details and you found an identical bug. Many test management tools can help by proactively listing out similar issues or issues found in the same test section or test module, which can significantly cut down on the dreaded “dupes.”

Putting it All Together

As a product manager I want my team moving fast and not wasting time on redundant documentation, infighting, or wading through a long list of bugs.  In addition to bothering developers, duplicate bugs often distract me and the management team too.  Looking through progress reports can be frustrating when time and again the bug count doesn’t seem to match up with reality.

Thanks for reading and let me know what you think.