Risk-Based Test Planning with Google Test Analytics
The trials and tribulations of testing by Brindusa Gabur
“The test plan is dead!”
This was the opening remark that James Whittaker (Engineering Director at Google) started with at a conference we attended last November. Among the audience, most of the people had at some point created test plans, but very few of them still used them even after a short period of time.
This scenario reminded me of a first test plan experience that I had some years ago while working on a massive document management system. It was my first week in a new job and I had to deliver a formal test plan for the customer to review and approve. I have to mention that this was in the waterfall era of my life, and at that time, formal testing documentation would make me nervous. After carrying out a bit of research on what a formal test plan should look like, I set about creating one. After 3 days of writing my plan, I sent the first draft for review and continued adding more information to it: details about the project, i.e. what needed to be tested and what needed to be left out, what testing methods were going to be used and where, testing environments, etc. Having not received any feedback for 5 days after that, I started asking the company for feedback on the test plan. Everyone was impressed by the size of the document. It seemed very thorough and full of useful information, so the test plan was sent to the customer and approved. After a while, during development, loads of variables from my plan were changed, which meant I had to spend time updating it. Also, I soon realized that I was one of the few users of that test plan, and if there were any questions or doubts, people would come up to me and ask me directly rather than open the document and search there. At some point, the energy for updating the test plan died along with the use of it.
As can be seen, you can spend a lot of time and energy writing test plans that are often long and full of details about a project, but are quickly abandoned.
How do we evolve and create leaner test plans?
How can we stick to the big picture of the product and not waste too much time on details?!
To evolve test plans from dead artifacts to useful and reliable documentation, they have to fulfill the following criteria:
- They must always be up to date
- The need to describe what the software should do and summarize how it will do it
- They need to create a snapshot of how the software and its major parts are structured
- The need to be easy to understand and reviewable by others
From a testing perspective, there are rules that need to be respected as well:
- It should not take long to create and must be easily modifiable
- It should describe what must be tested and what is not tested
- It should be useful in determining test progress and coverage gaps
With these criteria in mind, employees at Google set about creating a methodology that can replace a test plan which they called ACC:
- Attributes – adverbs and adjectives that describe the high-level concepts that testing is meant to ensure
- Components – nouns that define parts of the product (features, classes, module names)
- Capabilities – verbs that describe user actions and activities
Taking the basic idea of risk (impact x frequency), combining it with ACC, creates a powerful model for analyzing applications, and guides the tester’s thinking while testing.
An example using the test analytics tool
Test analytics is a tool that helps to create an ACC matrix step by step and provides a visualization of the matrix and risk associated with your ACC.
Let’s consider a very simple online shopping application.
The first step is to add the attributes of your product. These attributes are adverbs and adjectives that describe your product. They may simply be the answer to questions like “What are the advantages of the product?” or “Why is this interesting to users?”, or they can be found in requirements, advertising, packaging, etc.
Fig. 1 – Attributes
Continue with the components of your application. These components are nouns that state the major parts of your product. You can find them in design docs, areas in bug databases, area owners, names of classes or modules, etc.
Fig. 2 – Components
The capabilities of the product are verbs that are used to describe what the software actually does. For each intersection of attributes and components, a capability can be defined, if applicable. If the product does something that does not match an attribute/component intersection, ignore it or take it out.
Fig. 3 – Capabilities
Capabilities can be associated with inherited risk based on frequency and impact, which feeds the risk matrix.
Fig. 4 – Capabilities and inherited risks
There is also the option of importing bugs & tests and tracking code check-ins. Combining inherited risk with bugs, code churn and test coverage creates a realistic visualization of the highest risk areas of the application. As a result, it allows you to prioritize the hottest areas and to communicate the maximum and average risks when deciding to ship.
How is it different?
One big difference between the classical test plan and the ACC approach is the focus on the actual business traits of your product. It is harder to start with the attributes of your product when you start planning than initially thought, but ultimately it pays off as it focuses your testing on the core areas of your product by helping you to keep in mind the value that is delivered to the user.
From a visual perspective, ACC makes it possible to see test coverage and risk areas at a glance. You can find test cases associated with a component/attribute within one click, which makes maintaining and retrieving information much easier.
More information about the tool and ACC can be found at https://test-analytics.appspot.com.
Test planning and risk analysis is not the most exciting part of our job, but it is important. It allows you to get an overview of your product and helps you communicate it to others as well. Test planning needs to concentrate on what is actionable, and it needs to flow and have continuity. Most importantly, it needs to result in relevant test cases. It is vital to make the change from a heavy, never opened, out-of-date document to a fit and agile tool that everybody relies on.
ACC can be a way to achieve this, but it is not the ultimate silver bullet.
About the Author
Brindusa Gabur works as a Manager Quality Assurance in XING's Events team. She focuses on optimization of workflows, combining test automation, exploratory testing and training skills to support the team deliver the best quality product. XING Profile »