Assuring product quality? Here’s the QA scorecard you need
An INFJ personality wielding brevity in speech and writing.
We’ve consulted companies from across the industries for years and realize time and again that setting up an intensive testing process, reviewing it, and making improvements periodically is a sure-shot way to release products that resonate with the end-users and all the stakeholders involved.
One of the product quality owners at a firm, for instance, said they had been in the industry for a long time and had been quite adaptive to the trends such as test automation, setting up a continuous integration/continuous delivery model, etc., and yet they were still finding P0 P1 defects escaping into the production.
Though setting up a product QA looks reassuring, the end results were not what they were looking for.
We agree that no product or software is defect-free. But it can defect less.
And that’s just one example.
There are companies with different approaches to software quality, some doing so great, and others have distorted views of quality by measuring wrong metrics and resulting in bad decisions.
Big or small, incremental changes happen when you establish a basic software quality scorecard that measures the STLC process right from requirements till you hit the deploy button.
QA score card: What’s it?
A software QA scorecard is similar to your fit bit watches: Like the panel displays how many steps you have taken, sleep hours and other vital information, a product QA scorecard shows the KPI of your product quality – the metrics that will present how well your product quality is.
In short: Quickly identify areas of improvement, inefficient processes, or underperforming teams with QA scorecards.
Who uses it?
Organizations of any size, be it small or big, use QA Scorecards to improve the team’s performance and their QA strategies. An organization can use the QA Scorecard template with a list of checklists as the guideline for the team to meet the Quality and the standards as expected by their customers on a regular basis/cycle.
QA scorecards become helpful to the following people in general:
- QA analysts
- Team leaders
- Managers
- Supervisors
Engineering teams can prepare a QA scorecard by themselves or seek external help based on the need.
However, the QA team by themselves cannot assume the quality/performance of the team without evaluating against any checklists, guidelines that the team/organization is following.
QA score card benefits:
Following are the benefits that the QA team can infer from QA score card:
- Helps identify the areas of improvement in the quality roadmap
- Clear understanding of the status or quality based on the feedback
- Set up a benchmark and teams can learn best practices as well
- Helps to monitor progress of the testing activities.
Building a QA scorecard
As mentioned earlier, QA Score Card is a basically a template with the list of standard questions based on the project or the business. It consists of a limited set of questions, let’s say 10 to 20 to determine the quality.
Every question posed to the team comes with a briefing and detailed explanation of the expectations and standards to be met to maintain the quality, both functional & non-functional. Also, the weighing of the questions differs based on what the team wants to achieve.
For example, here’s an example of QA scorecard questionnaire you can use to measure your entire QA activities
| Activities | Definition |
|---|---|
| Requirements | |
| Clarity | Are the requirements provided clear with detailed acceptance criteria? |
| Impact analysis | Is impact analysis done for each story and tracked? |
| Dev Testing | |
| Unit & Integration tests | Are unit & integration tests automated and coverage analyzed? |
| Static code analysis | Is static code analysis done to uncover issues even before execution? |
| DevOps (CICD) | |
| Automated pipelines | Is the build process automated along with execution of automated tests? |
| Archiving artifacts | Are the builds archived & history maintained? |
| Test Strategy / Planning | |
| Types of testing | Are required types of testing identified and estimated during the Sprint/Release planning? |
| Risk | Are the risks related to Schedule Effort Environment Resources foreseen during planning? |
| Test Design | |
| Traceable to requirement | Are the tests designed traceable to the requirements? |
| Coverage | Are the peer review happening to ensure coverage? |
| Test Automation | |
| Planning & Execution | Is test automation planned, designed & executed consistently for different types of tests? |
| Quality | Is the effectiveness of the automation measured? |
| Test Artifacts | |
| Results | Are the test results captured and tracked to the requirement? |
| Sign-off | Is release Go-No Go substantiated with test sign-off report? |
| Functional Testing | |
| Positive + Negative flows | Are the functional tests covering both positive & negative flows? |
| Complex combinations | Are the functional tests covering complex combination of flows? |
| Non Functional Testing | |
| Planning & Execution | Are non-functional tests planned and executed as part of release testing? |
| Test Data & Environment | Test Data & Environment specific for non-functional tests available? |
| Regression Testing | |
| Impact areas per ticket | Are regressions test based on impact analysis identified and executed for each ticket? |
| Full per Sprint/Release | Complete regression test suite executed after code freeze for each sprint / release? |
| Production Readiness Testing | |
| Customer like E2E | Are End to End test cases similar to customer test identified & executed? |
| Test Data & Environment & Resource | Is Test environment along with proper test data, pheripherals/devices and other integrations available for the E2E testing? |
| Defect Analysis | |
| Parameters collected | Are the various parameters like Issue source, environment, component / functional area, root cause, source / identified & fixed version collected? |
| Analyzed & Acted | Are the metrics derived based on the above parameters, analyzed and converted to actionalable items? |
The scoring is usually done in numbers from 0-5 or with basic Yes/No option to arrive at the score. The final score is provided in terms of percentage out of 100.
Example of a QA scorecard:
| Target | Achieved | |
|---|---|---|
| Requirements | 5 | 4 |
| Dev Testing | 5 | 3.5 |
| DevOps (CICD) | 5 | 5 |
| Test Strategy / Planning | 5 | 3.5 |
| Test Design | 5 | 3 |
| Test Automation | 5 | 3.5 |
| Test Artifacts | 5 | 3.5 |
| Functional Testing | 5 | 1.5 |
| Non-Functional Testing | 5 | 4 |
| Regression Testing | 5 | 4 |
| Production Readiness Testing | 5 | 0 |
| Defect Analysis | 5 | 2 |
As you can see, this team scores well on CI/CD processes, but they still lag in the Production readiness testing and setting up a defect analysis system. Assuring a product/software quality depends on all of these parameters put together.
Scoring in detail:
| Activities | Target | Achieved |
|---|---|---|
| Requirements | ||
| Clarity | 5 | 4 |
| Impact analysis | 5 | 4 |
| Dev Testing | ||
| Unit & Integration tests | 5 | 4 |
| Static code analysis | 5 | 3 |
| DevOps (CICD) | ||
| Automated pipelines | 5 | 5 |
| Archiving artifacts | 5 | 5 |
| Test Strategy / Planning | ||
| Types of Testing | 5 | 4 |
| Risk | 5 | 3 |
| Test Design | ||
| Traceable to requirement | 5 | 4 |
| Coverage | 5 | 2 |
| Test Automation | ||
| Planning & Execution | 5 | 4 |
| Quality | 5 | 3 |
| Test Artifacts | ||
| Results | 5 | 4 |
| Sign-off | 5 | 3 |
| Functional Testing | ||
| Positive + Negative flows | 5 | 2 |
| Complex combinations | 5 | 1 |
| Non Functional Testing | ||
| Planning & Execution | 5 | 4 |
| Test Data & Environment | 5 | 4 |
| Regression Testing | ||
| Impact areas per ticket | 5 | 4 |
| Full per Sprint/Release | 5 | 4 |
| Production Readiness Testing | ||
| Customer like E2E | 5 | 0 |
| Test Data & Environment & Resource | 5 | 0 |
| Defect Analysis | ||
| Parameters collected | 5 | 3 |
| Analyzed & Acted | 5 | 2 |
| 120 | 76 | |
| 63.33% |
The objective of any product quality team is – continuous improvement and performance. A team can make use of this QA scorecard for two different types of projects.
- Existing project
- New project
Let’s take an example of an external QA vendor working with a customer who wants to arrive at the QA score for their product, the vendor’s team will have to come up with a clear goal before constructing the QA scorecard questionnaire.
In most cases, the customer wants to know the reason behind the issues in production, and missed bugs/defects. In such cases, the vendor will kickstart with an initial engagement, wherein they will coordinate closely with the customer’s team to get a grasp of their working model, test process, and work culture and arrive at the set of questions that will help meet the goals of the customer.
The benefit of doing this is to help understand if the bug leakage to production happens due to poor QA or all of their engineering practices combined right from the requirement to release.
Arriving at the final score involves weekly meetings with the team, studying their requirements, speaking with their engineering team, processes they follow, and going through test artifacts and defect tracking tools.
Analyzing and presenting the score usually spans 8 weeks followed by the sign-off process to the client.
For teams that are just starting out to build a product, the team can have the QA scorecard template as a checklist from the initial state to maintain the quality of the product.
Doing so would help them to bridge the process and knowledge gaps, and areas they need to work on to improve their performance and be successful in terms of delivering a seamless user experience.
Criteria for QA Score Card
What can QA teams infer from the scorecard?
QA teams infer the following from the Scorecard:
- Based on the score, they identify areas of improvement viz., process, documentation
- Root cause of the issue
- Quality level of the product
- Evaluate themselves against the standard procedures/metrics
Ways to improve QA Scorecard
Just like any practice, quality involves continuous improvement. Arriving at the QA scorecard should not be viewed as a one-time process. Following are a few ways to put the QA scorecard in a continuous improvement mode.
- Monitor systematically
- Collaborate among teams (Dev, QA, Ops)
- Rewards and recognition within the team
- Transparency and continuous feedback system
Looking to improve your test automation coverage? Take a look at Zuci’s test automation services and see how you can leverage Zuci for your business needs.
The article is co-authored by Vidhya Shanmugham, Test Lead @Zuci Systems.
Related Posts
