Important Software Test Metrics and Measurements – Explained with Examples and Graphs

Measuring the quality, cost, and effectiveness of software projects along with their associated processes is indispensable. Without correct measurement, it’s impossible to efficiently complete a project.

This piece will delve into the usage of software test metrics and measurements in the software testing process, supplemented with real-world examples and illustrative graphs.

There’s a popular adage that states, “What can’t be measured, can’t be managed.”

Within the realm of project management, this implies that project managers and leaders need to promptly discern any deviations from the test plan to apply appropriate remedial measures on time. Crafting test metrics that match the project’s requirements is vital for guaranteeing the quality of the software under test.

Grasping Software Testing Metrics

A metric is a quantifiable measure that indicates the extent of a particular attribute in a system, system component, or process.

In simpler words, metrics can be considered as benchmarks for measurement.

Software metrics are employed to assess the quality of a project. Fundamentally, a metric serves as a unit that portrays a particular attribute. Just like “kilogram” is a metric for weighing, in software testing, metrics can be related to questions such as “How many issues are detected in a thousand lines of code?”. Here, the number of issues and lines of code are quantified to establish a metric.

For instance, prevalent test metrics encompass:

• The cumulative number of defects within a module
• The number of test cases performed per individual
• The test coverage percentage

Understanding Software Test Measurement

Measurement denotes the quantitative representation of the scope, quantity, dimension, capacity, or size of a specific attribute of a product or process.

An exemplification of measurement in software testing is: determining the total count of defects identified.

For a clearer comprehension of the distinction between measurement and metrics, please refer to the diagram below.

Significance of Test Metrics

Producing software test metrics is a key obligation for test leaders and managers.

Test metrics are utilized to:

1. Dictate decisions concerning the subsequent stage of activities, like estimating the cost and schedule of future initiatives
2. Spot areas that need enhancement to guarantee project success
3. Decide on necessary alterations to processes or technology, among other aspects

Comprehending the importance of software testing metrics:

As discussed earlier, test metrics are crucial for gauging the quality of software. Without metrics, it would be onerous to appraise the work executed by a test analyst. Absence of metrics would result in their test report lacking crucial details needed to accurately evaluate the project’s status.

Metrics furnish the following insights:

1. The count of test cases designed for every prerequisite
2. The count of test cases that are yet to be designed
3. The count of test cases implemented
4. The count of test cases that have been successful, unsuccessful, or are obstructed
5. The count of test cases that are yet to be executed
6. The count and severity of detected defects
7. The count of test cases failed due to a particular defect, and so forth

These metrics, coupled with additional project-specific metrics, provide a comprehensive overview of the project’s status.

Crucial takeaways that can be inferred from these metrics include:

• The proportion of work completed
• The proportion of work that needs to be completed
• The estimated time required to complete the leftover tasks
• Whether the project is on track or experiencing delays

If the metrics manifest that the project isn’t progressing as per the schedule, the manager can issue an alert and elucidate the reasons for the delay to the client and other stakeholders, thereby preventing any unforeseen surprises.

Lifecycle of Metrics

Various Types of Manual Test Metrics

Testing metrics can be divided into two principal categories:

1. Base Metrics
2. Calculated Metrics

Base Metrics: These metrics are deduced from data assembled by test analysts during test case design and execution. Such data is monitored throughout the test lifecycle, recording details like the total count of test cases formulated for a project and the quantity of test cases that need to be implemented or have been successful/unsuccessful/blocked.

Calculated Metrics: These metrics are derived from the information acquired in base metrics. Test leaders and managers typically keep track of these metrics for reporting purposes related to testing.

Instances of Software Testing Metrics

Let’s illustrate various test metrics frequently utilized in software test reports with an example:

The ensuing table showcases the information accumulated from a test analyst who is actively engaged in testing:

Here are the interpretations and procedures for computing these metrics:

#1) Test Cases Executed Percentage: This metric offers insights into the execution status of test cases, represented as a percentage.

Test Cases Executed Percentage = (Quantity of Test Cases Executed / Total Quantity of Test Cases Composed) * 100

So, in this instance,
Test Cases Executed Percentage = (65 / 100) * 100 = 65%

#2) Test Cases Not Executed Percentage: This metric reveals the backlog in test case execution status as a percentage.

Test Cases Not Executed Percentage = (Count of Test Cases Not Executed / Total Count of Test Cases Composed) * 100

So, in this instance,
Test Cases Not Executed Percentage = (35 / 100) * 100 = 35%

#3) Test Cases Passed Percentage: This metric denotes the percentage of executed test cases that have been successful.

Test Cases Passed Percentage = (Number of Test Cases Passed / Total Number of Test Cases Executed) * 100

So, in this instance,
Test Cases Passed Percentage = (30 / 65) * 100 = 46%

#4) Test Cases Failed Percentage: This metric signifies the percentage of executed test cases that have been unsuccessful.

Test Cases Failed Percentage = (Number of Test Cases Failed / Total Number of Test Cases Executed) * 100

So, in this instance,
Test Cases Failed Percentage = (26 / 65) * 100 = 40%

#5) Test Cases Blocked Percentage: This metric depicts the percentage of executed test cases that are blocked. A comprehensive report should mention the reason behind the test case blockade.

Test Cases Blocked Percentage = (Number of Test Cases Blocked / Total Number of Test Cases Executed) * 100

So, in this instance,
Test Cases Blocked Percentage = (9 / 65) * 100 = 14%

#6) Defect Density: This metric ascertains the number of defects discovered per unit of measurement.

Defect Density = Number of Defects Identified / Size

In this case, “Size” pertains to a distinct requirement. Hence, defect density is computed as the number of defects found for every requirement. Defect density could also be figured as the number of defects detected per 100 lines of code or per module, etc.

So, in this instance,
Defect Density = (30 / 5) = 6

#7) Efficiency in Defect Removal (EDR): EDR is used to calculate the test effectiveness of a system.

Suppose 100 defects were found during development and QA testing, and during consecutive Alpha and Beta testing, the end user or client discovered an extra 40 defects that could have been detected during the QA testing phase. In such a case, EDR can be computed like:

EDR = (Count of Defects Discovered during QA Testing / (Count of Defects Discovered during QA Testing + Number of Defects Discovered by End User)) * 100

Therefore, in this instance,
EDR = [100 / (100 + 40)] * 100 = [100 /140] * 100 = 71%

#8) Leaked Defects: Defect leakage unveils the efficiency of QA testing by finding out the number of defects that were overlooked or evaded during the QA testing phase.

Defect Leakage = (Number of Defects Discovered in UAT / Number of Defects Discovered in QA Testing) * 100

So, in this instance,
Defect Leakage = (40 / 100) * 100 = 40%

#9) Defects by Priority: This metric exhibits the number of defects detected based on their intensity or priority, offering insights into the software’s quality.

Critical Defects Percentage = (Count of Critical Defects Found / Total Count of Defects Discovered) * 100
From the available data in the table:
Critical Defects Percentage = 6 / 30 * 100 = 20%

High Defects Percentage = (Count of High Defects Found / Total Count of Defects Discovered) * 100
From the available data in the table:
High Defects Percentage = 10 / 30 * 100 = 33.33%

Medium Defects Percentage = (Count of Medium Defects Found / Total Count of Defects Discovered) * 100
From the available data in the table:
Medium Defects Percentage = 6 / 30 * 100 = 20%

Low Defects Percentage = (Count of Low Defects Found / Total Count of Defects Discovered) * 100
From the available data in the table:
Low Defects Percentage = 8 / 30 * 100 = 27%

<a href