Cognitive Bias in Software Testing: Why Do Testers Miss Bugs?

Exploring Cognitive Bias in Software Testing: Are Testers Falling Prey to It?

In the ever-changing field of software testing, there’s a continuous evolution to keep pace with advancements in technology and the demand for “instantaneous quality.”

Buzzwords such as “Continuous Integration,” “Digital transformation,” “automation of life-cycle,” and “quality shift to the left to decrease costs” are common. However, these conversations invariably circle back to the lingering question, “Why and how were these flaws overlooked?” for which there seems to be no satisfactory resolution.

Even the most overt glitches can sometimes slip past the watchful gaze of testers.

But, what is the cause of this?

While we’d like to believe that our thinking is methodical, organized, and rational, the unfortunate reality is that we’re all susceptible to Cognitive Biases. These biases affect the way we think in both our personal and work lives, including in the realm of software testing.

A Brief Description of Cognitive Bias

As defined by Wikipedia, cognitive bias is the consistent deviation from logical thinking or usual standards in making judgments. People build their particular “subjective societal reality” based on their interpretation of input they receive.

In this context, cognitive biases can lead to skewed perceptions, irrational judgments, fallacious interpretations, and unreasonable behavior.

Despite this understanding, appreciating the effect of cognitive bias on thinking processes is essential for testers in today’s world of software testing.

How to Spot Different Types of Cognitive Biases in Software Testing

As software testers, it’s important to be aware of the various types of cognitive biases. Here are some of them:

#1) The Bias of Resemblance

People usually make judgements based on how similar a situation is to another one.

For instance, as testers, we generally predict that web applications and client-server applications will have varying types of mistakes.

Due to this inclination, we exclusively concentrate on expected errors, unintentionally sidestepping the most conspicuous ones.

#2) Congruence Bias

This bias causes us to discount alternative possibilities.

Testers focus only on validating the expected results, leading to missing out on vital negative validations.

While drafting test cases, we often encompass all necessities and their predicted outcomes, inadvertently leaving out negative scenarios. It’s not feasible to document all possible user actions, and not all negative situations are explicitly pointed out in the requirements.

#3) Bias of Confirmation

This bias influences us to seek and interpret data that reinforces our preexisting beliefs and suppositions.

In the world of testing, it’s quite common for us to assume that code written by a certain developer will have more defects than others. As a result, we dedicate a significant share of testing effort towards the modules developed by that individual.

Unfortunately, these convictions raise the chance of disregarding defects in modules developed by other individuals.

#4) The Bandwagon Impact

The bandwagon effect is the likelihood of people adopting behavior or beliefs just because they are widely accepted.

If a large number of people hold a belief, it becomes more likely for others to also adopt it. This occurrence can be seen frequently in our daily lives.

Similarly, in the world of software testing, when our peers deem a certain module to be without defects, we tend to unconsciously trust their judgment and reduce our focus during validation.

#5) In-Attentional Blindness

This bias refers to the testers’ habit of skipping glaring defects when they are not actively seeking them out.

For example, when you ask a group of individuals to count the number of people wearing a specific colored dress, they become so absorbed in counting that they fail to notice other relevant details around them.

Similarly, during a software enhancement project, testers might excessively concentrate on the newly developed screens while overlooking critical integrations.

#6) Bias of Negativity

Negativity bias is the tendency of humans to put more emphasis on negative experiences than positive ones.

In the world of testing, testers find it hard to approve a build for production as they primarily focus on the defects they discover. It’s almost impossible for them to vouch that a product is without defects. Therefore, the final decision to go live with a product usually lies with the Product or Business Manager, although test managers provide recommendations.

Conclusion

We trust that this excursion into the realm of cognitive bias in software testing has offered valuable perspectives on its influence and how to mitigate its impacts.

It’s important to acknowledge that we’re often blind to our own biases while being adept at identifying biases in others (a cognitive bias in itself). Nevertheless, by raising our awareness and consciously acknowledging these biases, we can navigate the testing landscape more effectively.

Have you observed cognitive biases influencing you? Have you noticed cognitive biases among your peers? We invite you to share your thoughts in the comments section below!

About the Author:

This thought-provoking article is penned by Geethanjali S, a certified Scrum Master and PMP with over 18 years of experience in Quality Assurance and Engineering. Geethanjali is a transformative leader who has led QA transformations, global rollout initiatives, and projects related to mergers and integrations.