What is Longevity Testing? How to Catch the Bugs Before the Customer Finds It

This article discusses the principles of “Longevity Testing” and its critical role in assessing the stability of a product or system while lessening the potential for customer-discovered defects. It aims to showcase the importance of “catching the glitches in-house before the customer encounters them”.

By the end of this piece, Managers, Leaders, and Testers in QA will have a comprehensive grasp on:

• The definition of Longevity Testing
• The importance of Longevity Testing
• The steps to design and conduct Longevity Tests
• The advantages and disadvantages of Longevity Testing

What is Longevity Testing?

Longevity Testing encompasses:

• Verifying the endurance and reliability features of a system or product over an extended duration, simulating it under genuine stress and loads conditions involving real-world traffic and applications.
• Minimizing the likelihood of bug discovery by the client on their site.

A flowchart detailing how client-reported concerns are handled (Fig. 1)

An Introduction to Longevity Testing

#1) Typically, at the client’s site, everything appears to function flawlessly in the initial weeks after a product deployment or an upgrade to the latest software release. But as time passes, the client may start encountering issues.

#2) A lot of these issues, seemingly simple, are difficult to duplicate in-house. Expert Teams need to invest substantial time and conduct in-depth examination to address them. Take Note: Time equates to money!!!

#3) The discovery of a bug by a customer can lead to one or more of the following scenarios (Fig. 1)

• The bug severity directly impacts the client’s business operations, which may lead to financial losses.
• A service ticket issued to the Technical Support Center incurs charges for the Product Engineering Organization.
• The Front-end Technical Support Teams are often unable to resolve the reported issues.
• Such queries or complaints necessitate escalation to the higher-level Support Team.
• Customer Ticket Escalation contributes to additional costs for the Organization.
• If the Escalation Team fails to address the concern, intervention from the Engineering Team (Development and QA) becomes imperative.
• The longer it takes to rectify the bug, the higher the odds of disappointed customers and potential loss of repeat orders. The customer may even switch to a competitor’s offering. These outcomes each result in revenue loss for the Product Engineering Organization.

4) The issues most reported by clients revolve around the stability of the system or product in terms of the client’s specific setup, infrastructure, traffic, and applications.

The Need for Longevity Testing

1) Any glitch surfacing from a client-generated concern is typically a Test Escape.

2) These glitches cause financial harm to both the client and the Engineering Organization providing the solutions and services.

3) In a typical scenario, such glitches should have been internally identified during the many testing cycles, which includes Regression Testing, carried out by one or more testers from the Testing Team, subject to the issue’s complexity.

4) Moreover, glitches emanating from client-reported issues signal a possibility that some testing scenarios or test instances might have been overlooked during the Test Plan execution stage.

5) There might be instances, experienced by many testers, where a specific feature fails on the client’s site despite having passed several in-house testing stages in different testbeds such as:

• Feature
• Regression
• Load
• Stress
• Performance
• System
• Solution
• Alpha
• Beta

6) Critical observations include:

• Throughout a software release phase, the System Under Test (SUT) or Device Under Test (DUT) in all testbeds frequently undergoes soft or hard reboots due to reasons like accommodating new code drops, verifying bugs, etc.
• Automated Regression Test Suites typically reboot or reset the SUT or DUT after implementing a particular test case script or a sequence of such scripts.
• However, the situation on the client’s site is much different. For productivity reasons, the client cannot afford frequent system reboots.
• Customers usually assign specific maintenance windows for conducting software upgrades or hardware replacements, which are entirely based on their internal policies and procedures.
• The frequency of these maintenance windows varies from quarterly to annually depending on the Customer Organization.
• In practice, the actual state of the system or product on the client’s site differs drastically from the testbeds during a Software Release cycle in any Product Engineering Organization.
• In sectors such as finance, healthcare, and government, many clients also require an approved quality document evidencing that the System or Product has cleared specific Vertical Model Testing.

Given the above-mentioned test gaps =>

• It is evident that the System or Product should be subjected to prolonged tests or Longevity Tests, simulating end-to-end scenarios at the client’s site or specific verticals.
• The recommended duration for Longevity Testing usually spans from 72-720 hours (3-30 days), or a suitable duration based on EFD or CFD data and specific client instances.
• QA Managers, Leaders, and Testers should consider Longevity Testing as a unique activity within a specific Software Release cycle.
• In conclusion, Longevity Testing is crucial for guaranteeing the stability of a System or Product, which has a direct impact on the Organization’s profitability.

Strategizing and Carrying Out Longevity Tests

Longevity Testing should be integrated into the complete Test Strategy by QA Managers, Leaders, and Testers.

Plan Development

• Engineering Organizations periodically organize in-house Test Escape Analysis (TEA) exercises for different products (both hardware and software). Some have implemented and automated mechanisms to gather Test Escape data, primarily based on Externally Found Defects (EFD) or Customer Found Defects (CFD) recorded by the Support Escalation Team.
• EFDs or CFDs should be closely analyzed from a holistic perspective, considering live client deployments, including not only the infrastructure but also the end-user devices, applications, and traffic behavior.

Knowing the Customer Verticals:

The following wider verticals typically include clients:

• Healthcare
• Retail
• Finance
• Education
• Transport
• Manufacturing
• Engineering
• Government (Federal)

Undertakings

#1) Put together a distinct Test Plan and Test Cases for Longevity Testing. This will assist in tracking test execution, logging bugs, and verification.

#2) Identify test cases based on insights from the Test Escape Analysis, usually involving scrutinizing bugs found in EFDs or CFDs.

#3) It’s vital for the QA team to duplicate one or more vertical testbeds that are relevant to the organization’s business.

#4) The Dedicated Test Beds should include:

• A Network Topology mirroring the target vertical(s)
• An Infrastructure with similar switches, routers, back-end servers, firewalls, etc.
• Application servers frequently used in the target vertical(s)
• End-user devices commonly used in the target vertical(s)

#5) Use the right tools to generate Stress, Load, and Real-time Traffic.

#6) Assign resources for human execution.

#7) Identify automation resources/methods for speedier and repeat execution.

#8) Determine the start and endpoints of Longevity Testing for a given release.

Two methods for defining the start and endpoints of Longevity Testing:

I) Method 1:

• Ensure Software code or Hardware is in a stable state.
• Begin after the Feature Test Completion.
• Conclude before the Code Freeze.

II) Method 2:

• Entertain somewhat unstable code.
• Commence after reaching a 70% completion of the Feature test cycle.
• Wrap up before the Code Freeze.

#9) Verify fixed defects.

#10) Incorporate Longevity Testing into the Regression phase for future Regression Testing runs.

Carrying out the Test

• Configure the Test Beds to simulate one or more Customer Verticals.
• Ensure that the back-end Infrastructure, Applications, and Databases imitate those used by the client.
• Guarantee the availability and use of end-user devices that are relevant to the client’s needs during the execution of the Test Plan.
• Employ appropriate tools to apply moderate Stress and Load on the System or Product.
• Execute the complete Test Suite from the Longevity Test Plan without triggering any soft or hard restarts of the SUT or DUT, back-end servers, and other related Infrastructure devices.
• Continuously run several tests in the above manner for a specified duration, which could range from 72 to 720 hours.
• Document the results.
• Log all uncovered bugs.
• Verify all bugs.

Balance of Longevity Testing: Pros and Cons

Advantages

• Aids in finding critical bugs before the client encounters them.
• Enhances the durability of the System or Product, which is key for the client’s efficiency and business.
• Boosts Customer Satisfaction.
• Substantially saves costs for the Organization. Remember, saving is equivalent to earning!!!
• The Longevity Testing report can serve as a Quality Certification proof for different verticals.

Disadvantages

• Entails initial expenditure to include Longevity Testing and related undertakings in a given release and Regression activities.
• Better suited for the Waterfall model.
• Agile/Scrum models necessitate adjustments in coverage and duration.

Wrapping Up

“Defects” that surface from client-raised issues primarily stem from Test Escapes. This vital point underscores the importance of developing, reviewing, covering, and executing Test Plans.

Externally Found Defects (EFD) or Customer Found Defects (CFD) have significant financial implications for both the client and the Product Institution.

Longevity Testing employs a unique technique to help Product organizations enhance Customer Satisfaction. It does this by identifying and solving defects before clients discover them, contributing to the stability of a System or Product—thus delivering solid quality.

About the author: This piece is penned by Vinayak, a prominent author at STH, boasting 12 years of experience in QA/testing in Fortune 500 enterprises.

Feel free to share your queries or suggestions about this article with us.