How to Perform Post-Release Testing Effectively and Minimize Impact of the Release to Live Clients

When I began my career as a Quality Assurance (QA) professional, I worked for a Software-as-a-Service (SaaS) company where production releases were critical, as they could potentially impact the functionality for live clients.

As our client base expanded, our QA team implemented the practice of post-release testing to manage the risk and minimize the impact of releases on live clients.

This concept was new to me, and I had numerous questions and doubts:

• What exactly is post-release testing?
• If I conducted thorough testing during the development phase, why is post-release testing necessary?
• Do I have to retest everything? What exactly does post-release verification entail?
• What should be done if an issue is discovered?

I am pleased to say that I found the answers to all my queries within my initial few production releases.

Now, I will share that knowledge with you, and I have chosen to write this article in a question and answer format to illustrate how I uncovered the answers.

What is Post Production Release Verification?

Post Production Release Verification refers to the process of testing a software release on the live/production environment after it has been deployed. The goal is to ensure that the released features meet the requirements.

Recommended read => How to Effectively Prepare “Test Environment” before beginning testing

While testing during the QA phase is crucial, post-production release testing is necessary due to several reasons:

• Data Issue – Variations in data between the production and test environments can lead to corner-case issues going unnoticed during testing.
• Deployment Issue – Manual build deployment processes can increase the likelihood of deployment issues such as missing configuration settings, DB script omissions, incorrect installation of dependencies, etc.
• Undetected Impact Areas – Some scenarios may not be correctly identified as areas of impact, leading to unanticipated issues upon release.
• Unknown Impact Areas – In complex architecture involving multiple products sharing a common architecture, even minor changes could cause functionality problems across multiple products.

What tasks and activities are included in the post-release verification phase?

The post-release verification phase typically involves the following tasks and activities:

• Post-production release verification
• Verification result reporting
• Reporting any issues discovered on the live/production environment
• Data cleanup after post-release verification
• Post-release monitoring (if applicable)

Do I need to test everything all over again?

It is not necessary to retest everything during post-release verification. The extent of testing depends on the nature of the release and impact analysis.

Detailed testing should be performed during the regular QA cycle. Post-release verification should follow a test plan derived from the full test plan for that release.

How do I formulate a post-production release verification strategy?

The formulation of a post-production release verification strategy should mirror the test flow followed during the QA cycle.

The strategy should include steps to test both new and major existing features, verify impact areas, and maximize functional coverage.

An effective post-production release strategy should:

• Include steps for testing new and major existing features
• Verify impact areas
• Maximize functional coverage
• Optionally, include any critical bugs discovered during testing
• Optionally, prioritize the test cases

Who creates the post-production release test plan?

The responsibility for creating the post-production release test plan varies across companies and depends on their organizational structure.

For instance, in a QA team organized as follows:

In this scenario, the QA professional working on a specific project would formulate the initial post-production release test plan.

Who approves the post-production release test plan?

The approval process for the post-production release test plan also varies across companies and depends on their organizational structure.

Considering the same organizational structure as previously mentioned, the post-production release test plan should be reviewed and approved by the Test Lead or QA Manager.

When do I create the post-production release verification plan?

The post-production release verification plan can be created during any phase of the software development cycle once the requirements, development scope, and impacted areas have been identified and finalized. It is advisable to create the plan midway into the sprint to allow for sufficient review and approval time.

It is good practice to include this test plan along with any formal QA sign-off documents before the project enters the deployment and release phase.

I completed the post-production release verification. What’s next?

Once the post-production release verification is complete, the following steps should be taken:

1) Communication of verification results – The verification results should be communicated to stakeholders, including any issues discovered during the verification process.

2) Reporting any issues found on production in the Defect Management tool – This facilitates root cause analysis and traceability.

3) Post-release verification data clean up – Data cleanup must be performed after the verification process is complete.

For example, if a release affects an e-commerce application, and a test order was created during the verification process, that test order should be canceled after verification to avoid impacting live client data.

4) Post-production release monitoring (if applicable) – Some releases require ongoing monitoring on the live/production environment.

For example, if the team made improvements to page load times, monitoring would be necessary to ensure the effectiveness of the improvements. The responsible individuals should be clearly identified and communicated.

What happens if I find an issue?

Any issues discovered should be reported in the Defect Management tool and communicated to stakeholders. If critical issues are found on the live/production environment, immediate communication is essential to determine whether a rollback is necessary for further investigation.

It is crucial to report all discovered issues in the Defect Tracking Tool. It is recommended to raise these as separate issue types (e.g., Post Production Bug) to distinguish them from bugs discovered during regular QA cycles. This facilitates easy filtering if root cause analysis is required.

What else do I need to know about the post-production release verification process?

In addition to the actual post-production release verification process, plan, and strategy, consider the following:

• Set clear expectations regarding the scope and purpose of post-release verification. Stakeholders (internal and external) should be aware that:
• It is not possible to test everything on the live environment.
• It is not feasible to condense days of testing into a few hours dedicated to post-release verification.
• Take caution when deciding the extent of post-production release testing. There are limitations to what and how much can actually be tested on the live environment. The live environment contains actual client data and must be handled with care. Additional planning should be done for changes involving data migration, updates, deletions, etc.
• Perform a root cause analysis for issues discovered during post-release verification to identify why the issue was not caught earlier and learn from past mistakes. This analysis can help fill gaps in implementation and prevent similar issues in the future. The Test Lead or QA Manager can conduct the root cause analysis with input from the project team.
• Consider using server logs for post-release monitoring. Server logs may reveal events or issues that are not visible to customers but can impact backend processes. Responsibility for this monitoring can be assigned to the Development Lead and DevOps team.

An Example:

Project Overview:

Changes are required for a social media application, specifically to the sign-up process:

1. The validation requirement for the last name field needs to be removed.
2. A toggle button must be implemented next to the email address field to allow users to set privacy settings for their email address on their profile.
3. Users should have the ability to choose their own avatar.
4. API calls during the sign-up process should be reduced to improve application performance.

Post-Production Release Verification Plan:

S.No.	Description	Expected Result	Status	Comments
1	Open the website homepage	The homepage should load successfully	Pass
2	Click on “Sign up” as a new user	The user should be redirected to the registration/sign-up page	Pass
3	Fill in the required fields and click on the “Register” button: – Enter “Lee” as the last name – Toggle the privacy button to “Do not Display” – Choose an avatar	– The user should be redirected to their profile page after successful registration. – The user’s phone number should not be shown. – The user’s selected avatar should be displayed.	Partial Pass	The avatar is not rendering properly and shows as a broken image. Reported in JIRA as BUG-1088.
4	Monitoring – Verify if the application performance has improved after this release	The reduction of API calls during the sign-up process should improve application performance	Ongoing	Action is on the Dev Lead and Dev Ops team to monitor the application for 24 hours
5	Post-release cleanup	Delete the test account created	Done

Conclusion

With the widespread adoption of Agile methodologies in most software companies, the frequency of production releases has increased. While a team following a Waterfall model might have a production release every 1.5 months, an Agile team may have releases every 2-3 weeks.

With every production release, there is a possibility of inadvertently impacting the functionality of live clients. Adopting post-production release verification immediately after deployment can provide additional confidence in the release and offer a safety net for rolling back the release before live clients encounter any issues.

For high-impact/risk projects, the post-production release verification plan can be structured based on the priority of the test scenarios. Critical priority tests can be executed first, and the results and any issues can be communicated to stakeholders. If no critical issues are found, post-production release verification can continue; otherwise, a decision needs to be made on whether to roll back the release to minimize application downtime and impact on live clients.

In addition, post-production release testing can be automated, and the test scripts can be run on demand as part of regression testing. However, caution must be exercised when running automated test scripts on the live environment to avoid impacting live client data and functionality.

Post-production release verification is the last line of defense for software companies. Failure to identify issues during this process can be devastating to a company’s reputation. To maintain the reliability of a product, it is essential to thoroughly test changes immediately after deployment.

About the author: This informative article was written by Neha B., a Quality Assurance Manager specializing in leading and managing in-house and offshore QA teams.

Share your post-production release testing strategy, tips, and experiences with our readers.