What is Regression Testing?
Regression Testing is a type of testing that is done to verify that a code change in the software does not impact the existing functionality of the product.
This is to ensure that the product works fine with new functionality, bug fixes or any changes to the existing feature. Previously executed test cases are re-executed in order to verify the impact of the change.
Regression Testing is a Software Testing type in which test cases are re-executed in order to check whether the previous functionality of the application is working fine and the new changes have not introduced any new bugs.
Regression test can be performed on a new build when there is a significant change in the original functionality that too even in a single bug fix.
Regression means retesting the unchanged parts of the application.
What You Will Learn:
- Tutorials Covered in This Series
- Regression Test Overview
- When to Perform This Test?
- Can Regression Testing Be Performed Manually?
- Automated Regression Testing Tools
- Why the Regression Test?
- Types of Regression Testing
- How Much Regression Is Required?
- What Do We Do In Regression Check?
- Regression Testing Techniques
- How To Select A Regression Test Suite?
- How To Perform Regression Testing?
- Regression in Agile
- Regression of GUI Application
- Difference Between Regression And Re-testing
- Regression Test Plan Template (TOC)
Tutorials Covered in This Series
Regression Test Overview
Regression test is like a verification method. Test cases are generally automated as test cases are required to be executed again and again and running the same test cases again and again manually is a time-consuming and tedious one too.
For Example, Consider a product X, in which one of the functionality is to trigger confirmation, acceptance, and dispatched emails when the Confirm, Accept and Dispatch buttons are clicked.
Some issues occur in the confirmation email and in order to fix the same, some code changes are made. In this case, not only the Confirmation emails need to be tested, but Acceptance and Dispatched emails also need to be tested to ensure that the change in the code has not affected them.
Regression Testing is not dependent on any programming language like Java, C++, C#, etc. This is a testing method which is used to test the product for modifications or for any updates being done. It verifies that any modification in a product does not affect the existing modules of the product.
Verify that the bug is fixed and the newly added features have not created any problem in the previous working version of the software.
Testers perform Functional Testing when a new build is available for verification. The intent of this test is to verify the changes made in the existing functionality and the newly added functionality as well.
When this test is done, the tester should verify whether the existing functionality is working as expected and the new changes have not introduced any defect in functionality that was working before this change.
Regression test should be a part of the Release Cycle and must be considered in the test estimation.
When to Perform This Test?
Regression Testing is usually performed after verification of changes or new functionality. But this is not always the case. For the release that is taking months to complete, regression tests must be incorporated in the daily test cycle. For weekly releases, regression tests can be performed when Functional Testing is over for the changes.
Regression checking is a variation of retest (which is simply to repeat a test). When Retesting, the reason can be anything. Say, you were testing a particular feature and it was the end of the day- you could not finish testing and had to stop the process without deciding if the test passed/failed.
The next day when you come back, you perform the test once more – that means you are repeating a test you performed before. The simple act of repeating a test is a Retest.
Regression test at its core is a retest of sorts. It is only for the special occasion that something in the application/code has changed. It might be code, design or anything at all that dictates the overall framework of the system.
A Retest that is conducted in this situation to make sure that the said change has not made an impact on anything that was already working before is called the Regression Test.
The most common reason why this might be conducted is because new versions of the code have been created (increase in scope/requirement) or bugs have been fixed.
Can Regression Testing Be Performed Manually?
I was just teaching one of these days in my class, and a question came to me – “Can regression be done manually?”
I answered the question and we moved on in the class. Everything seemed OK, but somehow this question nagged me for quite a while later.
Over the many batches, this question comes multiple times in various different ways.
Some of them are:
- Do we need a tool to perform the test execution?
- How is Regression Testing performed?
- Even after an entire round of testing– newcomers find it difficult to discern what exactly the Regression test is?
Of course, the original question:
- Can this Testing be performed manually?
To begin with, Test execution is a simple act of using your Test cases and performing those steps on the AUT, supplying the test data and comparing the result obtained on the AUT with the expected result mentioned in your test cases.
Depending on the comparison result, we set the status of the test case pass/fail. Test execution is as simple as that, there are no special tools necessary for this process.
Automated Regression Testing Tools
Automated Regression Test is a testing area where we can automate most of the testing efforts. We ran all the previously executed test cases on a new build.
This means that we have a test case set available and running these test cases manually is time-consuming. We know the expected results, so automating these test cases is time-saving and is an efficient regression test method. The extent of automation depends upon the number of test cases that are going to remain applicable overtime.
If test cases vary from time to time, the application scope goes on increasing and then automation of regression procedure will be a waste of time.
Most of the Regression testing tools are of record and playback types. You can record the test cases by navigating through the AUT (application under test) and verify whether the expected results are coming or not.
#1) Avo Assure
Avo Assure is 100% no-code and heterogeneous test automation solutions that makes regression testing simpler and faster.
Its cross-platform compatibility enables you to test across the web, mobile, desktop, Mainframe, ERPs, associated emulators, and more. With Avo Assure, you can run end-to-end regression tests without writing a single line of code and ensure rapid, high-quality delivery.
Avo Assure helps you to:
- Achieve >90% test automation coverage by executing end-to-end regression tests repeatedly.
- Easily visualize your entire testing hierarchy with a click of a button. Define test plans and design test cases through the Mindmaps feature.
- Leverage about 1500+ keywords and >100 SAP-specific keywords to deliver applications faster
- Execute multiple scenarios simultaneously using the Smart Scheduling and Execution feature.
- Integrate with a plethora of SDLC and Continuous Integration solutions like Jira, Sauce Labs, ALM, TFS, Jenkins, and QTest.
- Analyze reports intuitively with easy-to-read screenshots and videos of test case execution.
- Enable accessibility testing for your applications.
TimeShiftX gives companies a big advantage by making shorter test cycles, meeting deadlines, and reducing required resources which results in a shorter release cycle while providing high software reliability.
- Katalon Studio
- AdventNet QEngine
- Regression Tester
- Rational Functional Tester
Most of these are Functional and Regression test tools.
Recommended Reading => Check Here For The List Of Top Regression Tools
Adding and updating Regression test cases in an Automation test suite is a cumbersome task. While selecting an Automation tool for Regression tests, you should check if the tool allows you to add or update test cases easily.
In most cases, we need to update automated Regression test cases frequently due to frequent changes in the system.
WATCH THE VIDEO
For a more detailed explanation of the definition with an example, please check the following Regression Test video:
Why the Regression Test?
Regression is initiated when a programmer fixes any bug or adds a new code for a new functionality to the system.
There can be many dependencies in the newly added and existing functionality.
This is a quality measure to check whether the new code complies with the old code so that the unmodified code is not getting affected. Most of the time the testing team has the task to check the last minute changes in the system.
In such a situation, testing only affected the application area is necessary to complete the testing process on time by covering all the major system aspects.
This test is very important when there is a continuous change/improvement added in the application. The new functionality should not negatively affect the existing tested code.
Regression is required to find the bugs that occurred because of a change in the code. If this testing is not done, the product might get critical issues in the live environment and that indeed can lead the customer into trouble.
While testing any online website, the tester reports an issue that the Price of the Product is not showing correctly i.e., it shows a lesser price than the actual price of the Product, and it needs to be fixed soon.
Once the developer fixes the issue, it needs to be re-tested and Regression Testing is also required as verifying the price on the reported page would have got corrected but it might be showing an incorrect price on the summary page where the total is shown along with the other charges or the mail sent to the customer still has the incorrect price.
Now, in this case, the customer will have to bear the loss if this testing is not performed as the site calculates the total cost with the incorrect price and the same price goes to a customer by email. Once the customer accepts, the Product is sold online at a lower price, it will be a loss for the customer.
So, this testing plays a big role and is very much required and important as well.
Types of Regression Testing
Given below are the various types of Regression :
- Unit Regression
- Partial Regression
- Complete Regression
#1) Unit Regression
Unit Regression is done during the Unit Testing phase and code is tested in isolation i.e. any dependencies on the unit to be tested are blocked so that the unit can be tested individually without any discrepancy.
#2) Partial Regression
Partial Regression is done to verify that the code works fine even when the changes have been done in the code and that unit is integrated with the unchanged or already existing code.
#3) Complete Regression
Complete Regression is done when a change in the code is done on a number of modules and also if the change impact of a change in any other module is uncertain. The product as a whole is regressed to check for any changes because of the changed code.
How Much Regression Is Required?
This depends upon the scope of the newly added features.
If the scope of a fix or feature is too large, then the application area getting affected is also quite large and the testing should be performed thoroughly including all the application test cases. But this can be effectively decided when the tester gets input from a developer about the scope, nature, and the amount of change.
As these are repetitive tests, test cases can be automated so that a set of test cases alone can be easily executed on a new build.
Regression test cases need to be selected very carefully so that maximum functionality is covered in a minimum set of test cases. These set of test cases need continuous improvements for newly added functionality.
It becomes very difficult when the application scope is very huge and there are continuous increments or patches to the system. In such cases, selective tests need to be executed in order to save testing cost and time. These selective test cases are picked based on the enhancements done to the system and the parts where it can affect the most.
What Do We Do In Regression Check?
- Re-run the previously conducted tests.
- Compare the current results with previously executed test results
This is a continuous process performed at various stages throughout the software testing lifecycle.
A best practice is to conduct a Regression test after Sanity or Smoke Testing and at the end of Functional testing for a short release.
In order to conduct effective testing, a regression Test Plan should be created. This plan should outline the regression testing strategy and the exit criteria. Performance Testing is also a part of this test to make sure that the system performance is not affected due to the changes made in the system components.
Best practices: Run automated test cases every day in the evening so that any regression side effects can be fixed in the next day build. This way it reduces the release risk by covering almost all regression defects at an early stage rather than finding and fixing those at the end of the release cycle.
Regression Testing Techniques
Given below are the various techniques.
- Retest all
- Regression Test Selection
- Test case Prioritization
#1) Retest All
As the name itself suggests, the entire test cases in the test suite are re-executed to ensure that there are no bugs that have occurred because of a change in the code. This is an expensive method as it requires more time and resources when compared to the other techniques.
#2) Regression Test Selection
In this method, test cases are selected from the test suite to be re-executed. Not that the entire suite has been re-executed. The selection of test cases is done on the basis of code change in the module.
Test cases are divided into two categories, one is Reusable test cases and another one is Obsolete test cases. The reusable test cases can be used in future regression cycles whereas obsolete ones are not used in the upcoming regression cycles.
#3) Test Case Prioritization
Test cases with high Priority are executed first rather than the ones with medium and low priority. The priority of the test case depends on its criticality and its impact on the product and also on the functionality of the product which is used more often.
The hybrid technique is a combination of Regression Test Selection and Test case Prioritization. Rather than selecting the entire test suite, select only the test cases which are re-executed depending on their priority.
How To Select A Regression Test Suite?
Most of the bugs found in the production environment occur because of the changes done or bugs fixed at the eleventh hour i.e., the changes done at a later stage. The bug fix at the last stage might create other issues/bugs in the Product. That’s why Regression checking is very important before releasing a Product.
Below is a list of test cases that can be used while performing this Test:
- Functionalities which are frequently used.
- Test cases that cover the module where the changes have been made.
- Complex test cases.
- Integration test cases which include all the major components.
- Test cases for the core functionality or features of the Product.
- Priority 1 and Priority 2 test cases should be included.
- Test cases of frequently failed or recent testing defects were found for the same.
How To Perform Regression Testing?
Now that we have established what regression means, it is apparent that it is testing also – simply repeating in a specific situation for a specific reason. Therefore, we can safely derive that the same method applied for testing in the first place can be applied to this too.
Therefore, if testing can be done manually then Regression Testing can be done too. The use of a tool is not necessary. However, as time goes on applications get piled on with more and more functionality which keeps increasing the scope of regression. To make the most of the time, this testing is most often Automated.
Given below are the various steps involved in performing this Testing
- Prepare a Test suite for Regression considering the points mentioned in “How to select Regression Test suite”?
- Automate all test cases in the test suite.
- Update the Regression suite whenever it is required like if any new defect which is not covered in the test case is found, and a test case for the same should be updated in the test suite so that the testing is not missed for the same next time. The regression test suite should be managed properly by continuously updating the test cases.
- Execute the Regression test cases whenever there is any change in the code, the bug is fixed, new functionality is added, an enhancement to the existing functionality is done, etc.
- Create a test execution report which includes the Pass/Fails status of the executed test cases.
Let me explain this with an example. Please examine the situation below:
|Release 1 Statistics|
|No. of Requirements (Scope)||10|
|No. of Test Cases/Tests||100|
|No. of days it takes to Develop||5|
|No. of days it takes to Test||5|
|No. of Testers||3|
|Release 2 Statistics|
|No. of Requirements (Scope)||10+ 5 new Requirements|
|No. of Test cases/Tests||100+ 50 new|
|No. of days it takes to Develop||2.5 (since this half the amount of work than earlier)|
|No. of days it takes to Test||5(for the existing 100 TCs) + 2.5 (for new Requirements)|
|No. of Testers||3|
|Release 3 Statistics|
|No. of Requirements (Scope)||10+ 5 + 5 new requirements|
|No. of Test cases/Tests||100+ 50+ 50 new|
|No. of days it takes to Develop||2.5 (since this half the amount of work than earlier)|
|No. of days it takes to Test||7.5 (for the existing 150 TCs) + 2.5 (for new Requirements)|
|No. of Testers||3|
Given below are the observations that we can make from the above situation:
- As the releases grow, the functionality grows.
- Development time does not necessarily grow with releases, but the testing time does.
- No company/its management will be ready to invest more time in testing and less for development.
- We cannot even reduce the time it takes to test by increasing the test team size because more people means more money and new people also means lots of training and maybe also a compromise in quality as the new people might not be at par with the required knowledge levels immediately.
- The other alternative is clearly to reduce the amount of regression. But that could be risky for the software product.
For all these reasons, Regression Testing is a good candidate for Automation Testing, but it does not have to be done only that way.
Basic Steps to Perform Regression Tests
Every time the software undergoes a change and a new version/release comes up, given below are the steps that you can take to carry out this type of testing.
- Understand what kind of changes have been made to the software
- Analyze and determine what modules/parts of the software might be impacted – the development and BA teams can be instrumental in providing this information.
- Take a look at your test cases and determine if you will have to do a full, partial or unit regression. Identify the ones that will fit your situation
- Schedule a time and test away!
Regression in Agile
Agile is an adaptive approach that follows an iterative and incremental method. The product is developed in a short iteration called sprint which lasts for 2- 4 weeks. In agile, there are a number of iterations, hence this testing plays a significant role as the new functionality or code change is done in the iterations.
The Regression test suite should be prepared from the initial phase and should be updated with each sprint.
In Agile, Regression checks are covered under two categories:
- Sprint Level Regression
- End to End Regression
#1) Sprint Level Regression
Sprint Level Regression is done mainly for new functionality or enhancements that are done in the latest sprint. Test cases from the test suite are selected as per the newly added functionality or the enhancement that is done.
#2) End-to-End Regression
End-to-End Regression includes all the test cases that are to be re-executed to test the complete product end to end by covering all the core functionalities of the Product.
Agile has short sprints and as it goes on, it is very much required to automate the test suite, the test cases are executed again and that too needs to be completed in a short span of time. Automating the test cases reduces the time of execution and defect slippage.
Given below are the various advantages of the Regression test
- It improves the quality of the Product.
- This ensures that any bug fixes or enhancements that are done do not impact the existing functionality of the Product.
- Automation tools can be used for this testing.
- This will ensure that issues that are already fixed do not occur again.
Though there are several advantages, there are some disadvantages as well. They are:
- This has to be done for a small change in the code as well because even a small change in the code can create issues in the existing functionality.
- If in case automation is not used in the Project for this testing, it will be a time consuming and tedious task to execute the test cases again and again.
Regression of GUI Application
It is difficult to perform a GUI (Graphical User Interface) Regression test when the GUI structure is modified. The test cases written on the old GUI either become obsolete or need to be modified.
Re-using the regression test cases means GUI test cases are modified according to the new GUI. But this task becomes a cumbersome one if you have a large set of GUI test cases.
Difference Between Regression And Re-testing
Re-testing is done for the test cases which fail during the execution and the bug raised for the same has been fixed whereas Regression check is not limited to the bug fix as it covers other test cases as well to ensure that the bug fix has not impacted any other functionality of the Product.
Regression Test Plan Template (TOC)
1. Document History
3. Regression Test Plan
3.3. Test Strategy
3.4. Features to be tested
3.5. Resource Requirement
3.5.1. Hardware Requirement
3.5.2. Software Requirement
3.6. Test Schedule
3.7. Change Request
3.8. Entry/Exit criteria
3.8.1. Entry Criteria for this Testing
3.8.2. Exit Criteria for this Testing
3.10. Test Cases
3.11. Risk /Assumptions
Let’s take a look at each of them in detail.
#1) Document History
Document history consists of a record of the first draft and all the updated ones in the below-given format.
|2||DD/MM/YY||ABC||Updated for the added feature|
The References column keeps track of all the reference documents used or required for the Project while creating a test plan.
|1||SRS document||Shared drive|
#3) Regression Test Plan
This document describes the change/update/enhancement in the Product to be tested and the approach used for this testing. All code changes, enhancements, updates, and added features are outlined to be tested. Test cases used for Unit Testing and Integration Testing can be used to create a test suite for Regression.
The purpose of the Regression Test Plan is to describe what exactly and how testing would be performed to accomplish the results. Regression checks are done to ensure that no other functionality of the product is hampered because of the code change.
3.3. Test Strategy
Test Strategy describes the approach which will be used to perform this testing and that includes the technique that will be used, what will be the completion criteria, who will be performing which activity, who will write the test scripts, which regression tool will be used, steps to cover the risks like resource crunch, delay in production, etc.
3.4. Features to be tested
Features/components of the product to be tested are listed here. In regression, all the test cases are re-executed or the ones which affect the existing functionality are chosen depending on the fix/update or enhancement done.
3.5. Resource Requirement
3.5.1. Hardware Requirements:
Hardware Requirements can be identified here like computers, laptop, Modems, Mac book, Smartphone, etc.
3.5.2. Software Requirements:
Software Requirements are identified such as which Operating system and browsers will be required.
3.6. Test Schedule
The test schedule defines the estimated time for performing the testing activities.
For example, how many resources will perform a testing activity and that too in how much time?
3.7. Change Request
CR details are mentioned for which Regression will be performed.
|S.No||CR Description||Regression Test Suite|
3.8. Entry/Exit Criteria
3.8.1. Entry Criteria for this testing:
Entry criteria for the Product to start Regression check are defined.
- Coding changes/enhancement/addition of new features should be completed.
- Regression test Plan should be approved.
3.8.2. Exit Criteria for this testing:
Here are the exit criteria for Regression as defined.
- Regression testing should be completed.
- Any new critical bugs found during this testing should be closed.
- Test Report should be ready.
3.9. Test Cases
Regression Test cases are defined here.
Any risk & assumptions are identified and a contingency plan is prepared for the same.
Tools to be used in the Project are identified.
- Automation tool
- Bug Reporting tool
The names and designations of the people are listed here:
Regression Testing is one of the important aspects as it helps to deliver a quality product by making sure that any change in the code whether it’s small or large does not affect the existing or old functionality.
A lot of automation tools are available for automating the regression test cases, however, a tool should be selected as per the Project requirement. A tool should have the ability to update the test suite as the Regression test suite needs to be updated frequently.
With that, we are wrapping this topic up and hope there will be much better clarity on the subject from now on.
Please let us know your Regression related questions and comments. How did you tackle your Regression Testing tasks?