When To Opt For Automation Testing?

Should we consider automation testing for a project? When should we go for Automation Testing?

Testing is carried out to provide good quality deliverables to the end user. Testing Phase is one of the main aspects of STLC.

Any company focuses more on software testing as its quality brings in optimal customer satisfaction, but many of them still struggle in choosing which kind of testing to carry out, either with automated testing or manual testing. 

This article helps the reader to understand what is Automation Testing, when to go for it, and most importantly, when not to go for it. Also, learn optimal utilization of Automation Tools for Testing.

Whatever work is done, it should be carried out effectively and must be cost-effective as well. Moreover, it should make sense so that customer feels happy about the deliverables.

Software Testing and Cost Benefits

The Software Testing is normally carried out by a Software Tester. The difference between a tester and a real user is that the latter will know only a partial usage of the software that is used for their business or for their tasks and will not know the software completely. On the other hand, a tester will be aware of all the technical and functional requirements of the software. Based on the requirements provided by the client, test plans and test cases will have to be prepared.

A test plan is nothing but a detailed plan of the way in which the testing process is to be carried out. This will have complete details about the number of resources & sources involved in testing, what to do & when to do it, what will not be done, and the environment in which it will be carried out etc.

Test cases should be prepared after a clear understanding of the functional and technical aspect of the software. The tester must possess a keen observational capacity and complete knowledge about the software.

Moreover, cost plays an effective role here. Customers prefer to accept software with maximum quality at a minimal cost. When we go for manual testing, the process is more tedious and time-consuming as it is all done manually by a tester.

For Example, when we need ‘n’ number of testers to execute Regression Testing, it may take nearly 50 hours to execute all the test cases. And based on the resource availability, the test cases will be executed. But with less time for automated testing, optimal utilization of resources is carried out along with the maximum coverage of test cases when compared to manual testing.

Intelligence Behind Software Testing

It is very important for any organization to know when to start the testing process and when to exit it. We are supposed to know when to startup with testing because it’s useless to start testing when the development phase is done and when the criteria required is not met. It is always a best practice to start with the test design phase while the development is in progress.

Given below are the criteria for software testing entry and exit:

Entry Criteria

Once the design document has been signed off, the test plans have to be prepared in the planning phase. A test plan plays a vital role. The hardware required must be installed and configured properly and the functionality of the hardware needs to be checked. The functional requirements must be clear and approved. The developed code must be unit tested and signed off by the developers.

Test cases and test data must be prepared and approved. Test data and application should be available. The tester must possess significant and sufficient knowledge on the application. The resources should be well trained about tools and must be clarified with all the required functionalities.

The tester must be available. When any one of the criteria is not attained, the entry criteria of testing is withheld.

[Note: Click on any image for an enlarged view]

Exit Criteria

Only when at least 95% of the mandatory test cases are locked with a “pass” result, we can exit the testing phase for the product. However, it’s not so easy to determine when software testing can be stopped or if it still needs to be executed. And this kind of situation commonly arises as well.

The main criteria are given below:

• When all the bugs are fixed.
• When the deadline is reached.
• When the budget is exhausted or depleted.
• When all the test cases get passed.
• When the agreement is signed off.
• When a certain percentage of testing is done.
• When the Alpha and Beta testing ends up.

The exit criteria can be derived purely based on factors like risk, cost, etc. When testing of the main functional requirement has been achieved, then the testing will be stopped usually and they never look for minor bugs, which will create a trouble in the later periods.

Example: Software ABC is in a design phase. The development and testing construction generally occur at the same time. After the design has been frozen, the development of the software starts. Completion of the development of the software, as agreed upon, indicates the entry criteria. Deliverables here are from the development team. It includes release notes and known issues.

After few iterations of testing, when no major/blocker/show stoppers are pending resolution and 95% of testing have resulted in a pass, then it is referred as exit criteria.

Automation – Is It Really Essential?

When we need to decide if we require Automated Testing Technique or not, the question of available resources arises here.  The reasons we need to automate are in checking whether the data flow and the functionality developed are working as per the expectation without manual intervention or not. It is mainly used in places where the software will have changes in the form of multiple releases/cycles etc.

At the end of the development of each cycle, the testing of the currently added functionality will be done. Additionally, testing of the old functionality will be done to ensure that the old functionalities are not broken. This is the major part that has the scope for automation.

When verifying the code-driven logics and the GUI requirements, one can choose Automated testing, provided the risk factor is high.

Example: For the Software ABC, there are frequent upgrades, updates being sought by the client and provided by the developers. Hence as a part of testing, regression is done for the software which is already live and running in production. Irrespective of any number of releases, upgrades, and updates, the current version will be valid.

Say there are 10 days of manual efforts required for regression testing coverage, and then the utmost care for automating them must be taken. It can save at least 60% effort and 10*8 = 80 hrs manual work.

Automation can complete 80/24 = 3.33 days only.  This saves roughly 6.67.

Why Automation?

Automation can be chosen only when:

• The application has a very vast area with a high degree of investing effort in regression.
• Optimization in costs occurred due to manual errors.
• The software has multiple versions and releases.
• It is cost effective in long run.
• The risk factor is higher for a broader scope of test execution.
• Cost figures and mathematical calculations are included in the software functionality.
• There is a greater increase in the execution tempo, efficiency along with the software quality.
• There is a lesser turn around time, even for high-risk software testing.

Risk Factor

Risk factor becomes pre-dominantly common in the business where there are many dependencies on the time factor. The software that works based on the transactional systems and that works across multiple applications will require the software to act ideally as per the software design. In this case, there are many risks involved in getting the correct functional behavior recorded.

Here Automation will be very helpful in performing the functional transactions at a better pace as per the software mechanism.

For example, in the case of a Forex market indicator, the time factor is very important and critical. The changes in the stocks and commodities occur with respect to time, sometimes less than seconds. Here Automation can help in testing such software with high risk.

Example: Software ABC has multiple updates and upgrades. In order to save manual efforts and bring down the turn-around time for the testing phase, the base version, or the old functionalities, can be automated. This can become valid only when the base functionalities will remain unchanged.

The benefit in automation is that they can be run without any manual intervention. Even this can be performed in parallel with testing newer functionality. Hence automation saves much effort and a lot of time.

When should Automation not be PREFERRED?

There is a question among several organizations which is – Why 100% Automation is not possible?

The answer from experts is NO because skilled users are required to carry out Automated testing and they must be well trained as well. Automation cannot be carried out during the initial stage of the criteria and requirements of the applications will not be clear.

Usually, Automation is preferred from the second iteration of any software release. The user interface may be changed, which is more costly, and the script maintenance is also costlier.When the cost required for the automation tool exceeds the budget of the project we can say no.

Example: Software XYZ is a type of e-commerce site where the client requirements are not frozen and keep changing when required by the clients.

Here, in this case, automation cannot help the regression. This is because the old functionalities that are not valid should not be tested, and therefore, they have to be done manually. For instance, a client needs to have all list boxes in the base software to be changed as drop-down boxes.

Cost vs ROI for Automation

The ROI is very low when we go for automation initially because automation is expensive for the first time. ROI keeps increasing as the manual effort in testing the software, lowers from the iterations of the second release. We must be aware of the expected outcome of any test case before Automation.

Consider the design of the test cases more important when choosing the Automation and any tool to ensure it will not increase the cost.

Where can Automation render minimal to NO cost REDUCTION?

Even automation costs because the required tool for testing has to be purchased. The resources have to be trained with the particular tool. The tool chosen must be feasible to test all the areas of the software.

So the tool selection should be done carefully by the experts of Automation testing.

Example: Consider product XYZ which deals with insurance. To reduce the cost factor, the company used only manual testing, but when it comes to insurance, the risk factor is high and may cost the firm money when any one of the premium calculation goes wrong.The entire loss will be either for the management or to the end user. The end user will not bear loss whereas the company has to.

When the calculated premium amount mismatches with the original premium (i.e) when there is a difference in the front end and back end premium calculation, then a great problem arises between the customer and the product seller. It may contain many modules like automobiles, home and others as well.

When anything goes wrong, it is a complete loss. The difference in the calculation can make sense to the tester and may raise bugs. In this project, the manual testing can be done for the basic UI such as verifying the TIN number, social ID and other information related to the user portfolio and hence can be tested manually where the risk factor is low. The more the company would profit, the more they prefer automation for testing their software.

Conclusion

Both Automation and manual testing have advantages and disadvantages as well. Only when we are clear about the concepts and the requirements we will be able to choose what kind of testing to carry out.

No project can be tested with manual testing or automated testing alone. It depends upon the design, platform, and the technology with which the software has been developed with. So, when making a decision, one must be careful in choosing the method of testing and use the advice of experts.

In the above article, we might have missed few factors, kindly share the factors which you think are important while choosing automation or even tools for automation.

Meanwhile, feel free to share your comments/suggestions about this article.