Automated Fault-Tolerance Testing Using Katalon Studio

In this information, we’ll discover the idea of automated resilience testing to your take a look at suites utilizing maxFailedTests in Katalon Studio and TestOps. Learn methods to rapidly establish and deal with faults.

What does Resilience imply in Testing?

Are you conversant in the e book Fail Fast, Fail Often: How Losing Can Help You Win by Ryan Babineaux and John Krumboltz?

Well, in brief, the e book teaches us to deal with failures as alternatives for development and to be taught from them. When it involves software program testing, anticipating flawless releases or a very defect-free course of is just not life like. Instead, we are able to give attention to figuring out and managing minor bugs which have little affect on the appliance’s performance.

By doing so, groups can redirect their efforts and a spotlight to extra crucial points, comparable to system disruptions or malfunctions. If a part, system, or software program can proceed functioning as supposed regardless of the presence of defects, then these faults may be “tolerated”.

Automated Resilience Testing

As the saying goes: The earlier defects and failures are detected within the SDLC, the more cost effective they’re to repair.

Following the shift-left strategy in software program testing, receiving suggestions early and steadily can save money and time.

As bugs progress in direction of manufacturing, the price of fixing them will increase as a result of they turn out to be extra entangled with different code. Errors turn out to be more durable to breed, consuming growth time and resulting in missed deadlines.

Here are some practices adopted by software program growth groups:

• Using Continuous Integration (CI) and static evaluation to keep up clear code.
• Involving QA engineers within the early levels to know necessities and structure.
• Performing Root Cause Analysis to find out if defects had been brought on by a single situation or a series of occasions.
• Learning from failures and predicting software program conduct to cut back fault tolerance and enhance system reliability.

Setting a Failure Threshold

Determining the failure threshold depends upon the dimensions of your take a look at suites, the variety of take a look at instances, and the historic failure patterns noticed by your crew. This course of is much like Sprint Planning and story level estimation.

Team A and Team B might have totally different capabilities, leading to diverse story level estimations. Team A might need extra expertise dealing with complicated options, whereas Team B might have a wider vary of technical experience.

Therefore, the variety of story factors accomplished in a dash can’t be generalized throughout groups. Additionally, reordering the backlog for present and future sprints could also be essential.

There are a number of methods to find out an correct failure threshold:

• Count the entire variety of take a look at instances in your take a look at suite.
• Calculate the common variety of failed exams prior to now 90 days.

With these metrics, you possibly can know when to research the basis causes of failures with out ready for all of the exams to finish.

For instance, you probably have 1,000 take a look at instances, you possibly can set the failure threshold at 10%. This implies that if the variety of failed take a look at instances reaches 100, the execution of the remaining exams will likely be canceled.

In bigger initiatives, the variety of exams can attain a whole bunch or 1000’s, and relying solely on take a look at listeners is inadequate. When utilizing frameworks like Selenium, exams are executed one after the other with out the flexibility to alert the tester of a number of failures upfront.

In such instances, receiving early suggestions relating to defective code can save hours of ready for the whole take a look at suite to finish, solely to find points proper from the beginning.

Note: We will reveal methods to use Conditional Execution Termination in Katalon Studio.

Stopping Test Suite Execution After ‘X’ Failed Tests in Katalon Studio

Conditionally Terminating Execution

Step 1: Set the failure threshold to your Test Suites

To start, launch Katalon Studio and be sure that your undertaking, take a look at suites, and Katalon Runtime Engine are prepared.

*Note: Katalon Runtime Engine (KRE) is required to run exams in CLI mode.

Click the Build Command button positioned to the left of the Playback button.

Select your Organization and test the Terminate the execution as soon as the entire variety of take a look at failures reaches this threshold possibility.

Enter your required quantity and click on Generate Command.

This configuration ensures that Katalon Studio stops the take a look at suites instantly as soon as the required variety of failed take a look at instances is reached.

Step 2: Run Test Suites from the command line

Click Copy to Clipboard.

Then, paste the command with the -maxFailedTests=T parameter into your Terminal and press Enter.

Step 3: Execution terminated and Pass/Fail exams

Once the failure restrict is reached, you will note the next message.

You can get an summary of the handed and failed exams for every run.

Viewing Pass/Fail Trends in Katalon TestOps

Once you’ve got set your failure threshold, you need to use TestOps, Katalon Studio’s built-in take a look at reporting and orchestration platform, to investigate cross and fail traits.

Step 1: Set up an Organization and Project

Log in to analytics.katalon.com utilizing your Katalon account.

Step 2: Enable TestOps Integration in Katalon Studio

In Katalon Studio, go to Project > Settings (applies to each Mac and Windows).

• Click on Katalon TestOps, test the Override authentication field, and choose your Organization (in case your Organization is just not listed, click on Fetch Organization).
• Enter your Katalon e mail and password.
• Tick Enable Katalon TestOps Integration and select your Team > Project > Apply and shut.

Step 3: Push and think about Katalon Studio Test Suite Results on TestOps

Note: TestOps organizes take a look at runs by Test Suite. Ensure that your Test Cases are organized inside particular Test Suites for full recording.

Navigate to TestOps within the left menu > Test Runs > select a Test Run.

In TestOps, you possibly can analyze the cross/fail patterns of your take a look at runs.

TestOps additionally gives extra experiences on flaky exams and rancid take a look at runs for additional exploration.

Start your first undertaking with Katalon Studio.

For extra info on the fail-fast precept, confer with the complete article on Conditionally Terminating Execution in Katalon Studio.