Introduction to Sikuli GUI Automation Tool (Automate Anything You See on Screen) – Sikuli Tutorial #1

Our goal is to always provide new learning opportunities for our readers. Today, we will explore an interesting GUI automation tool called Sikuli.

“Automate anything you see” using Sikuli, a graphical user interface (GUI) automation tool. This complete beginner’s guide will help you quickly set up and start using the Sikuli Script tool, with in-depth Sikuli tutorials.

Sikuli automates any element you see on the screen by using image recognition to identify GUI elements. It allows users to automate GUI interaction by using screenshots.

List Of Tutorials In This Sikuli Series

The series is divided into 3 parts:

Tutorial #1: How It Works, How To Create A Simple Sikuli Project.
Tutorial #2: How Sikuli Can Be Used With Selenium Web Driver To Automate Webpages.
Tutorial #3: Automating Flash Based Applications Using Sikuli Tool

Sikuli GUI Automation Tool

Let’s begin with the first part of this series.

Sikuli is a tool that automates graphical user interfaces (GUI) using “Visual Image Match” method. In Sikuli, all web elements are treated as images and stored inside the project. Sikuli triggers GUI interactions based on image visual match, which is passed as a parameter along with all methods.

Sikuli is particularly useful for automating flash objects that do not have ID or name. It is effective when dealing with stable GUI components that do not change. Additionally, Sikuli can also be used to automate window-based applications. It provides a user-friendly Sikuli-script.jar that can be easily used in conjunction with Selenium WebDriver. Adobe Video/Audio player, as well as Flash Games on websites, can be automated using Sikuli. With its simple API, coding becomes easier.

Practical Uses

• Sikuli can be used to automate Flash Objects / Flash Websites.
• It is useful in automating Window based applications.
• It provides a simple API, with all methods accessible using screen class objects.
• It easily integrates with Selenium and other tools.
• Sikuli enables automation of desktop applications.
• Most automation testing tools do not support flash-object automation (e.g. Selenium), but Sikuli provides extensive support for automating flash objects.
• It uses a powerful “Visual Match” mechanism to automate desktop & flash objects.

Benefits

• Open-source Tool.
• Sikuli excels in automating Flash objects.
• It simplifies automation of Windows applications.
• When testing an application under development and the ID/name of elements is unknown, Sikuli can be used to check the appearance of images and interact accordingly.

Prerequisites:

Before getting started, you will need to download and install the following software:

• Any screenshot capturing tool (e.g., DuckCapture, or qSnap)
• JDK
• Eclipse (detailed steps here to install JDK and Eclipse)

Steps To Create The Sikuli Java Project

Step 1: Sikuli Download – Download Sikuli from here.

Step 2: Extract the downloaded zip file. It contains the Sikuli-script.jar file. Save this extracted file in your local file system.

Step 3: Open Eclipse.

Step 4: Create a Java project: File -> New -> Java Project

Step 5:

1. Right-click on the project
2. Go to Build Path -> Configure Build Path
3. Switch to the Libraries tab
4. Click the “Add External Jars” button and add Sikuli-Script.jar in the Build Path.
5. Click “Ok”

The Sikuli-script.jar file has been added to your project build path. Now, you can start writing Sikuli scripts within this project.

Some Sikuli Methods

#1) Creating an Object for the Screen Class

The screen is the base class provided by Sikuli. To access all the Sikuli methods, you need to create an object for this screen class.

Syntax:
Screen s = new Screen();

#2) Clicking on an Element

This method is used to click on a specific image on the screen.

Syntax:
s.click(“<<image name>>”);

For Example,
s.click(“test.png”);

#3) Right Clicking on an Element

This method is used to right-click on a specific image on the screen.

Syntax:
s.rightClick(“<<image name>>”);

For Example,
s.rightClick(“test.png”);

#4) Finding an Element

This method is used to find a specific element on the screen.

Syntax:
s.find(“<<image name>>”);

For Example,
s.find(“test.png”);

#5) Double Clicking on an Element

This method triggers a double click event on a specific image on the screen.

Syntax:
s.doubleClick(“<<image name>>”);

For Example,
s.doubleClick(“test.png”);

#6) Checking if an Element is Present on the Screen

This method is used to check whether a specified element is present on the screen.

Syntax:
s.exists(“<<image name>>”);

For Example,
s.exists(“test.png”);

#7) Typing a string on a Textbox

This method is used to enter text into a specified text box.

Syntax:
s.type(“<<image name>>”,”String to be typed”);

For Example,
s.type(“test.png”,”HI!!”);

#8) Performing Wheeling on a Particular Image

This method is used to perform a wheeling action on the specified image element.

Syntax:
s.wheel(“<<image name>>”,<<int position>>,<<int direction>>);

For Example,
s.wheel(“test.png”,25,0);

#9) Dragging and Dropping an Image/Element

This method is used to drag and drop a specified image from the source position to the target position.

Syntax:
s.dragDrop(“<<source image name>>”,”<<target image name>>”);

For Example,
s.dragDrop(“test.png”,”test1.png”);

#10) Performing Roll Hover on a Particular Image

This method is used to perform a roll hover event on the specified image.

Syntax:
s.hover(“<<image name>>”);

For Example,
s.hover(“test.png”);

#11) Pasting a Copied String

This method is used to paste text into the specified text box.

Syntax:
s.paste(“<<image name>>”,”test”);

For Example,
s.paste(“test.png”,”test”);

Sikuli Examples

#1) YouTube Video – Pause And Play A Video

Step 1) Open a YouTube video link and capture the play and pause element images using a screen capture tool.

Pause button (Note: filename is pause.png)

Play button (Note: filename is play.png)

Copy these images into your project.

Step 2) Create a package inside the Sikuli Java project and within that, create a class named “Youtube”.

Step 3) Type the following code inside that class.

 package com.test;

import org.sikuli.script.FindFailed;
 import org.sikuli.script.Screen;

public class Youtube {

public static void main(String[] args) throws FindFailed, InterruptedException {
 // TODO Auto-generated method stub

Screen s = new Screen();
 s.find("pause.png"); //identify pause button
 s.click("pause.png"); //click pause button
 System.out.println("pause button clicked");

s.find("play.png"); //identify play button
 s.click("play.png"); //click play button
 }

} 

Step 4) Right-click on the class and select Run As -> Java Application.

#2) Open Notepad And Type Some Text

Step 1) Capture a screenshot of the notepad icon on the desktop.

 

notepad_icon.png

notepad.png                                              

Step 2) Copy these images into your project.

Step 3) Create a class named “NotepadExample” and type the following code.

 package com.test;

import org.sikuli.script.FindFailed;
 import org.sikuli.script.Screen;

public class NotepadExample {

public static void main(String[] args) throws FindFailed {
 // TODO Auto-generated method stub

Screen s = new Screen();
 s.click("notepad_icon.png");
 s.find("notepad.png");
 s.type("notepad.png", "This is Nice Sikuli Tutorial!!!!");
 }

} 

Step 4) Open the screen to be tested before executing the code.
Execute this file by right-clicking and selecting Run As -> Java Application.

#3) Drag And Drop

Step 1) Take a screenshot of the required items on the screen and put them inside your Sikuli project.

[Note: here, the downloads icon is “source.png” and the flower image is “destination.png”]

Step 2) Put these pictures into your project.

Step 3) Create a class named “DragAndDrop” and write the following code.

 package com.test;

import org.sikuli.script.FindFailed;
 import org.sikuli.script.Screen;

public class DragAndDrop {

public static void main(String[] args) throws FindFailed, InterruptedException {
 // TODO Auto-generated method stub
 Screen s = new Screen();
 s.find("source.png");
 System.out.println("Source image found");
 s.find("target.png");
 System.out.println("Target image found");
 s.dragDrop("source.png", "target.png");
 }

} 

Step 4) Execute this script by right-clicking and selecting Run As -> Java Application.
After the execution of this script, the download icon will be dragged and dropped on the image indicated as the target.

Before Execution:

After Execution:

Drawbacks Of This Tool

• We cannot guarantee that the image match will always be accurate. Sometimes, if there are two or more similar images on the screen, Sikuli may select the wrong image.
• If the appearance of an image varies in pixel size, it may result in a “Find Failed” exception.
• There is the overhead of taking too many screenshots.
• If any of the screenshots are missing, it will affect the program’s execution.

More resources:

• Simple Sikuli example here.
• Detailed Sikuli documentation here.

Conclusion

Sikuli is extremely useful for automating flash objects and window-based applications. It offers a great way to interact with elements on a screen based on their visuals.

About the author: This is a guest post by Anitha Eswari. She is currently working as a senior test engineer with in-depth knowledge of manual and automation testing and various test management tools.

Next Tutorial: In the next part of this series, we will take a deep look at creating a Sikuli Maven project and integrating Selenium with Sikuli.

If you are already using this tool, please share your experience and tips. If you are just getting started and have any queries, let us know.