Introduction to Sikuli GUI Automation Tool (Automate Anything You See on Screen) – Sikuli Tutorial #1

We constantly strive to provide fresh learning resources for our readers. Today, we take an in-depth look at the intriguing GUI automation tool Sikuli.

Experience Sikuli, a GUI automation tool that allows you to “automate anything you see”. This step-by-step beginner’s overview will assist you in quickly installing and working with the Sikuli Script tool, supplemented by comprehensive Sikuli guide.

Sikuli empowers automation of any on-screen item via image recognition to identify GUI elements. This facilitates users to automate GUI engagement utilizing screenshots.

Guide Topics in This Sikuli Series

The series comprises 3 parts:

Guide #1: Understanding Functioning, Creating a Simple Sikuli Project.
Guide #2: Utilizing Sikuli with Selenium Web Driver for Webpage Automation.
Guide #3: Automating Flash-Based Applications with Sikuli Tool

Sikuli GUI Automation Tool

Let’s get started with the initial segment of this series.

Sikuli is a utility that controls graphical user interfaces (GUI) using the “Visual Image Match” method. In Sikuli, all on-screen elements are processed as images and saved within the project. Sikuli initiates GUI engagement based on visual image match, which is passed as a parameter together with all methods.

Particularly effective for automating flash elements that lack ID or name, Sikuli proves efficacious when interacting with stable GUI components that do not transform. In addition, Sikuli can be utilized for automating window-based applications. It offers a user-friendly Sikuli-script.jar that can be conveniently integrated with Selenium WebDriver. Adobe Video/Audio players and website-based Flash Games can be automated via Sikuli. Its straightforward API simplifies coding.

Practical Applications

• Sikuli facilitates automation of Flash Objects / Flash Websites.
• It proves beneficial in automating Window based applications.
• A straightforward API is provided, allowing access to all methods via screen class objects.
• Sikuli effortlessly integrates with Selenium and other utilities.
• It allows automation of desktop application.
• Most automation testing utilities do not support flash-object automation (for example, Selenium). However, Sikuli offers extensive support for flash object automation.
• It employs a potent “Visual Match” mechanism for automating desktop and flash objects.

Advantages

• Open-source software.
• Sikuli specializes in automating Flash elements.
• Simplifies automation of Windows software.
• Sikuli is an effective tool when testing a developing application with unidentified ID/name of elements, to verify image appearance and drive interaction accordingly.

Preparation:

Prior to starting, ensure download and installation of the following software:

• Any tool for capturing screenshot (for instance, DuckCapture, or qSnap)
• JDK
• Eclipse (follow the detailed steps here for installing JDK and Eclipse)

Establishing The Sikuli Java Project

Step 1: Sikuli Download – Fetch Sikuli from here.

Step 2: Unzip the downloaded zipped file. It includes the Sikuli-script.jar file. Save this unpacked file on your local storage system.

Step 3: Start Eclipse.

Step 4: Create a Java project: File -> New -> Java Project

Step 5:

1. Right-click on the project
2. Navigate to Build Path -> Configure Build Path
3. Switch to the tab labelled Libraries
4. Hit the “Add External Jars” button and add Sikuli-Script.jar in the Build Path.
5. Select “Ok”

The Sikuli-script.jar file is now integrated into your project build path. You can initiate writing Sikuli scripts within this project.

Key Sikuli Methods

#1) Establish an Object for the Screen Class

The screen is the core class rendered by Sikuli. To access all the Sikuli methods, it’s necessary to formulate an object for this screen class.

Syntax:
Screen s = new Screen();

#2) Interact with an Element

This method is employed to interact with a specific image on the screen.

Syntax:
s.click(“<<image name>>”);

For Instance,
s.click(“demo.png”);

#3) Right Interact with an Element

This method is employed to right-interact with a specific image on the screen.

Syntax:
s.rightClick(“<<image name>>”);

For Instance,
s.rightClick(“demo.png”);

#4) Discover an Element

This method is employed to discover a specific element on the screen.

Syntax:
s.find(“<<image name>>”);

For Instance,
s.find(“demo.png”);

#5) Double Interact with an Element

This method prompts a double interact event on a specific image on the screen.

Syntax:
s.doubleClick(“<<image name>>”);

For Instance,
s.doubleClick(“demo.png”);

#6) Verifying Element Presence on the Screen

This method is employed to verify if a specified element is visible on the screen.

Syntax:
s.exists(“<<image name>>”);

For Instance,
s.exists(“demo.png”);

#7) Input string in a Textbox

This method is employed to input text into a specified text box.

Syntax:
s.type(“<<image name>>”,”Text to be inputted”);

For Instance,
s.type(“demo.png”,”Hello!!”);

#8) Wheeling Action on Specific Image

This method is employed to perform a wheeling activity on the identified image element.

Syntax:
s.wheel(“<<image name>>”,<<int position>>,<<int direction>>);

For Instance,
s.wheel(“demo.png”,25,0);

#9) Drag and Drop Image/Element

This method is employed to drag and drop a specified image from source to destination.

Syntax:
s.dragDrop(“<<source image name>>”,”<<destination image name>>”);

For Instance,
s.dragDrop(“demo.png”,”demo1.png”);

#10) Initiating Roll Hover on Specify Image

This method is used to perform a roll hover event on the specified image.

Syntax:
s.hover(“<<image name>>”);

For Instance,
s.hover(“demo.png”);

#11) Pasting Copied Text

This method is used to paste text into the specified text box.

Syntax:
s.paste(“<<image name>>”,”text”);

For Instance,
s.paste(“demo.png”,”demo”);

Sikuli Examples

#1) YouTube Clip – Pause And Play Video

Step 1) Open a YouTube video URL and capture