Simplify Your Life With a Little UI Automation

Are you familiar with someone asking you to do things such as putting all of your clients into the new CRM or inserting 80 new users in a software product x? If so and if your software has no import feature, you probably found yourself copying and pasting all day long. A simple way to save you from this torture is using a UI automation tool like Sikuli Script.

What is Sikuli?

Sikuli automates UI functionalities like mouse, keyboard, and clipboard actions. It works with UI recognition to identify UI components such as buttons, input fields, or menus. To use this kind of recognition, Sikuli compares the actual screen with screenshots from the user.

You can use Sikuli via the Sikuli IDE, Sikuli Slides, or directly from Java code. The latter is our focus for now.

How to Use Sikuli in Your Java Program

To install Sikuli, visit www.sikuli.org, download the version for your OS, and unzip it. In the lib folder, you can find the sikuli-script.jar that you’ll need in your classpath.

Note: You may have to to unzip the whole downloaded file and install some requirements like OpenCV2.1 (Linux). Also you may need to add some environment variables.

Adding variables to the run config in Eclipse

As soon as everything is setup, you can start coding.

Example

Let’s go for a simple example: you have a list of URLs and for each you want to open a new tab in Firefox.

Read the list with simple Java I/O operations:

BufferedReader br = new BufferedReader(
                        new InputStreamReader(
                            new FileInputStream("urls.txt"), "UTF-8"));

String string = br.readLine();
while (string != null)
{
    // TODO do something with this line
    string = br.readLine();
}

br.close();

BufferedReader br = new BufferedReader(

new InputStreamReader(

new FileInputStream("urls.txt"), "UTF-8"));

String string = br.readLine();

while (string != null)

{

// TODO do something with this line

string = br.readLine();

}

br.close();

The next step is the automation. Start with a screen:

Screen s = new Screen();

1	Screen s = new Screen();

To automate the mouse usage, you need a screenshot of the button you want to press. To use this button, tell Sikuli to find it on your screen:

Pattern add = new Pattern("button.png");
Match match = s.find(add);

1 2	Pattern add = new Pattern("button.png"); Match match = s.find(add);

Now you’re able to use this button the exact same way as your mouse:

s.click(match);

1	s.click(match);

Keep in mind that, by default, you can just click in the center of the screenshot.

To insert data into an UI, you may also revert to your keyboard. With the type(String s) method from the screen you can type any string using the keyboard. If you decide to use the keyboard, keep in mind that you’ll have to deal with different keyboard layouts. To avoid this, you can simply use the clipboard:

s.paste(string);

1	s.paste(string);

Note: It’s always a good idea to give the software some time to react, so add some sleep time:

Thread.sleep(500);

1	Thread.sleep(500);

Since Firefox automatically places the cursor into the address bar, we don’t need to add another screenshot to click on. But of course we have to confirm the URL by pressing enter:

s.type(Key.ENTER);

1	s.type(Key.ENTER);

Hint: If you have to fill out multiple fields, use s.type(Key.TAB); instead of clicking into the fields. It’s faster and more reliable.

Running Sikuli from Java has one disadvantage: you have to minimize your IDE after starting the program because Sikuli can only use what it actually sees. To be sure you have enough time to do so, it’s recommended to add some sleep time. Three seconds should do.

Thread.sleep(3000);

1	Thread.sleep(3000);

Now the program is completed.

import java.io.BufferedReader;
import java.io.FileInputStream;
import java.io.InputStreamReader;

import org.sikuli.script.Key;
import org.sikuli.script.Match;
import org.sikuli.script.Pattern;
import org.sikuli.script.Screen;

public class TestSikuli
{
	public static void main(String[] args) throws Exception
	{
		BufferedReader br = new BufferedReader(new InputStreamReader(
								new FileInputStream("urls.txt"), "UTF-8"));

		Thread.sleep(3000);
		Screen s = new Screen();

		Pattern add = new Pattern("button.png");
		Match match = s.find(add);

		String string = br.readLine();
		while (string != null)
		{
			s.click(match);
			Thread.sleep(500);
			s.paste(string);
			s.type(Key.ENTER);

			string = br.readLine();
		}

		br.close();
	}
}

import java.io.BufferedReader;

import java.io.FileInputStream;

import java.io.InputStreamReader;

import org.sikuli.script.Key;

import org.sikuli.script.Match;

import org.sikuli.script.Pattern;

import org.sikuli.script.Screen;

public class TestSikuli

{

public static void main(String[] args) throws Exception

{

BufferedReader br = new BufferedReader(new InputStreamReader(

new FileInputStream("urls.txt"), "UTF-8"));

Thread.sleep(3000);

Screen s = new Screen();

Pattern add = new Pattern("button.png");

Match match = s.find(add);

String string = br.readLine();

while (string != null)

{

s.click(match);

Thread.sleep(500);

s.paste(string);

s.type(Key.ENTER);

string = br.readLine();

}

br.close();

}

Just run it, place Firefox in the top of your screen, and enjoy the show. Of course, these are only a few of the many possibilities Sikuli offers. However, to solve such simple but annoying tasks as we’ve outlined here, this is all you need.

Note: If you get some .dll errors in Windows, try to run it in debug mode.

Passionate Testing

Simplify Your Life With a Little UI Automation

What is Sikuli?

How to Use Sikuli in Your Java Program

Example

One thought on “Simplify Your Life With a Little UI Automation”