Simplify Your Life With a Little UI Automation

Are you familiar with someone asking you to do things such as putting all of your clients into the new CRM or inserting 80 new users in a software product x? If so and if your software has no import feature, you probably found yourself copying and pasting all day long. A simple way to save you from this torture is using a UI automation tool like Sikuli Script.

What is Sikuli?

Sikuli automates UI functionalities like mouse, keyboard, and clipboard actions. It works with UI recognition to identify UI components such as buttons, input fields, or menus. To use this kind of recognition, Sikuli compares the actual screen with screenshots from the user.

You can use Sikuli via the Sikuli IDE, Sikuli Slides, or directly from Java code. The latter is our focus for now.

How to Use Sikuli in Your Java Program

To install Sikuli, visit www.sikuli.org, download the version for your OS, and unzip it. In the lib folder, you can find the sikuli-script.jar that you’ll need in your classpath.

Note: You may have to to unzip the whole downloaded file and install some requirements like OpenCV2.1 (Linux). Also you may need to add some environment variables.

Adding variables to the run config in Eclipse
Adding variables to the run config in Eclipse

As soon as everything is setup, you can start coding.

Example

Let’s go for a simple example: you have a list of URLs and for each you want to open a new tab in Firefox.

Read the list with simple Java I/O operations:

The next step is the automation. Start with a screen:

To automate the mouse usage, you need a screenshot of the button you want to press. To use this button, tell Sikuli to find it on your screen:

Now you’re able to use this button the exact same way as your mouse:

Keep in mind that, by default, you can just click in the center of the screenshot.

To insert data into an UI, you may also revert to your keyboard. With the type(String s) method from the screen you can type any string using the keyboard. If you decide to use the keyboard, keep in mind that you’ll have to deal with different keyboard layouts. To avoid this, you can simply use the clipboard:

Note: It’s always a good idea to give the software some time to react, so add some sleep time:

Since Firefox automatically places the cursor into the address bar, we don’t need to add another screenshot to click on. But of course we have to confirm the URL by pressing enter:

Hint: If you have to fill out multiple fields, use s.type(Key.TAB); instead of clicking into the fields. It’s faster and more reliable.

Running Sikuli from Java has one disadvantage: you have to minimize your IDE after starting the program because Sikuli can only use what it actually sees. To be sure you have enough time to do so, it’s recommended to add some sleep time. Three seconds should do.

Now the program is completed.

Just run it, place Firefox in the top of your screen, and enjoy the show. Of course, these are only a few of the many possibilities Sikuli offers. However, to solve such simple but annoying tasks as we’ve outlined here, this is all you need.

Note: If you get some .dll errors in Windows, try to run it in debug mode.

One thought on “Simplify Your Life With a Little UI Automation”

Leave a Reply

Your email address will not be published. Required fields are marked *