Today’s article can be seen as a survey reflecting our thoughts on web test automation in general. It basically lists personal experiences that we were able to gain in customer projects or conclusions that we arrived at in recent discussions on this topic.

Feel free to use these ideas and thoughts as input for your own discourse on test automation, whether it is about starting it, keeping it running, or just questioning it every once in a while.

We do not claim that this is the only way of seeing things, so as always we are more than open to new input, arguments, or just a simple “Nah. Never!”.

Note that some of these thoughts are directly related to web UI automation – just in case you can’t match locators, CSS, or XPath to your current automation problem.

In General

• Automation mainly suits simple, straightforward testing that eats too much valuable time when done manually. The test area also shouldn’t be too sensitive to breakage due to different test constellations including pre- and postconditions.
• If it’s hard to automate and to maintain it, it’s not worth the effort.
• Test automation is very good at doing things humans can’t do or don’t want to do.

Keep in Mind

• Automated test cases usually don’t vary that much, the test is basically always the same. Change that! Varying test data helps you find additional defects, so you want to make sure not to be too static.
• Test automation can only detect problems it checks for. If you miss to check for certain things, the automation runs through and reflects a quality picture that doesn’t necessarily match reality.
• Test automation quality increases with its validation quality.
• Yes, running automation multiple times on the same code does not contribute to quality but may certainly be useful to identify test automation defects.
• Writing tests will let you find more defects than executing tests.
• Speed up the development process instead of trying to save a few dollars on testing.
• Developers should be able to run the tests before committing their code.
• Test automation is code. Code can have defects or can be wrong. Conclusion: Test automation requires both test and review.

Test Case Content

• Don’t squeeze too much into one test case!
• Have a test case that simulates the perfect world first.
• Afterwards have test cases that simulate deviations, such as incorrect data handling or creation of data that already exists.

Test Case Dependencies / Test Data Dependencies

• A test case should not depend on another test case.
  • It should create the data it needs in the beginning or participate in a global setup before all automation runs.
  • A failure of one test case shouldn’t block others from being executed.
• Test data should not be hardcoded in the test case. Rather use externalized test data to facilitate maintenance.
• Make sure you can rerun the test without having to reset data.
• If a test leaves a trail of data due to a failure or by definition, the next test should not be affected by that.

Comments

• Test automation is software development, so obey the rules your developers have to follow.

Data

• Try to use real and good data – this includes special characters because such a test case will cover both the normal case and the special character part simultaneously.
• Consider data-driven tests (possible with XLT JUnit approach) to execute the same test case multiple times with different data. This also simplifies the creation of larger data sets, such as 100 campaigns.

Locators

• Locators should be as short as possible and identify the element as unambiguously as possible.
• Prefer CSS over XPath because it’s easier to read.
  • //table[contains(@class,’additem')]//button would be css=table.additem button
• Avoid position statements such as //table//button[2] because they will break quickly.
• Try to use internal data to find elements since using the name of a button, for instance, causes you to depend on language. Granted, this may not be entirely avoidable in the first place but if you do find a nicely named CSS class, make sure to use it.

State and Timing

• Always wait after an interaction to have it completed and never before! That’s also known as the Ajax challenge.
• Wait for not longer than X amount of time for an action to complete. If you have to wait longer than X, consider the application broken.

Validation

• There is no need to validate an element before you use it. So if you want to input data, you can fail at the input because the element is missing. No need to fail at an assertion before that. Saves code and doesn’t change a thing!
• Always validate after an activity:
  • If you save something, validate that it was saved.
  • Filling out input fields doesn’t need any validation, but search while filling out input fields might need additional validation since it has the suggestion come up.

Misc

• Execute any test several times with several speed settings before you commit to your code repository. Both modified and completely new tests should be “tested” that way.
• If a test is too difficult to automate, chances are high it should remain a manual test.
• If a test requires a long wait between steps due to the wait for jobs or similar things, it might not be a good candidate for a test (yet).
• Not every test needs to run during build time or as part of an acceptance test. Maybe we can have another class of tests. Let’s call it automated/partially automated integration tests that include the following:
  • If a test requires manual intervention, it still can be partially automated to save time and make it predictable.
  • If the entire runtime of a test is too big, that is 5 minutes for a single test case.
  • If the test requires a manual data setup such as an site import that can take hours.

We hope these thoughts will help you improve, sell, or simply rethink automation, especially web UI automation. Comments are very welcome!