Should Load Tests Validate Functionality?

My answer to this question is a very strong “yes“. You might want to limit yourself a little in the overall validation, but checking response codes only is a strong fail in my opinion. Additionally, just checking the result by checking a single phrase or word is not enough.

Reasons and Examples

  • Modern web implementations often incorrectly return application status pages with response code 200.
  • How do you ensure that you got the entire page back and not only the first 75%?
  • Imagine an e-commerce search that breaks under load and instead of saying “I found 200 matches”, it returns a page saying “no matches found, did you mean …”. The latter is still a valid page but your load test will not discover the flaw.
  • If you implement real page flows, meaning you don’t have hard-coded URLs that you fire against the server and instead take the data from the page and apply randomness to get a broad sample, you might find pages that are incorrect or inconsistent, but which are hidden outside normal functional-testing click paths.
  • How can you distinguish a “login succeeded” page from a “login failed” page when you do not validate for content? Both are valid scenarios and “login failed” might not return an HTTP code different from 200. Even when it is designed to return “get lost”, aka HTTP code 401, it might break under load and incorrectly return 200.
  • Always assume the worst case: always assume that the application will behave differently under load. Not only timing-wise but also behaviorally. This is one reason for running load and performance tests.

A Tale from the Past

In my life before Xceptance, a colleague was testing something for a new web component. He said that the new implementation was better, because it got faster when he applied more load.

What? Well, he did not check the content of the returned page. Under higher load, he got error pages back and incorrectly assumed great performance instead of a broken application.

How We Handle Validation

Because I am with a web testing company (Xceptance) specialized in load and performance testing, here is what we generally do for all our load and performance testing:

  • Validate the response code
  • Validate the basic structure of the page: check that the main components are there, such as header, footer, search box, mini cart, user account link, etc.
  • Validate the completeness of the page: in other words, the closing HTML tag should be there or the JSON or XML should be complete (hopefully also valid)
  • Validate the most important points of the current activity, e.g:
    • if you search for an existing product, you have to find it
    • if you expect to get no search results, check that you get the correct message
    • if you add something to the cart, validate that the quantity changes correctly
    • if you check out, validate the order number format (under perfect conditions even the uniqueness or at least the change between checkouts); validate that all price information is a valid format (not necessarily calculated correctly, although that could be required under certain test conditions); check that the (random) user name used is applied during that visit (for billing, shipping, order, account, etc.)
    • if you log on, verify that from then on, you always see the name of the user on the page (the famous “Welcome Mr. Foo” or something like that)
    • verify that you do not see a “Welcome Mr. Foo” message before logging in, that you do not have a cart when you first arrive, and so on

We have so often found problems that are load related or often also functionality related that every single validation step really pays off again and again.
Only when applications return bad stuff and the implementation crew does not want to change that and cannot be convinced or forced to clean up the mess, do we apply exceptions and reduce the validation.

These are just examples, of course.

Verdict

You should extensively validate during load and performance testing. Always assume both that the application will break under load, and also that the failure is mostly not reflected in a status code 500. Status codes 404 and 500 are for the beginner; the real professionals find problems beyond that.

We usually find functional problems during our load and performance testing that would have stayed undetected without proper validation. Of course, don’t forget the classical erroneous performance behavior such as inconsistent response times, spikes, and everything else imaginable.

Or in other words: You do load and performance testing to find the unusual and unexpected. This is often not reflected in increased response times and error codes.

Of course there are tests when you really are hunting for something specific, and when you can easily skip most of the validations, but take it from us: never strip them out completely.

This article was created as response to a question on StackExchange: Should load test validate functionality?