Understand and choose a load test model

When running load tests it is all about the concept of how you setup and run a test. Previously we already talked about getting to numbers, today we will talk about the load models you can use to run your tests.

One of the decisions you have to make is for the right load model. Sounds complicated – but in fact isn’t. A load model basically describes what basic characteristic you can influence to reach a certain load and performance behavior.

Just one more thing before we talk about the models, because the definitions used in the following paragraph are not unique across the industry. So each vendor uses them slightly differently.

Transactions: “A transaction is an execution of exactly one test case or test scenario. In order to perform the scenario, the page flow is modeled in code. The test scenario is implemented as a test case, which itself executes a sequence of one or more actions.”

Action: “An action can be defined as one irreducible step within a test case. So an action interacts with the current page and – as a result – loads the next page. That page is associated with this action and becomes the current page for the next action in the test scenario. Generally, an action triggers one or more requests.”

Requests: “This level is equivalent to the HTTP request level used in web browsers or in any other application that relies on HTTP communication. You do not have to deal with requests directly because they are automatically generated by the underlying HtmlUnit framework when performing actions on HTML elements.”

Let’s check out an example. Before running the next marketing campaign your company’s online shop will have to be tested to make sure it won’t break under that much traffic. Two of your teammates are discussing the problem:

Head of IT: “The shop will have to handle 2000 concurrent users.”

Head of marketing: “With our campaign the the shop will have to handle 500 orders per hour.”

Both requirements can lead to two different load models, depending on the exact goal that is set.

  1. User Count Model: This is the approach of the IT guy. Define a certain number of concurrent users the system will have to handle. At any given time during the test, the target system has to handle 2000 concurrent users, no more, no less. The number of transactions that can be achieved depends on the time the target system needs to respond.

  2. Arrival Rate Model: The money driven marketing guy tends to another criteria: The number of transactions (or here: orders) per hour. The scenario will be performed 500 times, equally distributed across a period of one hour. As many concurrent users as necessary to fulfill the given arrival rate are used.

Okay. Understood. Head of IT is happy with the user count model. He’s pretty sure the shop will be stable with 2000 concurrent users.

So what will happen if the response time of the system increases during the test period?

When using the user count model simply less transactions will be finished. In comparison the arrival rate model will increase the number of concurrent users as well, to make sure the given number of orders will be performed.

You can see the difference here? The arrival rate model works feedback based and will react to response time changes during the test period. This way, the generated load is somewhat unpredictable – and that’s how the real world behaves.

Or to use a more common analogy, people will usually still enter the supermarket despite the fact that there are long line at the cashier. That what happens in your online shop when the response time increases and you use the arrival rate model. More user are coming in, because they do not know that there are already lines in the store.

The chances for picking the arrival rate model are getting higher, but the IT guy is just a little bit afraid of the aggressive behaviour of such a test. Just imagine:

  • response time increases
  • more concurrent users are used
  • response time will increase even more due to heavier load
  • more concurrent users are used
  • recursion here

Load Test Models: Arrival Rate Model

But there is a solution. To avoid a complete breakdown, you can define an upper limit to the number of concurrent users used in the arrival model. This can help you to restrict the total load on the system, if you want to avoid a total overload as a result of the feedback loop. That is not reality of course, but of the user feel a certain pain in reality, they might hold back as well.

Ok, the arrival rate model is going to be used for the next load test. But there are definitely scenarios in which the user count model could be a good choice? Here are some guidelines which model fits what purpose best.

The user count model is well-suited for:

  • a simple base line test (single-user test) to assess the base performance of the system under almost no load,
  • a real load or performance test to assess the performance under a high, but predictable aka stable load,
  • a test that should be easily repeatable and its load factor is not influenced by the system under test.

The arrival rate model is best used if the load test should prove that a system is indeed able to handle a certain number of transactions per hour. Since this is the primary purpose of load and performance tests, the arrival rate load model is the best choice for most of your test tasks.

So get testing and give all models a try. Load tests often start with a fixed user rate and once this runs fine, you move over to the more challenging arrival rate model.

Creating test automation: Where to start and how to select test cases?

So, your boss told you to create a test automation for your company´s online shop? The checkout process is a subject of frequent customer complaints referring to application errors. As a good employee of course you excitedly confirmed this task and promised to get into the subject immediately.

Manual testing as well as test automation is a present topic in your company, so you find all necessary tools as well as a documentation pretty easily. Everything is installed in no time and you are ready to go.

But where to start? And how?
Continue reading Creating test automation: Where to start and how to select test cases?