Tag Archives: charts

The Art of Reading Performance Test Charts

Powerful load and performance test tools don’t only retrieve pages of your website randomly with zillions of users at the same time, but they also cover realistic scenarios simulating the real-world user. It’s a given that they can deliver lots of useful information and plot interesting charts. To fully take advantage of these benefits, however, you need to be able to interpret this information and draw the right conclusions.

It is this need for the correct interpretation of test results, the mapping of all you see against actual application behavior that makes performance and load testing a non-trivial task. It requires much experience to decide on the right actions, make the right assumptions, or simply come up with a reasonable explanation of why something happened this way and not the other.

In today’s article, we’d like to present you a couple of charts displaying typical response time patterns, and discuss what they could indicate.

Disclaimer: Of course, the reasons for a certain behavior vary a lot, depending on your application and testing. However, as there’s no fixed manual for the interpretation of load testing charts, we want to provide you with a couple of basic guidelines to help you get better in interpreting them yourself and make the most out of your test results. Feel free to comment whether or not you agree with the ideas and explanations we come up with.

The Warm-up

These charts might indicate a system with a cold cache, for instance, when the system has just been started and the caches aren’t filled yet.

The basic characteristics of such a behavior are high response times in the beginning, followed by gradually lower response times until eventually a  certain degree of runtime stability is reached. This time frame is often referred to as the system’s warm-up period. Throughout this period, a couple of things can happen under the surface. If you know the system under test well, you’ll probably come up with the following: database and file system caches are filled, proxies learn about the data and store them, the system under test scales up because it sees traffic, page snippets are cached and so the computing overhead reduces… you name it.

Also keep in mind that it might be the testing process itself that causes such a response time profile. If the system is perfectly warmed up and you hit it, your sent traffic might be too uniform in the beginning. That being the case, randomization kicks in so that the traffic eventually distributes better over time. Furthermore, take into consideration that your load software and hardware are possibly not warmed up either.

The Caching

These charts depict either a typical cache clean or job patterns. In case of a cache clean, system-internal caches expire every half hour. If that’s not the case, the charts may indicate  a running background job draining power from the database or consuming lots of system bandwidth.

Both charts display the same test; however, this test has been executed for different time periods. While two spikes could signify a random event (despite the fact that the temporal distance of 30 minutes is suspicious), the longer test run seems to prove our first assumption: something is going on every half hour.

In any case, make sure that such a behavior is not produced by the test machines themselves, for example, because they’re busy writing or backing up data.

The Spiking

This is what we call a forest of spikes: many spikes that don’t seem to follow a comprehensible pattern; longer runtimes just occur occasionally, often caused by requests accessing certain data or URLs that produce long runtimes. To get behind that mystery, you have to dig into the results in more detail, find the calls behind the spikes, and derive a pattern based on the information you find. Often you’ll come across similar URLs, request parameters, or maybe response codes. Don’t forget any application logs you might have access to, such as web server, error, information, or debug logs. In a perfect world, your application under test offers the necessary tools to get to the bottom of this problem.

XLT lets you easily find this information. All test result data are accessible as CSV files that are quickly readable and documented. Feel free to work with this information and go beyond the scope of the reports available.

The worst outcome here is a non-identifiable pattern and no information on the server side as to what might have happened. In such a case, you have to repeat the test and narrow down your test setup later on to exclude as many variables as possible to find the cause. This is actually also a good time to ask for development or tech-ops support.

The 3rd-party Calls

The first chart is typical for issues with 3rd parties, especially in the field of e-commerce. We’re not talking about direct calls to 3rd parties here, such as analytics vendors or recommendation engines, but calls from one server to the other. Thus, the response time we see is the response time of two systems. Of course, it’s good to know the area where 3rd party calls typically happen, but you have to know the application under test anyway to test it efficiently. So when the final order steps start to act weird, you can easily narrow down the potential reasons.

The second chart looks more like the cache clean or expiration problem described above, but since you know the application, you also know that this area doesn’t use the typical caching logic but is highly dynamic instead. This means that the errors occurring every 50 minutes point into a different direction: as we know that 3rd parties are attached and called during shipping, we can conclude that the 3rd party failed on us.

Verdict

Knowing typical response time patterns helps you specify a certain problem so that you can give hints to the development or further shape the path of testing. If you can read charts and derive the right conclusions or at least know which questions you have to address, you’ll be ahead of the crowd. Be aware that knowledge on the system under test is very important – the production and measurement of a certain load doesn’t make much sense when you’re not able to actually interpret and explain what you’ve measured. Always remember: 42 is not a valid answer for everything. :)

XLT – Garbage Collector details visualized

Today we want to give you a small preview of an upcoming XLT feature. Most of you probably know that XLT features agent statistics. These statistics help you to keep an eye on the health of the test execution engines (agents) to ensure that you do not influence the test results by providing insufficient hardware or by applying no or incorrect settings.

Most modern programming languages are virtual machine based and these machines have knobs you can turn to adjust their behavior according to your requirements. XLT runs on Java and so all the things you might have already learnt from tuning your Java-based servers apply to XLT as well. If you do not have experience in tuning your Java-based servers, you will learn a lot that can be applied to your servers and help you to increase performance.
Continue reading XLT – Garbage Collector details visualized

Burn-in test of XLT 4.0

If you write a performance testing software, your first obligation is a performance test for and with that software. So before we can ship XLT 4.0, we have to make sure that it can hold up to its promises. Test software has to be tested too, so to speak. Today we provisioned a bunch of Amazon-EC2 boxes with 30 test agents in total. These charts are from a short test to see where we can get to, when ignoring any target numbers. This was a short and violent test. Just hammer the system under test and see if it can and will recover.

HitsPerSecond

A short ramp up period in the beginning of the test and afterwards we kept a steady load factor. For the steady phase we reached more than 11,000 hits per second.

Concurrent Users

The target system was seeing about 2,200 concurrent users during the peak of the test.

Received Bytes per Second

During that test, the network was transporting about 95 Mbytes/s inbound traffic, this is a network utilization of about 760 Mbit/s. Amazing that the EC2 boxes in the EU data center can handle that traffic. We used just 10 boxes and each box has a 100Mbit network, so the overall limit must have been reached we think.

By the way, the target system recovered easily and was able to serve its normal duties without any problems. But this test clearly showed the limits in terms of throughput. But this test did not show any limits of XLT 4.0, because neither the load boxes in terms of CPU nor the reporting had any problems with this test size.

Die neuen Reports in XLT 3.2

XLT 3.2.1 Beispiel ReportWir hatten ja noch einige Informationen zu XLT 3.2 versprochen. Heute gibt es einen Blick auf die neuen Reports, die nach einem Lasttest automatisch erzeugt werden.

Die Reports werden mittels XSLT aus XML-Daten erzeugt und sind pures XHTML 1.1 und CSS 2.1. Damit können sie leicht den eigenen Bedürfnissen angepasst werden. Auch die Größe der Diagramme ist veränderbar.

Man kann auch verschiedene Reports mit unterschiedlichen Inhalt gleichzeitig rendern lassen und so verschiedene Zielgruppen ansprechen. So zum Beispiel einen sehr detailierten für den Tester und einen Zusammenfassung für das Manager-Meeting am nächsten Morgen :)

Der komplette Report findet sich hier