Category Archives: XLT

Xceptance Releases XLT 4.3

Xceptance released version 4.3 of its load testing and test automation product Xceptance LoadTest. It features a wide range of improvements and new functionality.

As always, this update is free of charge for every user. You can use XLT freely for your daily automation work, regression testing, and performance validation. If you are an eligible Open Source project, you can get a full XLT license for your load and performance testing needs for free as well.

Script Developer

Enjoy more automation with less clicking. The new information panel summarizes the specifics of the currently selected item, so you do not need to open it anymore. Additionally, the new log panel lists all executed commands and their respective parameters.

If an element is styled with the CSS property text-transform, then the element’s text on the screen may have different character casing from what is defined in the page’s DOM tree. The WebDriver specification mandates to return the text as shown on the screen. Script Developer will now record text with the the character casing that appears on the screen and it will also take the CSS property text-transform into account when replaying text assertions.

Reporting

The XLT load test reports have been improved by adding more charts and data details, such as overall statistics for all timers. Arrival rate charts have been added to visualize the load development over time and make sure the desired load factor was reached.

Charts can now optionally be configured to be logarithmic. A capping can be added to hide larger spikes, which usually make charts hard to read, without removing the important information altogether.

The new networking section summarizes all general network-related statistics and charts on a separate page.

The error and event page has been redesigned and includes a new error chart that contains separate graphs for all transactions/actions/requests so that the temporal distribution of transaction/action/request errors is displayed in one chart. A new error summary table groups all errors by their error message to help you see which types of errors occurred and how many of them.

Load and Performance Testing

The master controller features two new commands to validate the availability of agents and  display their current configuration. A new command line option permits the skipping of result downloads when XLT is used as a load generator only.

Framework Extensions

A set of new commands is available in Script Developer and in the framework: commands such as assertAttribute, assertStyle, assertClass, assertEval, and their matching store and wait equivalents.

XLT can now send and receive IDs as part of the request to ease the correlation of server-side logs and test results. XLT may send a randomly generated alphanumeric ID as request header or extract such an ID from an arbitrary response header.

When dealing with different test environments, different load profiles, and/or different test data at the same time, managing different combinations of configuration settings can be challenging. The new property file include feature makes it easier now to predefine aspects and reuse them later in different configurations without copying and pasting.

When a test case reads a certain setting from the configuration, the framework uses a fallback strategy when doing the property look-up. This strategy performs an additional look-up step now, based on the transaction name (the short name to which the full class name is mapped). This additional step lets you parameterize different transactions differently, even if they are mapped to the same class and therefore share the same code.

The webdriver that will be used when executing functional tests can now be configured via properties. This allows greater flexibility and hardcoding is not necessary anymore.

When XLT executes XML script test cases with a WebDriver instance that is capable of taking screenshots, it can take a screenshot after each action, if desired.

Result Browser

New Result Browser UIA new look matches the overall styling of reports. Also, the navigation bar can now be resized, requests are color coded to visualize the content type, and the first page is displayed automatically. URLs are now active links, so you can click them easily.

If needed, the content of a response can be beautified for certain content types (HTML, JavaScript, JSON, CSS). This includes formatting and syntax highlighting.

Misc

  • XLT is shipped with an empty test suite project that can be used as a template for your own projects.
  • HtmlUnit has been upgraded to version 2.12.
  • WebDriver/Selenium has been upgraded to version 2.39.0.
  • EC2 admin console permits setting a tag name now.

Amazon Machine Images

The Amazon Machine Images (AMI) listed below are available for public use. Using these images is free of charge, but require that you own an Amazon Web Services (AWS) account. Please make sure that your EC2 security group permits communication on port tcp/8500. AMIs with Java 6 are no longer provided.

  • eu-west-1 : ami-ceae46b9
  • us-east-1 : ami-2510394c
  • us-west-1 : ami-e4ccfca1
  • us-west-2 : ami-9ebadeae
  • ap-southeast-2 : ami-617be45b

If you need XLT-AMIs in Tokyo, Singapore, or Sao Paulo, please let us know.

Purchase Licenses and Support Online

Licenses and support can be conveniently ordered through our XLT Self Service Center. You can instantly download licenses and purchase support right when it is needed. All your invoices and previous licenses (when purchased online) are accessible at any time.

We will notify you before your license or support runs out, so that you will never miss that again. This enables you to continue your daily automation and load testing work without interruption. Please note that we do not renew your contracts automatically, so no strings attached.

Visa and MasterCard are accepted. All credit card data is processed and secured by Wirecard.

Where to get it

More information about this release, the Quick Start Guide, and the Manual can be found in the release area. The full XLT 4.3.0 package can be downloaded here.

This upgrade is free of charge for everyone.

Xceptance LoadTest 4.2.10 released

We released Xceptance LoadTest 4.2.10. It addresses four defects and is a recommended update release.

  • Strange long delays when replaying test cases using Firefox 23
  • A sporadic I/O exception indicated static content download problems, but was caused by the internal connection management
  • JavaScript urls have been evaluated in the wrong context
  • Loading default frame content may have overwritten already created content

You can find more detailed information and the download link in the release notes for version 4.2.10.

Concurrent Users – The Art of Calculation

Most of you probably know the term Concurrent User. In the context of load and performance testing, this metric is often claimed the measure of all things, accompanied by the mentioning of astronomically high numbers we can’t really verify and that sometimes are simply used as sales argument for overpriced software products.

Today’s article is meant to shed some light on the concurrent user metric and the misunderstandings and myths surrounding it. Since Xceptance focuses on the internet and e-commerce, illustrations and examples will mainly refer to webshops; keep in mind, though, that the topic isn’t restricted to the domain of e-commerce load testing. Feel free to comment below, whether affirmative or critical.

Let’s start with a couple of key terms to help you understand what we’re talking about:

  • Visit: In general, a visit occurs when you send a request to a server and, as a response, the website you requested is displayed. Has a duration starting with the first page view and ends with the last. Consists of one or more page views.
  • Session: Technical term for a visit, basically the technical picture underlying it. Visit and session are often used synonymously.
  • Page view or page impression: A single complete page delivered due to a request of an URL; in a world of Ajax, intermediate logical pages can be considered an impression or view. Can lead to further technical requests (HTML, CSS, Javascript, images etc.)
  • Request: Submission of a request to a server, in the case of web applications mostly via HTTP/HTTPS protocols. Requested content may be HTML, CSS, Javascript as well as images, videos, Flash, or Silverlight applications – HTTP can deliver almost everything.
  • Think time: Time period between two page views of a visit.
  • Scenario: The course of a visit in terms of a use case (for example, to search something, to order something, or both). Representation of test cases meant to be run as load tests.
  • Concurrent User: We don’t exactly know about them yet…

Load and Performance Test

A load test wants to reflect present load conditions or anticipated load conditions. In either case, it’s impossible for a load test to cover all eventualities and be economical at the same time. There’s a myriad of ways you can go to explore a webshop. Thus, you decide on the most typical ones at first and make a scenario out of them afterwards. Most of the time, we consider a scenario an isolated visit repeating the steps of the test case and thus using defined data (note that also random data is defined data).

Let’s assume three scenarios: a visitor that is just looking (Browsing), a visitor that puts products into the cart (Add2Cart), and a visitor that checks out as a guest and wants their ordered items to be shipped to an address (Order).

The users have to go through the following steps to completely cover the scenario:

Browsing user

  1. Homepage
  2. Select a catalogue
  3. Select a subcatalogue
  4. Select a product

Add2Cart user

  1. Browsing 1.-4.
  2. Put a product into the cart

Order user

  1. Add2Cart 1.-2.
  2. Proceed to checkout
  3. Enter an address
  4. Select a payment method
  5. Select a delivery type
  6. Place the order

The first challenge is choosing the content for the single actions, that is should we always go for the same product, the same catalogue, should the number of items or the size of the cart vary, etc. Only these three scenarios offer infinite possibilities of variation already. But let’s stick with the basic steps and the simple Browsing for now.

Concurrent Users

When testing against a server, the single running of Browsing would be a visit consisting of 4 page views and possibly further requests for static content. A second execution of the test with all data and connections (cookies, HTTP-keep-alive, and browser cache) having been reset would result in another visit. If you now run these two visits simultaneously and independently from one another, you end up with two concurrent users. Note that the notion “user” is actually not the exact right term as we’re talking about concurrent visits here. We prefer the term visit in this context and the person performing it is the visitor.

Both of our visitors execute 4 page views each, thus resulting in a total of 8 page views. As each page view has a runtime on the server, let’s say 1 sec, one visit takes at least 4 sec. Therefore, if one user repeats their visits for one hour, he or she completes 3,600 seconds / (4 seconds per visit) = 900 visits / hour. Two concurrent visitors result in 1,800 visits in total leading to an overall total of 1,800 visits x (4 page views per visit) = 7,200 page views.

We just said “if one user repeats”. Of course, a single user would never repeat a visit that many times. Just look at the user here as the load test execution engine repeating that independently of other “users”.

Think times

Now, the majority of users isn’t that fast, of course, which is why usually think times get included. The average think time currently amounts to something between 10-20 seconds, depending on the web presence. It used to be 40 seconds but today’s users are more experienced and user guidance has improved a lot so that they can navigate through a website much faster. Let’s assume a think time of 15 sec for our example.

A visit would now take (4 page views each takes 1 sec) + (3 think times each 15 sec). It’s only 3 think times because there’s none after the last click that terminates the visit. Accordingly, our visit duration is 49 sec. If we now have a visitor repeat that for an hour, we’ll end up with a user completing 3,600 sec / (49 sec per visit) = 73.5 visits per hour.

If we want to test 1,800 visits again, we need 1,800 visits / (73.5 visits per hour per user) = 24.5 users, about 25. The number of page views stays the same since 1 visit equals 4 page views and the number of visits is constant. These 25 users need to complete their visits simultaneously and in parallel but still independent of one another.

Any Number of Concurrent Users is Possible

We now have 25 concurrent users that produce the exact same traffic simulation as 2 users without a think time. The exact same traffic? No, of course not – this is where extreme parallelism and the unpredictability of both testing and reality comes into play.

If the requirement was the simulation of 1,800 visits per hour and 7,200 page views per hour, we could now randomly pick a think time and by doing so, determine any number of concurrent visits aka users between 2 and x. With respect to our simulation period of 1 hour, we get a new session (begin of a visit) every two seconds on the server side – 3,600 sec / 1,800 visits as our visits are equally distributed.

You can also question the numbers by approaching the problem from another perspective: if 100 users are simultaneously active, then they can simultaneously request 100 page views. In the worst case (note that 1 page view takes 1 sec on the server side), however, this would amount to 100 * 3,600 sec = 36,000 page views per hour. Since the requirement of 100 concurrent users is actually never bound to a certain period, you therefore have to assume that these users could potentially click at any time.

From this point of view, you’ll soon realize that the number of concurrent users can basically mean anything: much traffic, little traffic, little load, much load. Only by knowing the test cases and additional numbers such as visits and page views per time unit can you a) define a number of concurrent users and b) check each number by means of calculation against the other numbers.

Oh, and needless to say that 42 is always a good number of concurrent users… ;-)

Why 100,000 Concurrent Users Aren’t 100,000 Visits

What we want to emphasize here is that a temporal dimension is absolutely necessary. The requirement of 300,000 users would always imply they could click simultaneously which would produce 300,000 visits at one blow. Now, you may want to argue that they aren’t coming simultaneously. However, if the users aren’t simultaneously active aka started a visit, they aren’t concurrent users anymore and then you don’t need to simulate them in the first place.

Provided an equal distribution and an average visit duration of 49 sec, 300,000 users per hour that are often identified with visits (business-wise) in most cases, would result in the following: a user completes 3,600 / 49 sec visit duration = 73.5 visits per hour so that you end up with 300,000 / 73.5 = 4,081 concurrent visits aka real concurrent users at any given second. 4,081 concurrent visits produce 4 page views in 49 sec (visit duration) each, that is in 49 sec we have 16,324 page views, thus 333 page views per sec (see next paragraph).

In terms of page views without think times this means: 300,000 users are 1,200,000 page views (for our example above). Thus, you need to complete 1,200,000 page views / 3,600 seconds = 333 page views per second. Without any think time you would therefore need 333 users for the simulation.

Regarding the final result, the simulation of 4,081 users and 15 sec think time therefore equals the simulation of 333 users without think time. On the server side, both will result in the identical number of visits per time period, the identical number of page views, etc.

That’s impossible…

You may raise some objections to this and they are actually valid since, in reality, the think time would never be exactly 15 sec and the response time would never always be 1 sec. This is where coincidence comes into play. With respect to our example, let’s assume the think time to vary between 10 and 20 sec. The arithmetic mean would still be 15 sec.

What happens now results in the following calculation: In the worst case, the duration of all visits is only 4 sec + 3 * 10 sec = 34 sec. With 34 sec, our server now has to deliver as many visits and page views as it delivered within 49 sec before. Thus, our test wouldn’t cover 300,000 users with 4,081 concurrent test users but 3,600 / 34 * 4,081 = 432.105 visits per hour.

That means you need to define target numbers you want to support, or measure what the server is currently able to deliver. As soon as you say you have a number of x visits that could vary in their duration, you end up with a higher maximum number of visits you need to support but that you actually don’t want to test.

Note that our sole focus is set on the load and performance test here. You want to know if you can cope with the traffic x where you assume x to be a constant worst case that applies to a longer period of time. If you want to measure the server side beyond the maximum “good” case, you don’t aim at the performance anymore but at the overload behavior. Then you focus on stability and a predictable way of “decline”. All tests that are normally run at first, which is absolutely correct, are tests that want to identify or verify the good case.

But still…

Of course, it can make sense to test 4,081 users instead of 333 although there’s the same number of visits and page views per time period on the server side. 4,081 users can be concurrent users for a very short time and claim, for example, 4,081 webserver threads or sockets, while 333 users will never reach this number. Even if you keep the think time for 4,081 users at a constant level, the traffic wouldn’t be as synchronous as you planned it to be in the beginning.

In the worst case, you can’t test at all now because each test run leads to a different result. With the restriction to 333 users with none or just minimal think time, you restrict the “movement” of the system at first to measure it. If the system delivers what it should, the test may expand in its width aka both the think times and the number of concurrent users go up.

Steady Load Vs. Constant Arrival Rate

To resolve this dilemma a) without having to consider the server side while b) still being able to measure accurately, you can choose between two typical load profiles:

  • Steady Load: Runs a fixed number of users that wait for the server, for instance, when it has long response times. This way you can’t reach the desired number of visits because users depend on the server’s response behavior. The profile is suitable for controlled measurements.
  • Constant Arrival Rate: Users arrive as new visitors regardless of what is happening on the server side. When the server is too slow, new users will still try to come in. If the server can handle the load, the system runs stable and you just need your user number x (according to our calculation, 4,081, for example). If there are problems on the server side, then the user number automatically increases to x + n (for example, to a total of 10,000 users). In the ideal case, that means you only need 4,081 users but when the server behaves unexpectedly, up to 10,000 users will be activated. This way you can also test the overload behavior at the same time.

Finally…

We hope you were able to follow and that the mess of numbers didn’t get too bad. At times, the concurrent user topic is getting downright absurd… Feel free to comment, any remark is appreciated.

New AWS-AMIs in Singapore

We just made XLT 4.2.9 AMIs available at AWS-EC2 ap-southeast-1/Singapore. Here is the list of all XLT-AMIs available. Happy testing!

  • Java 7
    • eu-west-1 : ami-71e1fc05
    • us-east-1 : ami-b03845d9
    • us-west-1 : ami-00a38a45
    • us-west-2 : ami-17fc6f27
    • ap-southeast-2 : ami-4514877f
    • ap-southeast-1 : ami-caa1e898
  • Java 6
    • eu-west-1 : ami-73e1fc07
    • us-east-1 : ami-be3845d7
    • us-west-1 : ami-7aa38a3f
    • us-west-2 : ami-3ffc6f0f
    • ap-southeast-2 : ami-4714877d
    • ap-southeast-1 : ami-c8a1e89a

Web Drivers in XLT: Basic Access Authentication

Today’s article of our WebDrivers series deals with HTTP authentication – a topic that, at first sight, seems to be very specific and of minor relevance. However, in the world of software testing it’s way more important than you’d think.

HTTP Authentification IE
HTTP Authentication IE
Often you will have an additional testing instance of a website to be tested. These instances are protected from abuse which is why they require credentials before you can access them. See below for an example in Internet Explorer:

This browser dialog appears just once. If you’ve entered the right credentials, you can access the related website as often as you like without further authentication – as long as you don’t reopen the browser. The latter is a critical issue for automated WebDriver testing.
Continue reading Web Drivers in XLT: Basic Access Authentication

Release Xceptance LoadTest 4.2.8

XLT_transparentWe just released a minor XLT update that brings you an improvement and a fix.

  • Script Developer: Mouse events dispatched by SD use target element coordinates now.
  • Load Test Environment: The result target directory can now be specified on the mastercontroller command line.

As always, this release update is free of charge and you can download it here: XLT Release 4.2.8. Do not forget to sign up to our release email list. To purchase licenses and support, please visit the XLT Portal.

XLT is free for up to five virtual users for load and performance testing. Regression testing usage is always free of charge.

WebDrivers in XLT: Test Case Design that Compensates for Inconsistencies across WebDrivers

Still remember the first post of our article series? It talked about how you can run XLT test cases in different browsers. If you’ve done so already, you might have noticed that the behavior of test cases developed in Script Developer sometimes differs depending on the WebDriver you’ve chosen. This article is meant to help you resolve such issues.

First of all, what’s the reason for these inconsistencies? Web browsers differ in their characteristics, such as site representation or functionality, due to their varying support of web technologies like CSS or HTML. You probably know that there is much more we could list, but the major point is pretty obvious already: using WebDrivers for test case execution calls real browser instances. Logically, you’re faced with the same differences as in real web browsers. It’s impossible to achieve a completely consistent web browser behavior; yet you can design your test case as outlined below to at least reduce the differences.

Time: Enlarging time properties can prevent timing problems with dynamically loaded content, like iframes or page content added with JavaScript. Script Developer lets you set an ‘Implicit Wait Timeout’. It allows to specify how long the test should wait for a command to fail while trying to find elements.

Continue reading WebDrivers in XLT: Test Case Design that Compensates for Inconsistencies across WebDrivers

Release Xceptance LoadTest 4.2.7

XLT LogoXLT 4.2.7 has been released. It delivers improvements as well as defect fixes. We recommend to upgrade as soon as possible to stay up to date.

Script Developer

  • Fix: Obsolete entries from the base url drop down can be removed now.
  • Improvement: With special characters in the link name, the Script Developer sometimes recorded an XPath locator instead of the more appropriate link locator.

Load Test Environment

  • Fix: Custom samplers stopped working for the rest of the load test when an exception was thrown. Now the sampler continues to run.
  • Fix: The Mastercontroller used a previously entered comment when the current comment was left empty.
  • Improvement: The Mastercontroller returns non-zero exit code for all error conditions now.

Framework

  • Fix: HtmlUnit did not sent the basic HTTP authentication header for follow-up requests correctly, only when challenged again.

Of course, we are providing a set of fresh Amazon Machines Images as well. You can find all information on our product page. Make sure you stay informed by subscribing to our XLT release email list.

WebDrivers in XLT: How to Run Test Cases in Multiple Browsers

Update: Before you continue reading, here is a new version of this test suite and article: Multi-Browser-Suite for Test Automation. It makes things a lot easier.

Have you heard of XLT Script Developer yet? If you have, you’ll probably agree that it’s a convenient tool to record and run automated test cases. However, with it being a Firefox plugin, you’re basically bound to run your test cases in Firefox. Wouldn’t it be nice to reuse Script Developer test cases in multiple browsers? You can actually do so by taking advantage of the WebDriver API that is part of the XLT framework.

If you don’t know much about WebDrivers, you should continue reading this article – the first one of a little article series on WebDrivers that hopefully gives you a good introduction to the topic.
Continue reading WebDrivers in XLT: How to Run Test Cases in Multiple Browsers

The Art of Reading Performance Test Charts

Powerful load and performance test tools don’t only retrieve pages of your website randomly with zillions of users at the same time, but they also cover realistic scenarios simulating the real-world user. It’s a given that they can deliver lots of useful information and plot interesting charts. To fully take advantage of these benefits, however, you need to be able to interpret this information and draw the right conclusions.

It is this need for the correct interpretation of test results, the mapping of all you see against actual application behavior that makes performance and load testing a non-trivial task. It requires much experience to decide on the right actions, make the right assumptions, or simply come up with a reasonable explanation of why something happened this way and not the other.

In today’s article, we’d like to present you a couple of charts displaying typical response time patterns, and discuss what they could indicate.

Disclaimer: Of course, the reasons for a certain behavior vary a lot, depending on your application and testing. However, as there’s no fixed manual for the interpretation of load testing charts, we want to provide you with a couple of basic guidelines to help you get better in interpreting them yourself and make the most out of your test results. Feel free to comment whether or not you agree with the ideas and explanations we come up with.

The Warm-up

These charts might indicate a system with a cold cache, for instance, when the system has just been started and the caches aren’t filled yet.

The basic characteristics of such a behavior are high response times in the beginning, followed by gradually lower response times until eventually a  certain degree of runtime stability is reached. This time frame is often referred to as the system’s warm-up period. Throughout this period, a couple of things can happen under the surface. If you know the system under test well, you’ll probably come up with the following: database and file system caches are filled, proxies learn about the data and store them, the system under test scales up because it sees traffic, page snippets are cached and so the computing overhead reduces… you name it.

Also keep in mind that it might be the testing process itself that causes such a response time profile. If the system is perfectly warmed up and you hit it, your sent traffic might be too uniform in the beginning. That being the case, randomization kicks in so that the traffic eventually distributes better over time. Furthermore, take into consideration that your load software and hardware are possibly not warmed up either.

The Caching

These charts depict either a typical cache clean or job patterns. In case of a cache clean, system-internal caches expire every half hour. If that’s not the case, the charts may indicate  a running background job draining power from the database or consuming lots of system bandwidth.

Both charts display the same test; however, this test has been executed for different time periods. While two spikes could signify a random event (despite the fact that the temporal distance of 30 minutes is suspicious), the longer test run seems to prove our first assumption: something is going on every half hour.

In any case, make sure that such a behavior is not produced by the test machines themselves, for example, because they’re busy writing or backing up data.

The Spiking

This is what we call a forest of spikes: many spikes that don’t seem to follow a comprehensible pattern; longer runtimes just occur occasionally, often caused by requests accessing certain data or URLs that produce long runtimes. To get behind that mystery, you have to dig into the results in more detail, find the calls behind the spikes, and derive a pattern based on the information you find. Often you’ll come across similar URLs, request parameters, or maybe response codes. Don’t forget any application logs you might have access to, such as web server, error, information, or debug logs. In a perfect world, your application under test offers the necessary tools to get to the bottom of this problem.

XLT lets you easily find this information. All test result data are accessible as CSV files that are quickly readable and documented. Feel free to work with this information and go beyond the scope of the reports available.

The worst outcome here is a non-identifiable pattern and no information on the server side as to what might have happened. In such a case, you have to repeat the test and narrow down your test setup later on to exclude as many variables as possible to find the cause. This is actually also a good time to ask for development or tech-ops support.

The 3rd-party Calls

The first chart is typical for issues with 3rd parties, especially in the field of e-commerce. We’re not talking about direct calls to 3rd parties here, such as analytics vendors or recommendation engines, but calls from one server to the other. Thus, the response time we see is the response time of two systems. Of course, it’s good to know the area where 3rd party calls typically happen, but you have to know the application under test anyway to test it efficiently. So when the final order steps start to act weird, you can easily narrow down the potential reasons.

The second chart looks more like the cache clean or expiration problem described above, but since you know the application, you also know that this area doesn’t use the typical caching logic but is highly dynamic instead. This means that the errors occurring every 50 minutes point into a different direction: as we know that 3rd parties are attached and called during shipping, we can conclude that the 3rd party failed on us.

Verdict

Knowing typical response time patterns helps you specify a certain problem so that you can give hints to the development or further shape the path of testing. If you can read charts and derive the right conclusions or at least know which questions you have to address, you’ll be ahead of the crowd. Be aware that knowledge on the system under test is very important – the production and measurement of a certain load doesn’t make much sense when you’re not able to actually interpret and explain what you’ve measured. Always remember: 42 is not a valid answer for everything. :)