Load and performance testing is not trivial. Load and performance testing microservices is even more challenging. Our article “Performance Tests of Microservices” in the iJUG-Magazin Java aktuell talks about approach and challenges when designing and executing performance test for microservices. So get yourself the Java Aktuell or, if you just want to check out the article, use this link to the PDF and enjoy. The article is in German.
Performancetests von Microservices
Eine Softwarearchitektur, die aus kleinen und unabhängigen Teilen zusammengesetzt ist, ist einfacher zu erstellen und zu warten. Microservices sind dadurch zum dominierenden Thema in der modernen Softwareentwicklung geworden. Während die Komplexität auf der Entwicklungsseite sinkt, steigt sie in Bezug auf die Themen Architektur, Performance und Zuverlässigkeit. Dieser Artikel diskutiert die Anforderungen an die Performance und die Planung von Performancetests von Microservices.
After our presentation at the JUG Saxony Day in 2019, we have been asked to turn the talk into an article for the iJUG-Magazin Java aktuell. The February 2020 issue just got published and features this article. So get yourself the latest Java Aktuell or, if you just want to check out the article, use this link to the PDF and enjoy. The article is in German.
High Performance Java – Hinter den Kulissen von Java
Die Zeiten, in denen Java der langsame Bytecode-Interpreter war, sind lange vorbei. Die JVM nutzt viele Tricks, um Code effizient auszuführen, und transformiert und optimiert Java-Code auf die ausführende Hardware. Mit etwas Wissen über diese Prozesse kann man vermeiden, dass man gegen die JVM arbeitet, und gleichzeitig mehr Geschwindigkeit erreichen. Selbst wenn man nicht die letzte Mikrosekunde jagt, ist es interessant zu sehen, welche Techniken die JVM einsetzt, um die Ausführungsumgebung besser zu verstehen.
Because we encountered another hidden encoding issue as part of test data, here are some information about BOM and why this might be interesting in general for everyone working with a computer beyond Excel and Word.
Before you educate yourself, here is the tool to own to see such a problem easily. Most of all editors hide that information and so you might scratch your head why some data is failing with strange error messages. Get xxd and you will see with other eyes:
Wolfgang Tiefensee, Thuringia’s Secretary of Commerce, in conjunction with the board of directors of the IT industry network ITNet Thuringia, awarded the first Thueringen Open-Source Prize to three companies, all of them software companies based in Jena: TRITUM, Xceptance and GraphDefined.
It is an honor for Xceptance to be the second-place winner of this competition. This result clearly demonstrates that open source as a component of commercial products can be a clear competitive advantage. XLT incorporates a number of open-source projects, including Apache HttpClient, Jetty, HtmlUnit, JUnit, and the Apache Commons libraries. As part of developing XLT, Xceptance is involved in testing and providing feedback for these projects, thus giving back to the open-source community.
TL;DR: The perceived loading time is what shapes the user’s impression of the speed of a website, but measuring perceived loading time is difficult. There are technical loading times available but it is not clear if these times can be used in any meaningful way. This master’s thesis verified whether the perceived loading time can be easily correlated with any technical loading time, such as FirstPaint for instance. The result shows that a page with a fast loading time from a technical point of view is not necessarily perceived as fast by the user and vice versa. Therefore it is not sufficient to rely only on technical measurements and disregard the user’s perception.
Client-side performance is a big deal. There are various studies on the relationship between loading time and critical success factors such as usability of online shops, customer loyalty and sales. Yet a page with an objectively long total loading time could still be perceived as fast by the user, as the visible part of the browser window has already loaded and invites the shopper to interact with the page.
In her master’s thesis “Client-side performance: Comparison of measured and perceived loading times of online platforms in the B2C sector” Bastienne Sauter scientifically evaluated this discrepancy between the technically measured and the perceived loading times. By implementing automated tests with the automation tool XLT (Xceptance LoadTest), the loading times (timestamps recorded: domLoading, firstPaint, domInteractive, domContentLoadedEventStart, domContentLoadedEventEnd, domComplete, loadEventStart and loadEventEnd) of fifteen German e-commerce stores were measured over a longer period of time, both for desktop and mobile. For each store and timestamp, a ranking of the average loading times was calculated for each device.
In order to determine the empirical values for the perceived loading time, an interaction study with test groups of students and of Xceptance employees was conducted. Each group consisted of 25 persons, so a total of 50 people took part in the study. The participants had to open predefined pages of an online store and assess the loading impression. The user’s perception was evaluated using suitable scales. In this manner a score value was determined for each store among the two test groups. Each participant evaluated only three of the 15 stores in order to keep the effort manageable. In other words a store was covered by five test persons of each group (according to Nielsen and Landauer, a total of five respondents per store is sufficient for usability tests to be meaningful and to cover the majority of usability results). To ensure comparability of the results, the interaction study was conducted on the same devices and at the same location to avoid different bandwidths of the internet connection. Furthermore the test session for every person was set up in a way to avoid stress and create a normal daily usage situation.
Comparing the measured technical loading times with the perceived ones from the study, it showed that they differ clearly from each other, with no significant correlation.
To get an impression of which timestamp could be most relevant for the perceived loading time, the distances of the positions in the rankings were calculated. The smallest deviations for the desktop showed the timestamp domContentLoadedEventEnd, while the timestamp firstPaint could be most relevant for mobile devices.
As a conclusion, it is neither sufficient nor purposeful to consider only the technical measurements for the evaluation of the client-side performance, because the user’s perception deviates significantly from the technical measurement. This proved to be true both for informed insiders (represented by company employees) and also the uninformed public (represented by the students).
The results raise further research questions, especially how quality key figures for the perceived loading time can be recorded and contribute to the evaluation of online stores. In the literature, user-centered key figures such as FirstMeaningfulPaint are cited for this purpose. Whether these figures are in fact useful to represent the perceived loading time, however, is unclear and requires further investigation.
Xceptance released version 4.12.2 of its load testing and test automation product Xceptance LoadTest. This is an improvement release. We recommend upgrading to this newest version.
Improvement: The result browser features a new tab that displays JSON responses in a tree-like view. The data can also be searched and filtered.
Improvement: The existing automatic request retry mechanism has been enhanced to retry failed requests in additional error situations such as connection resets. This should now behave similar to real browsers.
Improvement: When failed requests are retried, an event with diagnostic information is logged for each retry.
Improvement: The bundled Jetty library has been updated to the latest available version 9.4.14.
Improvement: Our public AWS machine images now come with OpenJDK 11.
Xceptance has released version 4.12.1 of its load testing and test automation product Xceptance LoadTest.
Fix: Our timer recorder extensions for Chrome and Firefox did sometimes report invalid request entries that could not be processed by the report generator. This could happen for requests that did not complete.
Fix: If a test case deliberately caught an exception / assertion error and afterwards ran to completion successfully, it might nevertheless be marked as failed in the load test report.
Improvement: Selenium has been updated to the latest version 3.141.59 and HtmlUnitDriver to version 2.33.3.
Improvement: The new AWS data center in Stockholm, Sweden (eu-north-1) is fully supported by ec2_admin now.
Xceptance released version 4.11 of its load testing and test automation product Xceptance LoadTest. This is a feature release. We recommend upgrading to this newest version.
Here is a selection of the most important changes:
HtmlUnit and Selenium have been updated
You can run a test case with only a single data set at development time
XLT picks a data set automatically when a test with multiple data sets is part of a load test
There is a new command-line tool to automatically evaluate a test report based on your defined success criteria
The XLT Jenkins plugin returns a result object now with all the details when used in a pipeline
The XLT Jenkins plugin can now create a comparison report
See our release notes for more details. As always, this upgrade is free and don’t forget, XLT itself is free as well. You don’t have an excuse to skip performance testing or rely on lame simple test cases anymore.
When using XLT for test automation or load testing, a visual and browsable log of your test case steps and their results will be created during test case execution. We call this log the XLT Result Browser and the following how-to will explain how to read it. Continue reading How to use the XLT Result Browser→
By clicking Opt-Out, we will place a non-personalized cookie on your machine that indicates that you don‘t wish to be tracked.