All posts by Rene

Neodymium 5.1 – Get more done faster

TL;DR: Neodymium 5.1 has been released with a host of impressive new features. These include full-page screenshots in reports, enhanced control over JSON assertions, accessibility testing via Google Lighthouse, and simplified session handling and URL validation.

The newest enhancements primarily focus on these key areas to boost testing efficiency and user experience.

Enhanced Reporting: We’ve significantly improved the quality and usability of test reports. You now get more informative and insightful results with features like full-page screenshots, enhanced JSON assertions for easier data comparison, and a streamlined report structure for improved readability.

Accessibility Testing: Recognizing the growing importance of web accessibility, we’ve integrated Google Lighthouse to automate accessibility checks. This allows you to easily identify and address accessibility issues within your web applications, ensuring compliance with WCAG standards.

Streamlined Workflows: We’ve introduced several features to streamline your testing processes. These include improved session handling for cleaner test environments, enhanced configuration options for greater flexibility, and robust URL validation to prevent unintended access to sensitive systems and ensure test stability.

These improvements are all about making your testing smoother and more dependable. You’ll get better results and learn more from them, so you can build even better software! So, let’s get into the details, shall we?

Enhanced Reporting

Neodymium 5.1 brings significant improvements to its reporting capabilities, providing you with more insightful and informative test results.

Improved Screenshots

You can now capture full-page screenshots with highlighted viewports for better visibility of the test context. Additionally, the last element, the test has interacted with, will be highlighted. This makes it easier to pinpoint the exact location of an interaction and understand the state of the application at that time.

Shows a screenshot of a report with an embedded screenshot showing the marking of the viewpport
Screenshot within Viewport Marking

Test Data Overview

Gain a deeper understanding of your test data and streamline the debugging process by accessing test data in JSON format for each test case. This detailed data analysis capability allows you to quickly identify and resolve potential issues.

Screenshot showing how test data is shown in an Allure report
Test Data within the Allure Report

JSON Assertions

The JSON file comparison feature has been enhanced. It now provides improved accuracy in identifying differences between expected and actual results.

A screenshot with test results on the left and a JSON comparison example on the right
JSON Comparison Within the Allure Report

Environment Information

We’ve added the browser used for testing to the report’s environment section to give you more context about the test results. Also, you can now add custom values to tell the difference between environments or profiles in your test runs, making it easier to figure out if any setup or deployment problems are causing issues.

Step Information and URLs

The clickable URLs included in each step of the report simplify and streamline the process to reproduce test failures and navigate to specific application pages at various points during test execution.

Reduced Report Clutter

The report now features fewer INFO sections, providing a cleaner and more concise overview with less visual noise, allowing for easier identification of critical information.

Accessibility Testing

Neodymium 5.1 now includes Google Lighthouse to help you identify and fix accessibility issues in your web applications. This integration supports your compliance with WCAG standards, and shows that we understand how important web accessibility is.

Shows a screenshot of a report with an embedded Google Lighthouse report
Lighthouse Report Embedded in Neodymium’s Allure Report

Additionally, you can assert against Lighthouse report data to track changes in state or outcome.

Furthermore, this enables WebVitals measurement and verification capabilities.

Workflow Enhancements

Several features help to streamline the testing process and improve overall test automation efficiency.

Popup Blocker

The improved popup blocker boosts test stability by minimizing disruptions from unexpected popups, which can lead to inconsistent test results.

Session Handling

Neodymium now utilizes a dedicated annotation to run @BeforeEach and @AfterEach methods in individual, fresh browser sessions. This guarantees a consistent environment for setup and tear down, and minimizes any influence from prior test steps.

Properties

Environment variables can now be used to override Neodymium properties, providing greater flexibility and simpler integration with CI/CD pipelines. This allows for more dynamic test adjustments based on the specific environment or build.

URL Validation

Restrict test access to certain sites only by defining include and exclude lists. This prevents accidental access to live sites or sensitive data, mitigating the risk of unintended modifications. Any attempt to access a URL not included in the defined list will result in an immediate test failure, providing early feedback and preventing further interactions.

Driver Command-Line Arguments

The configuration now allows for direct pass-through of driver command-line arguments, which streamlines driver setup, facilitates customization, and enables fine-tuning of driver behavior and test execution to meet your specific requirements.

Dependencies

We also updated numerous dependencies to keep them up-to-date and secure. Examples are Selenide, Allure, Log4J, and JUnit.

Summary

Neodymium 5.1 is now available! We value your feedback on new features and any defects encountered. Your input helps ensure Neodymium remains a powerful tool for your needs. You might even find a newer version of Neodymium when you read this article, depending on when you read it.

Performance Test Rating Criteria

TL;DR: Load and performance testing produces a vast amount of data. This data has to be interpreted and communicated. Because not every interested party speaks the same language, Xceptance developed a performance test rating and grading system. It evaluates response time, stability, and predictability and transforms three factors into a simple and communicable form. While doing that, it does not compromise on quality. It has been successfully used in more than 400 projects.

The Challenge

Load and performance testing is a key activity for making an online business successful. It validates that traffic and conversion expectations can be fulfilled. This of course applies to all kinds of Internet-based applications. Basically, as soon as there are expectations in terms of stability and performance, a test is mandatory to validate these. Expectations are usually set as requirements by different organizational groups such as sales, product management, engineering, and development teams.

Every group has a different understanding when it comes to results, goals, and success criteria. Some might be more concerned with the business impact, others are looking for technical implications of design decisions, and some just want to improve performance.

The group that is tasked with the evaluation of the requirements is faced with a very wide range of success definitions. In addition, it has to explain its technical measurements to all participating parties so that each party easily understands the state of testing.

Engineers rather look for detailed metrics including but not limited to the system behavior under test, while business-centric stakeholders just expect a clear yes or no. But performance testing typically does not deliver a clear result.

How can one reach all target groups without causing too much extra work to cater to all individual needs?

The Rating System

Xceptance developed a rating system that uses an American education system-like grading from A to F. The grades A to C symbolize a pass, while D and F are considered a fail. A grade B stands for an assumed average across similar customers and projects. It also stands for a good result. This leaves room in both directions to over or underperform.

Because performance results are not just shaped by response times, three factors are taken into account:

  • Response Times
  • Errors
  • Predictability

Before continuing a detailed discussion of the factors, this table explains each grade in one sentence. For the more ambitious customers, the A+ stretch goal was added.

Let’s talk about our three factors in detail now.

Continue reading Performance Test Rating Criteria

Data Science Day 2022

After a two-year break, on 18 May 2022, the 4th Data Science Day Jena took place. The Friedrich-Schiller University is hosting this mini-conference format annually.

Xceptance presented this year a closer look at the data that is collected and processed during load and performance testing. René Schwietzke, Managing Director of Xceptance, talked about the challenges to capture the right data as well as translate the collected data into meaningful results. 

A load test simulates millions of user interactions with a website and therefore is capturing huge amounts of data points. These have to be transformed into a few numbers to make the result of the test easy to communicate but still preserve important details. The talk started with typical business requirements and expectations of the target groups of a load test. It showed the data XLT captures and the dimensions which later drive the data reduction. A few example data series demonstrated the challenges behind the data reduction and what numbers are finally used to satisfy the requirements.

An example load test result illustrated the talk with real data. That example test run created about 17,500 data rows per second which contain about 293,000 data points. The entire test result consists of 3.2 billion data points. This massive data set is turned into a consumable report by XLT in less than six minutes. 

For everyone with an interest in data science, this presentation also offers ideas for research in regard to unsolved data challenges. There might be even some Master’s and Bachelor’s theses topics waiting for you. 

You can find a recording of the presentation below (courtesy of Thüringer Universitäts- und Landesbibliothek Jena).

This is the accompanying slide deck. It is a Reveal.js-based. You can navigate with the spacebar and the arrow keys.

Picture of the presentation


iJUG-Magazin Java aktuell – Performance Tests of Microservices

Load and performance testing is not trivial. Load and performance testing microservices is even more challenging. Our article “Performance Tests of Microservices” in the iJUG-Magazin Java aktuell talks about approach and challenges when designing and executing performance test for microservices. So get yourself the Java Aktuell or, if you just want to check out the article, use this link to the PDF and enjoy. The article is in German.

A software architecture that consists of small independent components, is easier to create and to maintain. As a result of that, microservices have turned into a dominating topic in terms of modern software development. While the complexity on the development site is decreasing, topics like architecture, performance, such as the planning of performance tests and reliability issues of microservices are getting more attention. This article discusses the performance requirements and performance test planning of microservices.

Comments are very welcome.

2020 – One Year, One Picture

One Picture Says It All

If we can only use one picture to symbolize 2020, this might just be it. Working from home and video conferences are now the new normal.

Happy Holidays, Merry Christmas, and a Happy and Healthy 2021, because we all need that.

iJUG-Magazin Java aktuell – High Performance Java

Java Aktuell Cover Issue 2/2020
Java aktuell 2/2020

After our presentation at the JUG Saxony Day in 2019, we have been asked to turn the talk into an article for the iJUG-Magazin Java aktuell. The February 2020 issue just got published and features this article. So get yourself the latest Java Aktuell or, if you just want to check out the article, use this link to the PDF and enjoy. The article is in German.

The days where Java was considered a slow bytecode interpreter have ended a long time ago. JVM is using many tricks in order to execute a code efficiently and to transform and optimize a Java code on the used hardware. With some knowledge about these processes, you can avoid working against the JVM system, while achieving more speed. Even if you are not chasing every last microsecond, it is still interesting to see what methods and techniques JVM uses for a better understanding of the runtime environment.

Comments are very welcome.

Xceptance LoadTest Goes Open Source

We have often been asked why XLT (Xceptance LoadTest) is free but not open source. Now, after considering our customers’ business needs and the advantages of using open source as opposed to free software, Xceptance has decided to open source XLT under the Apache License 2.0.

Within the next 30 days, we will restructure the XLT code base and move subprojects into independent open repositories. We will continue to expand and improve the documentation. All code will be published on GitHub.

Xceptance will continue to maintain and extend XLT as before. Releases will be built and published by Xceptance on a regular schedule. Libraries will be published to Maven Central as part of the open-sourcing process. The latest releases will be available from the XLT GitHub repository as well. Of course, the GitHub defect tracker will start to be your source for feature development and defect fixes.

If you are interested in training, projects, or support for XLT and load testing in general, we of course also offer those services, and will expand this offering further in upcoming months.

By open sourcing, we are opening a new chapter for XLT and we invite everyone to join us. More to come!

BOM – Byte Order Mark

Because we encountered another hidden encoding issue as part of test data, here are some information about BOM and why this might be interesting in general for everyone working with a computer beyond Excel and Word.

Before you educate yourself, here is the tool to own to see such a problem easily. Most of all editors hide that information and so you might scratch your head why some data is failing with strange error messages. Get xxd and you will see with other eyes:

The first marked bytes are the magic and now head over to the Wikipedia to read more about BOM: https://en.wikipedia.org/wiki/Byte_order_mark

Java Training Sessions

Today we are going to publish four of our Java training sessions so you can use the material and benefit from it.

Let’s get started with four direct links to extensive material that might help you to understand Java or code quality better or just help you to reflect on topics you already know.

  • The Java Memory Model: Why you have to know the JMM to understand Java and write stable, correct, and fast code.
  • Java Memory Management: Know more about the size of objects and how Java does garbage collection.
  • High Performance Java: All about the smart Java internals that turn your code into fast code and how you can leverage that knowledge.
  • High Quality Code: The anatomy of high quality code that supports longevity, cross-team usage, and correctness. This is not just about Java, this is about good code in general.

Show a little patience when loading the training, these are all large reveal.js based slide sets. Use the arrow keys or space to navigate. Because the slide sets are designed to be interactive sessions, in many cases, not the entire slide context is revealed at once but block by block.

We publish these training sessions because they are also based on openly shared material, it greatly helped us to advance and understand, as well of course advertise a little what Xceptance might be able to do for you.

We will release more of our material in the next weeks and month, so everybody can browse and learn. This won’t be limited to Java and also cover material about approaching load testing, how to come up with test cases, and more about the modern web and its quality and performance challenges. Of course there will be more Java material too. You can get a glimpse of it when you just follow this link and page through the slides: The Infinite Java Training. Please remember, not all material is complete yet.

If you like the material and you need an audio track aka a real presentation, please talk to us. If you see other training needs in the area of quality assurance, testing, and Java, please contact to us.

More to come.

Thuringia’s Open-Source Prize for XLT

Wolfgang Tiefensee, Thuringia’s Secretary of Commerce, in conjunction with the board of directors of the IT industry network ITNet Thuringia, awarded the first Thueringen Open-Source Prize to three companies, all of them software companies based in Jena: TRITUM, Xceptance and GraphDefined.

Open Source Prize Title Picture Second Place

It is an honor for Xceptance to be the second-place winner of this competition. This result clearly demonstrates that open source as a component of commercial products can be a clear competitive advantage. XLT incorporates a number of open-source projects, including Apache HttpClient, Jetty, HtmlUnit, JUnit, and the Apache Commons libraries. As part of developing XLT, Xceptance is involved in testing and providing feedback for these projects, thus giving back to the open-source community.

While XLT is itself not open source, Xceptance does provide the software free of charge and with virtually no usage restrictions, so for most applications there is no noticeable difference to open-source software.