Category Archives: Open Source

Open Source, Performance, XLT

Unlock Enhanced Performance Testing: JMeter and XLT

TL;DR: XLT now supports JMeter test suites, offering enhanced debugging, deeper reporting, and cost-effectiveness for JMeter users. It’s open-source and allows you to leverage XLT’s execution and evaluation features with your existing JMX files without vendor lock-in. Simply integrate your .jmx files into the XLT test suite, configure, and run for detailed performance insights.

Are you a JMeter user looking for more robust debugging, deeper insights, and a more streamlined performance testing workflow without abandoning your existing test assets? Do you find other commercial tools like BlazeMeter or OctoPerf to be beyond your budget? If so, we have exciting news! XLT, our powerful load and performance testing tool, now offers dedicated support for JMeter test suites, giving you the best of both worlds.

This integration isn’t about replacing JMeter; it’s about empowering your existing JMX files with XLT’s execution, evaluation, and reporting features. It allows you to protect your investment in JMeter scripts while providing a clear path to leverage XLT’s capabilities, whether you choose to gradually migrate or simply enhance your current JMeter-centric process.

And the best part? Both XLT and this JMeter integration are open source, licensed under the Apache License 2.0. We actively invite you to participate, contribute, and help us make performance testing even better for everyone.

Why Combine JMeter with XLT?

The core idea is simple: continue using JMeter for what it does best – recording, editing, and maintaining your tests. However, when it comes to execution and evaluation, XLT takes over. This seamless handover provides several significant advantages:

  • Enhanced Debugging: Gain crystal-clear insights into executed actions, requests, and responses, making troubleshooting much faster and more efficient.
  • Deeper Reporting: Leverage XLT’s comprehensive reporting capabilities for a more granular understanding of your performance tests.
  • Cost-Effectiveness & Open Source Freedom: Avoid the high costs associated with some commercial performance testing solutions while gaining enterprise-grade features. Being open source means transparency, flexibility, and no vendor lock-in.
  • Protect Your Investment: Your existing .jmx files remain valuable and fully usable within the XLT ecosystem.
  • Flexible Migration Path: You can continue developing in JMeter and execute with XLT, or gradually transition more logic into XLT over time as your needs evolve.

Getting Started

To get started, you’ll need a few prerequisites and then follow a straightforward process.

Requirements

  • Java Version 21
  • An existing .jmx file for test execution (created with JMeter).

Tip: You can find several example JMX files in <testsuite>/config/data/jmeter.

How To Use

  1. Clone the test suite: Get the dedicated XLT JMeter support test suite from its repository.
  2. Prepare your test plan: Build a new JMeter test plan or use an existing .jmx file.
  3. Save your .jmx file: Place your .jmx file into the <testsuite>/config/data/jmeter directory.
  4. Open in your IDE: Import the XLT test suite into your preferred Java IDE.
  5. Locate your file: Confirm your .jmx file is visible under config/data/jmeter.
  6. Integrate your test: Go to src/main/java/com/xceptance/loadtest/jmeter/tests and add your test.
  7. Map your test case(s): Add your new test case(s) to the XLT test case mapping.
  8. Configure active tests: Include your test case(s) in the list of active test cases within your test configuration.
  9. Run and review: Save and run your test. You’ll find the detailed browser results in <testsuite>/results.

This is an example of the simplest JUnit test case required to run a JMeter test with XLT.

Important Considerations

To get the most out of this integration, keep these points in mind:

  • Naming Conventions: Always provide meaningful names for thread groups, transactions, and requests in JMeter. These names directly translate to clearer reports in XLT. If no names are provided, XLT will generate default names.
  • One JMeter File Per Test Case: For optimal reporting and management, treat each .jmx file as a single test case in XLT and limit it to one thread group. 
  • Transaction Controllers & Reporting:
    • Transaction Controllers in your .jmx file will be listed as “Actions” in the XLT report.
    • If “Generate parent sample” is selected for a Transaction Controller, all requests under it will be grouped. If disabled, requests will be listed individually.
  • JMeter Dependencies: upgrade.properties, saveservice.properties, and jmeter.properties are taken from the default JMeter setup. If you have custom values, add these files under your created site.

Supported JMeter Functionality

The XLT JMeter integration offers broad support for common JMeter elements because it uses JMeter under the hood. We have not built an interpreter of JMX files, we put the JMeter engine into the test suite. Behind the scenes, we redirect the traffic to our HttpClient, capture the measured data, and control the traffic using the means of XLT.

  • Thread Group: While JMeter defines them, XLT handles thread management (loops, ramp-up, users) via XTC and properties. All thread groups in one .jmx file are executed sequentially (though one active thread group per .jmx is recommended for clarity).
  • HTTP Request: Supports simple HTTP requests and HTTP multipart.
  • Pre-Processors: Fully implemented as in JMeter.
  • Assertions: JMeter’s assertion checkers are implemented and trigger events in XLT. XLT also supports JMeter’s “continue” (events only) and “stop” functions (affecting ResultBrowser and errors).
  • Post-Processors: Fully implemented as in JMeter.
  • Loop Controller: Works with its internal counter.
  • While Controller: Works with its internal counter.
  • CSV Data: Supported. Place .csv files in <testsuite>/config/data/jmeter/data. Dynamic path resolution from JMeter is also supported by placing the file in the same folder as the .jmx file.

Limitations

While powerful, there are a few limitations to be aware of:

  • XPath2 Assertions: Not yet supported.
  • Load Configuration: Load test configurations (like ramp-up, users, duration) are not read from the JMeter file. These are managed the classical XLT way via property files. Think times and load are controlled by XLT.
  • Multiple Thread Groups: It is strongly recommended to have only one active thread group per scenario (.jmx file) for clearer reporting and separation into independent test scenarios.

The test suite is also a work in progress. Consider it in beta. We’re actively looking for your feedback, contributions, and suggestions.

Ready to Supercharge Your JMeter Tests?

This new integration provides a flexible and powerful way to elevate your performance testing. By combining the familiarity of JMeter with the advanced capabilities of XLT, you can achieve deeper insights, streamline your workflows, and conduct more effective load tests, all while protecting your existing investments.

As an open-source project under the Apache License, we encourage you to try it out, test it, and even contribute! Your feedback and participation are invaluable as we continue to develop and enhance this powerful integration.

Give it a try and discover how XLT can supercharge your JMeter performance testing today!

Links and Resources

Accessibility, Automation, Java, Open Source

Automated Accessibility Testing with Google Lighthouse and Neodymium

TL;DR: Accessibility is a crucial aspect of web development that ensures everyone, regardless of their abilities, can access and use your website. Web Content Accessibility Guidelines (WCAG) provide a standard for web accessibility, and adhering to them is not only ethically important but also increasingly becoming a legal requirement.

One valuable tool for checking WCAG compliance is Google Lighthouse. It can help identify accessibility issues during development. And when integrated into your test automation project with tools like Neodymium, it can significantly streamline your accessibility testing and monitoring process.

What is WCAG and Why is it Important?

WCAG, or Web Content Accessibility Guidelines, are a set of standards developed to make web content more accessible to people with disabilities. These guidelines cover a wide range of disabilities, including visual, auditory, motor, and cognitive impairments. By following WCAG, you ensure that your website is usable by a larger audience. Approximately 15% of the world’s population has some form of disability.

WCAG compliance is also gaining traction legally, with governments in the US and EU enforcing accessibility standards. Non-compliance can lead to lawsuits and of course to less revenue and visitors.

The Role of Test Automation

Automating WCAG checks using tools like Lighthouse can help find and fix accessibility flaws early in the development cycle, saving time and resources. It also reduces the burden on manual testers by automatically generating accessibility reports.

Most importantly, test automation maintains the achieved level of accessibility of the application through regression testing.

If you have an existing or planned test automation project, it is crucial to incorporate WCAG compliance into your testing strategy. This is why we have integrated Lighthouse into Neodymium, allowing you to easily extend your test cases with accessibility checks.

Continue reading Automated Accessibility Testing with Google Lighthouse and Neodymium →
Automation, Open Source, Testing

Neodymium 5.1 – Get more done faster

TL;DR: Neodymium 5.1 has been released with a host of impressive new features. These include full-page screenshots in reports, enhanced control over JSON assertions, accessibility testing via Google Lighthouse, and simplified session handling and URL validation.

The newest enhancements primarily focus on these key areas to boost testing efficiency and user experience.

Enhanced Reporting: We’ve significantly improved the quality and usability of test reports. You now get more informative and insightful results with features like full-page screenshots, enhanced JSON assertions for easier data comparison, and a streamlined report structure for improved readability.

Accessibility Testing: Recognizing the growing importance of web accessibility, we’ve integrated Google Lighthouse to automate accessibility checks. This allows you to easily identify and address accessibility issues within your web applications, ensuring compliance with WCAG standards.

Streamlined Workflows: We’ve introduced several features to streamline your testing processes. These include improved session handling for cleaner test environments, enhanced configuration options for greater flexibility, and robust URL validation to prevent unintended access to sensitive systems and ensure test stability.

These improvements are all about making your testing smoother and more dependable. You’ll get better results and learn more from them, so you can build even better software! So, let’s get into the details, shall we?

Continue reading Neodymium 5.1 – Get more done faster →
Automation, Open Source, Testing

A new magnetic force: Neodymium 5.0.0

TL;DR: We proudly announce the release of Neodymium 5.0.0. Neo is a Java-based test library for web automation that utilizes existing libraries (Selenide, WebDriver, Allure, JUnit, Maven) and concepts (localization, test multiplication, page objects) and adds missing components such as test data handling, starter templates, multi-device handling, and other small but useful everyday helpers.

Our R&D team has been busy brewing! So today we finally get back on the major release train and present you the new Neodymium 5.0.0. It comes with a lot of new little convenience features, giving you more control and possibilities on your test automation. A better browser control, improved configuration possibilities and a bunch of new annotations like @DataItem, @WorkInProgress, @RandomBrowser, will help you set up and configure your test automation to your specific needs. Even if a picture says more than 1000 words sometimes it’s a pain to see on a single screenshot, why the automation journey broke, to help you in such cases, we introduced video recording, to exactly see what happened during the whole user journey.

See below for what you can find inside the big box of updated code:

Continue reading A new magnetic force: Neodymium 5.0.0 →
Open Source, Performance, Software, Xceptance, XLT

Xceptance LoadTest Goes Open Source

We have often been asked why XLT (Xceptance LoadTest) is free but not open source. Now, after considering our customers’ business needs and the advantages of using open source as opposed to free software, Xceptance has decided to open source XLT under the Apache License 2.0.

Within the next 30 days, we will restructure the XLT code base and move subprojects into independent open repositories. We will continue to expand and improve the documentation. All code will be published on GitHub.

Xceptance will continue to maintain and extend XLT as before. Releases will be built and published by Xceptance on a regular schedule. Libraries will be published to Maven Central as part of the open-sourcing process. The latest releases will be available from the XLT GitHub repository as well. Of course, the GitHub defect tracker will start to be your source for feature development and defect fixes.

If you are interested in training, projects, or support for XLT and load testing in general, we of course also offer those services, and will expand this offering further in upcoming months.

By open sourcing, we are opening a new chapter for XLT and we invite everyone to join us. More to come!

Open Source, Software Development, Xceptance, XLT

Thuringia’s Open-Source Prize for XLT

Wolfgang Tiefensee, Thuringia’s Secretary of Commerce, in conjunction with the board of directors of the IT industry network ITNet Thuringia, awarded the first Thueringen Open-Source Prize to three companies, all of them software companies based in Jena: TRITUM, Xceptance and GraphDefined.

Open Source Prize Title Picture Second Place

It is an honor for Xceptance to be the second-place winner of this competition. This result clearly demonstrates that open source as a component of commercial products can be a clear competitive advantage. XLT incorporates a number of open-source projects, including Apache HttpClient, Jetty, HtmlUnit, JUnit, and the Apache Commons libraries. As part of developing XLT, Xceptance is involved in testing and providing feedback for these projects, thus giving back to the open-source community.

While XLT is itself not open source, Xceptance does provide the software free of charge and with virtually no usage restrictions, so for most applications there is no noticeable difference to open-source software.

Automation, Java, Open Source, Testing

Neodymium – An Open Source Framework for Web Testing

TL;DR: Neodymium is a Java-based test library for web automation that utilizes existing libraries (Selenide, WebDriver, Allure, JUnit, Maven) and concepts (localization, test multiplication, page objects) and adds missing components such as test data handling, starter templates, multi-device handling, and other small but useful everyday helpers.

Motivation

As a company focused on quality assurance and testing, Xceptance always needs test automation software, especially end-to-end automation software. Several years ago we built a Firefox add-on that was designed to create and run browser automation. The tool was primarily used by people who didn’t necessarily have a strong background in software development. Today, the landscape is a bit different: Mozilla cut the cord on the APIs we were using and standard programming languages have largely taken over test automation because they are more flexible and less proprietary. These changes convinced us it was time to implement an idea we had already hatched, namely our own Open Source test automation project: Neodymium. It is written in and utilizes the Java platform, it is MIT licensed, and of course you will find it on GitHub: https://github.com/Xceptance/neodymium-library

Basis

There are many libraries out there to aid web automation in Java, so developers are faced with the task of choosing ones they like and somehow making them work together. On top of that, there are tasks that require some custom code to work properly. We identified the overall tooling problem mostly as a hurdle in getting started and setting up a project. Finally, there are always things missing such as test data handling, concurrency, and common patterns which you don’t want to have to develop yourself. We chose JUnit, Selenide, WebDriver, Maven, and Allure for the base tooling.

Selenide provides an easy-to-use API to control Selenium WebDriver. Allure offers good mechanics to generate useful reports based on the assertions and actions you perform throughout your test cases. Maven is used to set up the build and execution environment for our framework and all the test projects. We decided to use JUnit as the test runner since it is the de facto standard in the Java world, but we enhanced the capabilities of JUnit to do even more. At its heart, Neodymium is a JUnit runner that wraps default JUnit behavior and adds significant useful functionality to it.

Multiple Browsers

You want to be able to run the same tests for different resolutions and/or browsers to simulate the browsers most common among your users. Additionally, you need to be able to implement small differences within your test execution to address variants such as responsive designs or progressive web apps. So we added a way to run web browsers with different configurations and retrieve the current device type and resolution from within the test.

Neodymium provides a Java annotation that can be added to your test case, in order to run different browser setups. Neodymium is very flexible in configuring browsers, allowing you to fully leverage the Chrome device emulation offerings.

Test Data

Another common task is the execution of a test case with different data sets, such as testing address forms with all the relevant variations. The basic idea is to have test data and data sets in structured files next to your code, preferably as JSON, XML, property style, or simply CSV. Hence, we introduced an easy-to-use API to access the current data set and retrieve basic types from it. Furthermore, you can configure specific scenarios running only a subset or even no data set at all by adding proper annotations. To complete the picture, Neodymium supports test data on a global and package-level scope.

Localization

Another recurring topic in modern software projects is localization. Most of the web sites that are in need of test automation also support several locales. We decided to provide an out-of-the-box solution.

Neodymium’s localization feature makes use of a central translation file written in YAML format. YAML helps to structure the translations. Additionally, we implemented a simple way to override specific translations for different locales. The localized text can be easily retrieved using Neodymium API methods that are globally available.

Development Support

As it is essential to understand what your test is doing, we added a feature that enables you to slow down the test execution and highlight elements that match the current selector. Since you can chain selectors using Selenide, any chain of elements is also represented by the highlighting. With this feature activated, a developer can track down the cause of test failures much more easily. In addition, we provide information on how to set up logging in your project should you need that. Finally, we decided to use the Page Object pattern to organize the website-related code to reduce the maintenance effort and increase reusability.   

Reporting

Allure is a widely used framework to generate reports. When using Neodymium with Selenide, your automation code also contributes report information. Your test classes and methods are listed as well as detailed Selenide automation commands. In case of errors, additional details such as screenshots and source code of the page in question are available. Neodymium also provides means to structure code blocks for reporting purposes.

Continuous Integration

Implementing principles of continuous integration will deliver more reliable software by increasing efficiency, and automation is nothing without a continuous integration environment.  Yet in almost every development cycle you will eventually end up needing varied settings due to differences in your setup, which can get complicated. Neodymium provides support for extra configuration files during development to override the standard production settings as needed. Furthermore, the framework supports overriding properties that change the configuration of your test execution by setting environment variables or simply passing Java arguments.

Because automation is supposed to run quickly, Neodymium provides support for parallel test execution and also demonstrates that setup as part of the sample test suite.

Documentation and Templates

Does Neodymium address some of your test automation challenges? Does it sound like a good entry point for your test automation?

Neodymium is hosted on GitHub (https://github.com/Xceptance/neodymium-library), where the accompanying project wiki (https://github.com/Xceptance/neodymium-library/wiki) provides extensive documentation to help you get started and answer your questions.

You might also want to take a look at the comprehensive example projects using Neodymium with Cucumber (https://github.com/Xceptance/neodymium-cucumber-example) or plain Java (https://github.com/Xceptance/neodymium-example). We’ve even provided a template project (https://github.com/Xceptance/neodymium-template) to get you started automating in no time.

License

Neodymium is licensed under the MIT License.

Who Are We

We are Xceptance. A software testing company with strong commerce knowledge and projects with customers from all around the world. Besides Neodymium, we have developed Xceptance Load Test (XLT), a load and performance test tool that is available free of charge and features an extensive range of awesome features to make the tester’s and developer’s life easier.

If you are looking for test automation that also covers the performance side of life, take a look at XLT. You can write and run load tests with real browsers including access to data from the Web Performance Timing API. In case browsers are too heavy, XLT has other modes of load testing to offer as well.

We offer professional support for Neodymium as well as implementation and training services.

Automation, Open Source, Testing, XLT

Our Demandware SiteGenesis Community Test Suite

The community test suite homepageToday, we are happy to announce the release of our new Demandware SiteGenesis Community Test Suite! It is a test suite for automated storefront testing of the Demandware SiteGenesis reference e-commerce storefront.

The test suite’s intention is to share experiences, transfer knowledge, and to demonstrate best practices in test automation. The test suite is built using XLT of course. XLT is freely available and extends concepts you already know from Selenium.
Continue reading Our Demandware SiteGenesis Community Test Suite →

Open Source, Security and Privacy, Xceptance

We Take Web Privacy Seriously

Internet SurveillanceWhile privacy laws are still a work in progress, the EU and Germany have made several statements about what is right or wrong. But the current state of regulation is unclear about what actually is permitted or has to be provided.

Update

In the light of the EU GDPR (DSGVO in German) that kicks in on 25 May 2018, this article is even more important, because you can ensure that tracking will only happen when a user has not opted out or he/she consent to tracking. Also our method automatically honors DNT settings by the user and won’t bother any visitor if this settings has been applied. And now… please read on…

Xceptance takes the privacy of our users seriously. We decided to implement tracking and privacy for our website xceptance.com differently than what is usually seen on the market: we provide users with more choices, and we’re stricter about respecting your decisions, yet without pestering you with lots of button clicking.
Continue reading We Take Web Privacy Seriously →

Open Source, Software, Xceptance

Xceptance.com Goes Open Source

GitHubLike most companies we use and benefit from open source projects from all around the world. To actively participate and give something back, we started our own open source project that others can hopefully benefit from as well: Our website Xceptance.com has been open sourced.

In building our new website, we faced different requirements and challenges, like easy to maintain pages, mostly static content, a modern design, multiple languages, multiple domains, and so on.
Continue reading Xceptance.com Goes Open Source →