Tag Archives: XLT

Data Science Day 2022

After a two-year break, on 18 May 2022, the 4th Data Science Day Jena took place. The Friedrich-Schiller University is hosting this mini-conference format annually.

Xceptance presented this year a closer look at the data that is collected and processed during load and performance testing. René Schwietzke, Managing Director of Xceptance, talked about the challenges to capture the right data as well as translate the collected data into meaningful results. 

A load test simulates millions of user interactions with a website and therefore is capturing huge amounts of data points. These have to be transformed into a few numbers to make the result of the test easy to communicate but still preserve important details. The talk started with typical business requirements and expectations of the target groups of a load test. It showed the data XLT captures and the dimensions which later drive the data reduction. A few example data series demonstrated the challenges behind the data reduction and what numbers are finally used to satisfy the requirements.

An example load test result illustrated the talk with real data. That example test run created about 17,500 data rows per second which contain about 293,000 data points. The entire test result consists of 3.2 billion data points. This massive data set is turned into a consumable report by XLT in less than six minutes. 

For everyone with an interest in data science, this presentation also offers ideas for research in regard to unsolved data challenges. There might be even some Master’s and Bachelor’s theses topics waiting for you. 

You can find a recording of the presentation below (courtesy of Thüringer Universitäts- und Landesbibliothek Jena).

This is the accompanying slide deck. It is a Reveal.js-based. You can navigate with the spacebar and the arrow keys.

Picture of the presentation


Xceptance LoadTest Goes Open Source

We have often been asked why XLT (Xceptance LoadTest) is free but not open source. Now, after considering our customers’ business needs and the advantages of using open source as opposed to free software, Xceptance has decided to open source XLT under the Apache License 2.0.

Within the next 30 days, we will restructure the XLT code base and move subprojects into independent open repositories. We will continue to expand and improve the documentation. All code will be published on GitHub.

Xceptance will continue to maintain and extend XLT as before. Releases will be built and published by Xceptance on a regular schedule. Libraries will be published to Maven Central as part of the open-sourcing process. The latest releases will be available from the XLT GitHub repository as well. Of course, the GitHub defect tracker will start to be your source for feature development and defect fixes.

If you are interested in training, projects, or support for XLT and load testing in general, we of course also offer those services, and will expand this offering further in upcoming months.

By open sourcing, we are opening a new chapter for XLT and we invite everyone to join us. More to come!

Thuringia’s Open-Source Prize for XLT

Wolfgang Tiefensee, Thuringia’s Secretary of Commerce, in conjunction with the board of directors of the IT industry network ITNet Thuringia, awarded the first Thueringen Open-Source Prize to three companies, all of them software companies based in Jena: TRITUM, Xceptance and GraphDefined.

Open Source Prize Title Picture Second Place

It is an honor for Xceptance to be the second-place winner of this competition. This result clearly demonstrates that open source as a component of commercial products can be a clear competitive advantage. XLT incorporates a number of open-source projects, including Apache HttpClient, Jetty, HtmlUnit, JUnit, and the Apache Commons libraries. As part of developing XLT, Xceptance is involved in testing and providing feedback for these projects, thus giving back to the open-source community.

While XLT is itself not open source, Xceptance does provide the software free of charge and with virtually no usage restrictions, so for most applications there is no noticeable difference to open-source software.

XLT 4.12.2 Release

Xceptance released version 4.12.2 of its load testing and test automation product Xceptance LoadTest. This is an improvement release. We recommend upgrading to this newest version.

Test Framework

  • Improvement: The result browser features a new tab that displays JSON responses in a tree-like view. The data can also be searched and filtered.
  • Improvement: The existing automatic request retry mechanism has been enhanced to retry failed requests in additional error situations such as connection resets. This should now behave similar to real browsers.
  • Improvement: When failed requests are retried, an event with diagnostic information is logged for each retry.
  • Improvement: The bundled Jetty library has been updated to the latest available version 9.4.14.

Load Testing

  • Improvement:  Our public AWS machine images now come with OpenJDK 11.

Make sure to read the full online release notes.

As always, this upgrade is free and don’t forget, XLT itself is free as well. You don’t have an excuse to skip performance testing or rely on lame simple test cases anymore.

XLT 4.12.1 Release

Xceptance has released version 4.12.1 of its load testing and test automation product Xceptance LoadTest.

Test Framework

  • Fix: Our timer recorder extensions for Chrome and Firefox did sometimes report invalid request entries that could not be processed by the report generator. This could happen for requests that did not complete.
  • Fix: If a test case deliberately caught an exception / assertion error and afterwards ran to completion successfully, it might nevertheless be marked as failed in the load test report.
  • Improvement: Selenium has been updated to the latest version 3.141.59 and HtmlUnitDriver to version 2.33.3.

Load Testing

  • Improvement: The new AWS data center in Stockholm, Sweden (eu-north-1) is fully supported by ec2_admin now.

Make sure to read the full online release notes.

As always, this upgrade is free and don’t forget, XLT itself is free as well. You don’t have an excuse to skip performance testing or rely on lame simple test cases anymore.

XLT 4.11 was released

Xceptance released version 4.11 of its load testing and test automation product Xceptance LoadTest. This is a feature release. We recommend upgrading to this newest version.

Here is a selection of the most important changes:

  • HtmlUnit and Selenium have been updated
  • You can run a test case with only a single data set at development time
  • XLT picks a data set automatically when a test with multiple data sets is part of a load test
  • There is a new command-line tool to automatically evaluate a test report based on your defined success criteria
  • The XLT Jenkins plugin returns a result object now with all the details when used in a pipeline
  • The XLT Jenkins plugin can now create a comparison report

See our release notes for more details. As always, this upgrade is free and don’t forget, XLT itself is free as well. You don’t have an excuse to skip performance testing or rely on lame simple test cases anymore.

XLT – Now Available Free Of Charge

Update: Please see Xceptance LoadTest Goes Open Source for the next step in the evolution of XLT.

Because it is always great to end the year with something pleasant and start the next year with something even better, we want to surprise you with the announcement that Xceptance LoadTest (XLT) will be free of charge for load and performance testing starting 1 January 2018. XLT was already free for use in test automation.

Why are we doing this? Well, we have seen many people working with other performance test tools over the years and found that often they chose one primarily because of the price or simply because it was already a familiar tool in the company. This is not necessarily bad but it is certainly not always optimal.

People run into seat limits, operating system limitations, debugging limits, extensibility limits, have to use poor reporting, or cannot adjust features or functionalities. We want to give the testing world a tool that will be chosen because it fits and not because the license is cheaper or has already been purchased. Or in other words: You have more choices now.

Xceptance and its customers have been using XLT for many years. What started as a quick workaround because commercial tools had no affordable license a consulting company could use, grew into a versatile tool with a focus on great reporting and diagnostic capabilities.

We will update the license portal to the new model in the next few days and also rework our website accordingly. We just didn’t want to wait to tell our existing customers and take one worry off the 2018 budget table.

Happy Holidays, Merry Christmas, and a Happy New Year!

New Free-License Terms

Obviously some context is needed to make clearer what you can do with XLT and its free-of-charge license now.

  • XLT is available free of charge but you still need a license file from us. Request this license from Xceptance via email (support@xceptance.de) or use our license portal, and we will issue you one without any user or time limits.
  • You can use XLT for commercial purposes as long as you do not sell XLT itself or create the impression that XLT is your own software. You can sell software components built on top of XLT.
  • You must not modify XLT in any way. But you can customize test suites, demos, templates, or other customizable components such as CSS or XSL as long as you distribute these components separately.
  • You cannot offer XLT for download, excluding your local Maven/Nexus repositories and company internal redistribution, of course.
  • You cannot integrate XLT into public SaaS or other commercial or noncommercial hosting offers without explicit permission from us. If you built your own company-internal cloud for load testing for instance, that is of course permitted.
  • You cannot share your license outside of your organization. If you are a consulting company or freelancer, we encourage your clients to request their own licenses, but we do not require it. All licenses are personalized with email and company name.

FAQ

Is XLT open source now?

No, XLT is still closed source; we just stopped charging you for using it.

Will updates be free, too?

Yes, updates stay free. Please make sure you sign up for our release newsletter. Purchasing a support contract allows us to support you faster and more directly.

Can I use XLT commercially?

Yes, you can. Just request a license. If you are not sure whether your usage pattern is covered by our free-license terms, please ask. If you have interesting business ideas or applications for XLT, we are here to help.

Will Xceptance continue to invest in XLT?

Yes, we will continue to invest in and enhance XLT because it is an integral part of our company’s load and performance testing services. Product development is funded by the support, training, and testing services provided by Xceptance.

Firefox 57 Changes and XLT

Firefox 57 is going to deliver a fundamental change that will affect XLT Script Developer.

As you might know, Mozilla decided to completely remove the support for XUL/XPCOM-based extensions (aka legacy extensions) in favor of extensions built upon the WebExtension API. This cut will take place on November 14, 2017, with the release of Firefox 57. Additionally, Mozilla will refuse to sign updates of legacy extensions at some point in the near future, although the exact date is not determined yet. See the Mozilla Add-ons Blog for an up-to-date timeline.

The consequence of this breaking change is that XLT Script Developer cannot be installed in Firefox 57 and higher. Also, an already installed XLT Script Developer cannot be used any longer.

We at Xceptance have been aware of this development. Over the course of the last year we have been busy evaluating several alternative options. As you might recall, we also conducted a survey back in the spring to collect your feedback.

Based on our findings and your input and after long discussions, we came to the conclusion that the feature set and comfort that had been offered by XLT Script Developer and its way of writing web automation cannot be recreated using the alternative Firefox APIs.

Therefore, we regret to announce that Script Developer is discontinued. Consequently, we don’t recommend starting new test projects based on Script Developer and XML script test cases. To be able to continue to use the features of XLT and the advantages it offers, we suggest you use XLT’s Java-based API.

If you are able to use Firefox 56 or Firefox 52/ESR, maybe in parallel to an updated version of Firefox, you can continue to work with XLT Script Developer as well as execute all your test cases as you have been.

We are aware that this decision might disappoint many of you and may leave open questions. For more information on what shaped this decision and what your options are for maintaining your existing XML script test suites in the future or migrating them to another base, please see the Q&A section below.

Q&A

Why did Mozilla decide to abandon legacy extensions?

By design, legacy extensions have not only full access to your local file system but also to the entire browser and thus to all the pages you visit. This has been causing privacy and security issues and hence Mozilla decided to abandon the API to address these problems.

Why isn’t XLT Script Developer being ported to the WebExtension API?

The possibilities offered by the WebExtension API are very limited. One such limitation, and the most important one, is that access to the local file system involves a user interaction for each and every file to save or read. This would make Script Developer simply impossible to use.

Further restrictions apply. Most of them are related to accessing and manipulating session and browsing data such as cookies or cache. In the end, Script Developer would only be able to support a very reduced feature set.

And last but not least, the outcome of our customer survey revealed needs which cannot be met by such a modest visual, non-programming approach to writing tests. See the next question for more details on the feedback we received.

Why won’t XLT Script Developer be ported to another platform?

The outcome of our survey was that users wanted a tighter integration with Java/IDEs, more commands, more ways to customize things, better flow control, and more flexibility with test data. At the end of the day, this is a full programming approach and turning the XLT Script Developer into another programming environment couldn’t address all these points.

We also believe that the concept of XML script test cases with their limited capabilities are no longer appropriate for modern testing needs. Therefore we have decided to give it up in favor of the opportunities a real programming language provides.

Can I continue to use XLT Script Developer with Firefox 56?

Yes, but at your own risk. Remember that your browser cannot be updated and therefore will not receive any security fixes. Furthermore, it may not be possible to install any Script Developer updates as this Firefox version accepts signed extensions only. And Mozilla might stop signing legacy extensions any day. See below for a better solution.

Can I continue to use XLT Script Developer with Firefox 52/ESR?

Yes, absolutely. That is the recommended way to go, at least for the next months. First, Firefox 52/ESR (Extended Service Release) will be updated with security fixes until May 2018. If you continue to use Script Developer after that date, though, you do so at your own risk. Make sure that you use this browser version for scripting only.

Furthermore, Firefox 52/ESR can be tweaked to install legacy extensions even if they are not signed by Mozilla. This way, you are still able to install Script Developer updates in case Xceptance provides some in the future.

Will the XLT framework still be able to replay XML script test cases?

Yes, XLT will continue to interpret and execute your existing XML script test cases as JUnit tests via their wrapper classes.

How to migrate existing projects?

Export all your existing XML script test cases to Java (Scripting API). From this point, you maintain your test cases in your favorite Java IDE instead of the XLT Script Developer. Since the concepts and commands are the same as in Script Developer, you will be on top of things quickly.

Instead of running your XML test cases in Script Developer, you would now run your Java-based tests as JUnit tests using your preferred WebDriver either from your IDE or using a continuous integration system.

Since your code base is plain Java now, you are free to add all the things that you might have missed in the past.

Note that XLT 4.10.0 will ship with some enhancements for Java-based test cases. For instance, Script Developer will provide an alternative export template that produces more compact code. Additionally, writing test cases directly in Java will be more comfortable as well. Stay tuned for the upcoming release and find all the details in the release notes.

Make sure you subscribe to our low-volume XLT release and news mailing list.

How long will you release XLT Script Developer updates?

TBD. Future updates of Script Developer will be bug-fix releases only (shipped as unsigned extensions once Mozilla stops signing legacy extensions). Don’t expect any new features.

Are there any other options?

Yes. There are forks of Firefox that promise to continue supporting legacy extensions while being kept up-to-date at the same time. For instance, Script Developer installs and runs nicely in Waterfox. However, we cannot predict how long this will actually work.

XLT 4.9 has been released

Xceptance released version 4.9 of its load testing and test automation product Xceptance LoadTest. This is primarily a 3rd party update release, but also delivers some improvements.

Here is a selection of the most important changes:

  • Script Developer supports Firefox 53
  • Selenium updated to version 3.4.0
  • Better reporting of JavaScript errors in script test cases
  • Master controller displays the configured load profile
  • Load test reports can be created for a subset of agents
  • Load test reports shows the number of entries in data tables and shows summary values when filtering the table
  • Demo app server ports can be reconfigured easily

Script Developer

Script Developer has been made compatible with the latest available Firefox version, while outdated versions are not supported any longer. Script Developer runs on Firefox 45/ESR up to 53 now.

Update Instructions: Firefox will not auto-update older versions of Script Developer to 4.9.0. You will need to do this manually. Please remove the currently installed version first and afterwards install the new version by simply dragging and dropping the file xlt-scriptdeveloper-4.9.0.xpi onto Firefox. Auto-updating within the 4.9.x product line will then work as usual again.

Alternatively you can visit the Mozilla Add-On marketplace and install Script Developer from there (https://addons.mozilla.org/en-US/firefox/addon/xceptance-script-developer/). Please make sure you uninstall your locally installed version first.

Framework

XLT now ships with Selenium 3.4.0. Make sure you update your locally installed driver binaries to the latest available version. This is especially true for geckodriver. In case you experience issues with geckodriver / Firefox, you might be better off running FirefoxDriver in legacy mode. The legacy mode is more mature.

All other core libraries have been updated as well. This also includes HtmlUnit for an improved browser emulation.

The XLT framework also comes with some functional improvements. In case a JavaScript expression in your script test case could not be evaluated successfully for any reason, the root cause will now be listed as part of the exception message. Libraries that make use of Java’s built-in logging framework do no longer log to the console, but to XLT’s log file.

Load and Performance Testing

The Mastercontroller now prints the configured load profile to the console when starting a load test and also when displaying the current status. This helps to spot test configuration mistakes earlier. Intermediate results downloaded, while a load test is still running, will now be flagged to distinguish them from final results.

The load test report shows the number of entries in a data table, and when filtering a data table, the footer row is updated accordingly. Load test reports may also be created for only a subset of the agents. You might remember that version 4.8 already delivered the ability to render reports for specific test cases only.

Last but not least, ec2_admin prints more details about running AWS machine instances and lets you review your choice before actually terminating running instances.

Demo App Server / Posters Store

The app server that hosts our demo applications uses ports 8080 and 8443 by default. Since these ports are often already used by other applications, you can now reconfigure them easily.

Our demo application Posters Store now runs with HTTPS only. Any HTTP request will be redirected to use HTTPS.

Sounds interesting? More details can be found in the release notes of XLT 4.9.

Release of XLT 4.8

XLT 4.8 is primarily a technology update release, but also comes with some new features.

First and foremost, XLT now ships with Selenium 3, the new version of the WebDriver library. All other core libraries have been updated as well. This also includes HtmlUnit for an improved browser emulation. Beginning with this release, XLT requires Java 8 to run.

The XLT framework also comes with some functional improvements. XLT now supports OperaDriver out of the box and can run FirefoxDriver in either the new Marionette mode using geckodriver or in the “old” legacy mode. When you drive Firefox via XltFirefoxDriver, you will get a much more detailed result browser now, with almost the same request and response details that you already know from XltDriver. Any values that you programmatically add to the newly introduced value log of a session are shown in the result browser as well. Furthermore, most of the XLT framework properties can now be configured not only globally, but also specifically for a certain test scenario.

For load testers, there is something in the box as well. If you use the AWS EC2 cloud a lot, you will be glad to hear that the new AWS data center in Ohio is now fully supported. The load test report has been tuned to become usable much faster, even with lengthy pages such as the Requests page.

Last but not least, the Poster Store demo application and the XLT Jenkins Plug-In have both been updated. If you ever wanted to load-test your WebDAV server, there is now a new demo test suite for that.

Sounds interesting? More details can be found in the release notes of XLT 4.8.