When using XLT for test automation or load testing, a visual and browsable log of your test case steps and their results will be created during test case execution. We call this log the XLT Result Browser and the following how-to will explain how to read it.
Continue reading How to use the XLT Result Browser
Category Archives: Automation
Firefox 57 Changes and XLT
Firefox 57 is going to deliver a fundamental change that will affect XLT Script Developer.
As you might know, Mozilla decided to completely remove the support for XUL/XPCOM-based extensions (aka legacy extensions) in favor of extensions built upon the WebExtension API. This cut will take place on November 14, 2017, with the release of Firefox 57. Additionally, Mozilla will refuse to sign updates of legacy extensions at some point in the near future, although the exact date is not determined yet. See the Mozilla Add-ons Blog for an up-to-date timeline.
The consequence of this breaking change is that XLT Script Developer cannot be installed in Firefox 57 and higher. Also, an already installed XLT Script Developer cannot be used any longer.
We at Xceptance have been aware of this development. Over the course of the last year we have been busy evaluating several alternative options. As you might recall, we also conducted a survey back in the spring to collect your feedback.
Based on our findings and your input and after long discussions, we came to the conclusion that the feature set and comfort that had been offered by XLT Script Developer and its way of writing web automation cannot be recreated using the alternative Firefox APIs.
Therefore, we regret to announce that Script Developer is discontinued. Consequently, we don’t recommend starting new test projects based on Script Developer and XML script test cases. To be able to continue to use the features of XLT and the advantages it offers, we suggest you use XLT’s Java-based API.
If you are able to use Firefox 56 or Firefox 52/ESR, maybe in parallel to an updated version of Firefox, you can continue to work with XLT Script Developer as well as execute all your test cases as you have been.
We are aware that this decision might disappoint many of you and may leave open questions. For more information on what shaped this decision and what your options are for maintaining your existing XML script test suites in the future or migrating them to another base, please see the Q&A section below.
Q&A
Why did Mozilla decide to abandon legacy extensions?
By design, legacy extensions have not only full access to your local file system but also to the entire browser and thus to all the pages you visit. This has been causing privacy and security issues and hence Mozilla decided to abandon the API to address these problems.
Why isn’t XLT Script Developer being ported to the WebExtension API?
The possibilities offered by the WebExtension API are very limited. One such limitation, and the most important one, is that access to the local file system involves a user interaction for each and every file to save or read. This would make Script Developer simply impossible to use.
Further restrictions apply. Most of them are related to accessing and manipulating session and browsing data such as cookies or cache. In the end, Script Developer would only be able to support a very reduced feature set.
And last but not least, the outcome of our customer survey revealed needs which cannot be met by such a modest visual, non-programming approach to writing tests. See the next question for more details on the feedback we received.
Why won’t XLT Script Developer be ported to another platform?
The outcome of our survey was that users wanted a tighter integration with Java/IDEs, more commands, more ways to customize things, better flow control, and more flexibility with test data. At the end of the day, this is a full programming approach and turning the XLT Script Developer into another programming environment couldn’t address all these points.
We also believe that the concept of XML script test cases with their limited capabilities are no longer appropriate for modern testing needs. Therefore we have decided to give it up in favor of the opportunities a real programming language provides.
Can I continue to use XLT Script Developer with Firefox 56?
Yes, but at your own risk. Remember that your browser cannot be updated and therefore will not receive any security fixes. Furthermore, it may not be possible to install any Script Developer updates as this Firefox version accepts signed extensions only. And Mozilla might stop signing legacy extensions any day. See below for a better solution.
Can I continue to use XLT Script Developer with Firefox 52/ESR?
Yes, absolutely. That is the recommended way to go, at least for the next months. First, Firefox 52/ESR (Extended Service Release) will be updated with security fixes until May 2018. If you continue to use Script Developer after that date, though, you do so at your own risk. Make sure that you use this browser version for scripting only.
Furthermore, Firefox 52/ESR can be tweaked to install legacy extensions even if they are not signed by Mozilla. This way, you are still able to install Script Developer updates in case Xceptance provides some in the future.
Will the XLT framework still be able to replay XML script test cases?
Yes, XLT will continue to interpret and execute your existing XML script test cases as JUnit tests via their wrapper classes.
How to migrate existing projects?
Export all your existing XML script test cases to Java (Scripting API). From this point, you maintain your test cases in your favorite Java IDE instead of the XLT Script Developer. Since the concepts and commands are the same as in Script Developer, you will be on top of things quickly.
Instead of running your XML test cases in Script Developer, you would now run your Java-based tests as JUnit tests using your preferred WebDriver either from your IDE or using a continuous integration system.
Since your code base is plain Java now, you are free to add all the things that you might have missed in the past.
Note that XLT 4.10.0 will ship with some enhancements for Java-based test cases. For instance, Script Developer will provide an alternative export template that produces more compact code. Additionally, writing test cases directly in Java will be more comfortable as well. Stay tuned for the upcoming release and find all the details in the release notes.
Make sure you subscribe to our low-volume XLT release and news mailing list.
How long will you release XLT Script Developer updates?
TBD. Future updates of Script Developer will be bug-fix releases only (shipped as unsigned extensions once Mozilla stops signing legacy extensions). Don’t expect any new features.
Are there any other options?
Yes. There are forks of Firefox that promise to continue supporting legacy extensions while being kept up-to-date at the same time. For instance, Script Developer installs and runs nicely in Waterfox. However, we cannot predict how long this will actually work.
Localisation Verifier – A Custom Java Module for Script Developer
Introduction
When employing XLT Script Developer you usually resort to automated or manual scripting to drive your testing. Sometimes though you will face a very specific or complex task that can not be expressed that easy with the standard scripting capabilities of Script Developer. For these types of scenarios Script Developer provides the option to integrate a custom Java module. With custom modules you have the full power of your Java runtime and are able to achieve virtually any testing objective.
The following blog post will describe a small custom Java module we created and used recently. By this example we will explain when to choose this route and demonstrate the creation and execution of Java modules. Ultimately you will be able to add Script Developer’s custom module option to your testing arsenal.
Continue reading Localisation Verifier – A Custom Java Module for Script Developer
Multi-Browser Support for Test Automation with XLT
Summary
In today’s post we will discuss the steps necessary to enhance an XLT-based test suite with multi-browser support. We will show how to tag your test cases to conveniently run them in different environments and execute the test suite in a local or remote fashion.
Introduction
Xceptance maintains a MIT licensed test suite at GitHub which demonstrates functional testing for large scale projects. With the suite we’ve put an emphasis on clear structures, naming and test case organization. Targeting Demandware’s SiteGenesis storefront at heart, the underlying concepts and mechanisms are valuable for everyone building test suites for comparable web applications with XLT. Next to being a template for test automation and best practices in test suite design, it can be a starting point ready to pick up in your very own projects. We regularly utilize it and want to encourage you to explore, employ and contribute.
A regular challenge in testing ecommerce applications is the variety of different browsers and platforms that are available today. As you probably know XLT, the test automation and load testing framework from Xceptance, is based on Selenium browser automation and the Webdriver API. Supporting multiple browsers therefore comes naturally. This blog post will demonstrate how XLT is able to streamline different testing environments directly in your test suite. You will learn how to execute your tests locally and remotely with the help of Sauce Labs and similar automated testing platforms. Along the way you will pick up some details about XLT as well as Script Developer and quickly find yourself equipped with a ready to use multi-browser test suite example.
Continue reading Multi-Browser Support for Test Automation with XLT
Special Characters in Script Developer
Recently we received a support request regarding special characters in Script Developer. Perhaps other XLT users stumble across a similar requirement, so it’s a good idea to make the discussion available to the public.
First of all, some bad news: Up to now, Script Developer does not have explicit support for special characters, such as Line Feed (\n), Horizontal Tab (\t), Backspace (\b) or similar. For example, typing multiple lines of text – each line delimited by a newline character – into an element on your page is not possible just like that. Upon loading your script, XLT Script Developer normalizes all white-space characters contained in the target or value field of any command.
Of course, we don’t want to leave you out in the rain but provide a feasible solution.
Continue reading Special Characters in Script Developer
Conditional Expressions in XLT
Motivation
Did you ever have to create multiple versions of your test cases to accommodate small differences of your test objects? Looking for a trade-off between good testing practice and minimizing project complexity. The following blog post reflects on this challenge and introduces you to a potential solution: Conditional Expressions.
Introduction
Xceptance introduced its test automation and load testing tool XLT 4.6 in February 2016 and with it we brought you conditionals. Today we want to shed some light on this new feature, the discussion that came along with it and why we finally decided to introduce it. This blog post will equip you with everything required to employ conditional expressions in your test scripts.
In computer programming, a condition or conditional expression performs an action depending on whether a given statement (the condition) evaluates to true or false. The programmer has the possibility to execute a part of the program only if certain circumstances are met. Now don’t worry, you do not need to become a full-fledged programmer to create your test cases with XLT Script Developer. But you should not skip on the possibilities this new feature is offering.
The Challenge
In testing typically you want your flow of execution to be linear, deterministic and transparent. The individual execution steps of your test case should be well-defined and yield the same results in a constant environment. If one execution step fails – e.g. an assertion does not check out – the whole test case always breaks and evaluates to failed. Run, rinse and repeat.
On the contrary often enough your real world test cases need to cater various scenarios. Think multi-region support of your page for example. Area specific content and functionality can quickly bring you into a catch-22 situation. Follow good practice in test case design, but deal with complexity and organizational nightmares in your test suite. Tiny differences in your test objects force you to keep multiple versions of your (already lengthy) test cases. Farewell maintainability!
Continue reading Conditional Expressions in XLT
Our Demandware SiteGenesis Community Test Suite
Today, we are happy to announce the release of our new Demandware SiteGenesis Community Test Suite! It is a test suite for automated storefront testing of the Demandware SiteGenesis reference e-commerce storefront.
The test suite’s intention is to share experiences, transfer knowledge, and to demonstrate best practices in test automation. The test suite is built using XLT of course. XLT is freely available and extends concepts you already know from Selenium.
Continue reading Our Demandware SiteGenesis Community Test Suite
Use XLT with Sauce Labs and BrowserStack
Sauce Labs and BrowserStack – What Are They and Why Use Them?
This approach still work fine, but we came up with a much better one. Head over to GitHub and see our Multi-Browser-TestSuite for XLT. It will make multi browser testing a breeze. By the way, all the code is licensed under the MIT license, so absolute flexibility for you.
Sauce Labs and BrowserStack allow you to run automated test cases on different browsers and operating systems. Both provide more than 200 mobile and desktop browsers on different operating systems. The benefit? You can focus on coding instead of having to maintain different devices. You can easily run your test cases written on iOS on an Internet Explorer without actually buying a Windows device; and last not least, you don’t need to worry about drivers or maintenance.
By the way, Internet Explorer even seems to run faster at Sauce Labs than on a desktop machine. Also note that Sauce Labs supports Maven builds.
Continue reading Use XLT with Sauce Labs and BrowserStack
TestSuite-NoCoding – Load Testing with CSV Files
We continue to share cool things with the community of software testers and developers. Today we are announcing the availability of our no coding test suite for XLT under the Apache License v2.0.
Introduction
You want to fire just a couple of URLs to load test your application? You have to investigate the performance problems of a feature and you need accurate measurements as well as a lot of load generated? You like XLT and its capabilities, but you don’t have the time to compile a sophisticated test suite from scratch? Whatever your motivation, our new test suite for XLT is the solution you are looking for.
Continue reading TestSuite-NoCoding – Load Testing with CSV Files
XCMailr – An Open Source Test Mail Forwarder
As a software tester, an episode such as the one below must be more than familiar to you and, let’s be honest, it has the potential of making the top ten of the most annoying things in our daily work routine:
- Pain in the neck: “Hey, I need more email addresses for testing, I just burnt all my own.”
- You: “Well, just use a fake one.”
- P: “Nah, I can’t, I need the activation emails.”
- Y: “Well, then, there are good disposable mailers out there.”
- P: “Very clever, but they aren’t protected by authentication and I signed an NDA for that project.”
- Y: “Here, use this one.”
- P: “But it wants to have a real email to sign me up and I don’t really feel like giving my real email away.”
- Y: “$§5$!51z1hhsks!”
Granted, it’s a matter of course that committed testers have many email addresses but what’s the use of them when you’re always limited to a certain number, when you can’t quickly change them, or deactivate them when an email service got hold of them?
Continue reading XCMailr – An Open Source Test Mail Forwarder