Tag Archives: test automation

Firefox 57 Changes and XLT

Firefox 57 is going to deliver a fundamental change that will affect XLT Script Developer.

As you might know, Mozilla decided to completely remove the support for XUL/XPCOM-based extensions (aka legacy extensions) in favor of extensions built upon the WebExtension API. This cut will take place on November 14, 2017, with the release of Firefox 57. Additionally, Mozilla will refuse to sign updates of legacy extensions at some point in the near future, although the exact date is not determined yet. See the Mozilla Add-ons Blog for an up-to-date timeline.

The consequence of this breaking change is that XLT Script Developer cannot be installed in Firefox 57 and higher. Also, an already installed XLT Script Developer cannot be used any longer.

We at Xceptance have been aware of this development. Over the course of the last year we have been busy evaluating several alternative options. As you might recall, we also conducted a survey back in the spring to collect your feedback.

Based on our findings and your input and after long discussions, we came to the conclusion that the feature set and comfort that had been offered by XLT Script Developer and its way of writing web automation cannot be recreated using the alternative Firefox APIs.

Therefore, we regret to announce that Script Developer is discontinued. Consequently, we don’t recommend starting new test projects based on Script Developer and XML script test cases. To be able to continue to use the features of XLT and the advantages it offers, we suggest you use XLT’s Java-based API.

If you are able to use Firefox 56 or Firefox 52/ESR, maybe in parallel to an updated version of Firefox, you can continue to work with XLT Script Developer as well as execute all your test cases as you have been.

We are aware that this decision might disappoint many of you and may leave open questions. For more information on what shaped this decision and what your options are for maintaining your existing XML script test suites in the future or migrating them to another base, please see the Q&A section below.

Q&A

Why did Mozilla decide to abandon legacy extensions?

By design, legacy extensions have not only full access to your local file system but also to the entire browser and thus to all the pages you visit. This has been causing privacy and security issues and hence Mozilla decided to abandon the API to address these problems.

Why isn’t XLT Script Developer being ported to the WebExtension API?

The possibilities offered by the WebExtension API are very limited. One such limitation, and the most important one, is that access to the local file system involves a user interaction for each and every file to save or read. This would make Script Developer simply impossible to use.

Further restrictions apply. Most of them are related to accessing and manipulating session and browsing data such as cookies or cache. In the end, Script Developer would only be able to support a very reduced feature set.

And last but not least, the outcome of our customer survey revealed needs which cannot be met by such a modest visual, non-programming approach to writing tests. See the next question for more details on the feedback we received.

Why won’t XLT Script Developer be ported to another platform?

The outcome of our survey was that users wanted a tighter integration with Java/IDEs, more commands, more ways to customize things, better flow control, and more flexibility with test data. At the end of the day, this is a full programming approach and turning the XLT Script Developer into another programming environment couldn’t address all these points.

We also believe that the concept of XML script test cases with their limited capabilities are no longer appropriate for modern testing needs. Therefore we have decided to give it up in favor of the opportunities a real programming language provides.

Can I continue to use XLT Script Developer with Firefox 56?

Yes, but at your own risk. Remember that your browser cannot be updated and therefore will not receive any security fixes. Furthermore, it may not be possible to install any Script Developer updates as this Firefox version accepts signed extensions only. And Mozilla might stop signing legacy extensions any day. See below for a better solution.

Can I continue to use XLT Script Developer with Firefox 52/ESR?

Yes, absolutely. That is the recommended way to go, at least for the next months. First, Firefox 52/ESR (Extended Service Release) will be updated with security fixes until May 2018. If you continue to use Script Developer after that date, though, you do so at your own risk. Make sure that you use this browser version for scripting only.

Furthermore, Firefox 52/ESR can be tweaked to install legacy extensions even if they are not signed by Mozilla. This way, you are still able to install Script Developer updates in case Xceptance provides some in the future.

Will the XLT framework still be able to replay XML script test cases?

Yes, XLT will continue to interpret and execute your existing XML script test cases as JUnit tests via their wrapper classes.

How to migrate existing projects?

Export all your existing XML script test cases to Java (Scripting API). From this point, you maintain your test cases in your favorite Java IDE instead of the XLT Script Developer. Since the concepts and commands are the same as in Script Developer, you will be on top of things quickly.

Instead of running your XML test cases in Script Developer, you would now run your Java-based tests as JUnit tests using your preferred WebDriver either from your IDE or using a continuous integration system.

Since your code base is plain Java now, you are free to add all the things that you might have missed in the past.

Note that XLT 4.10.0 will ship with some enhancements for Java-based test cases. For instance, Script Developer will provide an alternative export template that produces more compact code. Additionally, writing test cases directly in Java will be more comfortable as well. Stay tuned for the upcoming release and find all the details in the release notes.

Make sure you subscribe to our low-volume XLT release and news mailing list.

How long will you release XLT Script Developer updates?

TBD. Future updates of Script Developer will be bug-fix releases only (shipped as unsigned extensions once Mozilla stops signing legacy extensions). Don’t expect any new features.

Are there any other options?

Yes. There are forks of Firefox that promise to continue supporting legacy extensions while being kept up-to-date at the same time. For instance, Script Developer installs and runs nicely in Waterfox. However, we cannot predict how long this will actually work.

Release of XLT 4.7

Summary

Xceptance released version 4.7 of its load testing and test automation product Xceptance LoadTest. It features a wide range of improvements and new functionalities.

Here is a selection of the most important changes.

  • Script Developer fully supports test data sets now
  • Script Developer learned “post steps” for easy cleanup or tear down of test cases
  • Script Developer and the framework support the new echo command for enhanced debugging
  • Load testing supports a variable load factor function to easily and globally vary load quickly
  • Load test reports display DNS resolution time
  • Load test reports display error charts per error type and response code
  • Data tables and charts for custom values display more details
  • Data tables have sticky headers for easier data evaluation
  • Load test reports can be created for specific test scenarios by filtering to include or exclude specific scenarios during report creation
  • XLT can log method and POST data to its timer files
  • All core libraries have been updated for improved browser support and improved performance
  • The new Amazon data center in Mumbai, India (ap-south-1) has been added

See below for more details of the highlights. Make sure to read the full online release notes.

As always, this upgrade is free for everyone.

Continue reading Release of XLT 4.7

Conditional Expressions in XLT

Motivation

Did you ever have to create multiple versions of your test cases to accommodate small differences of your test objects? Looking for a trade-off between good testing practice and minimizing project complexity. The following blog post reflects on this challenge and introduces you to a potential solution: Conditional Expressions.

Introduction

Xceptance introduced its test automation and load testing tool XLT 4.6 in February 2016 and with it we brought you conditionals. Today we want to shed some light on this new feature, the discussion that came along with it and why we finally decided to introduce it. This blog post will equip you with everything required to employ conditional expressions in your test scripts.

In computer programming, a condition or conditional expression performs an action depending on whether a given statement (the condition) evaluates to true or false. The programmer has the possibility to execute a part of the program only if certain circumstances are met. Now don’t worry, you do not need to become a full-fledged programmer to create your test cases with XLT Script Developer. But you should not skip on the possibilities this new feature is offering.

The Challenge

In testing typically you want your flow of execution to be linear, deterministic and transparent. The individual execution steps of your test case should be well-defined and yield the same results in a constant environment. If one execution step fails – e.g. an assertion does not check out – the whole test case always breaks and evaluates to failed. Run, rinse and repeat.

On the contrary often enough your real world test cases need to cater various scenarios. Think multi-region support of your page for example. Area specific content and functionality can quickly bring you into a catch-22 situation. Follow good practice in test case design, but deal with complexity and organizational nightmares in your test suite. Tiny differences in your test objects force you to keep multiple versions of your (already lengthy) test cases. Farewell maintainability!
Continue reading Conditional Expressions in XLT

Xceptance Releases XLT 4.3

Xceptance released version 4.3 of its load testing and test automation product Xceptance LoadTest. It features a wide range of improvements and new functionality.

As always, this update is free of charge for every user. You can use XLT freely for your daily automation work, regression testing, and performance validation. If you are an eligible Open Source project, you can get a full XLT license for your load and performance testing needs for free as well.

Script Developer

Enjoy more automation with less clicking. The new information panel summarizes the specifics of the currently selected item, so you do not need to open it anymore. Additionally, the new log panel lists all executed commands and their respective parameters.

If an element is styled with the CSS property text-transform, then the element’s text on the screen may have different character casing from what is defined in the page’s DOM tree. The WebDriver specification mandates to return the text as shown on the screen. Script Developer will now record text with the the character casing that appears on the screen and it will also take the CSS property text-transform into account when replaying text assertions.

Reporting

The XLT load test reports have been improved by adding more charts and data details, such as overall statistics for all timers. Arrival rate charts have been added to visualize the load development over time and make sure the desired load factor was reached.

Charts can now optionally be configured to be logarithmic. A capping can be added to hide larger spikes, which usually make charts hard to read, without removing the important information altogether.

The new networking section summarizes all general network-related statistics and charts on a separate page.

The error and event page has been redesigned and includes a new error chart that contains separate graphs for all transactions/actions/requests so that the temporal distribution of transaction/action/request errors is displayed in one chart. A new error summary table groups all errors by their error message to help you see which types of errors occurred and how many of them.

Load and Performance Testing

The master controller features two new commands to validate the availability of agents and  display their current configuration. A new command line option permits the skipping of result downloads when XLT is used as a load generator only.

Framework Extensions

A set of new commands is available in Script Developer and in the framework: commands such as assertAttribute, assertStyle, assertClass, assertEval, and their matching store and wait equivalents.

XLT can now send and receive IDs as part of the request to ease the correlation of server-side logs and test results. XLT may send a randomly generated alphanumeric ID as request header or extract such an ID from an arbitrary response header.

When dealing with different test environments, different load profiles, and/or different test data at the same time, managing different combinations of configuration settings can be challenging. The new property file include feature makes it easier now to predefine aspects and reuse them later in different configurations without copying and pasting.

When a test case reads a certain setting from the configuration, the framework uses a fallback strategy when doing the property look-up. This strategy performs an additional look-up step now, based on the transaction name (the short name to which the full class name is mapped). This additional step lets you parameterize different transactions differently, even if they are mapped to the same class and therefore share the same code.

The webdriver that will be used when executing functional tests can now be configured via properties. This allows greater flexibility and hardcoding is not necessary anymore.

When XLT executes XML script test cases with a WebDriver instance that is capable of taking screenshots, it can take a screenshot after each action, if desired.

Result Browser

New Result Browser UIA new look matches the overall styling of reports. Also, the navigation bar can now be resized, requests are color coded to visualize the content type, and the first page is displayed automatically. URLs are now active links, so you can click them easily.

If needed, the content of a response can be beautified for certain content types (HTML, JavaScript, JSON, CSS). This includes formatting and syntax highlighting.

Misc

  • XLT is shipped with an empty test suite project that can be used as a template for your own projects.
  • HtmlUnit has been upgraded to version 2.12.
  • WebDriver/Selenium has been upgraded to version 2.39.0.
  • EC2 admin console permits setting a tag name now.

Amazon Machine Images

The Amazon Machine Images (AMI) listed below are available for public use. Using these images is free of charge, but require that you own an Amazon Web Services (AWS) account. Please make sure that your EC2 security group permits communication on port tcp/8500. AMIs with Java 6 are no longer provided.

  • eu-west-1 : ami-ceae46b9
  • us-east-1 : ami-2510394c
  • us-west-1 : ami-e4ccfca1
  • us-west-2 : ami-9ebadeae
  • ap-southeast-2 : ami-617be45b

If you need XLT-AMIs in Tokyo, Singapore, or Sao Paulo, please let us know.

Purchase Licenses and Support Online

Licenses and support can be conveniently ordered through our XLT Self Service Center. You can instantly download licenses and purchase support right when it is needed. All your invoices and previous licenses (when purchased online) are accessible at any time.

We will notify you before your license or support runs out, so that you will never miss that again. This enables you to continue your daily automation and load testing work without interruption. Please note that we do not renew your contracts automatically, so no strings attached.

Visa and MasterCard are accepted. All credit card data is processed and secured by Wirecard.

Where to get it

More information about this release, the Quick Start Guide, and the Manual can be found in the release area. The full XLT 4.3.0 package can be downloaded here.

This upgrade is free of charge for everyone.

Test Automation – Here are our Thoughts on it

XLT Script DeveloperToday’s article can be seen as a survey reflecting our thoughts on web test automation in general. It basically lists personal experiences that we were able to gain in customer projects or conclusions that we arrived at in recent discussions on this topic.

Feel free to use these ideas and thoughts as input for your own discourse on test automation, whether it is about starting it, keeping it running, or just questioning it every once in a while.
Continue reading Test Automation – Here are our Thoughts on it

Update: Test Automation for Demandware SiteGenesis

SiteGenesis Test Automation

We received a lot of positive feedback on introducing our sample test automation suite for Demandware SiteGenesis. Have a look at the original post containing a video tutorial and all information you will need to get started.

It’s our intention to keep the test automation suite up-to-date, so it covers new and additional features of SiteGenesis. The latest update for version 13.3 of SiteGenesis contains the following improvements:

  • Support of the new multi-ship feature of SiteGenesis
  • Better independence of SiteGenesis product data
  • Stability improvements to make the test suite much more stable with reference to timing issues
  • Support for testing in Chrome and Firefox remotely without Script Developer
  • Introducing of ANT support for easier integration into build automation

You can download the test suite here: Test Automation for Demandware SiteGenesis 13.3.

P.S. We do not provide instances of SiteGenesis for testing. You have to be a signed up partner or customer of Demandware to get your own test instance.

Web Drivers in XLT: Basic Access Authentication

Today’s article of our WebDrivers series deals with HTTP authentication – a topic that, at first sight, seems to be very specific and of minor relevance. However, in the world of software testing it’s way more important than you’d think.

HTTP Authentification IE
HTTP Authentication IE
Often you will have an additional testing instance of a website to be tested. These instances are protected from abuse which is why they require credentials before you can access them. See below for an example in Internet Explorer:

This browser dialog appears just once. If you’ve entered the right credentials, you can access the related website as often as you like without further authentication – as long as you don’t reopen the browser. The latter is a critical issue for automated WebDriver testing.
Continue reading Web Drivers in XLT: Basic Access Authentication

WebDrivers in XLT: Test Case Design that Compensates for Inconsistencies across WebDrivers

Still remember the first post of our article series? It talked about how you can run XLT test cases in different browsers. If you’ve done so already, you might have noticed that the behavior of test cases developed in Script Developer sometimes differs depending on the WebDriver you’ve chosen. This article is meant to help you resolve such issues.

First of all, what’s the reason for these inconsistencies? Web browsers differ in their characteristics, such as site representation or functionality, due to their varying support of web technologies like CSS or HTML. You probably know that there is much more we could list, but the major point is pretty obvious already: using WebDrivers for test case execution calls real browser instances. Logically, you’re faced with the same differences as in real web browsers. It’s impossible to achieve a completely consistent web browser behavior; yet you can design your test case as outlined below to at least reduce the differences.

Time: Enlarging time properties can prevent timing problems with dynamically loaded content, like iframes or page content added with JavaScript. Script Developer lets you set an ‘Implicit Wait Timeout’. It allows to specify how long the test should wait for a command to fail while trying to find elements.

Continue reading WebDrivers in XLT: Test Case Design that Compensates for Inconsistencies across WebDrivers

WebDrivers in XLT: How to Run Test Cases in Multiple Browsers

Update: Before you continue reading, here is a new version of this test suite and article: Multi-Browser-Suite for Test Automation. It makes things a lot easier.

Have you heard of XLT Script Developer yet? If you have, you’ll probably agree that it’s a convenient tool to record and run automated test cases. However, with it being a Firefox plugin, you’re basically bound to run your test cases in Firefox. Wouldn’t it be nice to reuse Script Developer test cases in multiple browsers? You can actually do so by taking advantage of the WebDriver API that is part of the XLT framework.

If you don’t know much about WebDrivers, you should continue reading this article – the first one of a little article series on WebDrivers that hopefully gives you a good introduction to the topic.
Continue reading WebDrivers in XLT: How to Run Test Cases in Multiple Browsers