Xceptance released version 4.5 of its load testing and test automation product Xceptance LoadTest (XLT). This release enables you to write better test documentation, delivers more scripting capabilities, and shows really cool charts.

Script Developer has been enhanced to let you maintain the documentation of your script library directly in Script Developer. This way you can create and maintain your test documentation together with the code and don’t need to be able to read the code to understand the test cases.

Load testing is even more fun now. XLT can send performance metrics to Graphite, a data collection and graphing tool. From now on you can thus watch the most important performance results in real time while a load test is still running.

We also did some fine-tuning of our reports. Most notably, the runtime charts now visualize the distribution of values and show how this distribution varies over time.

Users of Amazon’s EC2 cloud will be happy to hear that the ec2_admin tool can now also be used in scripted processes, making it possible to fully automate the starting and stopping of Amazon machines.

The scripting capabilities in Script Developer and the XLT framework have been extended. There are new commands to operate an application’s context menu. Package-level test data, a feature unique to XLT, which was previously accessible to test cases only, can now be used from script modules as well.

As usual for a major release, the technology stack the XLT framework is built upon has been updated and now provides the latest WebDriver support. Last but not least, the XLT Jenkins plug-in can now automatically start and stop agent machines in Amazon’s compute cloud.

As always, this update is free of charge for every user. You can use XLT freely for your daily automation work, regression testing, and performance validation. If you are an eligible Open Source project, you can get a full XLT license for your load and performance testing needs for free.

Download it now!

While privacy laws are still a work in progress, the EU and Germany have made several statements about what is right or wrong. But the current state of regulation is unclear about what actually is permitted or has to be provided.

Xceptance takes the privacy of our users seriously. We decided to implement tracking and privacy for our website xceptance.com differently than what is usually seen on the market: we provide users with more choices, and we’re stricter about respecting your decisions, yet without pestering you with lots of button clicking.

Our Approach in a Nutshell

There are no clear rules in Germany and across Europe that define if the user has to actively agree to cookies (opt-in) or if he or she has the choice to decline cookie usage (opt-out). Cookies are just one issue of course, because analytics works even without cookies.

On Xceptance.com we decided to use an intermediate approach that does not require actions when you agree, but still gives you the time to make a decision.

Visiting our page initially doesn’t include any analytics. You have the possibility to choose to be part of tracking or not. We offer an opt-out option to the user but also ensure we don’t track anything before you make a decision.

Everyone is Welcome

We’ve seen several versions of tracking agreements on websites. A common approach is to inform the user with a pop-up, overlay, or info box that the site uses cookies and tracking solutions. If the visitors don’t agree with this they are free to leave. In other words, accessing the website is only possible if you agree to be tracked. Additionally your first pageview was already tracked in most cases, so you do not have really a choice after all.

We decided to let people visit our site even if they don’t want to be tracked. If visitors don’t agree to be part of our analytics we do not gather any data, but they can still browse and enjoy our entire site. […]

Like most companies we use and benefit from open source projects from all around the world. To actively participate and give something back, we started our own open source project that others can hopefully benefit from as well: Our website Xceptance.com has been open sourced.

In building our new website, we faced different requirements and challenges, like easy to maintain pages, mostly static content, a modern design, multiple languages, multiple domains, and so on.

We decided to build a website which is statically generated, so we can avoid annoying software updates and security issues, get the fastest delivery possible, and have everything versioned in Git. The best solution for static content websites we found back then was Jekyll. There are so many solutions out now, but Jekyll is still the most popular.

Jekyll is a great tool, but it has its limitations, such as language-specific content and multi-domain usage. So you will find a couple of plugins we used in the code base, as well as some code in the templates to achieve a great menu.

For a clean, modern design we used other projects such as Bootstrap, Less, and FontAwesome.

On GitHub you can see the result of our work. This is a version that only deviates a little from our actual website. We watermarked all images and blurred the employee photos to make the copyright handling easier.

All the code is licensed under the MIT license. All content and the design is licensed under Creative Commons by Attribution-ShareAlike 4.0.

We invite you to learn from it, contribute to it, or simply use it as starting point for your own projects. But please obey the license.

On January 19-20, 2015, Xceptance held its first XLT Summit. About thirty representatives of well-known companies from a variety of fields, including finance, retail and eCommerce, all of whom use XLT to help them in developing web-based software applications, accepted our invitation to meet at Xceptance’s headquarters in Jena, Germany. The goal of the conference was to give XLT users the opportunity to discuss their experiences using our tool both with each other and with the internal XLT development team.

To kick things off, on January 19 we enjoyed a casual dinner in a local restaurant. The next day there was a varied program consisting of presentations contributed both by Xceptance colleagues and external users. Our customers reported on their practical experience using XLT, and had a chance to discuss their results with the other users. The many different applications of XLT–from load testing to test automation to application monitoring–were thus illustrated in light of actual day-to-day practice. A selection of these presentations can be found here: http://goo.gl/fPJj1Q, and parts of the presentation provided by our guests from OTTO.DE can be read here: http://dev.otto.de/.

In addition, the XLT development team presented the newest XLT features. Our guests had the opportunity to speak directly with the developers about their specific feature requests, some of which we were able to immediately agree to implement.

In short, the conference was a great success. In response to the overwhelmingly positive feedback from the participants, the XLT Summit will now be a regular annual event.

Last but not least, we would like to thank all the attendees, without whose engagement, interest and suggestions the event would not have been possible!

My answer to this question is a very strong “yes”. You might want to limit yourself a little in the overall validation, but checking response codes only is a strong fail in my opinion. Additionally, just checking the result by checking a single phrase or word is not enough.

Reasons and Examples

• Modern web implementations often incorrectly return application status pages with response code 200.
• How do you ensure that you got the entire page back and not only the first 75%?
• Imagine an e-commerce search that breaks under load and instead of saying “I found 200 matches”, it returns a page saying “no matches found, did you mean …”. The latter is still a valid page but your load test will not discover the flaw.
• If you implement real page flows, meaning you don’t have hard-coded URLs that you fire against the server and instead take the data from the page and apply randomness to get a broad sample, you might find pages that are incorrect or inconsistent, but which are hidden outside normal functional-testing click paths.
• How can you distinguish a “login succeeded” page from a “login failed” page when you do not validate for content? Both are valid scenarios and “login failed” might not return an HTTP code different from 200. Even when it is designed to return “get lost”, aka HTTP code 401, it might break under load and incorrectly return 200.
• Always assume the worst case: always assume that the application will behave differently under load. Not only timing-wise but also behaviorally. This is one reason for running load and performance tests.

A Tale from the Past

In my life before Xceptance, a colleague was testing something for a new web component. He said that the new implementation was better, because it got faster when he applied more load.

What? Well, he did not check the content of the returned page. Under higher load, he got error pages back and incorrectly assumed great performance instead of a broken application.

How We Handle Validation

Because I am with a web testing company (Xceptance) specialized in load and performance testing, here is what we generally do for all our load and performance testing:

• Validate the response code
• Validate the basic structure of the page: check that the main components are there, such as header, footer, search box, mini cart, user account link, etc.
• Validate the completeness of the page: in other words, the closing HTML tag should be there or the JSON or XML should be complete (hopefully also valid)
• Validate the most important points of the current activity, e.g:
  • if you search for an existing product, you have to find it
  • if you expect to get no search results, check that you get the correct message
  • if you add something to the cart, validate that the quantity changes correctly
  • if you check out, validate the order number format (under perfect conditions even the uniqueness or at least the change between checkouts); validate that all price information is a valid format (not necessarily calculated correctly, although that could be required under certain test conditions); check that the (random) user name used is applied during that visit (for billing, shipping, order, account, etc.)
  • if you log on, verify that from then on, you always see the name of the user on the page (the famous “Welcome Mr. Foo” or something like that)
  • verify that you do not see a “Welcome Mr. Foo” message before logging in, that you do not have a cart when you first arrive, and so on

We have so often found problems that are load related or often also functionality related that every single validation step really pays off again and again.
Only when applications return bad stuff and the implementation crew does not want to change that and cannot be convinced or forced to clean up the mess, do we apply exceptions and reduce the validation.

These are just examples, of course.

Verdict

You should extensively validate during load and performance testing. Always assume both that the application will break under load, and also that the failure is mostly not reflected in a status code 500. Status codes 404 and 500 are for the beginner; the real professionals find problems beyond that.

We usually find functional problems during our load and performance testing that would have stayed undetected without proper validation. Of course, don’t forget the classical erroneous performance behavior such as inconsistent response times, spikes, and everything else imaginable.

Or in other words: You do load and performance testing to find the unusual and unexpected. This is often not reflected in increased response times and error codes.

Of course there are tests when you really are hunting for something specific, and when you can easily skip most of the validations, but take it from us: never strip them out completely.

This article was created as response to a question on StackExchange: Should load test validate functionality?

The recent events of security problems with WordPress motivated us to speed up the effort to move our blog into the HTTPS only mode. This does not fix WordPress security problems, but makes it easier to spot possible problems later and might protect us against the negative outcome from time to time.

So we set the Strict-Transport-Security header to tell the browser that HTTP is off-limits for this site once and for all. We also turned all links to our blog from our website and internally in the blog including resources into HTTPS links.

Also we set .htaccess redirect rules to force all users to use HTTPS and only hit the domain that has the proper certificate.

###
# Send .de to .com
RewriteCond %{HTTP_HOST} xceptance\.de$ [NC]
RewriteRule ^(.*)$ https://blog.xceptance.com/$1 [L,R=301]

# Force https
RewriteCond %{HTTPS} !=on
RewriteRule ^ https://%{HTTP_HOST}%{REQUEST_URI} [L,R=301]

# Set the correct header to tell the browser secure only
Header set Strict-Transport-Security "max-age=31536000" env=HTTPS

The only thing open is to get our hoster convinced to properly support TLS 1.2.

We proudly announce that Xceptance has a new website. Our 10th anniversary made us look back on where we are coming from, what we have been doing and what experiences we gained throughout the past ten years. It was time to have a new web presence reflect all that!

We took advantage of Bootstrap, Less, Jekyll, Git, Font Awesome, and Jenkins to create a website that primarily wants to help our visitors quickly learn about Xceptance, our services and our product. We wanted it to be modern but plain so that we can communicate what we do in the most comprehensive and user-friendly way possible. No boasting, no bragging, and just a little bit about ourselves. To have it all look nice and work smoothly for the mobile users as well, we used Bootstrap.

Since we’re always looking for new people that want to join us, we added a comprehensive jobs page which lists current open positions in both our offices, Cambridge, MA, USA and Jena, Germany.

Go check it out for yourselves! As always, we appreciate any kind of feedback!

The B59, that’s the 16-story office building that accommodates the headquarters of Xceptance Germany, now has a new addition to its front. The red and black neon writing with Xceptance’s name and signature X was put up a week ago and thereby joins the logotypes of the Postbank and Allianz insurance.

The workers who installed the sign definitely had to be free from giddiness, since the sign overlooks the city of Jena from a height of approximately 98 ft. (30m). Several curious bystanders watched the installation process and were able to witness how, piece by piece, the name emerged on the wall.

So now, three years after Xceptance moved to the B59 office space, our flag is finally set. We think it’s been worth the wait!

There certainly are things out there that money can’t buy, just like the Beatles once sang. However, you might agree with us that money can do good (things), if managed appropriately.

With the company anniversary in mind we at Xceptance sought to combine both the good deeds and a great company gift. That’s how we found out about the idea of a ‘Wortpatenschaft‘ (engl.: “word sponsorship”), a campaign dedicated to the preservation and promotion of the German language (in cooperation with the German nonprofit association Deutsche Sprache e.V., Dortmund).

People interested in the German language can visit their website to look up their favorite word and, this is the best part, sign up to “adopt” it. For a fixed donation you get to be the sponsor of a word (provided that nobody else has claimed it already), you receive a certificate for the sponsorship via mail and you can also have your name listed on the website next to your word. Their database not only consists of common German vocabulary but also rather quaint expressions or idioms like ‘Kokolores’ (engl.: gibberish) or ‘Mutschekiepchen’ (engl.: ladybug). The library is constantly growing since the platform accepts new suggestions, as long as they are valid German words or sufficiently Germanized expressions. Thankfully we did not have to propose our favorite word, and it also wasn’t taken!

As a QA company we did not have to think twice about which word to choose, it just had to be a term that means a lot to us, that describes our passion best and reflects what we’re good at. In one word: Testen (engl.: testing)! Now we are the proud owners of a pretty certificate, neatly framed and hung up in the hallway of our German office. A link to our company homepage next to the sponsored word also tells people how to find out more about the sponsor.

Ten years and 230 projects later it’s time to look back at the beginnings of Xceptance. Let us take you on a quick trip down memory lane!

The Story

In the beginning there was a small group of four colleagues who decided to take a chance and try something new. They pooled their resources, knowledge and QA experience and struck out on their own. The beginnings of Xceptance are thus all about passion and commitment and the belief in making software better.

The very first company meetings were held in the living room of one of the founders, lending the enterprise a definite ‘start-up feeling’. The business plan proposed adding just two people in the first five years, but that didn’t quite work out as expected because five years later more than 20 people worked for Xceptance, the majority of whom are still with the company to this day.

The Name

There are several anecdotes about the origins of the company name as well. The name ‘Xceptance’ is based on the term ‘acceptance testing’ which stands for a certain way of testing to make sure the requirements of a contract or specification are met. As obvious as that might sound, this wasn’t the first choice. Names of constellations, stars, and animals were all on the shortlist. (Un)fortunately the domains for those names were all taken, and that’s how ‘acceptance’ became the front-runner. For a fancier version the ‘a’ became an ‘x’; that turned into ‘Xeptance’, then ‘Xcceptance’, until someone suggested that just one ‘c’ might be enough. That is how ‘Xceptance’ was born.

The Office

After graduating from meetings in cafés, kitchens and living rooms, we had a stint in a shared office with a befriended company and then Xceptance moved to its new quarters in the center of Jena. Once again a shared office, but at this point with Demandware.

In those days people at Xceptance had to close ranks; there was just one relatively small office space for almost a dozen employees, but it worked out well and the company kept growing. As a result, people at Xceptance had to wrap up their equipment once again to move to the current location a few blocks away from the old office. The movers had been booked for the next day and everybody expected to spend one last day working in the old office. That plan, however, got monkeywrenched by the internet service provider who cut services at the old office a day early. The lucky Xceptance people had no choice but to pack their essentials themselves and schlep everything over to the new office.

The People

Eighty percent of the employees who started in the first three years are still with us. They’ve led Xceptance to today’s success, and their loyalty means a lot to us. But we also would not be where we are now without all our more recent hires. They ensure that the company’s head and heart are constantly in motion, always developing, never resting. Plus we’re always looking for talents with what we call the tester gene. Check our open positions in Germany and the US. We’re looking forward to meeting you!

The Tool

In the beginning of Xceptance, we did not have a performance testing tool. If a customer needed testing, we had to buy or license one. At that point, we were still a startup ourselves and we worked for startups, so money was short and all the good tools were just too expensive. So we decided to start our own tool chain, called ‘Yart’ back then, for ‘Yet Another Regression Tool’. Once this little hack proved to be useful and promising, we created a product development department around it and a little later XLT (which stands for Xceptance LoadTest) surfaced as our product offering.

Today XLT is a player in the test automation and performance testing space with many unique features, great flexibility, and results you can trust.

The Facts

• 2004: Xceptance GmbH founded in Jena
• 2006: Xceptance Inc. founded in the U.S.
• 2007: Release of XLT 2.0
• 2008: Release of XLT 3.0, Xceptance joins the blogging society
• 2009: 5th anniversary of Xceptance Germany
• 2011: Release of XLT 4.0, 5th anniversary of Xceptance Inc., Move to our current office in Jena
• 2014: Release of XLT 4.3, 10th anniversary of Xceptance in Germany

As you can see, Xceptance has come a long way to offer you the best testing services possible! Ten years are an entire decade, but to us it feels like time just flew by and it was only yesterday that the business plan was first drafted. We hope to continue our hard work and we’d like to thank you, our clients and partners, for your confidence and support.

We’re looking forward to the next 10 years (at least)!