Category Archives: Testing

Multi-Browser Support for Test Automation with XLT

Summary

In today’s post we will discuss the steps necessary to enhance an XLT-based test suite with multi-browser support. We will show how to tag your test cases to conveniently run them in different environments and execute the test suite in a local or remote fashion.

Introduction

Xceptance maintains a MIT licensed test suite at GitHub which demonstrates functional testing for large scale projects. With the suite we’ve put an emphasis on clear structures, naming and test case organization. Targeting Demandware’s SiteGenesis storefront at heart, the underlying concepts and mechanisms are valuable for everyone building test suites for comparable web applications with XLT. Next to being a template for test automation and best practices in test suite design, it can be a starting point ready to pick up in your very own projects. We regularly utilize it and want to encourage you to explore, employ and contribute.

A regular challenge in testing ecommerce applications is the variety of different browsers and platforms that are available today. As you probably know XLT, the test automation and load testing framework from Xceptance, is based on Selenium browser automation and the Webdriver API. Supporting multiple browsers therefore comes naturally. This blog post will demonstrate how XLT is able to streamline different testing environments directly in your test suite. You will learn how to execute your tests locally and remotely with the help of Sauce Labs and similar automated testing platforms. Along the way you will pick up some details about XLT as well as Script Developer and quickly find yourself equipped with a ready to use multi-browser test suite example.
Continue reading Multi-Browser Support for Test Automation with XLT

Special Characters in Script Developer

Recently we received a support request regarding special characters in Script Developer. Perhaps other XLT users stumble across a similar requirement, so it’s a good idea to make the discussion available to the public.

First of all, some bad news: Up to now, Script Developer does not have explicit support for special characters, such as Line Feed (\n), Horizontal Tab (\t), Backspace (\b) or similar. For example, typing multiple lines of text – each line delimited by a newline character – into an element on your page is not possible just like that. Upon loading your script, XLT Script Developer normalizes all white-space characters contained in the target or value field of any command.

Of course, we don’t want to leave you out in the rain but provide a feasible solution.
Continue reading Special Characters in Script Developer

Tracking and Ad-Blocker Software

Summary in 64 words

Ad blockers are widely used tools these days. Users install this software to improve loading times of websites or to simply remove intrusive advertisement. But what does this mean for us as a testing company? We saw some sites break and are now including tests with enabled ad blockers. This helps our customers to see where problems might occur and to improve user experience.

Introduction

The amount of advertising increases and more and more users take this matter in their own hands and install ad blockers. Users also become privacy-conscious and do not want to expose their browsing behavior to an ever increasing number of companies. Facebook’s, Twitter’s, and Google’s ad or content pixels are on nearly every page nowadays. Adobe collects data as well with font services, Scene7, and Omniture. There are lots of big and small third parties that live on pages and are included as a service. We have seen websites with over 50 third party services on a single page of their web shop.

Continue reading Tracking and Ad-Blocker Software

Our Demandware SiteGenesis Community Test Suite

The community test suite homepageToday, we are happy to announce the release of our new Demandware SiteGenesis Community Test Suite! It is a test suite for automated storefront testing of the Demandware SiteGenesis reference e-commerce storefront.

The test suite’s intention is to share experiences, transfer knowledge, and to demonstrate best practices in test automation. The test suite is built using XLT of course. XLT is freely available and extends concepts you already know from Selenium.
Continue reading Our Demandware SiteGenesis Community Test Suite

Should Load Tests Validate Functionality?

My answer to this question is a very strong “yes“. You might want to limit yourself a little in the overall validation, but checking response codes only is a strong fail in my opinion. Additionally, just checking the result by checking a single phrase or word is not enough.

Reasons and Examples

  • Modern web implementations often incorrectly return application status pages with response code 200.
  • How do you ensure that you got the entire page back and not only the first 75%?
  • Imagine an e-commerce search that breaks under load and instead of saying “I found 200 matches”, it returns a page saying “no matches found, did you mean …”. The latter is still a valid page but your load test will not discover the flaw.
  • Continue reading Should Load Tests Validate Functionality?

SQE training week in Boston

SQE Training TableAt Xceptance we think that “you live and you learn”, so we were eager to read about an upcoming training week by SQE in Boston (March 24-28). The various tracks had promising headlines like ‘How to Break Software: Robustness Testing Unleashed’ or “Testing Under Pressure” but we decided to pick our favorites and chose “Exploring Usability Testing” and “Mobile Application Testing”.

The usability training was scheduled for just one day, the mobile training for two. We knew that it would be tricky to fit an overview of usability testing in just one day of training, and it turned out the instructor was well aware of these concerns.
Continue reading SQE training week in Boston

Use XLT with Sauce Labs and BrowserStack

Sauce Labs and BrowserStack – What Are They and Why Use Them?

This approach still work fine, but we came up with a much better one. Head over to GitHub and see our Multi-Browser-TestSuite for XLT. It will make multi browser testing a breeze. By the way, all the code is licensed under the MIT license, so absolute flexibility for you.

Sauce Labs and BrowserStack allow you to run automated test cases on different browsers and operating systems. Both provide more than 200 mobile and desktop browsers on different operating systems. The benefit? You can focus on coding instead of having to maintain different devices. You can easily run your test cases written on iOS on an Internet Explorer without actually buying a Windows device; and last not least, you don’t need to worry about drivers or maintenance.

By the way, Internet Explorer even seems to run faster at Sauce Labs than on a desktop machine. Also note that Sauce Labs supports Maven builds.
Continue reading Use XLT with Sauce Labs and BrowserStack

Why do women make good software testers?

The answer to this question is fairly simple: women make good software testers if they’re good at testing. And so do men. There would be no reason to go into this any further if it weren’t for the occasional blog posts that pop out here and there, carrying headlines like ‘Are women better testers than men?’ or ‘Is software testing women’s work?’ (if you don’t believe that, go ahead and ask Google!). We are aware that this is a tricky topic, and bringing it up usually seems to imply that sexism and discrimination are just around the corner. But instead of jumping right at it and listing possible skills that may or may not make women better testers (who hasn’t heard of ‘multi-tasking’, ‘emotional intelligence’ or ‘an eye for detail’?), we’d rather tell you about our own experiences.

At Xceptance we just think our team is great the way it is. There are women and there are men, and everyone contributes to the company’s success in their own way. Our employees do a great job listening to customer requests, they passionately discuss appropriate testing strategies and they sometimes make phone calls while simultaneously updating browser versions on testing devices and formatting the latest bug report. Some of our employees love snowboarding in their free time while others like cooking and crafts, and we can assure you that there is nothing gender-related about all that. So if there are more women than men in software testing, that’s great! But it doesn’t mean anything besides those bare statistics.

Image by kevinshine under CC-BY-2.0Of course now you could ask: if gender doesn’t matter, why make a blog post of it? Admittedly, that is a good question. But with all those articles out there about women in software testing and the accompanying stereotypes we just felt like we’d have to take a stand and outline our own opinion. We’re not big fans of the ‘you say it best when you say nothing at all’ attitude, so we figured we could as well go for it. And with all of the above being said, there’s just one thing we have to confess: every year on March 8th all of our female employees receive a chocolate treat in honor of International Women’s Day.

Photo by kevinshine under CC-BY-2.0.

Another Six Things You Should Never Say to a Software Tester

Inspired by this UTest article 6 Things You Should Never Say to a Software Tester, we came up with additional six comments a software tester does not really like to hear:

But it works on my development machine!

We don’t doubt that, really. But a sandbox is called a sandbox for a reason. It’s a playground, a simulation of reality, with its own rules and regulations. Kids fight about sand castles first before moving on to arguing about real estate in the grown-up world, so please throw away your shovel for a second and join us in the real world!

You broke it.

“No, I didn’t!” – “Yes, you did!” – “NO, I DID NOT!” and so on… That was me and my friend in kindergarten, at the age of 4. The toy was in pieces and none of us could (or would) recall whose fault it was. And it didn’t really matter at this point. The more important question was how to hide the broken thing from the teacher to avoid any unpleasant consequences. Once we had figured that out, we were done with the argument and started working together. A good example for how you can learn from past mistakes.

Your test report says you executed all of the test cases, so there won’t be any new bugs from now on, right?

“You’ve eaten all the pasta, that means you’ll never be hungry again, right?” – If you think this is an absurd example, read the third statement again. See the analogy?

You’re testing e-commerce sites? Oh, so you’re basically shopping online all day long?

Yes. And no. I don’t need 100 luxury bathrobes embroidered with profanities, and I also don’t want them. I also don’t have a second home in Hawaii and I know that entering incomplete billing information won’t take me to the next step in checkout. And yet I simulated all these scenarios while testing. Testing an e-commerce site is different from ordinary online shopping, especially since the term “ordinary online shopping” has yet to be defined. Am I the grandmother who just got an iPad for Christmas and is now looking for bargains? Am I the hipster who is annoyed by all the promotions that are thrown at him while he just wants to get his shopping done? Is the website prepared for the most challenging checkout attempts? It is our job to find that out, and it has little to do with “online shopping”. Though I did try to pay my last purchase at the farmer’s market with a test credit card, but that’s another story…

I see your point, but it’s not a requirement.

So it’s not a requirement that the password field should be cleared out after submitting the form? Instead of treating the documentation as a bible, we’d appreciate the use of common sense more often. We know that there’s the tight schedule and the even tighter budget and that those things don’t allow for many extras. But, believe us, a password that floats around unattended on a page is not an extra, but it can hit you hard when it comes back to you later.

This is no longer a bug, please see the updated requirements.

This is even worse if it comes along with the previous statement. It should not be an acceptable solution to turn a defect into a requirement. If it happens and the alleged bug turns into a butterfly, that is if the requirements change, please! let! the! testers! know! We don’t like to spend our time hunting fake bugs, therefore we need to know about any updates as soon as possible. And yes, this also includes design changes!

What else comes to your mind?

TestSuite-NoCoding – Load Testing with CSV Files

Our test suite on GitHubWe continue to share cool things with the community of software testers and developers. Today we are announcing the availability of our no coding test suite for XLT under the Apache License v2.0.

Introduction

You want to fire just a couple of URLs to load test your application? You have to investigate the performance problems of a feature and you need accurate measurements as well as a lot of load generated? You like XLT and its capabilities, but you don’t have the time to compile a sophisticated test suite from scratch? Whatever your motivation, our new test suite for XLT is the solution you are looking for.
Continue reading TestSuite-NoCoding – Load Testing with CSV Files