Category Archives: Testing

Should Load Tests Validate Functionality?

My answer to this question is a very strong “yes“. You might want to limit yourself a little in the overall validation, but checking response codes only is a strong fail in my opinion. Additionally, just checking the result by checking a single phrase or word is not enough.

Reasons and Examples

  • Modern web implementations often incorrectly return application status pages with response code 200.
  • How do you ensure that you got the entire page back and not only the first 75%?
  • Imagine an e-commerce search that breaks under load and instead of saying “I found 200 matches”, it returns a page saying “no matches found, did you mean …”. The latter is still a valid page but your load test will not discover the flaw.
  • Continue reading Should Load Tests Validate Functionality?

SQE training week in Boston

SQE Training TableAt Xceptance we think that “you live and you learn”, so we were eager to read about an upcoming training week by SQE in Boston (March 24-28). The various tracks had promising headlines like ‘How to Break Software: Robustness Testing Unleashed’ or “Testing Under Pressure” but we decided to pick our favorites and chose “Exploring Usability Testing” and “Mobile Application Testing”.

The usability training was scheduled for just one day, the mobile training for two. We knew that it would be tricky to fit an overview of usability testing in just one day of training, and it turned out the instructor was well aware of these concerns.
Continue reading SQE training week in Boston

Use XLT with Sauce Labs and BrowserStack

Sauce Labs and BrowserStack – What Are They and Why Use Them?

This approach still work fine, but we came up with a much better one. Head over to GitHub and see our Multi-Browser-TestSuite for XLT. It will make multi browser testing a breeze. By the way, all the code is licensed under the MIT license, so absolute flexibility for you.

Sauce Labs and BrowserStack allow you to run automated test cases on different browsers and operating systems. Both provide more than 200 mobile and desktop browsers on different operating systems. The benefit? You can focus on coding instead of having to maintain different devices. You can easily run your test cases written on iOS on an Internet Explorer without actually buying a Windows device; and last not least, you don’t need to worry about drivers or maintenance.

By the way, Internet Explorer even seems to run faster at Sauce Labs than on a desktop machine. Also note that Sauce Labs supports Maven builds.
Continue reading Use XLT with Sauce Labs and BrowserStack

Why do women make good software testers?

The answer to this question is fairly simple: women make good software testers if they’re good at testing. And so do men. There would be no reason to go into this any further if it weren’t for the occasional blog posts that pop out here and there, carrying headlines like ‘Are women better testers than men?’ or ‘Is software testing women’s work?’ (if you don’t believe that, go ahead and ask Google!). We are aware that this is a tricky topic, and bringing it up usually seems to imply that sexism and discrimination are just around the corner. But instead of jumping right at it and listing possible skills that may or may not make women better testers (who hasn’t heard of ‘multi-tasking’, ‘emotional intelligence’ or ‘an eye for detail’?), we’d rather tell you about our own experiences.

At Xceptance we just think our team is great the way it is. There are women and there are men, and everyone contributes to the company’s success in their own way. Our employees do a great job listening to customer requests, they passionately discuss appropriate testing strategies and they sometimes make phone calls while simultaneously updating browser versions on testing devices and formatting the latest bug report. Some of our employees love snowboarding in their free time while others like cooking and crafts, and we can assure you that there is nothing gender-related about all that. So if there are more women than men in software testing, that’s great! But it doesn’t mean anything besides those bare statistics.

Image by kevinshine under CC-BY-2.0Of course now you could ask: if gender doesn’t matter, why make a blog post of it? Admittedly, that is a good question. But with all those articles out there about women in software testing and the accompanying stereotypes we just felt like we’d have to take a stand and outline our own opinion. We’re not big fans of the ‘you say it best when you say nothing at all’ attitude, so we figured we could as well go for it. And with all of the above being said, there’s just one thing we have to confess: every year on March 8th all of our female employees receive a chocolate treat in honor of International Women’s Day.

Photo by kevinshine under CC-BY-2.0.

Another Six Things You Should Never Say to a Software Tester

Inspired by this UTest article 6 Things You Should Never Say to a Software Tester, we came up with additional six comments a software tester does not really like to hear:

But it works on my development machine!

We don’t doubt that, really. But a sandbox is called a sandbox for a reason. It’s a playground, a simulation of reality, with its own rules and regulations. Kids fight about sand castles first before moving on to arguing about real estate in the grown-up world, so please throw away your shovel for a second and join us in the real world!

You broke it.

“No, I didn’t!” – “Yes, you did!” – “NO, I DID NOT!” and so on… That was me and my friend in kindergarten, at the age of 4. The toy was in pieces and none of us could (or would) recall whose fault it was. And it didn’t really matter at this point. The more important question was how to hide the broken thing from the teacher to avoid any unpleasant consequences. Once we had figured that out, we were done with the argument and started working together. A good example for how you can learn from past mistakes.

Your test report says you executed all of the test cases, so there won’t be any new bugs from now on, right?

“You’ve eaten all the pasta, that means you’ll never be hungry again, right?” – If you think this is an absurd example, read the third statement again. See the analogy?

You’re testing e-commerce sites? Oh, so you’re basically shopping online all day long?

Yes. And no. I don’t need 100 luxury bathrobes embroidered with profanities, and I also don’t want them. I also don’t have a second home in Hawaii and I know that entering incomplete billing information won’t take me to the next step in checkout. And yet I simulated all these scenarios while testing. Testing an e-commerce site is different from ordinary online shopping, especially since the term “ordinary online shopping” has yet to be defined. Am I the grandmother who just got an iPad for Christmas and is now looking for bargains? Am I the hipster who is annoyed by all the promotions that are thrown at him while he just wants to get his shopping done? Is the website prepared for the most challenging checkout attempts? It is our job to find that out, and it has little to do with “online shopping”. Though I did try to pay my last purchase at the farmer’s market with a test credit card, but that’s another story…

I see your point, but it’s not a requirement.

So it’s not a requirement that the password field should be cleared out after submitting the form? Instead of treating the documentation as a bible, we’d appreciate the use of common sense more often. We know that there’s the tight schedule and the even tighter budget and that those things don’t allow for many extras. But, believe us, a password that floats around unattended on a page is not an extra, but it can hit you hard when it comes back to you later.

This is no longer a bug, please see the updated requirements.

This is even worse if it comes along with the previous statement. It should not be an acceptable solution to turn a defect into a requirement. If it happens and the alleged bug turns into a butterfly, that is if the requirements change, please! let! the! testers! know! We don’t like to spend our time hunting fake bugs, therefore we need to know about any updates as soon as possible. And yes, this also includes design changes!

What else comes to your mind?

TestSuite-NoCoding – Load Testing with CSV Files

Our test suite on GitHubWe continue to share cool things with the community of software testers and developers. Today we are announcing the availability of our no coding test suite for XLT under the Apache License v2.0.

Introduction

You want to fire just a couple of URLs to load test your application? You have to investigate the performance problems of a feature and you need accurate measurements as well as a lot of load generated? You like XLT and its capabilities, but you don’t have the time to compile a sophisticated test suite from scratch? Whatever your motivation, our new test suite for XLT is the solution you are looking for.
Continue reading TestSuite-NoCoding – Load Testing with CSV Files

XCMailr – An Open Source Test Mail Forwarder

As a software tester, an episode such as the one below must be more than familiar to you and, let’s be honest, it has the potential of making the top ten of the most annoying things in our daily work routine:

  • Pain in the neck: “Hey, I need more email addresses for testing, I just burnt all my own.”
  • You: “Well, just use a fake one.”
  • P: “Nah, I can’t, I need the activation emails.”
  • Y: “Well, then, there are good disposable mailers out there.”
  • P: “Very clever, but they aren’t protected by authentication and I signed an NDA for that project.”
  • Y: “Here, use this one.”
  • P: “But it wants to have a real email to sign me up and I don’t really feel like giving my real email away.”
  • Y: “$§5$!51z1hhsks!”

Granted, it’s a matter of course that committed testers have many email addresses but what’s the use of them when you’re always limited to a certain number, when you can’t quickly change them, or deactivate them when an email service got hold of them?
Continue reading XCMailr – An Open Source Test Mail Forwarder

Test Automation – Here are our Thoughts on it

XLT Script DeveloperToday’s article can be seen as a survey reflecting our thoughts on web test automation in general. It basically lists personal experiences that we were able to gain in customer projects or conclusions that we arrived at in recent discussions on this topic.

Feel free to use these ideas and thoughts as input for your own discourse on test automation, whether it is about starting it, keeping it running, or just questioning it every once in a while.
Continue reading Test Automation – Here are our Thoughts on it

Simplify Your Life With a Little UI Automation

Are you familiar with someone asking you to do things such as putting all of your clients into the new CRM or inserting 80 new users in a software product x? If so and if your software has no import feature, you probably found yourself copying and pasting all day long. A simple way to save you from this torture is using a UI automation tool like Sikuli Script.

What is Sikuli?

Sikuli automates UI functionalities like mouse, keyboard, and clipboard actions. It works with UI recognition to identify UI components such as buttons, input fields, or menus. To use this kind of recognition, Sikuli compares the actual screen with screenshots from the user.
Continue reading Simplify Your Life With a Little UI Automation

Web Drivers in XLT: Basic Access Authentication

Today’s article of our WebDrivers series deals with HTTP authentication – a topic that, at first sight, seems to be very specific and of minor relevance. However, in the world of software testing it’s way more important than you’d think.

HTTP Authentification IE
HTTP Authentication IE
Often you will have an additional testing instance of a website to be tested. These instances are protected from abuse which is why they require credentials before you can access them. See below for an example in Internet Explorer:

This browser dialog appears just once. If you’ve entered the right credentials, you can access the related website as often as you like without further authentication – as long as you don’t reopen the browser. The latter is a critical issue for automated WebDriver testing.
Continue reading Web Drivers in XLT: Basic Access Authentication