All posts by Andrea

Update: Test Automation for Demandware SiteGenesis

SiteGenesis Test Automation

We received a lot of positive feedback on introducing our sample test automation suite for Demandware SiteGenesis. Have a look at the original post containing a video tutorial and all information you will need to get started.

It’s our intention to keep the test automation suite up-to-date, so it covers new and additional features of SiteGenesis. The latest update for version 13.3 of SiteGenesis contains the following improvements:

  • Support of the new multi-ship feature of SiteGenesis
  • Better independence of SiteGenesis product data
  • Stability improvements to make the test suite much more stable with reference to timing issues
  • Support for testing in Chrome and Firefox remotely without Script Developer
  • Introducing of ANT support for easier integration into build automation

You can download the test suite here: Test Automation for Demandware SiteGenesis 13.3.

P.S. We do not provide instances of SiteGenesis for testing. You have to be a signed up partner or customer of Demandware to get your own test instance.

This is Just a Data Issue

Image © Juja Schneider

Many projects have their closed or “won’t fix” bugs commented with remarks such as:

  • “Content-related issue”,
  • “Won’t happen on live system”, or
  • “This is just a data issue”.

Both the development of new features and data maintenance may be accomplished in different environments. For testers, it’s thus difficult to decide on the cause for a certain issue: is a required text missing because pages aren’t implemented the right way or is it simply not there because the corresponding product data aren’t available in this particular test system? Very often testers work on development systems without having access to latest data. The following is a typical scenario:

  1. Testers report a bug.
  2. The development resolves this bug as content-related and won’t fix.
  3. Testers report further bugs of that sort.
  4. The development gets irritated: “We’ve told them that these are content issues!”
  5. The relationship between testers and developers might really go downhill from there, seriously threatening the project’s success: the credibility of the test team is damaged, developers  don’t take reported bugs seriously anymore, testers get overly cautious and stop reporting content-related problems…

Inevitably, potential bugs don’t get reported and reported bugs never get retested before the project goes live. This is even more true in poorly managed projects because no one really feels responsible to solve this problem.

Continue reading This is Just a Data Issue

Successful Software Testing – Communication is Everything

Image © Juja Schneider

Continue reading this article if one of the following statements applies to your software testing projects:

  • Comments about incompetent developers and nit-picking testers are normal.
  • What test data should be used and where it can be found is always unclear, regardless of how often it has been explained before.
  • The same questions are raised again and again by different people.
  • Features known to be incomplete are tested.
  • People report the same defect over and over again.

All these points may indicate that you communicate poorly in your project. As in real life, communication both within the test team and between the test and development department is the key to successful projects. After all, everybody has the same goal: creating software of high quality. Here are some useful tips that will help you to get the most out of your teams:

Communication between Testers and Developers

  1. Get in touch
    Especially in virtual teams often the only point of interaction between developers and testers are bug tracking system; yet it is especially the written word that causes misunderstandings and leads to pointless extra work. Regular calls and actually talking to each other can work miracles here.
  2. Clarify expectations
    Which information do developers exactly need when you find bugs? Which test areas are of highest priority right now? To avoid extra work, try to settle on some basic guidelines right from the beginning and make sure to communicate or change them whenever necessary.
  3. Create a culture of constructive criticism
    That´s probably the most important point of all. There is one goal and everybody is working on achieving it. Thus, each detected and eliminated bug is a step towards a better piece of software and should be seen as a  great team achievement.

Communication within Test Teams

  1. Get to know each other
    This might sound easy, but it´s surprising how little information testers have about each other, even if they work for the same company. This gets even worse if virtual teams are involved! Sharing information about competences and experiences will result in much more efficient work.
  2. Talk
    Another simple but often neglected point. Testing in a team will lead to great success if you share your knowledge and your findings. Group chats can work for some teams, but routinized and beneficial meetings for others.
  3. Share your test data
    Have you ever found yourself in a situation where testing a specific feature took a lot of time just because you had to ask for test data you could use? This certainly happens every once in a while, and it’s actually not a problem if only you need to do so. However, it does become an issue as soon as everyone else in the test team has to request the required data. So find a way to share you test data with all team members – it´s worth it.
  4. Instantly share news
    There is a new release? There is a temporary problem on one of your test systems? Make sure everybody knows about it.

Last but not least, a major point for everyone involved:

Speak the same language
There is bound to be misunderstandings if everybody talks about the same things in different words. The core of successful team communication is a mutual agreement on the terminology used within your project. Try to write it down, even if it causes extra work. It will help new people get into the subject.

A variety of tools makes communication much easier nowadays and doesn’t require much setup: use shared online documents, create mailing lists, and take advantage of chat systems.

To cut a long story short: cultivate a constructive work environment, make sure to always communicate, and share your knowledge. It´s that easy!

Test Case Design – The Unfinished Discussion

This is a topic whole generations of software testers have debated about. Test case design is a very subjective matter, depending hugely on the preferences of the test engineer and the requirements of the project. Although exploratory techniques are increasingly popular in the testing world, scripted manual testing used in many projects.

Image © Juja Schneider

When you are in charge of designing test cases you will have to decide which level of detail you want to apply to your test cases. This level of detail reaches from the vague description of a given scenario to a list of every single click a tester should perform. You can cover several actions in one test step or split them up into several ones. The more details you provide, the bigger your test suite will be at the end.

But which approach is the right one? This answer might not really help you, but: It depends!

Let’s talk about some advantages of detailed test cases.

  • The test case are very detailed.
  • You can ensure that less obvious features and scenarios are covered.
  • Even less experienced testers can perform them easily as a precise sequence of steps is specified. It is often stated that this becomes a candidate for outsourcing.
  • Testers who are not familiar with the target application will get a detailed overview about the system’s features.
  • Complex and critical key features can be tested very thoroughly based on the requirements and specifications.
  • There will be no room for interpretation whether a test case is failed or not.

Sounds good? It is… often. However we wouldn’t have started the discussion again if this is the answer. So what are the disadvantages?

  • The test case are very detailed. Yes, this sentence is supposed to be here.
  • You have to invest much more time and effort into creating the test cases.
  • The maintenance of your test cases is a much larger topic. Every time a small detail changes, you have to adjust the test cases.
  • You may miss creative testing approaches of skilled testers. Test case extensions while testing are unlikely to happen.
  • Re-using test cases in future projects is less likely as test cases are more project specific.
  • Testers may develop kind of a tunnel view. You may know this from your own experience: If you are forced to follow a specific path to reach your target, you may lose sight for details along the way.
  • The test results depend heavily on the person who created the test cases. If he/she didn’t do a good job, the executing testers won’t make up for this during testing. If the test case designed missed important steps, the testers will miss them too.
  • Providing detailed steps without room for alternative paths will magically merge a group of testers into one single tester. While they will follow precisely the steps when detailed test cases are provided, they might use a completely different approach when they have the freedom to look around.
  • If the results are precisely defined, nobody will question what he or she sees on the screen, so you miss the testing of testing.
  • Or taken to an extreme: Detailed aka step by step test cases will turn your testers into human test automation engines with the advantage of additional pairs of eyes. If you get lucky, your testers will report things they see but which are not stated in the tests, such as broken layouts for instance.

So what is right and what is wrong? Some guidelines would help, wouldn’t they? As there won’t be THE ultimate solution, here are a couple of hints about how to find the right way for your project.

  1. Keep the subject of your project in mind
    If you test a medical device, you probably have to have very detailed test cases to satisfy the regulations and document everything as good as possible, because there are lives at stack. Assuming you test just a DVD player, you might go with 20 relatively free and 5 detailed test cases instead.
  2. Consider the skills of your testers
    Highly skilled testers won’t need detailed instructions for all test areas. You may want to define test cases in a more pragmatic way.
  3. Think of your future user base
    If this is an extremely large group, where all levels of expertise exist, you want to reflect that in your testing without driving up the time needed to test. If your future user base is extremely experienced and have to go by the books anyways, such as pilots, you can reflect that in your testing.
  4. Specify the goal and not the way
    Try to avoid the step by step descriptions. Instead of sending someone down a menu by saying click here, here, here, here, and there, you just state: Navigate to X using the application menu.
  5. Challenge your application
    The later real world user will make mistakes you cannot anticipate, so add that jitter to your testing. Give your team room for creativity and for going individual paths.

Regardless if your project subject requires detailed test cases: Try to harvest the power of randomness, give your testers enough power to question anything anytime while keeping control of what has to be done.

The Bipolar Life Of A Software Tester – Continued

Eric Jacobsen from started this little rant about his bipolar life as a tester. You should read it, it is very entertaining. It describes precisely what we feel from time to time. So we felt encouraged to continue.

Image © Juja Schneider

Cool, developers have marked most of my bugs as resolved. Maybe we will be able to launch the project in time!

No, wait…

I will be busy doing retests most of the day. This sucks. I won’t be able to continue my scheduled test cases for today. Test management won’t be too happy with me. I hate doing all these retests.

I’m so proud, because I have completed all my test cases in record time and found so many bugs! I’m a testing ninja! I might be able to go home early.

No, wait…

What’s that? Test management has assigned a whole bunch of new test cases to me? Is this my reward for working quickly? Life is not fair.

But I found this really big bug minutes ago. Wohooo! No regular user will be able to work with that feature. It’s a usability nightmare! Hope they will fix this soon. No way they can go live with this one. …I’m a representative of “a regular user”, right? I won’t even look at the specification. This cannot be right!

No, wait…

I took a look at the specification. It is expected to work like that. The design agency sold this as “visionary approach”. What do do now?

Poor developers! I feel honest sympathy for them. All these bugs I submit really cause a lot of work.

No, wait…

Why can’t they build it right from the beginning? I have so much more work to do, just because they deliver a buggy system.

No, wait…

Would I have a job, if all developers would create perfectly well running software? I should be happy that they are a little sloppy sometimes.

Wait a moment, is this a bug? The weather forecast mentioned “light snow in the afternoon”. I would rather call it “heavy-snowish” – and it is pretty late too.

No, wait…

Maybe this is a “Works as designed”?

P.S: Feel free to continue.

Our Top 6 Software Testing Trends 2013

Happy New Year to everybody! Time to think about 2013 and the work ahead of us.

The ecommerce market is growing and becoming more competitive every day. This means, the customer experience is going to play an even bigger role in 2013. Online shops are expected to be stylish and beautifully designed – but customers are getting more demanding in terms of performance and usability on multiple devices.

The following topics are our point of view on the most important issue that will keep us busy in 2013.
Continue reading Our Top 6 Software Testing Trends 2013

XLT 4.3 Preview

We are looking forward to release XLT 4.3 soon. A lot of work has been invested to improve XLT and to provide a much easier way to gather all information from our powerful report engine. Have a look at some of the new features below.


New summary statistics directly on the overview page

It´s easier than before to get a quick overview of the performance of a system in general. Right on the overview page you will find information about summary statistics calculated over all transactions, actions, requests and custom timers. In addition to these statistics you´ll find a summary graph on the respecting category pages:

Trend reports will also contain summary charts, summary statistics will be available in comparison reports.

Arrival rate charts

We have already explained in detail why using the arrival rate model is best suited to simulate realistic system behavior. It was about time that we are offering dedicated charts to observe the arrival rate of a test run.

New error overview table

In addition to the detailed error table there will be a brand new reduced overview table which provides a quick overview about occurred errors:

Better readability for charts with high peaks

Charts with peaks (for example, caused by timeouts) can be hard to interpret as the relevant value area is squeezed together. We will offer two ways to improve this:

  • The report generator can produce charts with a logarithmic scale instead of a linear scale.
  • The maximum displayed y-value can be limited to a certain upper value, using a configurable factor multiplied by the mean value.

Both ways can be controlled within XLT settings and can be combined as well.

More report improvements

  • It will be possible to switch to the same chart tab for all shown timers
  • Added sorting feature to certain report tables such as the error and the response code table
  • Overview about all responses grouped by their content type

Under the hood

  • XLT 4.3 will use HtmlUnit v2.11
  • It will be possible to ping agent controllers to briefly check whether all configured agent controllers are alive and run the same version of XLT.
  • Improved error messages
  • Added include feature to configuration files which makes it much easier to create flexible test suite configurations for different environments and with different load profiles.

There are many more improvements and several bugfixes. Stay tuned!

Statistics, Facts and Figures: Web Performance on Black Friday and Cyber Monday

According to CyberMonday 2012 was the biggest buying and spending day ever!

Whether you are a fan of facts and figures or find such data rather boring – statistics published on are really interesting:

Short Summary

  • Most of the shop seemed well prepared. Congratulations!
  • Big players like Amazon and Apple came in with good results and obviously did their homework.
  • Barnes & Noble should review their site really carefully: Up to 262 requests per page and about 10 seconds until a page was fully loaded – this upsets even patient customers.

Now imagine you run a retail site that doesn’t handle the increased traffic on such an important day. Without doubt this will bring revenue down big time, which leads to an upset CEO and rolling heads. Not to mention all the nasty comments on Twitter and Facebook.

But how to prepare for THE day?

Option 1

You could hire thousands of students (each of them owning a notebook, smartphone or tablet) and let them shop your site. If you want to repeat this scenario, you obviously have to pay them twice or three times or…

Option 2

You could automate common page flows using a good test automation tool and run load tests frequently, watch the trends, and really run your own BlackFriday before BlackFriday.

It´s up to you! Afraid we can’t help you with option 1…

Understand and choose a load test model

When running load tests it is all about the concept of how you setup and run a test. Previously we already talked about getting to numbers, today we will talk about the load models you can use to run your tests.

One of the decisions you have to make is for the right load model. Sounds complicated – but in fact isn’t. A load model basically describes what basic characteristic you can influence to reach a certain load and performance behavior.

Just one more thing before we talk about the models, because the definitions used in the following paragraph are not unique across the industry. So each vendor uses them slightly differently.

Transactions: “A transaction is an execution of exactly one test case or test scenario. In order to perform the scenario, the page flow is modeled in code. The test scenario is implemented as a test case, which itself executes a sequence of one or more actions.”

Action: “An action can be defined as one irreducible step within a test case. So an action interacts with the current page and – as a result – loads the next page. That page is associated with this action and becomes the current page for the next action in the test scenario. Generally, an action triggers one or more requests.”

Requests: “This level is equivalent to the HTTP request level used in web browsers or in any other application that relies on HTTP communication. You do not have to deal with requests directly because they are automatically generated by the underlying HtmlUnit framework when performing actions on HTML elements.”

Let’s check out an example. Before running the next marketing campaign your company’s online shop will have to be tested to make sure it won’t break under that much traffic. Two of your teammates are discussing the problem:

Head of IT: “The shop will have to handle 2000 concurrent users.”

Head of marketing: “With our campaign the the shop will have to handle 500 orders per hour.”

Both requirements can lead to two different load models, depending on the exact goal that is set.

  1. User Count Model: This is the approach of the IT guy. Define a certain number of concurrent users the system will have to handle. At any given time during the test, the target system has to handle 2000 concurrent users, no more, no less. The number of transactions that can be achieved depends on the time the target system needs to respond.

  2. Arrival Rate Model: The money driven marketing guy tends to another criteria: The number of transactions (or here: orders) per hour. The scenario will be performed 500 times, equally distributed across a period of one hour. As many concurrent users as necessary to fulfill the given arrival rate are used.

Okay. Understood. Head of IT is happy with the user count model. He’s pretty sure the shop will be stable with 2000 concurrent users.

So what will happen if the response time of the system increases during the test period?

When using the user count model simply less transactions will be finished. In comparison the arrival rate model will increase the number of concurrent users as well, to make sure the given number of orders will be performed.

You can see the difference here? The arrival rate model works feedback based and will react to response time changes during the test period. This way, the generated load is somewhat unpredictable – and that’s how the real world behaves.

Or to use a more common analogy, people will usually still enter the supermarket despite the fact that there are long line at the cashier. That what happens in your online shop when the response time increases and you use the arrival rate model. More user are coming in, because they do not know that there are already lines in the store.

The chances for picking the arrival rate model are getting higher, but the IT guy is just a little bit afraid of the aggressive behaviour of such a test. Just imagine:

  • response time increases
  • more concurrent users are used
  • response time will increase even more due to heavier load
  • more concurrent users are used
  • recursion here

Load Test Models: Arrival Rate Model

But there is a solution. To avoid a complete breakdown, you can define an upper limit to the number of concurrent users used in the arrival model. This can help you to restrict the total load on the system, if you want to avoid a total overload as a result of the feedback loop. That is not reality of course, but of the user feel a certain pain in reality, they might hold back as well.

Ok, the arrival rate model is going to be used for the next load test. But there are definitely scenarios in which the user count model could be a good choice? Here are some guidelines which model fits what purpose best.

The user count model is well-suited for:

  • a simple base line test (single-user test) to assess the base performance of the system under almost no load,
  • a real load or performance test to assess the performance under a high, but predictable aka stable load,
  • a test that should be easily repeatable and its load factor is not influenced by the system under test.

The arrival rate model is best used if the load test should prove that a system is indeed able to handle a certain number of transactions per hour. Since this is the primary purpose of load and performance tests, the arrival rate load model is the best choice for most of your test tasks.

So get testing and give all models a try. Load tests often start with a fixed user rate and once this runs fine, you move over to the more challenging arrival rate model.