Tag Archives: Testing

Test Case Design – The Unfinished Discussion

This is a topic whole generations of software testers have debated about. Test case design is a very subjective matter, depending hugely on the preferences of the test engineer and the requirements of the project. Although exploratory techniques are increasingly popular in the testing world, scripted manual testing used in many projects.

Image © Juja Schneider

When you are in charge of designing test cases you will have to decide which level of detail you want to apply to your test cases. This level of detail reaches from the vague description of a given scenario to a list of every single click a tester should perform. You can cover several actions in one test step or split them up into several ones. The more details you provide, the bigger your test suite will be at the end.

But which approach is the right one? This answer might not really help you, but: It depends!

Let’s talk about some advantages of detailed test cases.

  • The test case are very detailed.
  • You can ensure that less obvious features and scenarios are covered.
  • Even less experienced testers can perform them easily as a precise sequence of steps is specified. It is often stated that this becomes a candidate for outsourcing.
  • Testers who are not familiar with the target application will get a detailed overview about the system’s features.
  • Complex and critical key features can be tested very thoroughly based on the requirements and specifications.
  • There will be no room for interpretation whether a test case is failed or not.

Sounds good? It is… often. However we wouldn’t have started the discussion again if this is the answer. So what are the disadvantages?

  • The test case are very detailed. Yes, this sentence is supposed to be here.
  • You have to invest much more time and effort into creating the test cases.
  • The maintenance of your test cases is a much larger topic. Every time a small detail changes, you have to adjust the test cases.
  • You may miss creative testing approaches of skilled testers. Test case extensions while testing are unlikely to happen.
  • Re-using test cases in future projects is less likely as test cases are more project specific.
  • Testers may develop kind of a tunnel view. You may know this from your own experience: If you are forced to follow a specific path to reach your target, you may lose sight for details along the way.
  • The test results depend heavily on the person who created the test cases. If he/she didn’t do a good job, the executing testers won’t make up for this during testing. If the test case designed missed important steps, the testers will miss them too.
  • Providing detailed steps without room for alternative paths will magically merge a group of testers into one single tester. While they will follow precisely the steps when detailed test cases are provided, they might use a completely different approach when they have the freedom to look around.
  • If the results are precisely defined, nobody will question what he or she sees on the screen, so you miss the testing of testing.
  • Or taken to an extreme: Detailed aka step by step test cases will turn your testers into human test automation engines with the advantage of additional pairs of eyes. If you get lucky, your testers will report things they see but which are not stated in the tests, such as broken layouts for instance.

So what is right and what is wrong? Some guidelines would help, wouldn’t they? As there won’t be THE ultimate solution, here are a couple of hints about how to find the right way for your project.

  1. Keep the subject of your project in mind
    If you test a medical device, you probably have to have very detailed test cases to satisfy the regulations and document everything as good as possible, because there are lives at stack. Assuming you test just a DVD player, you might go with 20 relatively free and 5 detailed test cases instead.
  2. Consider the skills of your testers
    Highly skilled testers won’t need detailed instructions for all test areas. You may want to define test cases in a more pragmatic way.
  3. Think of your future user base
    If this is an extremely large group, where all levels of expertise exist, you want to reflect that in your testing without driving up the time needed to test. If your future user base is extremely experienced and have to go by the books anyways, such as pilots, you can reflect that in your testing.
  4. Specify the goal and not the way
    Try to avoid the step by step descriptions. Instead of sending someone down a menu by saying click here, here, here, here, and there, you just state: Navigate to X using the application menu.
  5. Challenge your application
    The later real world user will make mistakes you cannot anticipate, so add that jitter to your testing. Give your team room for creativity and for going individual paths.

Regardless if your project subject requires detailed test cases: Try to harvest the power of randomness, give your testers enough power to question anything anytime while keeping control of what has to be done.

The Bipolar Life Of A Software Tester – Continued

Eric Jacobsen from testthisblog.com started this little rant about his bipolar life as a tester. You should read it, it is very entertaining. It describes precisely what we feel from time to time. So we felt encouraged to continue.

Image © Juja Schneider

Cool, developers have marked most of my bugs as resolved. Maybe we will be able to launch the project in time!

No, wait…

I will be busy doing retests most of the day. This sucks. I won’t be able to continue my scheduled test cases for today. Test management won’t be too happy with me. I hate doing all these retests.

I’m so proud, because I have completed all my test cases in record time and found so many bugs! I’m a testing ninja! I might be able to go home early.

No, wait…

What’s that? Test management has assigned a whole bunch of new test cases to me? Is this my reward for working quickly? Life is not fair.

But I found this really big bug minutes ago. Wohooo! No regular user will be able to work with that feature. It’s a usability nightmare! Hope they will fix this soon. No way they can go live with this one. …I’m a representative of “a regular user”, right? I won’t even look at the specification. This cannot be right!

No, wait…

I took a look at the specification. It is expected to work like that. The design agency sold this as “visionary approach”. What do do now?

Poor developers! I feel honest sympathy for them. All these bugs I submit really cause a lot of work.

No, wait…

Why can’t they build it right from the beginning? I have so much more work to do, just because they deliver a buggy system.

No, wait…

Would I have a job, if all developers would create perfectly well running software? I should be happy that they are a little sloppy sometimes.

Wait a moment, is this a bug? The weather forecast mentioned “light snow in the afternoon”. I would rather call it “heavy-snowish” – and it is pretty late too.

No, wait…

Maybe this is a “Works as designed”?

P.S: Feel free to continue.

Our Top 6 Software Testing Trends 2013

Happy New Year to everybody! Time to think about 2013 and the work ahead of us.

The ecommerce market is growing and becoming more competitive every day. This means, the customer experience is going to play an even bigger role in 2013. Online shops are expected to be stylish and beautifully designed – but customers are getting more demanding in terms of performance and usability on multiple devices.

The following topics are our point of view on the most important issue that will keep us busy in 2013.
Continue reading Our Top 6 Software Testing Trends 2013

Burn-in test of XLT 4.0

If you write a performance testing software, your first obligation is a performance test for and with that software. So before we can ship XLT 4.0, we have to make sure that it can hold up to its promises. Test software has to be tested too, so to speak. Today we provisioned a bunch of Amazon-EC2 boxes with 30 test agents in total. These charts are from a short test to see where we can get to, when ignoring any target numbers. This was a short and violent test. Just hammer the system under test and see if it can and will recover.


A short ramp up period in the beginning of the test and afterwards we kept a steady load factor. For the steady phase we reached more than 11,000 hits per second.

Concurrent Users

The target system was seeing about 2,200 concurrent users during the peak of the test.

Received Bytes per Second

During that test, the network was transporting about 95 Mbytes/s inbound traffic, this is a network utilization of about 760 Mbit/s. Amazing that the EC2 boxes in the EU data center can handle that traffic. We used just 10 boxes and each box has a 100Mbit network, so the overall limit must have been reached we think.

By the way, the target system recovered easily and was able to serve its normal duties without any problems. But this test clearly showed the limits in terms of throughput. But this test did not show any limits of XLT 4.0, because neither the load boxes in terms of CPU nor the reporting had any problems with this test size.

Speed matters for your ranking

Nati Shalom discusses in one of his latest blog entries the changes Google made to its page ranking algorithm and how it influences your Google page ranking.

Last month Google added Website speed to its site ranking algorithm: It’s Official: Google Now Counts Site Speed As A Ranking Factor… The rationale behind this move by Google is fairly straightforward:

Slow web sites lead to a poor user experience, and therefore should not appear at the top of the search list even if they contain relevant content.

This emphasizes once more the influence of performance on your daily business. A simple change to your site can now affect your entire page ranking and how users find your content. Continuous performance testing is now even more important than ever.

Andere Blogs rund ums Testen

Natürlich gibt es noch andere Blogs, die sich mit Testen und Qualitätssicherung beschäftigen. Einige davon möchte ich heute mit einem kurzen Kommentar vorstellen:

  • DevelopSense von Michael Bolton (nicht der Sänger). Für Michael ist der Fakt wichtig, dass ein Tester viel wertvoller ist, als jede Automation. Das Hirn eines Testers ist sein Werkzeug. Er nennt es auch: Checking is not testing.
  • James Bach vertritt eine ähnliche Meinung und verurteilt die blinden Bestrebungen, alles unkontrolliert zu automatisieren. Sein Kernbotschaft dreht sich um Exploratory Testing. Das intuitive, aber nicht zufällige Testen.
  • Eric Jacobson legt sich nicht auf Gebiete fest, sondern kommentiert alles Querbeet.
  • Nicht zu vergessen das Google Testing Blog. Hier berichten Google Tester aus ihrer täglichen Arbeit und den Herausforderungen von großer Software.

Wer kennt weitere empfehlenswerte Blogs?

Lasttesten mit Webdriver und Ruby

Zur Zeit arbeiten wir intensiv an unseren nächsten Xceptance LoadTest Version.  Sie wird einige interessante Neuerungen mitbringen, die man bisher auf dem Markt noch nicht so gesehen hat:

  • Als Scriptsprache steht jetzt neben Java auch Ruby zur Verfügung. Wer also die schnelle agile Entwicklung mit Ruby mag, der kann jetzt nahtlos auch in Ruby Regressions- und Lasttests erstellen.
  • Google hat sich die Mühe gemacht und eine einheitliche API zur Programmierung von Webregressiontests ins Leben gerufen – Google Webdriver. XLT spricht jetzt Webdriver. Damit lassen sich schnell Webtests schreiben und, im Gegensatz zu den anderen Tools, auch als Lasttest ausführen. Wir denken, dass damit die Einführung von XLT deutlich schneller geht und sich damit noch besser für Projekte mit Rapid-Prototyping-Charakter eignet.
  • Zwei Lasttest-Läufe lassen sich jetzt innerhalb von Sekunden vergleichen und das Ergebnis zeigt schnell und deutlich, wo die Änderungen liegen.
  • Wer sich mehr für die langfristige Entwicklung der Performance interessiert ist, dem wird der neue Trendreport eine grosse Hilfe sein. Eine beliebige Menge von Testläufen lässt sich zueinander in Relation setzen und man kann daraus einfach die Entwicklung des Performancetrends ablesen.

Wir freuen uns schon auf die Fertigstellung von XLT 3.3. Zu den einzelnen Neuigkeiten wird es demnächst mehr Blogeinträge geben.

Buchtipp: Lessons Learned in Software Testing

Nach langer Zeit habe ich mein Softwaretest-Lieblingsbuch mal wieder aus dem Schrank gezerrt: Lessons Learned in Software Testing. Übrigens eine Empfehlung und ein Geschenk von guten Freunden.

Für mich ist dieses Buch immer wieder eine Motivation mit QA und Testen weiterzumachen, weil es bestätigt und aufbaut. Es ist kein Wälzer mit theoretischen Lehren, sondern es enthält 293 Lektionen/Schnipsel mit wissenswerten Tipps und Ideen. Wer testet oder über das Testen Bescheid wissen muss, für den ist das Buch aus meiner Sicht Pflichtlektüre. Alle anderen Wälzer sind viel zu theoretisch und an der Praxis vorbei.

Lesson 11: You don’t assure quality by testing.


Heute habe ich einen interessanten Ansatz für eine Verbesserung von JUnit gefunden – TestedBy.

In a nutshell TestedBy aims to change the point of view regarding test classes and classes under test. What we would obtain is to put class under test (the most important classes of your projects) on the centre and link from there your test class and test method.

Ist einen Blick wert, da es scheinbar erstmal das Management der Tests verbessert.