Title: When entering special characters in the zip code field a pop-up appears that says “yo!”

Image © Juja Schneider

Continue reading this article if one of the following statements applies to your software testing projects:

• Comments about incompetent developers and nit-picking testers are normal.
• What test data should be used and where it can be found is always unclear, regardless of how often it has been explained before.
• The same questions are raised again and again by different people.
• Features known to be incomplete are tested.
• People report the same defect over and over again.

All these points may indicate that you communicate poorly in your project. As in real life, communication both within the test team and between the test and development department is the key to successful projects. After all, everybody has the same goal: creating software of high quality. Here are some useful tips that will help you to get the most out of your teams:

Communication between Testers and Developers

1. Get in touch
   Especially in virtual teams often the only point of interaction between developers and testers are bug tracking system; yet it is especially the written word that causes misunderstandings and leads to pointless extra work. Regular calls and actually talking to each other can work miracles here.
2. Clarify expectations
   Which information do developers exactly need when you find bugs? Which test areas are of highest priority right now? To avoid extra work, try to settle on some basic guidelines right from the beginning and make sure to communicate or change them whenever necessary.
3. Create a culture of constructive criticism
   That´s probably the most important point of all. There is one goal and everybody is working on achieving it. Thus, each detected and eliminated bug is a step towards a better piece of software and should be seen as a  great team achievement.

Communication within Test Teams

1. Get to know each other
   This might sound easy, but it´s surprising how little information testers have about each other, even if they work for the same company. This gets even worse if virtual teams are involved! Sharing information about competences and experiences will result in much more efficient work.
2. Talk
   Another simple but often neglected point. Testing in a team will lead to great success if you share your knowledge and your findings. Group chats can work for some teams, but routinized and beneficial meetings for others.
3. Share your test data
   Have you ever found yourself in a situation where testing a specific feature took a lot of time just because you had to ask for test data you could use? This certainly happens every once in a while, and it’s actually not a problem if only you need to do so. However, it does become an issue as soon as everyone else in the test team has to request the required data. So find a way to share you test data with all team members – it´s worth it.
4. Instantly share news
   There is a new release? There is a temporary problem on one of your test systems? Make sure everybody knows about it.

Last but not least, a major point for everyone involved:

Speak the same language
There is bound to be misunderstandings if everybody talks about the same things in different words. The core of successful team communication is a mutual agreement on the terminology used within your project. Try to write it down, even if it causes extra work. It will help new people get into the subject.

A variety of tools makes communication much easier nowadays and doesn’t require much setup: use shared online documents, create mailing lists, and take advantage of chat systems.

To cut a long story short: cultivate a constructive work environment, make sure to always communicate, and share your knowledge. It´s that easy!

This is a topic whole generations of software testers have debated about. Test case design is a very subjective matter, depending hugely on the preferences of the test engineer and the requirements of the project. Although exploratory techniques are increasingly popular in the testing world, scripted manual testing used in many projects.

Image © Juja Schneider

When you are in charge of designing test cases you will have to decide which level of detail you want to apply to your test cases. This level of detail reaches from the vague description of a given scenario to a list of every single click a tester should perform. You can cover several actions in one test step or split them up into several ones. The more details you provide, the bigger your test suite will be at the end.

But which approach is the right one? This answer might not really help you, but: It depends!

Let’s talk about some advantages of detailed test cases.

• The test case are very detailed.
• You can ensure that less obvious features and scenarios are covered.
• Even less experienced testers can perform them easily as a precise sequence of steps is specified. It is often stated that this becomes a candidate for outsourcing.
• Testers who are not familiar with the target application will get a detailed overview about the system’s features.
• Complex and critical key features can be tested very thoroughly based on the requirements and specifications.
• There will be no room for interpretation whether a test case is failed or not.

Sounds good? It is… often. However we wouldn’t have started the discussion again if this is the answer. So what are the disadvantages?

• The test case are very detailed. Yes, this sentence is supposed to be here.
• You have to invest much more time and effort into creating the test cases.
• The maintenance of your test cases is a much larger topic. Every time a small detail changes, you have to adjust the test cases.
• You may miss creative testing approaches of skilled testers. Test case extensions while testing are unlikely to happen.
• Re-using test cases in future projects is less likely as test cases are more project specific.
• Testers may develop kind of a tunnel view. You may know this from your own experience: If you are forced to follow a specific path to reach your target, you may lose sight for details along the way.
• The test results depend heavily on the person who created the test cases. If he/she didn’t do a good job, the executing testers won’t make up for this during testing. If the test case designed missed important steps, the testers will miss them too.
• Providing detailed steps without room for alternative paths will magically merge a group of testers into one single tester. While they will follow precisely the steps when detailed test cases are provided, they might use a completely different approach when they have the freedom to look around.
• If the results are precisely defined, nobody will question what he or she sees on the screen, so you miss the testing of testing.
• Or taken to an extreme: Detailed aka step by step test cases will turn your testers into human test automation engines with the advantage of additional pairs of eyes. If you get lucky, your testers will report things they see but which are not stated in the tests, such as broken layouts for instance.

So what is right and what is wrong? Some guidelines would help, wouldn’t they? As there won’t be THE ultimate solution, here are a couple of hints about how to find the right way for your project.

1. Keep the subject of your project in mind
   If you test a medical device, you probably have to have very detailed test cases to satisfy the regulations and document everything as good as possible, because there are lives at stack. Assuming you test just a DVD player, you might go with 20 relatively free and 5 detailed test cases instead.
2. Consider the skills of your testers
   Highly skilled testers won’t need detailed instructions for all test areas. You may want to define test cases in a more pragmatic way.
3. Think of your future user base
   If this is an extremely large group, where all levels of expertise exist, you want to reflect that in your testing without driving up the time needed to test. If your future user base is extremely experienced and have to go by the books anyways, such as pilots, you can reflect that in your testing.
4. Specify the goal and not the way
   Try to avoid the step by step descriptions. Instead of sending someone down a menu by saying click here, here, here, here, and there, you just state: Navigate to X using the application menu.
5. Challenge your application
   The later real world user will make mistakes you cannot anticipate, so add that jitter to your testing. Give your team room for creativity and for going individual paths.

Regardless if your project subject requires detailed test cases: Try to harvest the power of randomness, give your testers enough power to question anything anytime while keeping control of what has to be done.

Eric Jacobsen from testthisblog.com started this little rant about his bipolar life as a tester. You should read it, it is very entertaining. It describes precisely what we feel from time to time. So we felt encouraged to continue.

Image © Juja Schneider

Cool, developers have marked most of my bugs as resolved. Maybe we will be able to launch the project in time!

No, wait…

I will be busy doing retests most of the day. This sucks. I won’t be able to continue my scheduled test cases for today. Test management won’t be too happy with me. I hate doing all these retests.

I’m so proud, because I have completed all my test cases in record time and found so many bugs! I’m a testing ninja! I might be able to go home early.

No, wait…

What’s that? Test management has assigned a whole bunch of new test cases to me? Is this my reward for working quickly? Life is not fair.

But I found this really big bug minutes ago. Wohooo! No regular user will be able to work with that feature. It’s a usability nightmare! Hope they will fix this soon. No way they can go live with this one. …I’m a representative of “a regular user”, right? I won’t even look at the specification. This cannot be right!

No, wait…

I took a look at the specification. It is expected to work like that. The design agency sold this as “visionary approach”. What do do now?

Poor developers! I feel honest sympathy for them. All these bugs I submit really cause a lot of work.

No, wait…

Why can’t they build it right from the beginning? I have so much more work to do, just because they deliver a buggy system.

No, wait…

Would I have a job, if all developers would create perfectly well running software? I should be happy that they are a little sloppy sometimes.

Wait a moment, is this a bug? The weather forecast mentioned “light snow in the afternoon”. I would rather call it “heavy-snowish” – and it is pretty late too.

No, wait…

Maybe this is a “Works as designed”?

P.S: Feel free to continue.

Happy New Year to everybody! Time to think about 2013 and the work ahead of us.

The ecommerce market is growing and becoming more competitive every day. This means, the customer experience is going to play an even bigger role in 2013. Online shops are expected to be stylish and beautifully designed – but customers are getting more demanding in terms of performance and usability on multiple devices.

The following topics are our point of view on the most important issue that will keep us busy in 2013.

1. Mobile Usability

Mobile shopping is growing really fast. You might think that every company is ready for this, but that’s not what we see. Many mobile shops display basically the standard shop but smaller. This is often a disaster in terms of usability and wastes a lot of potential.

2. Client Side Performance

No need to discuss the importance of lightning fast websites. Considering the huge success of powerful JavaScript libraries and the integration of third party apps, the client side performance will suffer. So the focus will be on testing the browser performance impression in combination to the server performance.

3. Automated Cross-Browser Testing

In a perfect world, every site behaves the same, no matter which browser is used. But we do not live in a perfect word. So not only test automation is important, but being able to perform them on different platforms running different browsers.

4. Mobile Performance

Okay, we have mentioned the mobile sector before, but nonetheless in times of infinite bandwidth not everybody keeps in mind that mobile surfing is usually much slower than browsing the web with a regular computer. Standard performance tests will be extended to mobile sites covering bandwidth as well as client site issues.

5. More Realistic Load Tests

Do real users do the same stuff again and again on a web site? Do customers really search for one specific product, add it to the cart, and order it repeatedly?

We are convinced that load tests have to simulate the real world as best as possible. Let the users browse freely and order random products with random quantities. User behaviour variance during load tests is the key to detailed insights such as cache patterns, memory behavior, and load distribution. Additionally it makes tests more robust, because the data dependencies becoming smaller.

6. Exploratory Testing

Our world is moving faster, agile software development cycles are getting more and more popular. Is it about time to adjust testing to the more exploratory approach? The ability to deliver creative and quick results without having to set up a full-blown test project is becoming increasingly important. Test design and execution are performed simultaneously.

Exploratory testing requires highly skilled testers who are able to produce results quickly and consistently. Testers who work structured and cover all bases but do not spent days or weeks organizing and maintaining test cases. Testers who can apply their knowledge to any application in no time. Testers who are pair with the developers. We feel well prepared for that.

Your comments are highly appreciated. What topics do you see in 2013? What will be the next big thing in testing? What should be the tester’s focus?

Happy Holidays, Merry Christmas, and a Happy New Year to all our customers and readers. Happy testing in 2013!

We are looking forward to release XLT 4.3 soon. A lot of work has been invested to improve XLT and to provide a much easier way to gather all information from our powerful report engine. Have a look at some of the new features below.

Highlights

New summary statistics directly on the overview page

It´s easier than before to get a quick overview of the performance of a system in general. Right on the overview page you will find information about summary statistics calculated over all transactions, actions, requests and custom timers. In addition to these statistics you´ll find a summary graph on the respecting category pages:

Trend reports will also contain summary charts, summary statistics will be available in comparison reports.

Arrival rate charts

We have already explained in detail why using the arrival rate model is best suited to simulate realistic system behavior. It was about time that we are offering dedicated charts to observe the arrival rate of a test run.

New error overview table

In addition to the detailed error table there will be a brand new reduced overview table which provides a quick overview about occurred errors:

Better readability for charts with high peaks

Charts with peaks (for example, caused by timeouts) can be hard to interpret as the relevant value area is squeezed together. We will offer two ways to improve this:

• The report generator can produce charts with a logarithmic scale instead of a linear scale.
• The maximum displayed y-value can be limited to a certain upper value, using a configurable factor multiplied by the mean value.

Both ways can be controlled within XLT settings and can be combined as well.

More report improvements

• It will be possible to switch to the same chart tab for all shown timers
• Added sorting feature to certain report tables such as the error and the response code table
• Overview about all responses grouped by their content type

Under the hood

• XLT 4.3 will use HtmlUnit v2.11
• It will be possible to ping agent controllers to briefly check whether all configured agent controllers are alive and run the same version of XLT.
• Improved error messages
• Added include feature to configuration files which makes it much easier to create flexible test suite configurations for different environments and with different load profiles.

There are many more improvements and several bugfixes. Stay tuned!

You have to work with HP Quality Center (HPQC), but you don’t want to execute all the test cases manually. You automated some tests using XLT Script Developer and like the outcome. You want to use the Script developer much more but you face one last problem: You still have to enter the test results manually into HPQC. This renders some of the test automation advantages useless.

The following example can mitigate that problem. HPQC offers an API called Quality Center Open Test Architecture API (OTA API).
Using this interface, you can set test results automatically.

This is the way to do it: Run the test cases as a Script Developer Batch Test, export the results, parse them and submit them to HPQC. Completely without touching the HPQC web interface at all.

The following Python command line script does the parsing and submitting:

# -*- coding: utf-8 -*-
import win32com
from win32com.client import Dispatch
from datetime import datetime
from bs4 import BeautifulSoup
import re

# global vars
result_file_name = 'X:\\path\\to\\xlt-result.html'

# the regular expression for the id that can be found in test case names, 
# the id has to be enclosed in brackets so that the regex can contain things 
# that are not part of the id
# examples: XLT id: r'(XLT[0-9]{4})[^0-9]', whole test case name: r'(^.*$)', r'(T[0-9]{2})[^0-9]'
tc_id_regex = re.compile(r'(T[0-9]{2})[^0-9]')

# value for strftime to generate the name for the run
run_name_pattern = "Run_%Y-%m-%d-%H-%M-%S"

# hpqc credentials
qcServer = "https://hpqc.server"
qcUser = "username"
qcPassword = "password"
qcDomain = "domain"
qcProject = "project_name"
qcTestCasePath = "Path\to\test\case"
qcTestCaseNode = "test case node name"

# test case status names in hpqc
hpqc_status_pass = 'Passed'
hpqc_status_fail = 'Failed'
hpqc_status_no_run = 'No Run'
hpqc_status_not_completed = 'Not Completed'
hpqc_status_na = 'N/A'

# status names in the XLT result file
xlt_result_pass = 'OK'
xlt_result_fail = 'FAILED'


# parse the result file
# the processed html structure looks like this:
#<html>
#<body>
#<table></table>
#<h2>Details</h2>
#<table>
    #<thead></thead>
    #<tbody>
        #<tr>
            #<td>Tests.XLT0001_First_Test_Case_Name</td>
            #<td class="success">OK</td>
            #<td class="number">0.355</td>
            #<td></td>
        #</tr>
        #<tr>
            #<td>Tests.XLT0002_Second_Test_Case_Name</td>
            #<td class="error">FAILED</td>
            #<td class="number">2.255</td>
            #<td>Assertion &apos;assertElementPresent(provoke error)&apos; at line 1 has failed. =&gt;</td>
        #</tr>
    #</tbody>
#</table>
#</body>
#</html>
result = BeautifulSoup(open(result_file_name))
resultArr = {}
for tr in result.select("table")[1].select("tbody tr"):
    tc_name = tr.select("td")[0].string[6:]
    tc_res = tr.select("td")[1].string
    tc_id_mo = tc_id_regex.search(tc_name)
    
    if tc_id_mo:
        tc_id = tc_id_mo.group(1)
        if tc_id in resultArr:
            print("ERROR: there has already been a result for tc_id " + tc_id + " tc_name " + tc_name + " tc_res: " + tc_res)
        else:
            if tc_res == xlt_result_pass:
                resultArr[tc_id] = hpqc_status_pass
            else:
                resultArr[tc_id] = hpqc_status_fail
            print(tc_id + " - " + tc_name + " tc_res: " + tc_res)
    else:
        print("ERROR: no valid id found in tc_name " + tc_name + " tc_res: " + tc_res)

# login to hpqc
td = win32com.client.Dispatch("TDApiOle80.TDConnection.1")
td.InitConnectionEx(qcServer)
td.Login(qcUser,qcPassword)
td.Connect(qcDomain,qcProject)
if td.Connected:
    print("Logged in to " + qcProject)
else:
    print("ERROR: Connect failed to " + qcProject)

# navigate through the test set tree
tsTreeMgr = td.TestSetTreeManager
tsFolder = tsTreeMgr.NodeByPath(qcTestCasePath)
tsList = tsFolder.FindTestSets(qcTestCaseNode)

for tsItem in tsList:
    tsTestList = tsItem.TSTestFactory.NewList("")
    
    # loop through all test cases in this list
    for tsTest in tsTestList:
        print("Test case name: " + tsTest.TestName)
        runFactory = tsTest.RunFactory
        run_name = datetime.today().strftime(run_name_pattern)
        run = runFactory.AddItem(run_name)
        run.CopyDesignSteps() # copies the test steps into this run
        
        # extract the test case id from the test name
        tc_id_mo = tc_id_regex.search(tsTest.TestName)
        if tc_id_mo:
            tc_id = tc_id_mo.group(1)
        else:
            tc_id = ''
            print("ERROR: no valid id found in name " + tsTest.TestName)
        
        # get the results from resultArr and set the test result accordingly (identical for test case and all test steps)
        if tc_id in resultArr:
            status = resultArr[tc_id]
        else:
            status = hpqc_status_no_run
        
        run.Status = status
        
        # set the status for each test step
        tsStepList = run.StepFactory.NewList("")
        for tsStep in tsStepList:
            print("    Test step name: " + tsStep.Name)
            tsStep.Status = status
            tsStep.Post()
        
        # submit this run to hpqc
        run.Post()
        print("Run added: status: " + status + " run_name: " + run_name)

# close the connection to hpqc
if td.Connected:
    td.Disconnect
    td.Logout
    print("Logged out from " + qcProject)

td = None

The processed part of the Script Developer result file looks like this:

<html>
<body>
<table></table>
<h2>Details</h2>
<table>
    <thead></thead>
    <tbody>
        <tr>
            <td>Tests.XLT0001_First_Test_Case_Name</td>
            <td class="success">OK</td>
            <td class="number">0.355</td>
            <td></td>
        </tr>
        <tr>
            <td>Tests.XLT0002_Second_Test_Case_Name</td>
            <td class="error">FAILED</td>
            <td class="number">2.255</td>
            <td>Assertion &apos;assertElementPresent(provoke error)&apos; at line 1 has failed.</td>
        </tr>
    </tbody>
</table>
</body>
</html>

It needs to run in a Windows environment because it is importing win32com. You have to install:

• Python (version 3 is needed)
• pywin32 (Python for Windows extensions)
• Beautiful Soup 4 (a HTML parser for Python) (after extracting the archive, cd into it and run ‘setup.py install’ to install it)

Now adjust the variables in the script and you are ready to go. The following screenshot depicts the parts of HPQC that the script is using.

In the first td there is the name of the test case (always starting with “Tests.”) and the second td contains the status of the test case.

Disclaimer: This is an example only. Use at your own risk.

The script has been inspired by Generating Custom Reports with Quality Center OTA using Python.

When Google+ brought the community feature online, we immediately knew, that could be it to get testers together and discuss test automation, learn from each other, and share knowledge. Google+ gathered a more technical crowd compared to Facebook and so we will give it a try.

Feel invited and we hope to see you soon: Test Automation Community at Google+.

According to onvab.com CyberMonday 2012 was the biggest buying and spending day ever!

Whether you are a fan of facts and figures or find such data rather boring – statistics published on dzone.com are really interesting: http://css.dzone.com/articles/black-friday-cyber-monday-2012

Short Summary

• Most of the shop seemed well prepared. Congratulations!
• Big players like Amazon and Apple came in with good results and obviously did their homework.
• Barnes & Noble should review their site really carefully: Up to 262 requests per page and about 10 seconds until a page was fully loaded – this upsets even patient customers.

Now imagine you run a retail site that doesn’t handle the increased traffic on such an important day. Without doubt this will bring revenue down big time, which leads to an upset CEO and rolling heads. Not to mention all the nasty comments on Twitter and Facebook.

But how to prepare for THE day?

Option 1

You could hire thousands of students (each of them owning a notebook, smartphone or tablet) and let them shop your site. If you want to repeat this scenario, you obviously have to pay them twice or three times or…

Option 2

You could automate common page flows using a good test automation tool and run load tests frequently, watch the trends, and really run your own BlackFriday before BlackFriday.

It´s up to you! Afraid we can’t help you with option 1…