Image © Juja Schneider

This article is not about setting up a sophisticated environment for automated tests since the relevance and importance of test automation is covered elsewhere. I rather want to give you my personal answer on how to speed up scripted testing.

Many of our e-commerce projects contain test scenarios such as the following: search for an item, select some refinements, put the product into the cart, and proceed to checkout where you need to enter your data and test another feature on the receipt page. In fact, this isn’t very complicated but it certainly takes a lot of time – even the more so when you have to go through this procedure not only once but many times a day which, on top of that, might get pretty boring in the long run.

Based on my experiences as a tester, I can honestly say that test automation tremendously facilitates scripted testing. I don’t use it to actually work off test cases assigned to me, but it always turns out to be a convenient tool that saves me from having to repeat the same steps over and over again.

Here are my pros of using test automation for scripted testing:

• I can center my attention on the most important features in the current test case.
• I save much time – time that I can well use for additional, exploratory testing.
• I’m able to stay focused much longer because I don’t have to bother with monotonous and repeating test steps.
• The minimal effort of test automation. I use XLT Script Developer that works as a Firefox Add-on and lets me record and replay test steps easily. If the project requires testing in Firefox only, I’m almost instantly done. Most projects include testing in multiple browsers, however. See here for a possible solution: WebDrivers in XLT: How to Run Test Cases in Multiple Browsers
• My helpers can be a nice starter for future test automation.
• It’s actually not a big deal if the previous point doesn’t apply as the automation tool still served its purpose of making my test life a little easier.

And here are some tips to take it easy:

• Don’t be shy to throw away your helper after the project finished.
• Don’t over engineer your scripts. They are meant to help you right now and not in the long run.
• Don’t get lost in trying to get it to work. If you cannot get it working, return to manual testing even though it is painful at this point in time.

Image © Juja Schneider

When discussing possible load test setups with our clients, we usually need to refer to these key terms: visits, sessions, requests, hits, page impressions, and page views. Actually, we don’t need to discuss all of them, but some are occasionally brought up by the customer, some are requested by us, depending on the context (and complex enough to be discussed in a separate article).

Many clients of ours have told us that it’s sometimes hard not to confuse these terms as they seem to denote the exact same thing. Today’s article is thus meant to give you an overview on their definition and help you distinguish them. Here it comes.

Visits

Basically, a visit occurs when you send a request to a server and, as a response, the website you requested is displayed. The display of the page is what we call the page view (which is covered in more detail below). Take, for instance, www.amazon.com: you enter the URL, that is you send a request to that server. As a result, you arrive on the shop’s homepage. In doing so, you generate a visit and the website owner now knows that someone visited their site. Accordingly, a visit consists of at least one page view. Typically, though, you’ll decide to browse through the shop, thus producing further page views with every single one of them adding up to your visit regardless of what you actually do while staying on the site.

Your visit ends when you become inactive for whatever reason, for example, when you stop clicking links or when you overall close your browser. The server then deletes or deactivates the data that has been collected during your visit. Depending on the web application, the time period before this actually happens varies from 30 minutes up to 24 hours (see below for more details). Compared to the real world, a virtual visit is therefore not much different from a visit in an actual store: regardless of how you spend your time in there, the store owner will consider you a visitor and they will eventually forget about you when you decide to leave their store for a longer period of time.

A visit defines three metrics that are important for us to know: the visit duration, the number of page views per unit of time, and the time period between two page views (thinktime). By the way, the Internet is crowded by an insane number of machines that also generate visits and that cause statistics to get messed up. There are a couple of technical measures trying to filter out these irrelevant visits. Let me know if you’re interested in how that can be accomplished and I’ll talk about it in another article.

Page view

As mentioned above already, a page view, or page impression, is the display of a website you trigger by sending a request to a server. It was not long ago that one request was sufficient for the page you requested to be displayed; via further requests, the browser then added images, CSS, and Javascript. In the modern web, the page view term is used in broader contexts as AJAX and recent user concepts don’t exchange complete websites but only modify small parts anymore. Thus, a page view is actually an interaction or action most of the time.

But let’s stick with the old-fashioned page view for now. It gets initiated by a request and terminated by the response from the server. This sequence of events is also referred to as hit. A site may contain other elements, such as images and CSS. Each of these elements is again processed by a request/response sequence and leads to another hit. Thus, a page view is made up of an HTML component and many embedded elements with each element and the HTML component causing further hits. Important metrics are, for instance: page load time, page size, and view time, namely the time period until the next click.

Hit is often the business term for a request that, in turn, is the technical term for a single data transfer.

To sum this up: A visit consists of one or more page views and has a certain duration while a page view has a runtime and consists of elements referred to as request or hit.

Sessions

Now, how does a session differ from a visit? In simple terms, a session is the technical picture behind a visit. The software you take advantage of while browsing through a webshop has to remember which requests belong together so that functionalities such as a login or a cart can actually do the things they’re supposed to do.

Sessions consist of data that summarize certain information concerning your visit which is why they are also called session information. Usually, these information have a limited lifetime as they are subject to a session-timeout. In most cases, this time-out amounts to something between 30 minutes to 2 hours. As soon as your visit ends, time is ticking down and, upon reaching the session-timeout, all data is deleted. If you continue your visit before the time has elapsed, the countdown resets to 0. To illustrate this with our real-world example, imagine you decided to buy something in the store you walked into (visited) but at the register you suddenly realize you didn’t bring any money. You can leave your cart at the register for a while to go get some cash but if you don’t return in time, the cashier will assume you’re not coming back and empty your cart.

Technically speaking, the number of sessions equals the number of visits. Due to business-related aspects, however, visits are counted differently than sessions so that the number of visits is usually lower than the number of sessions per unit of time.

Summary

In a few words:

• Visits mostly equal sessions, where session is the technical term and visit is the business term. A visit consists of at least on page view.
• Hits and requests are the same, request is the technical, hit is the business term.
• A page view is a single complete page delivered. At least one request is needed to serve it, mostly many requests are fired to assemble a page. Nowadays, page views tend to be interactions because often full pages aren’t loaded anymore; instead, only pieces are dynamically changed (the famous Ajax magic).

Confused? Feel free to ask!

Image © Juja Schneider

Many projects have their closed or “won’t fix” bugs commented with remarks such as:

• “Content-related issue”,
• “Won’t happen on live system”, or
• “This is just a data issue”.

Both the development of new features and data maintenance may be accomplished in different environments. For testers, it’s thus difficult to decide on the cause for a certain issue: is a required text missing because pages aren’t implemented the right way or is it simply not there because the corresponding product data aren’t available in this particular test system? Very often testers work on development systems without having access to latest data. The following is a typical scenario:

1. Testers report a bug.
2. The development resolves this bug as content-related and won’t fix.
3. Testers report further bugs of that sort.
4. The development gets irritated: “We’ve told them that these are content issues!”
5. The relationship between testers and developers might really go downhill from there, seriously threatening the project’s success: the credibility of the test team is damaged, developers  don’t take reported bugs seriously anymore, testers get overly cautious and stop reporting content-related problems…

Inevitably, potential bugs don’t get reported and reported bugs never get retested before the project goes live. This is even more true in poorly managed projects because no one really feels responsible to solve this problem.

Here are a couple of guidelines we advise you to keep in mind if the descriptions above somehow sound familiar to you:

For testers:

• A bug is a bug is a bug. When you see it, report it. Let your test or project manager know that you’ll report these kind of issues even if they are potentially content-related. Only this way, you can be sure not to miss anything important.
• Draw conclusions from previously reported issues and comments.
• Try to get a feeling for your project and make sure not to address problems you’re actually causing yourself.

For project and test managers:

• Don’t dismiss these issues as minor troubles you don’t want to bother with. They can easily turn into big ones at any stage of your project. Before go-live, you want everything to be fixed, not only functional problems.
• Problems that appear to be data-related at first sight might in fact be functional. Handling them within the regular test process facilitates their correct fixing.
• Don’t hesitate to tell your testers if they are causing the problem themselves. Granted, this might be an uncomfortable situation at first, but it will definitely trigger an Aha! effect everyone in the team can benefit from.
• Give content-related issues a low severity. This way, you can focus on troubles that are more pressing and, at the same time, ensure that nothing gets lost and QA will still retest everything.
• Inform your test team about the system’s architecture and how content and date influence its behavior.
• Also let your test team know when certain data aren’t available yet and when you don’t want the resulting problems reported. Testing is way easier when having such information in advance.

Still remember the first post of our article series? It talked about how you can run XLT test cases in different browsers. If you’ve done so already, you might have noticed that the behavior of test cases developed in Script Developer sometimes differs depending on the WebDriver you’ve chosen. This article is meant to help you resolve such issues.

First of all, what’s the reason for these inconsistencies? Web browsers differ in their characteristics, such as site representation or functionality, due to their varying support of web technologies like CSS or HTML. You probably know that there is much more we could list, but the major point is pretty obvious already: using WebDrivers for test case execution calls real browser instances. Logically, you’re faced with the same differences as in real web browsers. It’s impossible to achieve a completely consistent web browser behavior; yet you can design your test case as outlined below to at least reduce the differences:

Time: Enlarging time properties can prevent timing problems with dynamically loaded content, like iframes or page content added with JavaScript. Script Developer lets you set an ‘Implicit Wait Timeout’. It allows to specify how long the test should wait for a command to fail while trying to find elements.

Implicit Wait Timeout

A corresponding setting is also available for WebDriver test execution in the XLT properties files. The following property defines the WebDriver’s implicit wait timeout used when finding elements:

com.xceptance.xlt.scripting.defaultImplicitWaitTimeout=10000

Another way to control the execution speed is to set the action think time, that is a delay between the execution of two subsequent actions. Note that this property is only relevant when using the XltDriver.

com.xceptance.xlt.thinktime.action=500

Incomplete database processes or unfinished AJAX operations may also cause certain problems. If you encounter such problems with the XltDriver, check whether the following property is set to default:

com.xceptance.xlt.js.backgroundActivity.waitingTime=0

This is the time (in ms) to wait for JavaScript background jobs to finish.
When this time has elapsed, all pending jobs will be removed. Default value is 0, if not set. If set to -1, the engine will neither wait for running jobs to finish nor remove pending jobs. Note that the other WebDrivers don’t have this property. That’s why you should prevent possible problems by taking care of them when you actually design your tests with Script Developer. waitFor… commands are a very good way to achieve this. They’ve been developed for XLT and wait much longer than other assertions. You can configure their waiting time, or command timeout, in both the Script Developer settings and in the XLT properties for WebDriver test execution:

Command Timeout

com.xceptance.xlt.scripting.defaultTimeout=30000

Make sure to use waitFor… commands for dynamically loaded content, even if, for instance, an assert… command appears to be sufficient in Script Developer. If you want the test execution to wait for a specific time period, you can also define a pause. Avoid using …andWait commands for dynamically loaded content because these commands will wait for a new page load which will not happen when content is loaded dynamically.

CSS/link selectors: Using CSS-selectors may be comfortable and is also supported by Script Developer. Keep in mind, though, that the CSS interpretation is different for each WebDriver. For example, the CSS text-transform feature modifies the text specified in HTML. A text validation might fail if this CSS feature is supported in one browser only. Link locators may also cause problems in this context. For now, it’s best for you to avoid these selection strategies when creating test cases in Script Developer and instead use XPath referencing where possible.

Redundant clicks: Text and number validation in forms is often done using the onBlur event that is fired when you leave the input field by clicking somewhere outside the form.

Imagine you are testing input field validation. A JavaScript function displays an error message onBlur, you leave a field with an invalid input, and click somewher else so that the onBlur event is fired. If this click hits the submit button, Script Developer and the XltDriver will forward to the next page whereas WebDrivers need two clicks, one for firing the event and another for submitting the page. A helpful workaround is the seemingly redundant click on a static page element to force the JavaScript event right before submitting. This also helps in case of hiding UI-elements, that mask other elements. This scenario is a known issue in ChromeDriver.

Conclusion:

Being third-party products, WebDrivers aren’t consistent and never will be. Nevertheless, modifying certain XLT properties as outlined above may help you cope with these differences and reduce them whenever you want to go for multiple browsers.

Powerful load and performance test tools don’t only retrieve pages of your website randomly with zillions of users at the same time, but they also cover realistic scenarios simulating the real-world user. It’s a given that they can deliver lots of useful information and plot interesting charts. To fully take advantage of these benefits, however, you need to be able to interpret this information and draw the right conclusions.

It is this need for the correct interpretation of test results, the mapping of all you see against actual application behavior that makes performance and load testing a non-trivial task. It requires much experience to decide on the right actions, make the right assumptions, or simply come up with a reasonable explanation of why something happened this way and not the other.

In today’s article, we’d like to present you a couple of charts displaying typical response time patterns, and discuss what they could indicate.

Disclaimer: Of course, the reasons for a certain behavior vary a lot, depending on your application and testing. However, as there’s no fixed manual for the interpretation of load testing charts, we want to provide you with a couple of basic guidelines to help you get better in interpreting them yourself and make the most out of your test results. Feel free to comment whether or not you agree with the ideas and explanations we come up with.

The Warm-up

These charts might indicate a system with a cold cache, for instance, when the system has just been started and the caches aren’t filled yet.

The basic characteristics of such a behavior are high response times in the beginning, followed by gradually lower response times until eventually a  certain degree of runtime stability is reached. This time frame is often referred to as the system’s warm-up period. Throughout this period, a couple of things can happen under the surface. If you know the system under test well, you’ll probably come up with the following: database and file system caches are filled, proxies learn about the data and store them, the system under test scales up because it sees traffic, page snippets are cached and so the computing overhead reduces… you name it.

Also keep in mind that it might be the testing process itself that causes such a response time profile. If the system is perfectly warmed up and you hit it, your sent traffic might be too uniform in the beginning. That being the case, randomization kicks in so that the traffic eventually distributes better over time. Furthermore, take into consideration that your load software and hardware are possibly not warmed up either.

The Caching

These charts depict either a typical cache clean or job patterns. In case of a cache clean, system-internal caches expire every half hour. If that’s not the case, the charts may indicate  a running background job draining power from the database or consuming lots of system bandwidth.

Both charts display the same test; however, this test has been executed for different time periods. While two spikes could signify a random event (despite the fact that the temporal distance of 30 minutes is suspicious), the longer test run seems to prove our first assumption: something is going on every half hour.

In any case, make sure that such a behavior is not produced by the test machines themselves, for example, because they’re busy writing or backing up data.

The Spiking

This is what we call a forest of spikes: many spikes that don’t seem to follow a comprehensible pattern; longer runtimes just occur occasionally, often caused by requests accessing certain data or URLs that produce long runtimes. To get behind that mystery, you have to dig into the results in more detail, find the calls behind the spikes, and derive a pattern based on the information you find. Often you’ll come across similar URLs, request parameters, or maybe response codes. Don’t forget any application logs you might have access to, such as web server, error, information, or debug logs. In a perfect world, your application under test offers the necessary tools to get to the bottom of this problem.

XLT lets you easily find this information. All test result data are accessible as CSV files that are quickly readable and documented. Feel free to work with this information and go beyond the scope of the reports available.

The worst outcome here is a non-identifiable pattern and no information on the server side as to what might have happened. In such a case, you have to repeat the test and narrow down your test setup later on to exclude as many variables as possible to find the cause. This is actually also a good time to ask for development or tech-ops support.

The 3rd-party Calls

The first chart is typical for issues with 3rd parties, especially in the field of e-commerce. We’re not talking about direct calls to 3rd parties here, such as analytics vendors or recommendation engines, but calls from one server to the other. Thus, the response time we see is the response time of two systems. Of course, it’s good to know the area where 3rd party calls typically happen, but you have to know the application under test anyway to test it efficiently. So when the final order steps start to act weird, you can easily narrow down the potential reasons.

The second chart looks more like the cache clean or expiration problem described above, but since you know the application, you also know that this area doesn’t use the typical caching logic but is highly dynamic instead. This means that the errors occurring every 50 minutes point into a different direction: as we know that 3rd parties are attached and called during shipping, we can conclude that the 3rd party failed on us.

Verdict

Knowing typical response time patterns helps you specify a certain problem so that you can give hints to the development or further shape the path of testing. If you can read charts and derive the right conclusions or at least know which questions you have to address, you’ll be ahead of the crowd. Be aware that knowledge on the system under test is very important – the production and measurement of a certain load doesn’t make much sense when you’re not able to actually interpret and explain what you’ve measured. Always remember: 42 is not a valid answer for everything.

Image © Juja Schneider

Continue reading this article if one of the following statements applies to your software testing projects:

• Comments about incompetent developers and nit-picking testers are normal.
• What test data should be used and where it can be found is always unclear, regardless of how often it has been explained before.
• The same questions are raised again and again by different people.
• Features known to be incomplete are tested.
• People report the same defect over and over again.

All these points may indicate that you communicate poorly in your project. As in real life, communication both within the test team and between the test and development department is the key to successful projects. After all, everybody has the same goal: creating software of high quality. Here are some useful tips that will help you to get the most out of your teams:

Communication between Testers and Developers

1. Get in touch
   Especially in virtual teams often the only point of interaction between developers and testers are bug tracking system; yet it is especially the written word that causes misunderstandings and leads to pointless extra work. Regular calls and actually talking to each other can work miracles here.
2. Clarify expectations
   Which information do developers exactly need when you find bugs? Which test areas are of highest priority right now? To avoid extra work, try to settle on some basic guidelines right from the beginning and make sure to communicate or change them whenever necessary.
3. Create a culture of constructive criticism
   That´s probably the most important point of all. There is one goal and everybody is working on achieving it. Thus, each detected and eliminated bug is a step towards a better piece of software and should be seen as a  great team achievement.

Communication within Test Teams

1. Get to know each other
   This might sound easy, but it´s surprising how little information testers have about each other, even if they work for the same company. This gets even worse if virtual teams are involved! Sharing information about competences and experiences will result in much more efficient work.
2. Talk
   Another simple but often neglected point. Testing in a team will lead to great success if you share your knowledge and your findings. Group chats can work for some teams, but routinized and beneficial meetings for others.
3. Share your test data
   Have you ever found yourself in a situation where testing a specific feature took a lot of time just because you had to ask for test data you could use? This certainly happens every once in a while, and it’s actually not a problem if only you need to do so. However, it does become an issue as soon as everyone else in the test team has to request the required data. So find a way to share you test data with all team members – it´s worth it.
4. Instantly share news
   There is a new release? There is a temporary problem on one of your test systems? Make sure everybody knows about it.

Last but not least, a major point for everyone involved:

Speak the same language
There is bound to be misunderstandings if everybody talks about the same things in different words. The core of successful team communication is a mutual agreement on the terminology used within your project. Try to write it down, even if it causes extra work. It will help new people get into the subject.

A variety of tools makes communication much easier nowadays and doesn’t require much setup: use shared online documents, create mailing lists, and take advantage of chat systems.

To cut a long story short: cultivate a constructive work environment, make sure to always communicate, and share your knowledge. It´s that easy!

This is a topic whole generations of software testers have debated about. Test case design is a very subjective matter, depending hugely on the preferences of the test engineer and the requirements of the project. Although exploratory techniques are increasingly popular in the testing world, scripted manual testing used in many projects.

Image © Juja Schneider

When you are in charge of designing test cases you will have to decide which level of detail you want to apply to your test cases. This level of detail reaches from the vague description of a given scenario to a list of every single click a tester should perform. You can cover several actions in one test step or split them up into several ones. The more details you provide, the bigger your test suite will be at the end.

But which approach is the right one? This answer might not really help you, but: It depends!

Let’s talk about some advantages of detailed test cases.

• The test case are very detailed.
• You can ensure that less obvious features and scenarios are covered.
• Even less experienced testers can perform them easily as a precise sequence of steps is specified. It is often stated that this becomes a candidate for outsourcing.
• Testers who are not familiar with the target application will get a detailed overview about the system’s features.
• Complex and critical key features can be tested very thoroughly based on the requirements and specifications.
• There will be no room for interpretation whether a test case is failed or not.

Sounds good? It is… often. However we wouldn’t have started the discussion again if this is the answer. So what are the disadvantages?

• The test case are very detailed. Yes, this sentence is supposed to be here.
• You have to invest much more time and effort into creating the test cases.
• The maintenance of your test cases is a much larger topic. Every time a small detail changes, you have to adjust the test cases.
• You may miss creative testing approaches of skilled testers. Test case extensions while testing are unlikely to happen.
• Re-using test cases in future projects is less likely as test cases are more project specific.
• Testers may develop kind of a tunnel view. You may know this from your own experience: If you are forced to follow a specific path to reach your target, you may lose sight for details along the way.
• The test results depend heavily on the person who created the test cases. If he/she didn’t do a good job, the executing testers won’t make up for this during testing. If the test case designed missed important steps, the testers will miss them too.
• Providing detailed steps without room for alternative paths will magically merge a group of testers into one single tester. While they will follow precisely the steps when detailed test cases are provided, they might use a completely different approach when they have the freedom to look around.
• If the results are precisely defined, nobody will question what he or she sees on the screen, so you miss the testing of testing.
• Or taken to an extreme: Detailed aka step by step test cases will turn your testers into human test automation engines with the advantage of additional pairs of eyes. If you get lucky, your testers will report things they see but which are not stated in the tests, such as broken layouts for instance.

So what is right and what is wrong? Some guidelines would help, wouldn’t they? As there won’t be THE ultimate solution, here are a couple of hints about how to find the right way for your project.

1. Keep the subject of your project in mind
   If you test a medical device, you probably have to have very detailed test cases to satisfy the regulations and document everything as good as possible, because there are lives at stack. Assuming you test just a DVD player, you might go with 20 relatively free and 5 detailed test cases instead.
2. Consider the skills of your testers
   Highly skilled testers won’t need detailed instructions for all test areas. You may want to define test cases in a more pragmatic way.
3. Think of your future user base
   If this is an extremely large group, where all levels of expertise exist, you want to reflect that in your testing without driving up the time needed to test. If your future user base is extremely experienced and have to go by the books anyways, such as pilots, you can reflect that in your testing.
4. Specify the goal and not the way
   Try to avoid the step by step descriptions. Instead of sending someone down a menu by saying click here, here, here, here, and there, you just state: Navigate to X using the application menu.
5. Challenge your application
   The later real world user will make mistakes you cannot anticipate, so add that jitter to your testing. Give your team room for creativity and for going individual paths.

Regardless if your project subject requires detailed test cases: Try to harvest the power of randomness, give your testers enough power to question anything anytime while keeping control of what has to be done.

Eric Jacobsen from testthisblog.com started this little rant about his bipolar life as a tester. You should read it, it is very entertaining. It describes precisely what we feel from time to time. So we felt encouraged to continue.

Image © Juja Schneider

Cool, developers have marked most of my bugs as resolved. Maybe we will be able to launch the project in time!

No, wait…

I will be busy doing retests most of the day. This sucks. I won’t be able to continue my scheduled test cases for today. Test management won’t be too happy with me. I hate doing all these retests.

I’m so proud, because I have completed all my test cases in record time and found so many bugs! I’m a testing ninja! I might be able to go home early.

No, wait…

What’s that? Test management has assigned a whole bunch of new test cases to me? Is this my reward for working quickly? Life is not fair.

But I found this really big bug minutes ago. Wohooo! No regular user will be able to work with that feature. It’s a usability nightmare! Hope they will fix this soon. No way they can go live with this one. …I’m a representative of “a regular user”, right? I won’t even look at the specification. This cannot be right!

No, wait…

I took a look at the specification. It is expected to work like that. The design agency sold this as “visionary approach”. What do do now?

Poor developers! I feel honest sympathy for them. All these bugs I submit really cause a lot of work.

No, wait…

Why can’t they build it right from the beginning? I have so much more work to do, just because they deliver a buggy system.

No, wait…

Would I have a job, if all developers would create perfectly well running software? I should be happy that they are a little sloppy sometimes.

Wait a moment, is this a bug? The weather forecast mentioned “light snow in the afternoon”. I would rather call it “heavy-snowish” – and it is pretty late too.

No, wait…

Maybe this is a “Works as designed”?

P.S: Feel free to continue.

Happy New Year to everybody! Time to think about 2013 and the work ahead of us.

The ecommerce market is growing and becoming more competitive every day. This means, the customer experience is going to play an even bigger role in 2013. Online shops are expected to be stylish and beautifully designed – but customers are getting more demanding in terms of performance and usability on multiple devices.

The following topics are our point of view on the most important issue that will keep us busy in 2013.

1. Mobile Usability

Mobile shopping is growing really fast. You might think that every company is ready for this, but that’s not what we see. Many mobile shops display basically the standard shop but smaller. This is often a disaster in terms of usability and wastes a lot of potential.

2. Client Side Performance

No need to discuss the importance of lightning fast websites. Considering the huge success of powerful JavaScript libraries and the integration of third party apps, the client side performance will suffer. So the focus will be on testing the browser performance impression in combination to the server performance.

3. Automated Cross-Browser Testing

In a perfect world, every site behaves the same, no matter which browser is used. But we do not live in a perfect word. So not only test automation is important, but being able to perform them on different platforms running different browsers.

4. Mobile Performance

Okay, we have mentioned the mobile sector before, but nonetheless in times of infinite bandwidth not everybody keeps in mind that mobile surfing is usually much slower than browsing the web with a regular computer. Standard performance tests will be extended to mobile sites covering bandwidth as well as client site issues.

5. More Realistic Load Tests

Do real users do the same stuff again and again on a web site? Do customers really search for one specific product, add it to the cart, and order it repeatedly?

We are convinced that load tests have to simulate the real world as best as possible. Let the users browse freely and order random products with random quantities. User behaviour variance during load tests is the key to detailed insights such as cache patterns, memory behavior, and load distribution. Additionally it makes tests more robust, because the data dependencies becoming smaller.

6. Exploratory Testing

Our world is moving faster, agile software development cycles are getting more and more popular. Is it about time to adjust testing to the more exploratory approach? The ability to deliver creative and quick results without having to set up a full-blown test project is becoming increasingly important. Test design and execution are performed simultaneously.

Exploratory testing requires highly skilled testers who are able to produce results quickly and consistently. Testers who work structured and cover all bases but do not spent days or weeks organizing and maintaining test cases. Testers who can apply their knowledge to any application in no time. Testers who are pair with the developers. We feel well prepared for that.

Your comments are highly appreciated. What topics do you see in 2013? What will be the next big thing in testing? What should be the tester’s focus?

When Google+ brought the community feature online, we immediately knew, that could be it to get testers together and discuss test automation, learn from each other, and share knowledge. Google+ gathered a more technical crowd compared to Facebook and so we will give it a try.

Feel invited and we hope to see you soon: Test Automation Community at Google+.