3 scenarios when we should automate (or not)

This is a guest article by Shuchi.

Shuchi is a principal consultant in a multi national corporation and has ten years of development/testing experience in the IT industry. She has worked closely with test automation tools and has been involved in test strategy-related decisions.

In her off-work hours she writes about cryptic crosswords. Check out her cryptic crossword guide on the blog, Crossword Unclued.

Why Automate?

A prevalent belief among IT folk is that automation, by default, will enhance the quality of testing.

How true is that belief?

“There is a time and place for everything”, as the popular idiom goes, and that holds true for test automation too.

Certain scenarios in which automation is generally a good option:

1. Repetitive Tests

A system that needs the same tests executed periodically is a good candidate for automation. Take for example, support projects that must perform regression tests for every PR/CR. Manual effort in such cases tends to be huge and error-prone. Automation can prove very efficient in such cases.

It makes sense to opt for a testing tool with a version controlled repository to manage the test scripts.

The cost of automation and the initial effort investment for script creation is usually greater than for manual testing, and it must be analyzed how far this investment will pay off in the end. A few pointers to help make that decision:

• How many repetitions of tests are expected, for how long? A longer “life” of the test scripts tips the scale in favor of automation.
• How expensive is the tool?
• How much time/effort will it take to create the test scripts? Automation may not simply be record-and-play, a lot of design and coding may have to go in to achieve the desired result.
• Are incremental changes expected to the scripts?
• Do testers on the team require training on the tool?

“Repetitive tests => automation” is a fair thumb rule but not an inviolable one. A costly automation setup on one hand and quick-to-build on-the-fly manual tests on the other, and you might just find that automation will not be useful even if you expect a certain number of repetitions.

2. Manually Infeasible Tests

Before an application is deployed in the production environment, it might be critical to determine how the application will behave under huge user load or when dealing with millions of records.

Such test scenarios are usually best simulated with automation. In high risk conditions, where the cost of missing a test might be disastrous, even a single-time use of automation might be a worthwhile decision.

3. Low Severity/Probability Of “Human” Bugs

Automation can perform some jobs far better than a human can hope to do. Give the tool and the human tester two excel sheets with 10,000 rows of data each to reconcile with each other. No prizes for guessing who can do it faster, without errors.

But then, humans are far better at another sort of testing. Humans can notice oddities beyond documented tests. Does the UI look too cluttered? Does the mouse flicker strangely when moved over the button? Did the screen behind the modal window just blink?

Automation is limited to testing what it is programmed to do. Not everything that the human mind can observe/analyze can be programmed, and so, automation will miss out on capturing some bugs. The question to ask is: how important are those missed-out bugs? An unstable system might not be ready for automation, as severe “manual” bugs may get ignored. Automation is better suited to systems in which the “human” bugs are expected to be fewer and are of low severity.

In Closing

A point worth remembering is that automation vs. manual does not have to be an either-or decision. A mix can work. Take a multi-step workflow – a couple of steps might be fit for automation, not the entire flow. Automate only those steps that will benefit from automation.

Ankur: Thanks Shuchi for the insightful article. I’m sure it will be of great help to the automation testing community.

If you wish to share your experience/insights related to test automation or have discovered some cool new tip related to QTP, contact me using this contact form. [Please select 'Article' in the dropdown.]