There has been many a blog written about both the positives and negatives of automation. My own personal take on this matter, is that the positives do outweigh the negatives, but it all depends on how it is being used. It is a unique tool in your arsenal but be wary of over-valuing! Automation should be done in conjunction with manual testing NOT replace it!
I’ve worked in situations where the whole focus was on automating. Development is done, Test Planning is done, now ‘Script, Damn You, Script’! This extreme focus takes you away from other valuable testing that you could be doing.
Some positives to automation are:
- Complex, time-consuming or mundane data setup or test execution can be automated to save time when you need to run it over and over again. Big plus!
- With a very stable feature, it provides a good ‘checking’ mechanism to ensure certain behavior still works as expected
- Is a great change management tool for picking up change that occurs in your system/product
- For a very large product/project, it allows you to have some coverage in an area, where due to time and personnel resources, you may not get to again. You might call that ‘Better Than Nothing Testing (BTNT?)’
Some negatives to automation are:
- Provides a false sense of ‘quality’ of your product, when too much emphasis is put into the scripted tests
- Difficult to maintain, especially with a volatile product that is in flux
- Time spent investigating possible failures quite often results in ‘change’ not ‘defects’ (cost vs. benefit)
- Tests may indicate ‘PASS’ but may no longer be valid or applicable anymore (see #1)
If the amount of time spent automating, maintaining, re-working, scrapping, reviewing scripts far outweigh the benefits, it makes sense to constantly review your process and make adjustments. Automation should provide you with useful information and have value. If it, consistently over time, is not providing worthwhile information (and/or defects), it is time for a change. It should also not be an all-consuming process!
Change can be a very daunting thing.
How do you balance manual and automated testing? Very good question, and is one that you should constantly be striving to find the answer to. Constantly review what you are trying to accomplish with the automation and adjust accordingly.
Since being exposed to context-driven testing, exploratory testing, agile, and the whole testing community as a whole, it certainly has opened my eyes to the future of testing. It is an exciting thing to be involved in, and to help invoke change in the way things are done.