Home » Uncategorized » Automation – is it really worth it?

Automation – is it really worth it?

There has been many a blog written about both the positives and negatives of automation. My own personal take on this matter, is that the positives do outweigh the negatives, but it all depends on how it is being used. It is a unique tool in your arsenal but be wary of over-valuing! Automation should be done in conjunction with manual testing NOT replace it!

I’ve worked in situations where the whole focus was on automating. Development is done, Test Planning is done, now ‘Script, Damn You, Script’! This extreme focus takes you away from other valuable testing that you could be doing.

Some positives to automation are:

  1. Complex, time-consuming or mundane data setup or test execution can be automated to save time when you need to run it over and over again. Big plus!
  2. With a very stable feature, it provides a good ‘checking’ mechanism to ensure certain behavior still works as expected
  3. Is a great change management tool for picking up change that occurs in your system/product
  4. For a very large product/project, it allows you to have some coverage in an area, where due to time and personnel resources, you may not get to again. You might call that ‘Better Than Nothing Testing (BTNT?)’

Some negatives to automation are:

  1. Provides a false sense of ‘quality’ of your product, when too much emphasis is put into the scripted tests
  2. Difficult to maintain, especially with a volatile product that is in flux
  3. Time spent investigating possible failures quite often results in ‘change’ not ‘defects’ (cost vs. benefit)
  4. Tests may indicate ‘PASS’ but may no longer be valid or applicable anymore (see #1)

If the amount of time spent automating, maintaining, re-working, scrapping, reviewing scripts far outweigh the benefits, it makes sense to constantly review your process and make adjustments. Automation should provide you with useful information and have value. If it, consistently over time, is not providing worthwhile information (and/or defects), it is time for a change. It should also not be an all-consuming process!

Change can be a very daunting thing.

How do you balance manual and automated testing? Very good question, and is one that you should constantly be striving to find the answer to. Constantly review what you are trying to accomplish with the automation and adjust accordingly.

Since being exposed to context-driven testing, exploratory testing, agile, and the whole testing community as a whole, it certainly has opened my eyes to the future of testing. It is an exciting thing to be involved in, and to help invoke change in the way things are done.

Advertisements

8 thoughts on “Automation – is it really worth it?

  1. Do checks (automated tests) that pass offer value to a company? I find they often offer a false sense of confidence in the software, let alone addressing the quality of the problems the software is meant to solve.

    In my opinion, the value of checks really comes out when they do not pass. This is where they give opportunity to obtaining new information. If a check fails and provides no useful information to any stakeholder, then remove that check: it is more of a cost in maintenance than it is beneficial.

    What are your thoughts on the benefits of passing checks?

    • Do tests that pass, provide useful valuable information? Sure they do. They help provide coverage for certain paths through your product (happy?) and help detect change. As I mentioned in my blog, it also provides at a least some coverage that you otherwise may not get around to. As you mentioned, they do also provide a false sense of assurance of quality as well. To say we have 100% code coverage or 96% of automated tests pass, are kind of meaningless numbers on their own. The key is to make accurate, informed decisions as to what the automation coverage is really attempting to do and help supplement it with other modes of testing.

      How to report on all that is a whole other matter 🙂

  2. Good point. They do provide coverage of fixed paths through the software.

    I like Martin Fowler’s definition of “enough checks”: You have enough in place when you rarely get bugs that escape into production, and you are rarely hesitant to change some code for fear it will cause production bugs.

    As for reporting… I agree, that is an area in need of further rumination (especially for a product so large it is difficult to define its scope 🙂 )

  3. Pingback: Test Automation – is it really worth it?

  4. Pingback: Testing Bits – 2/23/14 – 3/1/14 | Testing Curator Blog

  5. Good article John.
    I would recommend having a look at Michael Larsen at CAST 2012:

    “Get the Balance Right: ATDD, GUI Automation and Exploratory Testing”

    I’ve actually seen it twice, once by myself a year back, and just last week as a “coffee and cake”-video with my current team of developers.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s