Release testing and releasing your product, are like peanut butter with jam, eggs with bacon, or beer with nuts. They kinda just go together. Even though you can have one by itself, we always tend to associate them together. But do they have to be?
Here are some questions you should ask yourself before blindly doing release testing and spending inordinate amounts of time, to determine whether a test is worth executing:
1. When was it executed last?
2. What has changed since then?
3. Is it manual or automated?
4. How long does it take to execute?
5. Has it ever failed before?
6. Do many of our users use this feature or not?
7. Do we have the expertise available for executing and analyzing the test?
8. What capacity do we have?
9. Is the test still valid?
Let’s take each of the questions and break them down:
1. When was it executed last – If this test was executed 2 weeks prior to your release candidate build being created, is there a lot of value in re-executing again? Maybe, maybe not…
2. What has changed since then – In an area that has had little to no new development or bug fixes since the last time it was executed – do you really need to run it?
3. Is it manual or automated – If manual, should you invest a full day of a testers time to execute this test? If automated, do we have resources available to handle failures, unexpected change, or script updates that you do find?
4. How long does it take to execute – In general terms, is it worth the time spent to do it?
5. Has it ever failed before – Alas, the tests that get run over and over and over and over and always PASS. Stop doing these if manual. Exploratory testing may be a way better alternative!
6. Do many of our users use this feature or not – If no one uses it, stop it. If very few use it, think about stopping it.
7. Do we have the expertise available for performing and analyzing the test – Testers are not quality gatekeepers. If you are not knowledgeable in the area, the technology or the product, you should not be doing release testing for this feature. Find someone who is more capable, tester or not.
8. What capacity do we have – Capacity planning should always be factored in prior to doing release testing activities. How many testers do I have, what do they know, how long do we have to test, what else is everyone working on? These should always drive your testing effort.
9. Is the test still valid – The curse of the out-dated testing documentation. If the test has not been modified in a long time, question it’s validity or appropriateness.
Performing testing activities should never be done just because it is best-practice or what you’ve always done. I’m a big proponent of testing what makes sense at the time, and what is or could be the biggest risk(s). Release testing does help give people the warm and fuzzy feeling that there is a certain level of quality in your product which is nice. I’m not saying to do it or not to do it, just do what makes sense with the time and resources you have and factoring in the potential risks.