Testing a new product – what should be the main focus?

Often, as a tester we can be too focused on the technology, the business requirements, the design, the data, the use cases, the user workflows/stories and the like.

When a new product is about to be built, we should ask ourselves “What ‘claims’ and ‘value adds’ are we are making about this new product?”

Does this not seem like the number one thing we should be concerned about testing? Should this not drive your testing efforts? I believe it should.

Whenever a new product or service goes to market, there are always the big marketing splashes:

  • “Faster than any other product on the market”
  • “Increases productivity of users by 50%”
  • “Allows collaboration of various entities for ease of operation”
  • “Integrates seamlessly with various other tools”

Yadda, Yadda, Yadda

Marketing messages and sales claims should be the major drivers for testing. These messages can also lead to other ‘value adds’ as well that aren’t explicitly stated or even implied. How can these claims be made without testing them explicitly?

Before testing commences, there should be consultations with senior level management, product managers, marketing and sales about how we are going to validate these statements and claims and prove the value add statements! Also, what do they really mean? What are specific metrics?

Think of it as a pyramid or a flowchart of some sort. Each ‘value add’ statement should be at the top. As you work your way down, you can get more involved and detailed about specifics like business requirements, technology/tools, use cases, etc. But they should all lead back to each of the value adds defined for the feature.

These should drive your test efforts and be your main focus. Should we be focused on finding bugs? Yes. Should you validate business requirements? Yes. Should you validate the design, write up various tests and test cases, Yes, Yes, Yes. But all of those are secondary, if it is not as fast as we said it would be, does not increase productivity like we said it would, does not integrate seamlessly, or does not help users collaborate with each other easier.

Thinking of these, will help keep you focused on what matters, and not simply how many testcases are passing/failing.

Automation – is it really worth it?

There has been many a blog written about both the positives and negatives of automation. My own personal take on this matter, is that the positives do outweigh the negatives, but it all depends on how it is being used. It is a unique tool in your arsenal but be wary of over-valuing! Automation should be done in conjunction with manual testing NOT replace it!

I’ve worked in situations where the whole focus was on automating. Development is done, Test Planning is done, now ‘Script, Damn You, Script’! This extreme focus takes you away from other valuable testing that you could be doing.

Some positives to automation are:

  1. Complex, time-consuming or mundane data setup or test execution can be automated to save time when you need to run it over and over again. Big plus!
  2. With a very stable feature, it provides a good ‘checking’ mechanism to ensure certain behavior still works as expected
  3. Is a great change management tool for picking up change that occurs in your system/product
  4. For a very large product/project, it allows you to have some coverage in an area, where due to time and personnel resources, you may not get to again. You might call that ‘Better Than Nothing Testing (BTNT?)’

Some negatives to automation are:

  1. Provides a false sense of ‘quality’ of your product, when too much emphasis is put into the scripted tests
  2. Difficult to maintain, especially with a volatile product that is in flux
  3. Time spent investigating possible failures quite often results in ‘change’ not ‘defects’ (cost vs. benefit)
  4. Tests may indicate ‘PASS’ but may no longer be valid or applicable anymore (see #1)

If the amount of time spent automating, maintaining, re-working, scrapping, reviewing scripts far outweigh the benefits, it makes sense to constantly review your process and make adjustments. Automation should provide you with useful information and have value. If it, consistently over time, is not providing worthwhile information (and/or defects), it is time for a change. It should also not be an all-consuming process!

Change can be a very daunting thing.

How do you balance manual and automated testing? Very good question, and is one that you should constantly be striving to find the answer to. Constantly review what you are trying to accomplish with the automation and adjust accordingly.

Since being exposed to context-driven testing, exploratory testing, agile, and the whole testing community as a whole, it certainly has opened my eyes to the future of testing. It is an exciting thing to be involved in, and to help invoke change in the way things are done.

Finding bugs that matter

There is nothing more frustrating then spending an inordinate amount of time researching a potential bug, scouring the repository to see if it already exists, running through various scenarios to try and reproduce, talking to SMEs, documenting the issue, setting up meetings and phone calls to discuss, etc, only to find out… the issue does not matter!

Some examples may be:

a) Being rejected by the stakeholder as not important, and thus will not get fixed
b) Falling into the cesspool, that is the bug repository, where it will sit, sight unseen for eons
c) Lowering the severity to where it ‘may’ be fixed at some point
d) Being cast as ‘something that a user would never do’ and dismissed

Often, what we think may be a show-stopper or a higher priority issue, is not viewed that way by others. As testers, we have a unique outlook on things, and can bring a different perspective. Certainly from a UX level, we can provide valuable input.

Here are a few suggestions that may help:

1. Sell yourself and the issue – I have found many instances of issues that I’ve raised, that were decided to never be fixed, only to have customers raise them as high-priority issues at a later date. Don’t be afraid to stand up, be counted, and explain why this issue matters.
2. Consult a second opinion – what may not be a big deal to one, may be a big deal to another. There are always multiple irons in the fire. Create a discussion. It is better to be fixed and found now and certainly much less costlier then doing it later.
3. Review the issue at a later date – what may have been dismissed earlier, may be taken more seriously as the product is/has evolved.
4. Accept it as what it is – sometimes we need to do this. We can’t fix every bug, and it may very well be that it is not important to be fixed, or will be back-burnered until the end of time.

Don’t be offended, sell yourself, sell the issue, consult second opinion(s), review later, accept.

It can be hard to do all these things, but in the end, each has its place.