Before throwing all of your resources at building a new feature for your product, you need to establish that you’re pursuing the right idea.
Testing isn’t always the easiest thing to do — there are plenty of situations where it can seem easier to build whatever feature you have in mind and just see what actually happens — but creating a consistent testing strategy helps determind that you’re using your resources efficiently.
Even better, the right approach to testing can let you speed up the process of working through your product roadmap. Depending on the results your tests turn up, you'll discover new opportunities to add to your process.
1. Choose What to Test
Step 1: Small Changes
As you’re adding new features or improvements to your products, you need to make sure that you’re offering them in the best way possible. You need to test both demand and the execution of what you’re offering to your customers. That requires having something to test. The MVP (minimum viable product) model often offers the best approach to creating something worth testing.
Whether or not you already have an existing product to work with, or not, think in terms of the smallest change that will give you valuable data. What will allow you to get actionable data without sinking a ton of time into a change that may turn out to be not worth your while? Of course, a minimal feature may still be a ton of work, but the goal is to have as little wasted work as possible overall.
Step 2: Testing Parameters
Since you can probably guess what it will take to build your product, at least if you’re experienced in your field, it’s important to test the things that you can’t make educated guesses about. Unless you’ve got mind-reading skills, that means understanding how your clients respond to what you’re building. Tinkering around with technologies may be fun, but it shouldn’t be the priority.
You need to think about what will constitute a successful test.
As you’re looking at a particular item on your product roadmap, consider what the core elements of the feature you want to add are: What are the parts a customer actually sees and interacts with? How can they be separated out into concrete pieces?
If, for instance, you’re creating a new shovel, the customer interacts with the handle, the shovel blade and the stick in between. If, instead, you’re creating a new piece of software, the customer’s interaction with what you’ve built will be far more limited.
You need to think about what will constitute a successful test. If you have customers already using an existing product, will getting fifty percent of them to try out a new feature be enough to establish a need for the next feature you want to build? Do you need them to keep using the feature to consider it a successful test? And if you are looking for new users, what sort of retention rates are necessary to establish good data?
In short, you want to be able to look at the data you’ve collected on a certain date and know automatically if you should move forward with fully building the feature or if you should scrap the idea. You should know what success looks like ahead of time.
Step 3: Hands-on Testing
Once you’ve broken down what you could put in front of a customer, consider the easiest way to build out that interaction, so that you can test it. If you need to order a small batch of a test version of your product, what’s the smallest order you can place? Is there possibly a way to modify your existing product to add on a feature — even if doing so yourself isn’t scalable in the long run?
Even if offering your new feature for a test comes down to taking a call from each individual customer yourself and then going out and modifying their product by hand, that level of work may be well worth your time. At the very least you’re likely to be able to handle at least a few such orders for less time and money than you’d have to invest to build out even a minimal feature.
2. Look for Real World Feedback
Step 1: Realistic Testing
The more realistic your tests, the more valuable you will find the data you collect. If you can’t arrange for the same sort of users who will rely on your product when you release it, you can’t assume that the data you get will accurately reflect reality. With that in mind, focus on how you can gather more information specifically from the types of users you rely on.
Of course, you need to focus on collecting data that you’ll actually use — running hundreds of customer surveys that you never look at again is a waste of your time and money. Have a plan in place for dealing with any information you put together, including a way to process the bits and pieces into a cohesive analysis that you can act on.
Step 2: Test with Pricing
Make a point of charging for your feature, even if you aren’t charging the full price you expect to down the line. Otherwise, you’re not going to get the highest quality data. If you expect your customers to eventually pay for the feature you want to build, you need to know if the mere existence of a price tag changes how they use the feature. It’s impossible to test every single variable out there, but the big ones, like price, need to be built into your tests every time.
Step 3: Make Your Results Easy to Access
Store your data in such a way that you can refer back to it. While you’ll want to make a practice of running tests every time you are considering a new feature for your product, there’s no need to reinvent the wheel if you’re working on something similar. You may be able to find a wealth of new information in old data as you find new questions worth asking.
3. Decide if Your Test was a Success
Step 1: Line in the Sand
While you’ve hopefully run your tests with an idea of the answers you need already in mind, you need to be able to act on the information you’ve brought together.
You have to commit that you’ll fully build the feature if you get a certain level of positive response, but you’ll scrap it if you don’t get enough good feedback. This is a clear line in the sand you can work from and gauge if your test is a success. It’s a matter of checking if you got the appropriate level of response.
Step 2: How Much Room to Maneuver?
Unfortunately, more often than you would like, you’ll prove that you shouldn’t pursue a feature that you’re personally in love with.
There may be some difficulty if you see a set of responses that almost meet your expectations, but don’t quite hit the mark. In such situations, you have to decide how much wiggle room you’re willing to allow your data. If you just picked the number you wanted to reach out of thin air, than not quite hitting it isn’t a particularly big deal.
If you put a lot of thought into that number — say, perhaps calculating at what point adding a new feature is actually financially viable — than you may need to stick to your guns, even if the data is close enough to that line to make you reconsider adjusting your initial numbers. That’s the point of a test. Unfortunately, more often than you would like, you’ll prove that you shouldn’t pursue a feature that you’re personally in love with.
Step 3: Experimenting Versus Building
After you’ve run your initial test, you may find that you need to run further tests to clarify certain details. You should continue testing as long as you get a clear advantage out of doing so. But there will come a point where you won’t benefit any more from building minimum viable versions of your product and running experiments.
If you’ve reached a point where running yet another test brings requires as much effort from you as actually building a product, it may be time to end the cycle of testing and actually start building. There are no hard and fast rules about such situations, but be mindful of the time you’re investing in each stage of creating a new product.
Keep Testing in the Long Run
Depending on your business, you may have a list of improvements you may want to add on to your product a mile long. Building a strong testing strategy can help you sort through the different paths you can take far more quickly, letting you sink your resources into the options that will do you the most good.
That strategy can be a key factor in differentiating yourself from your competition. If you’re always testing a new option — and acting on the data you collect through your tests — you’ll move much faster than someone who is iterating based on anecdotal information, or someone who isn’t bothering to consider improvements at all. You can make sure that you’re offering the best possible version of your product to your customers at any given time.