Qualitative research methods, such as interviews and observation, reveal what you don’t know. Quantitative research methods, such as surveys and experiments, reveal how many.
Product and marketing teams seem to love surveys. Surveys grab a bunch of data quickly. However, the problem with surveys and experiments is they are best for validation rather than discovery. You don’t know what you don’t know. Interviews and observation are critical to understanding WHAT and HOW before you get into HOW MANY. That’s why I say, “Interview a few; measure a bunch.” Interviews give you stories; surveys give you data. Sure, you’ll need data to know IF you should pursue your idea but you’ll need stories to understand HOW to pursue your idea. How to begin
First determine what you need to understand. Whether for a product or a business, you have many ideas—hypotheses that need to be proven. Any aspect of your business canvas is likely to be an assumption.
Include these types of inquiries:
What problems do our buyers have? (Ensures you're solving real problems)
Are those problems urgent to the buyer and important to the user? (Helps determine if your idea will sell)
Are they willing to pay you for a solution? (Helps with pricing)
Which features are critical and which are merely nice-to-have? (Defines your product features)
How do I access the buyer? Do they read blogs, attend conferences, subscribe to podcasts. (These will help you develop your promotion plan)
What other assumptions have you made? Each can be validated with some form of market research.
Qualitative and Quantitative
There are two types of research: qualitative and quantitative. Qualitative translates to anecdotes and stories; quantitative results in data. Both are used in two ways: to discover or to validate.
Here are some common methods and how they can inform your decisions. Interviews are best for discovery. Experiments are best for validation. Here's the thing: Qualitative gives you insight; quantitative gives you facts. Discovery helps you learn about problems; validation helps ensure you understand the problems and have developed the right solution. Although many people swear by them (and many more swear at them), surveys are almost never a good idea. Look for automated validation when you can, using experiments such as A/B testing or other data-based approaches. But before you think about data, think about anecdotes for insight. Set up some phone interviews; do some onsite observation. Really understand your personas and their problems. Dirty Data A retail store was trying to determine where to open a new location. They asked all shoppers to provide their zip code. In today’s suspicious world, many would decline but the field was required by the form in the point of sale system. There was no option to indicate the client refused so clerks would type in 00000. But the form creators had anticipated “bad behavior” and had forbidden 00000 and 99999 and these were disallowed too. So the clerks put in their personal zip code. As a result, the survey data was filled with erroneous data but no one could tell the good from the bad. If they had allowed 00000, then the analyst could simply remove those records. Based on this research, the store placed the second location near where most of the employees lived, not where most of their customers lived. And within a year, the store failed. When designing a survey, consider all the use scenarios. Not just what you want to know as a researcher but how the client will respond and what error conditions to anticipate. Treat a survey the same way you would treat a product feature. What are you trying accomplish? What are the typical scenarios? Is research safe? Don't worry about someone stealing your idea; worry that no one knows you have the idea. The internet is a big place—your problem isn’t safety, it’s obscurity. Yes, it’s safe to create a mockup or landing page—an experiment. And like any web site, you’ll need to do some promotions to drive people to it. There, you’ll ask folks to sign up for more information. And now you have data. You don’t necessarily need to reveal too many details on the landing page. Focus instead on the problem you can solve and for whom. Check out the way projects are done on Kickstarter. You don’t imply that the product is already available so there’s no problem if you decide not to offer it. Good product decisions are made when human judgement is combined with good data. Use data to validate; use personal experience to discover.
Discovery and Validation research is one of the many modern methods taught in the Fundamentals of Managing Products series available from Product Growth Leaders. Learn more here.