The Persistence of Quantitative Research

Big data, social listening, machine learning and AI are all affecting the market research industry in new and exciting ways. The mind-bogglingly large datasets generated by the digitization of our lives are presenting market researchers the richest data trove ever created by humanity. The sheer abundance of data has prompted many to question whether we’ve entered a “post-survey era” where it no longer makes sense to field quantitative surveys with hundreds or thousands of respondents when data is available on millions.

We at ThinkNow Research like big datasets as much as anyone else and our company collects lots of data through opt-in passive mobile tracking, social media monitoring, and organic search studies. Those tools are terrific for taking the temperature of a population or topic but don’t always arrive at concrete, usable answers. Simply making an observation and arriving at a conclusion as to why it happened without asking the observed population whether that conclusion is correct can be dangerous. We all have biases based on our personal experiences so assuming that we know why a population is or is not doing something online can be misleading.

Online transactional data is good but not every question is transactional

Observing behavior is good and sometimes ideal, especially if it’s transactional data. Amazon became the juggernaut that it is today by harnessing the power of past purchases and online behavior. Getting people to do more of the same thing by uncovering the triggers that make us purchase items online can be found on Amazon’s servers. This type of transactional data is highly predictive for future purchases of the same or similar types of products. However, getting people to engage in new activities or buy previously unpurchased products is less predictable.

Digital data is also limited by the fact that it’s well…digital. Even with so much of our lives spent on our smartphones surfing social media we still live our lives mostly offline. If you purchase a new car, it’s possible that you saw something about that model online, but it’s more likely that one drove past you on the street or you saw it in a magazine, television commercial or billboard ad. There were no digital breadcrumbs to follow. That offline stimulus is something Silicon Valley will likely be able to measure in the future, but we’re not quite there yet. Once interest is piqued and individuals start generating online data through searches, that data can be used to target ads. Ads for people not in the market, however, don’t work.

Social media: Consumer, Bot, or Foreign Agent?

Another issue that is becoming problematic is the “noise” present in large online datasets. Measuring online support for certain causes, products or services does not necessarily indicate market success. Lots of products, television shows and services with positive online data streams ultimately fail. Negative commentary isn’t always what it seems either. Remember that most people had an unfavorable opinion of the Affordable Care Act when it was first introduced. Liberals didn’t like it because they felt it didn’t go far enough and conservatives thought it went too far.

Additionally, with the advent of chatbots that write comments on behalf of companies and foreign governments that wish to influence online discussions, it can be difficult to discern what data represents true public opinion. Even when social media comments are made by credible individuals, they are often aspirational (I love the Lamborghini Aventador!) and not necessarily reflective of offline behavior.

Analyzing social media comments may hold predictive value when determining the beliefs of particular individuals and may be useful if the comments address a specific issue being studied by a marketer, but most individuals rarely mention brands in their natural online conversations and seldom address the specific market research questions marketers need answered.

Sometimes the best way to find something out is to ask

How then, can we find out what someone might do in the future? What solution is out there that can yield this valuable information? Asking questions. It may seem antiquated to some but asking direct questions is often the best way to get direct answers. Asking directly through well-designed quantitative surveys is even better since the answers are projectable to larger populations.

We see this play out in our day to day lives where miscommunication is common. Ask any wife whether her husband understands her, and you’ll get an earful. Most interpersonal miscommunication, however, eventually gets corrected. “Yes, I actually did want you to make reservations for our anniversary when I told you I’d like to spend it at that restaurant.” The ambiguity in the online world, however, doesn’t have that same opportunity for self-correction nor does passive online data have the specificity quantitative researchers have perfected when answering marketing related questions. Years of study and field experience have taught us to look out for double-barreled questions, telescoping, social desirability bias, primacy effects and many, many other issues that affect the accuracy of the answers we’re seeking.

Final Word

Companies have always had access to “big data” such as information on historical sales and trends. They’ve turned to consumer insights, however, to uncover the motivations behind those sales. We can try looking for motivations in online datasets, but those motivations do not always leave clean data trails or worse, may over or under-represent the variables we’re trying to understand. Determining the importance of one motivator over another is something a well-designed quantitative survey can do. While it might be possible to find out what your mother wants for her birthday by snooping through her search history, or social media profiles, sometimes simply asking saves time, money and potential embarrassment.

This blog post was originally published on Quirk's Media