 |
| Photo by Shannon Kringen |
People who worked for Steve Jobs sometimes talked about his “reality distortion field” that forced them to look at the world differently, often challenging them to do things that they previously thought impossible. I’d like to thank Janet F-H for recently turning my attention to a different reality distortion field that surrounds social media research.
A recent article discussed how social media analysis did a poor job of predicting the results of Super Tuesday in the Republican primary. Some of the research flaws could be characterized as sample frame bias, such as not knowing whether the authors of social media posts are likely to vote in the primary, or even if they are Republicans. Thus, it’s hard to know which sentiments gleaned from social media are relevant to predicting a primary vote. Furthermore, the demographics of social media usage are quite different from the demographics of voters in the Republican primary.
So far, we have characterized the people who have access to social media. However, active participation in social media is a different story. Have you noticed that 90% of the posts on your Facebook wall seem to come from 10% of your friends? Obviously, some people are much more active in posting than others, which means that the demographic/psychographic profile of sampling posts could be very different from the profile of sampling people in social media.
Wouldn’t some demographic weighting help? A former colleague did a study, several years ago, where he showed that the demographics and psychographics of active online participants were quite different from the general population, sampled by more traditional methods, such as telephone and mail research. (Although web usage is more mainstream today, the same principles apply.) When he used demographics to weight the results to the general population, he found that the psychographics skewed even worse. Some biases cannot be simply weighted away.
Even if sampling problems could be solved, there is a fundamental flaw in sentiment analysis. Sentiment analysis is often automated by computational algorithms, but even the best algorithms are only about 70% accurate in predicting positive vs. negative sentiment. That’s still useful if you’re using a social media dashboard to focus your attention on negative comments as part of a routine scan of social media to manage PR. As a research tool, this is inadequate, so sentiment analysis should be performed by humans who can understand humor, sarcasm, nuance, and online abbreviations.
Social media analysis is a useful new tool, but it complements other forms of marketing research, rather than replacing the latter. As professional marketing researchers, we know that qualitative studies may be useful for generating hypotheses, yet their small sample sizes are not necessarily representative of the general population. Social media analysis can be even less representative at times, despite the false comfort of sampling large numbers of posts.
For several years, studies of marketing executives have revealed a strong desire to incorporate social media into the marketing toolbox, but also anxiety about not knowing how to effectively harness social media. In future articles, I shall explore the ways that marketing research can work hand-in-hand with social media and digital marketing.
Mario
Read more...