Past, Present, & Future of Text Analytics
As comments and reviews left on social media and online platforms increasingly influence the purchasing habits of consumers, the ability to accurately reveal trends in textual data becomes more important than ever for companies looking to enhance their brand presence.
This might explain why the global text analytics market is predicted to exhibit exponential 17.6% CAGR through 2024—increasing in value from $2.82 bn in 2015 to a predicted $12.16 bn by 2024. The heart of text analytics is that it allows us to transform unstructured data into quantifiable chunks we can digest, without losing the text’s honest and raw feedback.
The benefits of open end response
Customers writing reviews or answering open ended survey questions are able to express their opinions literally in their own words. Open ended questions help to prevent inaccurate data, because the respondent is not limited to the answer choices provided by the surveyor. Comments allow for more nuanced and complex insights, and let the respondent address questions the surveyor may not have even thought to ask. For all of these reasons, comments are the clearest avenue to Voice of the Consumer, and therefore at the heart of VoC insights. While quantitative responses give us quick benchmarks and metrics that can be easily compared across time periods or brands, comments provide the story behind these numbers.
Historical challenges and present opportunities
However, the challenge with analyzing this type of data is due exactly to the prime benefits we just laid out. Identifying overarching trends in text responses has historically been both human and time intensive. Sifting through thousands or even hundreds of thousands of comments to assess consumer perception was so daunting, that analyses were compromised or even abandoned altogether, leaving a wealth of insights buried in comments and reviews. Modern text analytics programs give us a starting point to codify unstructured text answers into structured data, using statistical, linguistic, and machine learning techniques.
With today’s technology, we can tag comments with distinct topics and the sentiments regarding those topics, and even assess sentiment intensity. For example, the sentence, “I love the brand’s clothing, but the service is horrible,” will be ranked as positive for the category ‘Brand’ but negative for the category ‘Customer Service.’ Now take the comment, “I am OBSESSED with the brand’s styles, and the service is pretty good.” Most text analytics programs will also tag this comment as positive regarding ‘Brand,’ and rank it as even more positive in this topic than the previous comment. It is obvious how this kind of software can make text analytics drastically more efficient, especially for marketing professionals.
As far as machine learning and AI have advanced, turning this data into actionable insights is still a human task. While text analytics software can point us in the right direction to find what customers are talking about in the highest volumes and even the associated sentiments, this data becomes exponentially more powerful when we can extract from it practical business recommendations.
Take this example
Say you are a Fortune 500 clothing brand assessing the company’s success in a foreign market, where sales have been markedly lower than in other countries. Satisfaction metrics across different topics such as Merchandise and Brand are low, so you turn to text analytics to understand why. Within Merchandise, your text analytics program tells you that in terms of volume, customers are discussing with higher frequency ‘models’ and ‘descriptions’ and you can even see that these comments are generally negative. So you know the problem is lying in these themes and sub-themes, but that’s pretty much as far as you can go. Without looking at the comments themselves, you would never know that customers are most dissatisfied by the fact that the pictured models don’t reflect the local look, and that sizing descriptions are stated in foreign units difficult to convert quickly to the local standard. These factors are deterring customers in the foreign market from buying your clothing. Making these relatively easy adjustments can have drastic effects on foreign sales and improve the company’s performance in international markets.
The persistent need for human insight
Often it’s not enough to know that a frequently discussed theme is generally positive or negative—you need to know why and what you can do about it. This is the added value of human insight to any text analytics program.
Additionally, sometimes the themes provided by algorithms do not accurately reflect the story on the ground, and analysts need to retag or recombine portions of the data to make it relevant for the analysis aims. As an example, for a rewards program deep dive we once performed at OGC, we needed to analyze all of the comments related to rewards, rewards points, discounts, promotions, etc. Who would have thought the word ‘point’ would be so problematic? The analysis required removal of any instance where ‘point’ was used in the context of ‘pointing out’ an item, a price ‘point’, or ‘point’ as in the purpose of an action. Without this adjustment, our analysis of rewards comments would have looked very different.
Though text analytics will most likely improve in this particular regard in the future, for now human judgment is necessary to verify and assess many algorithmic results. Technical difficulties with translation from languages other than English, as well as data security, also hinder the seamless functionality of present text analytics programs.
With text comments more accessible than ever, we can now use text analytics to shed light on the quantitative metrics. While NPS and overall satisfaction scores tell us how we are performing, consumer comments hold the key as to why. Text analytics programs make it easier to identify where to dig for meaningful topics, by assessing comment volume and broad sentiment. When converting unstructured data to structured, we are faced with a constant trade-off between processing efficiency and preserving the Voice of the Consumer. Leveraging text analytics allows us to optimize our efforts and get the results most reflective of reality, leading to actionable business decisions backed up by consumers in their own voices.
Rebecca Glanzer, Business Analyst