Sentiment analysis: Is it accurate enough?

In our latest post we’ve discussed briefly the questions about the accuracy of sentiment analysis. Are the current sentiment analysis techniques accurate enough? Are humans 100% accurate in performing sentiment analysis? In this follow-up post I would like to discuss these questions more thoroughly. While discussing this, I will refer to a number of blog posts in which the accuracy of sentiment analysis is discussed.

When we are talking about ‘accuracy’ and sentiment analysis, it gives us an idea about the quality of the methods of sentiment analysis. If a method for sentiment analysis would be 100% accurate, it will return 100% correct results. How accurate is sentiment analysis at this moment? For starters, there are many different methods to perform sentiment analysis with their own specific accuracy, so talking about the accuracy of sentiment analysis is generalizing anyhow. In this blog post, I will discuss sentiment analysis methods in general, so the accuracy numbers between the specific methods may vary. Also, the accuracy of a sentiment analysis method is difficult to measure. Results will differ depending upon the test data (statements about products are for instance easier to classify than political statements). On top of that, no exact numbers are available. Suppliers of sentiment analysis techniques are wary of informing their customers about the exact accuracy of their techniques. These factors make it difficult to estimate at what level sentiment analysis currently is regarding accuracy. If we believe the experts (pro and con sentiment analysis) the accuracy is below 100%, let’s say about 70-80%.

Is 70-80% accuracy good enough? The people who write about this can be divided in two groups regarding the answer to this question. The first group of people say that it isn’t enough and they use arguments like this:

In thinking about how to work with a sentiment analysis tool the analogy with online banking comes to mind. Would you continue to pay your bills online or use an ATM if you knew you lost 30 cents for every dollar you spent? Certainly 70 percent accuracy is not good enough for my money. How can it be good enough to reflect the hard earned efforts of an ongoing PR program, which ultimately comes down to money as well? (Carol Holden)

Or as Andy Beal very amusingly compares sentiment analysis with canaries in a coal mine:

After all, would you walk into a coal mine with a bird that has a 30% chance of getting it wrong about dangerous gas levels? I know I wouldn’t.

Interesting with these statements is that 100% accuracy is used as the standard. If a system were 100% accurate, it would be good enough for a banking system or a system for the equivalent of a canary in a coal mine. But is 100% the standard for sentiment analysis as well? As Seth Grimes suggests (he is a ‘member’ of the other group) humans can’t even perform sentiment analysis at 100% accuracy. Can we expect sentiment analysis techniques to do so? For instance, what is the general sentiment of the following sentences?

            “He is the prettiest of all ugly people.”

            “For the best comedian in the world, she made a pretty lousy joke.”

Maybe accuracy is not the only aspect on which the value of sentiment analysis should be based. Grimes and also Christine Sierra point out the importance and value of automating the process of classifying information based upon sentiment.

Automation advantages typically include speed, reach, consistency, and cost. (Seth Grimes)

If you are processing a lot of information and need to streamline the process by concentrating on the extremes, then explore what automated systems can do for you. (Christine Sierra) 

Grimes and Sierra focus on other aspects of sentiment analysis to determine whether or not sentiment analysis is valuable. They focus on what sentiment analysis can do, instead of what it can’t do. As I’ve suggested in my previous blog, I too like to focus on this positive approach. Sentiment analysis isn’t perfect (yet), but can still be very valuable in your Competitive Intelligence environment. Human interference will still be necessary, not only to filter out the flaws. Performing analysis, adding value to information and transforming information in ‘intelligence’, these are all activities that can’t (and shouldn’t) be automated. It is the combination of automated processes and human assets that counts!

Links
Is Sentiment Analysis Reliable? by Andy Beal
Competitive Intelligence functionality: Sentiment Analysis by Anne van den Brink
Expert Analysis: Is Sentiment Analysis an 80% Solution? by Seth Grimes
Is “Automated” Costing You Results? by Carol Holden
Automated doesn’t always mean perfect but it doesn’t always mean wrong either by Christine Sierra

Share and Enjoy:
  • Google Bookmarks
  • LinkedIn
  • del.icio.us
  • Technorati
  • Twitter
Comments: Comments Off

Comments are closed.