Sentiment analysis – together with machine learning techniques – is a powerful tool to boost a brand’s performance and profit from successful customer experiences.
Customer focus often dictates that businesses need to spend big on research to form an effective marketing strategy, from the feedback analysis and competitors’ study to product fit in the new markets. Given that, it’s understandable that data is key in developing strategies, tools and techniques to make a company stand out. With the amount of unstructured data available, any effort to organize, sort, understand, and even monetize, seems like a daunting task.
Sentiment analysis is one of the best ways to unlock the massive potential of this information. Using this technology, companies can tap into the great potential of market trends, customers’ attitudes, people’s inclinations and influences. How can businesses effectively embed sentiment analysis algorithms for marketing projects? Let’s explore this matter step by step with Unicsoft’s big data and machine learning experts.
Sentiment Analysis Definition
As the name suggests, sentiment analysis aims to detect sentiments, or the polarity of people’s emotions in the text. It is also referred to as “opinion mining.” Sentiment analysis can be used to analyze any type of text, from customers’ feedback to social media feeds and survey responses, and define the general attitude towards the brand, a new product, pricing changes, customer service, etc.
Sentiment analysis focuses on:
- polarity – positive vs. negative
- emotions – happy, frustrated, angry, etc.
- intentions – interested vs. not interested
Sentiment Analysis Components
Sentiment analysis is perfect for processing marketing data. It can be:
- rule-based or lexicon-based – a set of rules is developed by the linguists, in which all words are classified as positive or negative
- machine learning-based, where ML algorithms are trained to recognize the polarity, emotions and intentions in a supervised, unsupervised, or reinforced manner
- a hybrid one, which leverages the mix of two approaches
Let’s define key sentiment analysis applications.
Common Sentiment Analysis Applications in Various Industries
Sentiment analysis is a technique that supports brand monitoring and reputation management, among other things. Businesses use big data analysis & machine learning to gain a competitive advantage in their business domains. Machine learning is the backbone for accurate sentiment analysis and valid business decisions, from building long-term trends to composing the perfect words to make customers love your product instantly.
Here are a few, prominent sectors where sentiment analysis is beneficial:
- Customer experience: consumer voice, brand reputation, e-commerce, advertising.
Social media analysis is often used to monitor a brand’s reputation on Facebook, Twitter and Instagram. One example is real-time tweet analysis. - Politics: voting advice, adjustment of politicians’ programs.
Campaign managers use sentiment analysis to understand how people feel about certain issues in the politician’s program or rhetorics, how they react to speeches and actions, etc. - Public relations: event monitoring, policies, transportation, legal matters.
By analyzing social media content and news feeds, authorities can optimize traffic flow, ensure better public security and resolve issues before they become too pressing. - Finance: financial risk, the evolution of stocks and shares.
By analyzing articles, news and social media info about public companies, data analysts can assign scores for trading systems – such as Stock Sonar – and form stock prices.
Let’s focus on the business domain and explore how companies leverage sentiment analysis applications.
Examples of Business Values Generated with Sentiment Analysis
Customer experiences are driving business results these days, causing companies to continue increasing R & D budgets in this field. Sentiment analysis is worth investing in, as it helps decision-making in product development and analysis, marketing campaigns, trend creation, advanced support and many other areas.
Here’s how businesses leverage this tool.
Enhanced Brand Loyalty
Every time a customer mentions a brand, they do it in a specific context and with a personal intent. Brands should pay attention since instances like these provide valuable insight into the customer’s attitudes and loyalty. Based on this information, companies can tune product features, adjust marketing campaigns, correct mistakes and improve conversions.
Recently, Unicsoft developed a big data SaaS solution for a tobacco company (whose name is under NDA). The customer wanted to analyze and predict customer reactions to various types of ads, visual materials, design elements and so on. By leveraging machine learning algorithms, our customer can now define behavior and purchasing activity fueled by promotional activities. As a result, they reported an increase in customer response and, consequently, a revenue increase.
Secured Social Media Reputation
Social media is probably the most potent source of opinions and attitudes and is, therefore, perfect for sentiment analysis. With this automatic tool, one can analyze thousands of comments, tweets and video comments easily, then categorize urgent issues to prioritize necessary improvements.
Opinion listening across social media channels helps uncover influencers who can support a solid marketing strategy: 40% of customers say they purchased a product after seeing it on Twitter, Youtube or Instagram. By tracking sentiment, a brand can detect influencers talking about their product and engage with their fans as well.
Listen to the Voice of the Customer
Capturing the ‘voice of the customer’ means defining your target audience accurately, formulating a value proposition and changing it according to the needs of your customer. However, there’s no ready solution. The company needs to form customer voice based on various sources across multiple platforms. Sentiment analysis can help capture the “voice of the customer” and sort everything out effectively.
Comprehensive Market Research
Whether you are launching a new product or exploring a new market, sentiment analysis can help keep an eye on customers’ reactions. This allows your MVP or product to be changed and improved before it becomes too costly. By segmenting your product’s features through sentiment analysis, you can create marketing campaigns to target certain groups who have shown interest in that specific feature. Also, sentiment analysis allows you to analyze your competitors and use this information to your advantage.
Winning Customer Service
90% of US citizens consider customer service an essential factor when deciding whether or not to do business with a company. And 49% of Americans switch to competitors because of poor customer service. With the strong influence of social media on the modern consumer, one bad experience can go viral within a few hours.
With sentiment analysis, businesses can undertake risk management, create emergency plans and provide customer support teams with the best tools to face problems. Mining opinions can also help to understand how people feel at different stages of the user journey and where their biggest concerns lie.
Applying sentiment analysis to customer feedback, Unicsoft machine learning experts helped a significant e-commerce business detect the tone and temperament of customers’ social posts and categorize those sentiments. Their business goal was to increase customer loyalty, drive business changes, and deliver real return on investment. As a result, the analytical solution created by Unicsoft professionals assisted customers in developing a data-driven marketing and sales strategy, which resulted in a 10% revenue increase within one year of deployment.
Now it’s time to go deeper into how sentiment analysis and machine learning algorithms actually work.
Key Tasks Performed by Sentiment Analysis when Combined with Machine Learning and AI
Manual processing of tweets, lets say, to rate the sentiment of each statement and then draw various conclusions requires an enormous amount of manpower. Instead, by applying sentiment analysis API, one can create datasets based on certain keywords and then perform text mining. Sentiment analysis runs on top of an AI suite and can leverage machine learning algorithms. The most popular APIs for sentiment analysis are:
- IBM Watson Natural Language Understanding
- Amazon Comprehend
- Microsoft Azure Text Analytics
- Google Cloud Natural Language
- Lexalytics Semantria API
Let’s dive deeper into the tasks that sentiment analysis fulfills.
Sentiment Analysis: Sub-Tasks and Approaches
Differentiating between factual and opinionated text (Subjectivity Classification)
Subjectivity classification detects various sentiments, emotions, evaluations, etc., based on specific words and context. It is more complicated than determining polarity as various types of text like news, videos or political documents require the identification of both the topic and the attitude holder. So, for subjectivity classification, the algorithms must recognize opinionated language and distinguish it from the objective text. Though ML is more efficient, the lexicon-based approach is used as well.
Determination of positive, negative, or neutral text (Sentiment Classification)
Using ML, one can classify sentiment accurately. There are two approaches, supervised and unsupervised.
- In supervised sentiment classification, you have to teach machine learning algorithms how each word influences the overall sentence being positive, negative, or neutral. This requires manually tagged words or libraries of pre-trained NLP models with sentiment scores.
- An unsupervised approach presupposes that you provide data, and the AI has to learn the structure of the given data on its own.
The lexicon-based approach is also used for sentiment classification – in a dictionary (with positive-negative labels on the words) or corpus-based (defining sentiment in conjunction with a set of positive or negative words it is used with) way.
Definition of positive vs negative sentiment (Polarity Determination)
Polarity determination is a primary task of sentiment analysis and can be performed with machine learning, lexicon-based and hybrid approaches. The determination of positive, negative or neutral attitudes is applied for any kind of text, from product reviews to forums and social media. Video and image content can be analyzed also.
Differentiating emotions and sarcasm in the text (Resolving Ambiguity in Opinionated Text)
Since the majority of texts are emotion-colored and include rhetorics, metaphors, sarcasm, comparison, etc., the detection and understanding of these nuances is a challenging task of opinion mining.
The AI model is trained with specific sets of preprocessed data where words and sentences are given specific scores: positive/negative words, level of pleasantness, a humorous property of the sentence, etc. There are also attempts to “classify” sarcasm and ugliness.
Cross-language analysis (Multilingual Sentiment Analysis)
This kind of sentiment analysis utilizes both machine learning and hybrid approaches.
In cases where cross-language analysis is needed, machine translation using Google Translator API, for instance, precedes actual sentiment analysis. Also, analysts use labeled lexicons – bilingual dictionaries with polarity or sentiment tags (positive, negative, neutral, offensive, etc). The text is then classified into datasets and analyzed.
Detection of fake reviews (Opinion Spam Detection)
Marketing strategies can involve writing fake reviews and piling up positive feedback. Spam comments can be used both to promote a low-quality product or discredit a good one. Spam detection aims to confirm whether a review is written by an actual customer or not. For this to happen, one must analyze three features:
- text: NLP and machine learning algorithms can uncover deception in the content
- meta-data: valid user ID, location, IP address
- real-life knowledge: brand monitoring and competitor analysis, with the help of sentiment analysis
Types of Sentiment Analysis
Sentiment analysis depends on well-structured training data. Sentiment analysis is a common NLP task but without training data, it can’t be efficient. While NLP accurately distinguishes the feelings in sentences and classifies text using machine learning, part-of-speech tagging is vital to sentiment analysis because it assigns nouns, verbs, adjectives, etc. to words in the sentence. It’s almost impossible to conduct sentiment analysis without part-of-speech tagging.
Fine-grained
Fine-grained sentiment analysis involves expanding polarity categories to their most detailed. We use this concept most often when we need maximum accuracy in our analysis. For example, 5-star ratings can be interpreted like this:
5 stars – Very Positive
4 stars – Positive
3 stars – Neutral
2 stars – Negative
1 star – Very Negative
This approach allows us to assess sentiment with greatest accuracy. On the other hand, it’s more time-consuming and costly compared to other methods.
Emotion detection
Emotion detection allows you to analyze a text according to its emotional temperament, such as joy, anger, sadness, or frustration. Emotion detection sentiment analysis pinpoints the feelings that color the vocabulary of any given text. In other words, it identifies the emotions the objects described evoke in the author. The method requires lexicons (lists of words and corresponding emotions) or complex sentiment analysis machine learning algorithms alongside support vector machines.
Lexicons are suitable for detecting overt emotions. For example, in the sentence “I expected the download to be faster”, the analysis will reveal not only a negative sentiment, but also disappointment. In the sentence “I’m worried that the next episode of The Avengers won’t have my favorite superhero”, the analysis will detect a negative feeling and concern. In the sentence “I can’t wait for my new iPhone!”, the analysis will reveal a positive sentiment and hope.
Sophisticated machine learning for sentiment analysis is much more efficient than lexicons. There are many examples where the same word in different contexts shows different emotions. For example, the word “kill” in the sentence “your dog has killed him” expresses a negative, while in the sentence “yes, you are killing the opponent!” convey joy. For that reason, it’s preferable to employ complex machine learning algorithms.
Aspect-based
We use this approach most often to analyze product reviews, as it allows us to determine the basic spectrum of emotions people reveal in their comments. Thus in the sentence “It would be better if the phone battery had a longer life, so I’m not satisfied”, the aspect-based method will detect that it’s a negative review. It helps to understand what customers think about specific product attributes.
The aspect-based analysis provides important information about the different attitudes customers have toward products and services. Let’s say you own a well-known restaurant. With the help of aspect-based analysis, you may find that your customers don’t like a certain sauce in your burgers, and some of them refuse to buy it solely for that reason.
Intent Analysis
This approach helps to understand a person’s intentions. For example, intent analysis can show whether someone intends to buy your product or not. We can trace the intentions of every regular or potential customer by forming a pattern, and then using it for marketing and advertising.
The basic idea is that intent analysis can find out all intentions. It can provide us with many insights we won’t find anywhere else. Intent analysis frees employees from mundane tasks, improves workflow, and allows them to pay attention to the goals and tasks that really matter.
Sentiment Analysis Challenges
Sentiment analysis in machine learning is still rapidly evolving. The learning process of algorithms becomes easier year on year.. In fact, the use of AI and ML has become a leading competitive advantage in all spheres. In spite of this, sentiment analysis faces several difficulties. Experts from all over the world are trying to solve these to make the use of sentiment analysis more optimal, accurate, and functional.
Comparisons
Comparisons complicate the learning process of sentiment analysis because they introduce additional elements of assessment into a text. Such additional elements are sometimes tricky to identify. Separating the judgements they make is also a challenge. For example, in a review of the film Avatar, a reviewer might mention two other films:
“The movie is great. It’s won a lot of awards. But I wouldn’t say it’s better than James Cameron’s earlier films. I liked his 2002 film “Solaris” much better and I raved about his even earlier 1997 film “Titanic”.
For the machine learning-based sentiment analysis, this example is not the most difficult, as the reviewer expresses similar feelings about all of the films they mention. A more complex situation arises in the following passage from a restaurant review:
“I visited this place quite often in winter and was delighted. Everything was at its best – the atmosphere was friendly, the service was super… Went there on a recent weekend, and was disappointed”
There are a lot of positive words, but these don’t apply to the current visit to the restaurant. In addition, it’s difficult to determine that this review contains a comparison, as it compares between objects, but rather makes a semantic comparison between different elements in the text.
Irony and Sarcasm
Online communication often contains sarcastic and ironic phrases that humans find difficult to identify, let alone natural language processing algorithms, with their bag of words model. The bag of words model is used in document classification methods where training sets are necessary. Handling irony and sarcasm presents serious classification problems for sentiment analysis systems, because the tone of an ironic or sarcastic statement differs from its literal meaning.
At the heart of irony lies the discrepancy between the literal meaning of a sentence and its implied meaning. Often, a positive-looking statement, containing more positive evaluative words or an equal number of positive and negative words, hides a negative opinion. For example, “This bank has the largest network of non-working ATMs in Europe”. Spotting irony and sarcasm in a text is a complex task for sentiment analysis using machine learning.
Factors of “unreal” context in sentiment analysis
When analyzing sentiment, it’s important to consider how much the tone we are evaluating matches reality. For example, the sentence “We’d hoped we would like the movie” uses the word “like” in a positive tone, but it doesn’t tell us whether we actually liked the movie. In the automatic sentiment analysis model, we should not take the word ‘like’ into account as evidence of a positive attitude towards the film.
Polarity modifiers: negation, intensifiers, etc.
In many texts, modifiers will accompany opinion words. These modifiers might intensify (‘very’, ‘more’), reduce (‘too’, ‘less’) or convert to the reverse meaning (‘not’, ‘no’). Thus when analyzing sentiment, the system must consider such modifiers and have a numerical model that modifies the original polarities of the word. One common model for treating tonality modifiers assigns them coefficients, which the system handles as multipliers of the polarity of the modified words.
Another important problem is deciding the scope of a polarity modifier in a particular sentence, such as negation. For example, in the sentence “I don’t like the design of the new model, but it has some interesting features”, “does not” refers only to the word “like”. It doesn’t modify the polarity of the word “interesting”.
Explicit and implicit evaluations
We usually assume that sentiment is expressed through evaluative vocabulary, which is an explicit way of saying what we feel. However, we can also express our sentiments in an implicit way, sharing facts that suggest a value judgement or using words with connotations.
For example, in restaurant reviews, there may be sentences like “waited a long time” or “a fly was floating in the soup”, which, on the one hand, describes what is happening (i.e., they report real facts), and on the other, evaluates the event.
An implicit opinion is an objective statement from which an evaluation follows. An implicit opinion communicates a desirable or undesirable fact, while connotations are evaluative associations of words. The appearance of words with positive or negative connotations correlate with corresponding evaluations in the text. Thus in movie reviews, names of famous actors usually have positive connotations.
Obviously, it’s tricky to analyze such implicit types of connotations, as it depends to a large extent on the subject matter, and evaluation facts can’t be collected in advance. Finding and extracting these facts from texts is also challenging because their expression varies
The multiple meanings of evaluative vocabulary. Context determines sentiment.
Difficulties may also arise with the interpretation of explicit evaluative vocabulary. Words can have several meanings, and they can be neutral in one sense and negative or positive in other senses. For example, the word “fresh” in the phrase “fresh water” is neutral, perhaps with some positive connotations. While in the phrase “fresh comments”, this word carries a negative sentiment.
A word can change its polarity or lose its polarity depending on the subject matter or the current context. The words “mean” and “betrayal” are not evaluative in film reviews because they can’t be used to judge the quality of films. The word “funny” is likely to be negative in the realm of politics, but positive when it comes to comedies. When characterizing film genres, the word can be either positive or negative.
Within a subject area, evaluation words can convey a positive or negative tone, depending on the characteristics of the object to which they apply. For instance, the word “long” can be both negative and positive in the subject area of digital cameras: “they say that a battery lasts a long time” – positive sentiment, “they say it takes a long time to adjust the focus” – negative sentiment.
Summing It Up
Recognizing that customer experience drives business performance, brands are taking a smarter approach to market research and sales strategies. Financial institutions and political parties also recognize the importance of collecting and analyzing opinions.
Combined with machine learning, sentiment analysis is a powerful tool with multiple applications across different industries. It is already influencing the way brands approach marketing, and the impact will be even more visible as AI becomes smarter and ML algorithms become more advanced. By understanding what makes your customers tick, you can resolve the pain points, anticipate wishes and predict problems.
Do you want to see how Unicsoft can apply the combination of machine learning and sentiment analysis in your business domain to make your brand performance more data-driven & client-oriented? Contact me today!
