2.6 Project: Finding the Tweets with Negative Sentiment about a Product
Contents
2.6 Project: Finding the Tweets with Negative Sentiment about a Product#
Now that we know the basics of using Hugging Face, let’s do some simple projects leveraging pre-trained models from the Hub. In this project, we’ll find the tweets with positive and negative sentiment regarding specific products.
Why Looking for Tweets with Negative Sentiment about a Product#
Companies spend a lot of money on making their customer happy, especially with customer care services. Unhappy customers express themselves in several ways, and one of them is writing on social media. Bad posts about a company’s products on social media have negative consequences, because it makes the products or services appear to be of low quality and that little importance is given to the customers.
Consequently, intercepting posts with negative sentiment about products or services is very useful because it allows the company to identify dissatisfied customers and give them immediate support. Also, tracking the number of posts with positive and negative sentiment allows you to get an idea of the results of your customer care efforts
Sentiment Classification on Tweets#
In this project, we assign a sentiment to tweets about a specific product. For example, let’s try to do this with tweets containing the words “macbook” or “iphone”.
Look for Suitable Pre-trained Models#
As the first step, let’s look for pre-trained models on the Hugging Face Hub that can compute the sentiment of tweets. Here are three popular models:
distilbert-base-uncased-finetuned-sst-2-english: It’s a checkpoint ofDistilBERT-base-uncasedfine-tuned on SST-2. DistilBERT is a smaller and faster version of BERT obtained by knowledge distillation. SST-2 is the Stanford Sentiment Treebank dataset, suitable for binary sentiment classification and made of movie reviews.cardiffnlp/twitter-roberta-base-sentiment-latest: It’s aroBERTa-basemodel trained on ~124M tweets from January 2018 to December 2021, and finetuned for sentiment analysis with the TweetEval benchmark.ProsusAI/finbert: FinBERT is a pre-trained model to analyze the sentiment of financial text, fine-tuned on the finance domain using the Financial PhraseBank financial corpus. It performs financial sentiment classification.
We won’t use the ProsusAI/finbert model as we want to compute the sentiment of generic tweets without a focus on the financial domain. The distilbert-base-uncased-finetuned-sst-2-english model has been trained on movie reviews, which are different from tweets but may still produce rather good results. Last, the cardiffnlp/twitter-roberta-base-sentiment-latest has been trained specifically on tweets, so it’s probably the best for our use case.
In case we didn’t find a suitable pre-trained model, we would have had to train one, using a suitable dataset. In this chapter, we will see what can be done using only pre-trained models that we can find in the Hub. In a later chapter, we will learn how to fine-tune these models.
Install and Import Libraries#
Let’s install the necessary libraries. The dataset library allows to easily download datasets from the Hub.
pip install transformers datasets
Then, we import the load_dataset function, the pipeline function, and pandas.
from datasets import load_dataset
from transformers import pipeline
import pandas as pd
Get Tweets about MacBooks#
In a real-world scenario, we would use the Twitter API to download the latest tweets posted containing specific keywords. In this sample project, we download the test split of the Twitter Sentiment Analysis Training Corpus.
# download the tweets dataset
dataset = load_dataset("carblacac/twitter-sentiment-analysis", split="test")
print(dataset)
Dataset({
features: ['text', 'feeling'],
num_rows: 61998
})
The load_dataset returns an instance of the Dataset class. We’ll learn more about it in a future lesson.
The dataset has ~62k tweets, whose content is in the “text” field and whose ground-truth sentiment is in the “feeling” field. Let’s convert this dataset to a pandas dataframe and drop the “feeling” column.
# convert dataset to pandas dataframe
df = pd.DataFrame(dataset).drop("feeling", axis=1)
df.head()
| text | |
|---|---|
| 0 | @justineville ...yeahhh. ) i'm 39 tweets from ... |
| 1 | @ApplesnFeathers aww. Poor baby! On your only ... |
| 2 | @joeymcintyre With my refunded $225 (Australia... |
| 3 | It's fine. Today sucks just because me those t... |
| 4 | Im just chilling on psp and stuff, but sitting... |
We can now filter the tweets and keep only the ones whose text contains the “macbook” word.
# keep only the tweets with a specific filter word. Let's try with "macbook"
filter_word = "macbook"
df_subset = df[df["text"].str.contains(filter_word)].reset_index(drop=True)
df_subset.head()
| text | |
|---|---|
| 0 | @BergenLarsen the max virtualbox will allow. m... |
| 1 | "practicing" on my daughter's macboo... |
| 2 | @HighTechDad havent tried that yet.. only had ... |
| 3 | Nofx new album, uh yes please. Also, why cant ... |
| 4 | @Livingdeadpingu What's your point? lol I had ... |
Using the Tweets Sentiment Classification Model#
We are now ready to download the pre-trained sentiment classification model. Let’s leverage the Pipeline class we learned about in the previous lessons. We also pass the parameter device=0 to leverage the GPU (in case it’s CUDA-compatible) and make computations faster. If you can’t use a GPU, delete the device parameter or use it as device=-1 to use the CPU.
# download pre-trained tweet sentiment model
model = pipeline("sentiment-analysis", model="cardiffnlp/twitter-roberta-base-sentiment-latest", device=0)
Then, we use the model to compute the sentiment of each tweet. For each tweet, the model returns a dictionary containing the predicted label (i.e. one of POSITIVE, NEGATIVE, or NEUTRAL) and it’s score.
# compute the sentiment of each tweet using the model
all_texts = df_subset["text"].values.tolist()
all_sentiments = model(all_texts)
df_subset["sentiment_label"] = [d["label"] for d in all_sentiments]
df_subset["sentiment_score"] = [d["score"] for d in all_sentiments]
df_subset.head()
| text | sentiment_label | sentiment_score | |
|---|---|---|---|
| 0 | @BergenLarsen the max virtualbox will allow. m... | neutral | 0.549103 |
| 1 | "practicing" on my daughter's macboo... | neutral | 0.490027 |
| 2 | @HighTechDad havent tried that yet.. only had ... | negative | 0.556181 |
| 3 | Nofx new album, uh yes please. Also, why cant ... | positive | 0.662267 |
| 4 | @Livingdeadpingu What's your point? lol I had ... | positive | 0.383863 |
That’s it! Let’s show some tweets whose sentiment label is NEGATIVE.
negative_tweets = df_subset[df_subset["sentiment_label"] == "negative"]
top_negative_tweets = negative_tweets.sort_values(by="sentiment_score", ascending=False)
top_negative_tweets = top_negative_tweets.reset_index()
print("Negative tweets:")
for i, row in top_negative_tweets.iterrows():
print(f"- {row['text']} ({row['sentiment_score']:.3f})")
if i == 10:
break
Negative tweets:
- I think the harddrive on my beloved macbook just died. Ill be in mourning for 40 days. (0.926)
- So Apple just rerouted my new macbook AND cancelled my 3GS pre-order AFTER my phone was shown as being upgraded so I can't reorder. WTF (0.877)
- my macbook is starting to crack at the bottom.. is this normal !?!?! (0.871)
- Just about to 2 plug in new macbook when biggest bolt of lightning flashed. Can't destroy 2 in 1 week so still on crappy old Toshiba. (0.825)
- @HighTechDad havent tried that yet.. only had a few minutes to play on the macbook at home this morning and cant run tweetdeck at work (0.556)
- I wish I had a macbook. I wanna be on the computer but I wanna lay in bed what a dilemma (0.469)
Once you have the list of tweets with negative sentiment, there are many actions you could take. For example, the most recurring problems could be answered through your official support account, or you could even just analyze the volumes and discover weaknesses of your products.
Let’s see some tweets whose sentiment label is POSITIVE as well.
positive_tweets = df_subset[df_subset["sentiment_label"] == "positive"]
top_positive_tweets = positive_tweets.sort_values(by="sentiment_score", ascending=False)
top_positive_tweets = top_positive_tweets.reset_index()
print("Positive tweets:")
for i, row in top_positive_tweets.iterrows():
print(f"- {row['text']} ({row['sentiment_score']:.3f})")
if i == 10:
break
Positive tweets:
- happy 16th too me... Got a macbook :O (0.985)
- out around town with my macbook pro...i love unprotected wifi networks (0.960)
- Hanging out with my cousin in my room. She has a macbook So envy her. (0.857)
- @novelisa ok the u can buy the macbook, just buy ok don`t worry, u have all the money (0.813)
- Nofx new album, uh yes please. Also, why cant macbooks have more usb ports (0.662)
- messing around withh my friends macbook pro. i don't want to put it downn! haha. (0.656)
- @Livingdeadpingu What's your point? lol I had a macbook too but I killed it with fire! (or salad cream to be accurate ) (0.384)
Even with tweets with positive sentiment, action can be taken. For example, you could wish a happy birthday to someone happy that they bought your product for their birthday.
Test with Tweets about iPhones#
Let’s repeat the whole process with tweets containing the keyword “iphone”. Read the tweets with negative and positive sentiment and think about actions that could be taken!
# keep only the tweets with a specific filter word. Let's try with "iphone"
filter_word = "iphone"
df_subset = df[df["text"].str.contains(filter_word)].reset_index(drop=True)
# compute the sentiment of each tweet using the model
all_texts = df_subset["text"].values.tolist()
all_sentiments = model(all_texts)
df_subset["sentiment_label"] = [d["label"] for d in all_sentiments]
df_subset["sentiment_score"] = [d["score"] for d in all_sentiments]
# show tweets with negative sentiment
negative_tweets = df_subset[df_subset["sentiment_label"] == "negative"]
top_negative_tweets = negative_tweets.sort_values(by="sentiment_score", ascending=False)
top_negative_tweets = top_negative_tweets.reset_index()
print("Negative tweets:")
for i, row in top_negative_tweets.iterrows():
print(f"- {row['text']} ({row['sentiment_score']:.3f})")
if i == 10:
break
print()
# show tweets with positive sentiment
positive_tweets = df_subset[df_subset["sentiment_label"] == "positive"]
top_positive_tweets = positive_tweets.sort_values(by="sentiment_score", ascending=False)
top_positive_tweets = top_positive_tweets.reset_index()
print("Positive tweets:")
for i, row in top_positive_tweets.iterrows():
print(f"- {row['text']} ({row['sentiment_score']:.3f})")
if i == 10:
break
Negative tweets:
- @benjorg you are a traitor. how the eff am I supposed to bbm you now? Unless your stupid iphone made a stupid app for it. ;) ha. (0.963)
- ok so this sucks i guess i have to have an iphone or an itouch to win tickets. soo sad (0.936)
- yeah #iphone relase is tomorrow..not happy jan...Please explain Apple (0.918)
- @tysiphonehelp I don't have any requests. I am on vacation with my friends family and I don't get mine until tomorrow night depressing. (0.917)
- @peterfacinelli I don't have an iphone...that makes me sad.... (0.910)
- ugh iphone 3.0 not installing (0.909)
- Tried to replace the screen protector myself on my iphone....Cocked it up three times!!! Waste of ã7.99..now no screen protection... (0.898)
- All my calendar data between my iphone and ical/entourage has disappeared so annoying, I wish they would just sort this issue out. (0.896)
- @reddawn lost my iphone ... sob sob ~ (0.890)
- waited in line for 2 hrs to find that my dad needs 2 b there for me to get a new iphone. saddest story ive ever heard. there were tears. (0.882)
- Why hasn't my iphone shipped yet!! (0.865)
Positive tweets:
- Another day of sun, fun and riding the bike. So gonna sit in park later and use shiny, new iphone, yay! (0.987)
- playing with emixs iphone to funny emex i luv u *_* (0.976)
- @joniVanBogaert this book: http://bit.ly/dHsto, and also obj C and iphone dev of the same serie happy reading (0.971)
- OMG! New iphone software! Get crazy! Get psyched! Get the app now! .... I don't have an iphone (0.961)
- found a big list of free iphone apps! (i know they are on my apps button but easier to scan here...) http://bit.ly/KMJuN (0.948)
- Got my iphone and have to call Apple - just like the first one. Once they're activated; they're great. (0.946)
- I'm seriously loving TweetDeck. I'm actually using Twitter properly now! Still no #iphone 3.0 #jailbreak. Good night! (0.945)
- Anyone giving away an iphone lol i really want one anyone generous =] (0.924)
- I want an iphone! (0.903)
- is loving the new Twitteriffic iphone app. I just paid for and was getting used to tweetie, but now it's back to Twitterific... $3 gone (0.902)
- Good afternoon people about to head to nj than going to get me my new iphone station (0.902)
Code Exercises#
Questions and Feedbacks#
Have questions about this lesson? Would you like to exchange ideas? Or would you like to point out something that needs to be corrected? Join the NLPlanet Discord server and interact with the community! There’s a specific channel for this course called practical-nlp-nlplanet.