Sentiment Analysis of Twitter Data Using Python

I’m sure this has been covered by many people before but I wanted to explore the current state of natural language processing and have chosen as an example of this to pull some data relating to various stocks from twitter and from there explore how we can automatically evaluate average user sentiment.

Creating a Twitter Developer Account and App

The Twitter Developer site can be found here. From there you can register an app by clicking the ‘Create New App’ button and filling in the required fields. Registration of an app gives you the various keys you need to pull data automatically from Twitter. We’ll come back to how we use them a bit later.

Google Natural Language API

This seems like a really good solution. I’d trust Google to provide a best in class service here. According to Google the first 5K request are free, then you are billed $1 per month for the next million or so. I’m probably going to exceed 5K per month just testing this small test-app alone, but I can afford a dollar per month, so that’s fine.

We can setup access to this API from here. Google require billing details in order to activate this service. So far, so good, until we try to setup the billing details, only to find that Google Cloud Services are only available for business users. I’m not a business user :-(.

So I’m stuck looking for another solution.

Stanford CoreNLP

This is available from here. It takes the form of a Java app that spins up a server on your PC that local processes can then access via an API.

To run this, there is a large download, after which you need to install the JDK from here, and finally spin up the server by executing “java -mx4g -cp “*” edu.stanford.nlp.pipeline.StanfordCoreNLPServer” from the command line.

For testing, to check the service is running, you can access the service from the browser using http://localhost:9000/ which should present you with a nice UI and and various visual representations of the analysis that’s carried out.

Python

I’m going to build a simple app using Python. There are API’s for all the services I want to use that can be accessed easily from python so it’s a great way of bringing them together. For reference I’m using python 3.5.1.

To access to Twitter data I’m using python-twitter from here. That can be installed easily, along with a module that allows my app to talk to the CoreNLP server, by executing the following commands:

python -m pip install python-twitterpython

python -m pip install pycorenlp

Working Example #1

This program will download the most recent 100 English tweets relating to Microsoft stock and print them out. Replace the XX fields with values taken from the Twitter application page earlier.

import twitter
from pprint import pprint
from pycorenlp.corenlp import StanfordCoreNLP

api = twitter.Api(
 consumer_key='XX',
 consumer_secret='XX',
 access_token_key='XX',
 access_token_secret='XX'
 )

host = "http://localhost"
port = "9000"
nlp = StanfordCoreNLP(host + ":" + port)

search = api.GetSearch(term='$MSFT', lang='en', result_type='recent', count=100, max_id='')
for t in search:
 
 text = t.text.encode('utf-8')
 print(text)
 
 output = nlp.annotate(
    t.text,
    properties={
        "outputFormat": "json",
        "annotators": "depparse,ner,entitymentions,sentiment"
    }
 )

 for s in output["sentences"]:
  print("%d: '%s': %s" % (
   s["index"],
   s["sentimentValue"].encode('utf-8'), 
   s["sentiment"].encode('utf-8')))

Working Example #1 – Results

In short the results are not great. Almost everything is classified as either Neutral or Negative for some reason. The very few that turn positive are typically garbage posts that have nothing to do with the subject at all. For example I’ve fed the app the tickers of the top 5 tech companies and in each case sentiment was heavily negative. Microsoft fared worst, and didn’t have a single positive post, though that might be because of the negative headlines surrounding the current WannaCrypt outbreak, but generally this doesn’t feel right, and doesn’t feel representative of actual user sentiment.

With a bit more research I can see that I’m not the first person to hit this problem. Apparently the Stanford NLP service is trained using movie reviews and the resulting analysis doesn’t fit twitter feeds very well.

Again, I’m looking for another solution.

VADER

VADER is a python module specifically written to perform twitter analysis. On the face of it, it’s a less impressive piece of technology when compared to the Google and Stanford offerings, but being more focused on the task in hand I’m hoping for a good result.

The library is documented on it’s GitHub page here, and can be installed by executing the following command:

python -m pip install vaderSentiment

Working Example #2

As before, this example will grab the most recent 100 posts relating to Microsoft stock and hand them to the sentiment analyzer. This time as well as gathering per post scores we also compute and print an average.

import twitter
from vaderSentiment.vaderSentiment import SentimentIntensityAnalyzer

api = twitter.Api(
 consumer_key='XX',
 consumer_secret='XX',
 access_token_key='XX',
 access_token_secret='XX'
 )

scores = []

search = api.GetSearch(term='$MSFT', lang='en', result_type='recent', count=100, max_id='')
for t in search:
 
 text = t.text.encode('utf-8')
 print(text)

 sentence = t.text
 
 analyzer = SentimentIntensityAnalyzer()
 vs = analyzer.polarity_scores(sentence)
 print("{%s}" % format(str(vs)))

 scores.append(vs['compound'])

average = 0.0
for score in scores:
 average += score
average /= len(scores)
print("Average = %f" % average)

Working Example #2 – Results

The results are much better. The scores seem to for the most part be a genuine reflection of sentiment! So far so good…

Working Example #3 – Charting the results

As a further experiment I’m going to see what it would take to create charts from the results. There is a library for Python called matplotlib that seems to be more or less what I’m after. You can feed it a data set and with just a few commands it produces a chart of the data.

I’ve install version 1.5.1 of the module because the latest version seems to be broken and fails with missing dependencies on Freetype and PNG modules, but this version seems to work just fine and can be installed like so:

python -m pip install matplotlib==1.5.1

This code shows how we would adapt the previous example code to produce a chart. In this case we are pulling data for each day across a specified time-frame. Then for each day we produce an average of the sentiment scores. Finally we hand the data to the charting module to plot sentiment against time, with dates along the bottom.

import time
import twitter
from time import mktime
from datetime import datetime
from datetime import timedelta
from vaderSentiment.vaderSentiment import SentimentIntensityAnalyzer
import matplotlib.pyplot as plt
import matplotlib.dates as mdates

api = twitter.Api(
 consumer_key='XX',
 consumer_secret='XX',
 access_token_key='XX',
 access_token_secret='XX'
 )

start_date = "2017-05-05"
stop_date = "2017-05-13"
start = datetime.strptime(start_date, "%Y-%m-%d")
stop = datetime.strptime(stop_date, "%Y-%m-%d")

scores = []
dates = []

while start < stop:

 start = start + timedelta(days=1)  # increase day one by one
 start_p1 = start + timedelta(days=1)  # increase day one by one

 start_str = start.strftime("%Y-%m-%d")
 start_p1_str = start_p1.strftime("%Y-%m-%d")

 day_score = 0.0
 day_count = 0

 search = api.GetSearch(term='$MSFT', lang='en', result_type='recent', count=1000, max_id='',
  since=start_str, until=start_p1_str)

 for t in search:
 
  analyzer = SentimentIntensityAnalyzer()
  vs = analyzer.polarity_scores(t.text)

  day_score += vs['compound']
  day_count += 1

 if day_count > 0:

  day_score = day_score / day_count

  print("%s : %f" % (start_str, day_score))

  scores.append(day_score)
  dates.append(start)

x = dates
y = scores

plt.gca().xaxis.set_major_formatter(mdates.DateFormatter('%m/%d/%Y'))
plt.gca().xaxis.set_major_locator(mdates.DayLocator())
plt.plot(x, y)
plt.gcf().autofmt_xdate()
plt.show()

Working Example #3 – Results

The result looks something like this!

Interestingly Microsoft’s stock price climbed during this period and then fell somewhat on the 11th May. The sentiment chart doesn’t seem too far off, though in this case at best it seems to be a trailing rather than leading indicator of performance, or maybe it doesn’t mean anything at all.

Leave a Reply

Your email address will not be published. Required fields are marked *