Twitter Cashtag NLP Project Part 1: Getting Started with the Twitter API!

Kicking off a new natural language processing project using the Twitter API

Graphic created by the author

Hello there friends! I’m excited to kick off a new project here that I think will be a fun one for learning purposes. Like many people in the data world, I have been equally curious if we can use data science in some capacity to be able to predict stock market trends. Naturally, many people have taken the approach of using time series forecasting against historical stock data, and as you might guess, markets are so volatile that time series forecasting is unfortunately not super predictive here.

Now, I’m a pretty active Twitter user (my handle is “dkhundley”), and it’s always fascinated me how Twitter seems to be like this mass global consciousness for any given topic. That’s obviously a bit of an exaggeration, but if you are familiar with stock market stories like GameStop or Bed Bath & Beyond, social media has definitely had some level of influence on the stock market for better or worse. So the question on my mind is, can we use natural language processing’s sentiment analysis capabilities to see if we can assess if Twitter really does have an impact on the stock market?

Before going on, I think I should put out a disclaimer: This project is NOT actually trying to predict the stock market and should NOT be used for any fiduciary reasons! I’m putting out this disclaimer partially because I feel obligated to, but I’m mostly stating this because I don’t actually think our end product will actually be that predictive. 😂 Like I noted above, many people have tried to use data science / machine learning to predict the stock market, and I’m not naive enough to believe that my approach here will be any better than other people’s approaches. (In fact, I would be willing to bet somebody has done a very similar project with tweets.)

Anyway, this project is intended to be both fun and a good stage for helping us to learn some new skills in the natural language processing (NLP) world! You’re probably already familiar with a hashtag (e.g. #NLP, #TwitterRocks), which are essentially tags that you can later click on and see what other users have used the same hashtag. What you may not know is that there is…