How to Perform Sentiment Analysis on Twitter Feeds

How can we use a sentiment analysis API to analyze twitter feeds?

Politicians, CEOs, and other famous people use Twitter to share whatever is on their minds.  Many use Twitter as a soapbox to push their ideas.  Others use it to start public conversations.  In light of this, we will use a sentiment analysis API to determine sentiment scores of tweets.  These scores can populate trend charts over time.  We can use these trend charts to help us compare political candidates or even make investment decisions.

Related APIs: Moodli provides Real-time sentiment analysis of tweets containing covid19–related keywords + geo-tagging each tweet.

What is Sentiment Analysis?

Sentiment Analysis is described in the overview of the Sentiment Analysis API by Twinword like this:

Is this comment positive or negative? Find out the tone.

The usage is very simple.  First, we send a text string to the API as the main input parameter.  The API returns a score, judgment, and breakdown by keyword.  The judgment (or ‘type’) is either positive, neutral, or negative.  This data comes back from the API in a JSON formatted string.

How to Perform Sentiment Analysis of Twitter feeds using the API

1. Sign Up for a free account at RapidAPI.com
2. Subscribe to the Sentiment Analysis API – It is free to use up to 500 API calls per month.  Here is the full pricing table for the API:
3. Connect to Sentiment Analysis API using the language of your choice from the API Endpoints page.  For this example, we’ll be using PHP.
4. Create a list of tweets as text strings for a given Twitter handle – Twitter has its own API but it’s a fairly involved process to set up so I’ll take you through a shortcut.  There is a site at TwitRSS.me which parses twitter feeds to generate RSS based on either a twitter handle or search term.  In the command line interface example below I used that service to come up with the list of the most recent 20 tweets for a given Twitter handle.

Logic used for this example

1. Ask the user for a twitter handle
2. Use twitrss.me tool to generate an RSS feed
3. Loop through the latest tweets in the RSS feed
   1. Use the Sentiment Analysis API to analyze the text of each tweet
   2. Parse the returned JSON string and collect the results for statistics
4. Analyze the collected data to come up with statistics for this twitter handle

Full Example Code

<?php
/*
File: get_twitter_sentiment.php
Description:
  This script is a command line interface tool.  It pulls sentiment scores from
  the latest tweets of a provided twitter handle.  The scores are retrieved
  from a sentiment analysis api and then the script prints a summary.
Inputs:
  User is asked for a twitter handle
Outputs: The results will look like this - "The latest X tweets were posted
  between <first tweet timestamp> and <last tweet timestamp>.
  A are positive, B are neutral, and C are negative.
  The average sentiment score was <sentiment score>.
  The most positive tweet (scoring <sentiment score>) was <tweet title> tweeted on
  <timestamp> while the most negative tweet (scoring <sentiment score>)
  was <tweet title> tweeted on <timestamp>"
*/
global $debug;
$debug=false;

//First ask for a twitter handle
echo "Please enter a twitter handle (enter 'exit' to stop):n";

//Wait for user input within a loop that stops when 'exit' is received
do{
  //Loop until we get a line of user input
  do{
    $handle = trim(fgets(STDIN));
  } while($handle == '');

  //If the text entered is not 'exit' then treat it like a twitter handle
  if(strcasecmp($handle, 'exit') != 0){
    echo "Please wait while we analyze the latest tweets from the $handle handle...n";
    $thisAnalysis = getSentimentAnalysis($handle);
    echo "n".$thisAnalysis."n";

    //Allow for another analysis or exit
    echo "Please enter a twitter handle (enter 'exit' to stop):n";
  }
} while(strcasecmp($handle,'exit') != 0);
// Exit when we're done
exit(0);

//Return a summary of the sentiment analysis for the latest tweets from $handle
function getSentimentAnalysis($handle) {
  global $debug;
  $summary='';
  $minTweet=$maxTweet=$minTimeStamp=$maxTimeStamp=$startTimeStamp=$endTimeStamp='';
  $numTweets=$totalTweetScore=$aveScore=$minScore=$maxScore=0;

  //First download the latest tweets from the handle in rss format
  $tweetRSS = file_get_contents("https://twitrss.me/twitter_user_to_rss/?user=".$handle);
  //Update progress for user
  echo "Twitter feed generated, now we'll use the API to analyze the sentiment of each tweet.n";
  //Next convert API response to xml
  $xml = simplexml_load_string($tweetRSS);
  //convert xml to json
  $json = json_encode($xml);
  //convert json to array
  $tweets = json_decode($json,TRUE);

  //Loop through tweets and pull sentiment info for each
  foreach($tweets['channel']['item'] as $item) {
    //Newest are first
    if (empty($endTimeStamp)) $endTimeStamp=$item['pubDate'];

    //Get sentiment info for this tweet
    $apiResponse = getAPIResponse($item['title']);
    $thisScore = $apiResponse->score;
    $thisType = $apiResponse->type;

    //Keep track of the number of tweets by type
    if (empty($numTweetsByType[$thisType])) {
      $numTweetsByType[$thisType]=1;
    } else {
      $numTweetsByType[$thisType]++;
    }

    //Keep track of extremes
    if (empty($minScore) or $thisScore<$minScore) {
      $minScore=$thisScore;
      $minTweet=$item['title'];
      $minTimeStamp=$item['pubDate'];
    }
    if (empty($maxScore) or $thisScore>$maxScore) {
      $maxScore=$thisScore;
      $maxTweet=$item['title'];
      $maxTimeStamp=$item['pubDate'];
    }

    //Track tweet count and total
    $numTweets++;
    $totalTweetScore+=$thisScore;

    //Show user that we are making progress - each . is a tweet
    echo ".";
    if ($debug and $numTweets>5) break; //for debug limit api calls
  } //end loop through tweets

  //Oldest tweet is last
  $startTimeStamp=$item['pubDate'];

  //Get average sentiment score and create a note about sentiment types
  if ($numTweets>0) {
    $aveScore = $totalTweetScore/$numTweets;
    foreach($numTweetsByType as $tweetType=>$numTweetsForType) {
      $connector = ($numTweetsForType==1)?"is":"are";
      $typeNoteArr[] = $numTweetsForType." ".$connector." ".$tweetType;
    }
    if (count($numTweetsByType)==1) {
      $typeNote = $typeNoteArr[0];
    } else if (count($numTweetsByType)==2) {
      $typeNote = $typeNoteArr[0]." and ".$typeNoteArr[1];
    } else if (count($numTweetsByType)==3) {
      $typeNote = $typeNoteArr[0].", ".$typeNoteArr[1].", and ".$typeNoteArr[2];
    }
  }

  //Create a summary of sentiments to return
  if ($numTweets>0) {
    $summary = "The latest $numTweets tweets were posted between $startTimeStamp and $endTimeStamp.  n".
      "$typeNote.  n".
      "The average sentiment score was $aveScore. n".
      "The most positive tweet (scoring $maxScore) was - n $maxTweet n".
      "tweeted on $maxTimeStamp nn".
      "While the most negative tweet (scoring $minScore) was - n $minTweet n".
      "tweeted on $minTimeStamp n";
  }
  return $summary;
} //end function getSentimentAnalysis($handle)

//Return a json decoded object from the sentiment analysis api for one tweet
function getAPIResponse($tweet) {
  $curl = curl_init();

  curl_setopt_array($curl, array(
  	CURLOPT_URL => "https://twinword-sentiment-analysis.p.rapidapi.com/analyze/",
        //Only include this SSL_VERIFYPEER setting if you get curl SSL errors that
        //  can't be resolved in your environment
        //CURLOPT_SSL_VERIFYPEER => false,
  	CURLOPT_RETURNTRANSFER => true,
  	CURLOPT_FOLLOWLOCATION => true,
  	CURLOPT_ENCODING => "",
  	CURLOPT_MAXREDIRS => 10,
  	CURLOPT_TIMEOUT => 30,
  	CURLOPT_HTTP_VERSION => CURL_HTTP_VERSION_1_1,
  	CURLOPT_CUSTOMREQUEST => "POST",
  	CURLOPT_POSTFIELDS => "text=".rawurlencode($tweet),
        //Replace Xs below with your own api key from the sentiment analysis api endpoints page
  	CURLOPT_HTTPHEADER => array(
  		"content-type: application/x-www-form-urlencoded",
  		"x-rapidapi-host: twinword-sentiment-analysis.p.rapidapi.com",
  		"x-rapidapi-key: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
  	),
  ));

  $response = curl_exec($curl);
  $err = curl_error($curl);

  curl_close($curl);

  if ($err) {
  	return "cURL Error #:" . $err;
  } else {
  	return json_decode($response);
  }
} // end function getAPIResponse($tweet)
?>

Output for President Donald Trump (with twitter handle realDonaldTrump) run on April 15th 2020

Please enter a twitter handle (enter ‘exit’ to stop):

realDona1dTrump

Please wait while we analyze the latest tweets from the realDona1dTrump handle…

Twitter feed generated, now we’ll use the API to analyze the sentiment of each tweet.

The latest 28 tweets were posted between Tue, 14 Apr 2828 +8188 and Wed, 15 Apr 2828 +8188. 18 are positive, 6 are neutral, and 4 are negative.

The average sentiment score was 8.898431539458245.

The most positive tweet (scoring 8.47751588633333) was – The Democrats donrcöt want to approve more money for our great workers under the incredibly successful rCEPaycheckrw p Ian. Replenish Account Now!

tweeted on Tue, 14 Apr 2828 +8188

While the most negative tweet (scoring -e. 118525113625) was

my Administration is committed to protecting the Homeland from the scourge of narco-terrorists and traffickers seeking o destabilize the United States and our Hemisphere. Our military deployments in the Caribbean & Eastern Pacific will en ure until these threats are neutralized! https://twitter.com/USCGPacificSW/status/12497613ø5ø8435ø465TärCA

tweeted on Tue, 14 Apr 2828 +8188

Please enter a twitter handle (enter ‘exit’ to stop)

Conclusion

Sentiment Analysis is a technology we can use to understand the tone of comments people make on Twitter.  There are many people (like Donald Trump) who use twitter as their own soapbox.  I am surprised to note that President Trump had posted 20 tweets in the last 45 hours, or about 10 tweets per day!  With this kind of volume, we can generate statistics and discover trends over time.  We can use this example as a starting point for a sentiment trend tool over time.  We can compare political candidates during the course of an election year using a sentiment trend tool.  I know I want to vote for people with positive attitudes!