Quick References
In this article, we’ll go over how to use the Bing News Search API with the Aylien Text Analysis API to perform News Summarization and Sentiment Analysis.
Introduction
Nowadays, APIs are an important part of the IT industry. The application programming interface is the way to access data or even deal with some particular task. The APIs are developed mainly for usage by different applications.
You can also consider APIs as building blocks of other applications. Let’s say, you are developing a website, and you want to show the weather data on the site. You can use a weather API to fetch the needed data. However, this is the simplest example; today’s APIs can perform very complex tasks. For instance, you can take the image uploaded by the user in the mobile app, and send it for analysis to some of the image processing APIs. On their end, they will process the image, probably even apply some machine learning techniques, and return a pretty structured response to the mobile app. This response may include information about the class label of the image, detected objects, main colors, etc. The transformed image also can be returned to the mobile app.
There are many APIs, and usually, they are focused on some particular subject matter. For example, there are weather APIs, financial, sports, food-related, SMS and email APIs, APIs for image, sound, text processing, for retrieving a different kind of data, and so on. When you are building an application and there is a need to perform some specific task or get some data, you should think about whether there is an API which can deal with this efficiently. Instead of creating all the features needed by yourself, when you decide to use an API, all you need to do is to create an infrastructure of sending requests to the API and receiving the responses from it.
As we have mentioned earlier, there are a lot of APIs today. And the number of different APIs is continuously growing. So, how not to get lost in these jungles? One of the possible ways to do this is to use a service for the APIs aggregation.
RapidAPI is an API marketplace where many different APIs are gathered in one place. This allows a quick and efficient search for the API which is perfectly matched with your needs. When choosing the right API, you will not only see its name, description, possible endpoints, examples of calls and responses but also will be able to check its average health metrics and popularity score. If you become interested in some of the API, you can quickly check how it is working directly in your browser, without even leaving the RapidAPI website.
Also, as we mentioned earlier, the developers using APIs should think about the infrastructure for communication with the APIs in their applications. This is better than doing all the work directly in the app, without any APIs usage. However, this can take some time and effort, because you will need to get familiar with the documentation of each of the used APIs. RapidAPI provides a single SDK for dealing with all APIs. This makes development even easier. RapidAPI supports a range of popular languages (Python, NodeJS, PHP, Ruby, Java, C#, etc.).
One more important feature of the RapidAPI marketplace is a dashboard creation. RapidAPI aggregates the information about API usage into the set of interactive visualizations, which are available for further analysis. You can get a clear visual picture of how your apps use APIs, what is the latency, errors share, billing information and so on.
In this article, we want to show you how to use Bing News Search API in a bundle with Aylien Text Analysis API to perform news summarization and sentiment analysis. We will interact with both APIs via RapidAPI platform. We assume that you already have a RapidAPI account.
Bing News Search API
“Freemium” means that there is a quota for free usage. You should pay for the usage only above this quota. You should make a subscription to start using this API. Go to Pricing section and press Subscribe button.
The most interesting section is Endpoints. The screen is split into 3 large areas. From the left, there is a list of all available tasks for performing, alongside with the HTTP method needed for each of them. In the case of Bing News Search, there are 3 tasks: retrieving news in a category, search, and trending topics retrieval.
News in Category endpoint allows retrieving the news in the specified category. There is a list of available categories, and to avoid getting an error, you should use only supported categories.
Search endpoint is probably the most popular. It provides an ability to search for news by the search query. We will use this endpoint later in our example application.
Trending Topics endpoint responds with the currently trending topics from the social networks.
If we select a particular task, the middle and right side of the screen will be updated and start to display information about specific endpoint:
If you want to test an API call, fill in the required parameters and, if needed, optional parameters. The changes will be immediately displayed at the top right corner of the window in the code. If you press the Test Endpoint button, then an API call will be sent, and the response will be displayed at the right part of the page.
If you click on the name of the programming language in the Request Snippet section, the context menu with all available languages will appear. You can see, that Bing News Search API is available through RapidAPI for NodeJS, PHP, Python, Ruby, Objective-C, Java, and C#:
For example, if we will choose NodeJS, the following request snippet will be generated:
unirest.get("https://microsoft-azure-bing-news-search-v1.p.rapidapi.com/search?q=football")
.header("X-RapidAPI-Host", "microsoft-azure-bing-news-search-v1.p.rapidapi.com")
.header("X-RapidAPI-Key", "40aeaxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx")
.end(function (result) {
console.log(result.status, result.headers, result.body);
});
For Python the code is different:
response = unirest.get("https://microsoft-azure-bing-news-search-v1.p.rapidapi.com/search?q=football",
headers={
"X-RapidAPI-Host": "microsoft-azure-bing-news-search-v1.p.rapidapi.com",
"X-RapidAPI-Key": "40aeaxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
}
)
Bing News Search API PHP:
$response = UnirestRequest::get("https://microsoft-azure-bing-news-search-v1.p.rapidapi.com/search?q=football",
array(
"X-RapidAPI-Host" => "microsoft-azure-bing-news-search-v1.p.rapidapi.com",
"X-RapidAPI-Key" => "40aeaxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
)
);
Now, let’s move to the Text Analysis API exploration.
Aylien Text Analysis API
Text Analysis API is a powerful tool for information retrieval and natural language processing. It provides many useful features. Let’s briefly describe them.
- Article Extraction endpoint takes an article URL as an input and responds by the text of the article, as well as any embedded media. If possible, it also returns the author name, publication date, and the name of the article.
- Summarization endpoint can be used to get the most important few sentences which reflect the main idea of an article. This endpoint expects URL or text of the article as inputs.
- Classification endpoint determines the main categories of an article (for example, sports, politics, etc.).
- Concept Extraction endpoint searches for the core concepts in the article using the public knowledge bases (for example, Wikipedia).
- Hashtag Suggestion endpoint can be used to generate a list of possible hashtags for an article.
- Entity Extraction endpoint can be used to extract named entities from an article: people, organizations, places, products, dates, email addresses, telephone numbers, currency amounts, links, percentages, etc.
- Language Detection endpoint is able to detect the language of an article among 62 different languages.
- Sentiment Analysis endpoint can return the polarity of an article (positive, neutral, negative) as well as the confidence score. For short texts (with mode = “tweet” parameter) the subjectivity and subjectivity confidence score is also returned by this endpoint.
The HTTP GET method is used in all endpoints. Nevertheless, for most endpoints, the POST method is also supported.
Text Analysis API is available in a range of languages. Below you can see several examples for different languages (for Article Extraction endpoint).
Text Analysis API NodeJS:
unirest.get("https://aylien-text.p.rapidapi.com/extract?url=http%3A%2F%2Ftechcrunch.com%2F2014%2F02%2F27%2Faylien-launches-text-analysis-api-to-help-developers-extract-meaning-from-documents%2F")
.header("X-RapidAPI-Host", "aylien-text.p.rapidapi.com")
.header("X-RapidAPI-Key", "40aeaxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx")
.end(function (result) {
console.log(result.status, result.headers, result.body);
});
Text Analysis API Python:
response = unirest.get("https://aylien-text.p.rapidapi.com/extract?url=http%3A%2F%2Ftechcrunch.com%2F2014%2F02%2F27%2Faylien-launches-text-analysis-api-to-help-developers-extract-meaning-from-documents%2F",
headers={
"X-RapidAPI-Host": "aylien-text.p.rapidapi.com",
"X-RapidAPI-Key": "40aeaxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
}
)
Text Analysis API Java:
HttpResponse response = Unirest.get("https://aylien-text.p.rapidapi.com/extract?url=http%3A%2F%2Ftechcrunch.com%2F2014%2F02%2F27%2Faylien-launches-text-analysis-api-to-help-developers-extract-meaning-from-documents%2F")
.header("X-RapidAPI-Host", "aylien-text.p.rapidapi.com")
.header("X-RapidAPI-Key", "40aeaxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx")
.asJson();
Text Analysis API C#:
Task<HttpResponse<MyClass>> response = Unirest.get("https://aylien-text.p.rapidapi.com/extract?url=http%3A%2F%2Ftechcrunch.com%2F2014%2F02%2F27%2Faylien-launches-text-analysis-api-to-help-developers-extract-meaning-from-documents%2F")
.header("X-RapidAPI-Host", "aylien-text.p.rapidapi.com")
.header("X-RapidAPI-Key", "40aeaxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx")
.asJson();To learn more about using different endpoints in different APIs, see the documentation available on the RapidAPI website.
Let’s go to the creation of the example application using the APIs described above.
Example Application
We want to demonstrate the basic example of how you can use APIs from the RapidAPI platform in your applications.
Consumers of the APIs can be different (websites, mobile applications, desktop applications, various devices, production systems, etc.). The context of use is different for different consumers. Nevertheless, the core structure is similar. For demonstration purposes, we won’t develop an entire app but prepare a Python script which can be considered as a basic frame for the complex application. However, for simplicity, we want to call our script as “application”.
Here is the general idea of the application.
The user will enter the main page of the application and type in some search query. Then, the application should use Bing News Search API to find 10 news for this query. After retrieving the news for each article, it should use Text Analysis API to get the sentiment (polarity) of the text. The extracted polarities are then averaged, and the overall polarity for the entire batch of articles should be displayed to the user.
Besides this, for each news article, we also want to perform summarization and show to the user the names of articles near the summaries. Eventually, when a user clicks on some of the articles, the entity extraction and hashtag suggestion should be performed for this article.
Before starting to work directly with the code, we need to make some preparation steps. If you carefully looked at the examples from the previous chapters of this article, you could notice that there are 3 parameters from the header of the request which are similar in each API call. This header parameters are used to interact with RapidAPI interface.
X-RapidAPI-Host parameter is used to specify what API we want to call in this request. This parameter can look similar to this: microsoft-azure-bing-news-search-v1.p.rapidapi.com.
RapidAPI Project and X-RapidAPI-Key are the parameters which are used to distinguish between your applications that can consume APIs and to perform authentication on behalf of the specific application. You can think about these parameters as about something like username and password.
Let’s create a new app. Click on the Dashboard link at the head of the page. This will open a new page. On this page, click on the Add New App button in the menu from the left side of the window:
However, in order to fulfill requests to the API, you also need the X-RapidAPI-Key. You can find this key if you click on the Security link that corresponds to the specific application. You can find it in the menu on the left side of the screen in the My Apps section:
If you follow this link, you will find yourself on a different page. Notice App Credentials section:
Click on the (SHOW) toggle to make the key visible. This is the key you should include in each API call belonging to this application (News_Analysis).
Now we are ready to go further and write the code for our example application.
The user interaction with our application begins with the fact that he or she comes to the main page. There user sees a greeting, initial instructions for use, and an offer to enter a search query. After the user enters the search query and presses the search button, some backend logic must be started. First, we need to get the data to work with. In our case, the data is the news relevant to the search query. To get this news, we will use the Bing News Search API. You can find the code below:
def search_news(query):
apicall = "https://microsoft-azure-bing-news-search-v1.p.rapidapi.com/search??count=10&q="
query_words = query.split()
for i in range(len(query_words)):
if i != len(query_words)-1:
apicall += query_words[i] + "+"
else:
apicall += query_words[i]
rapidapi_host = "microsoft-azure-bing-news-search-v1.p.rapidapi.com"
rapidapi_key = "251fdxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
headers = {
"X-RapidAPI-Host": rapidapi_host,
"X-RapidAPI-Key": rapidapi_key
}
r = requests.get(apicall, headers=headers)
body = json.loads(r.text)
articles = [{'name': i['name'], 'url': i['url']} for i in body['value']]
return articlesNotice, that here and below, we use the requests library to perform the HTTP requests to the APIs, instead of the unicast library, which is listed on the RapidAPI website. This is due to the fact that in this example Python 3 is used as the programming language. The unicast library does not support the third version of Python. In fact, you can use any convenient way for you to execute HTTP requests.The search_news() function takes a query string as input. This is a search query that came from the frontend of our application. Next is the preparation of the API call. Pay attention to the use of variables rapidapi_host and rapidapi_key. As we mentioned earlier, you need them to identify your application correctly and conduct correct analytics and billing. At the output, the function returns a list of dictionaries, each containing two fields – a name and a link. This list will be transferred for processing to other functions on the backend, as well as, if necessary, can be transferred to the frontend (for example, to display the names of the articles found).
The next building block of our application is the component responsible for the sentiment analysis. We should use at least 2 functions here. Take a look at them:
def sentiment_analysis(articles):
polarities = []
for i in range(len(articles)):
url = articles[i]['url']
apicall = "https://aylien-text.p.rapidapi.com/sentiment?mode=document&url=" + url
rapidapi_host = "aylien-text.p.rapidapi.com"
rapidapi_key = "251fdxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
headers = {
"X-RapidAPI-Host": rapidapi_host,
"X-RapidAPI-Key": rapidapi_key
}
r = requests.get(apicall, headers=headers)
body = json.loads(r.text)
polarities.append({'polarity': body['polarity'], 'score': body['polarity_confidence']})
return polarities
def average_sentiment(polarities):
score = 0
for i in range(len(polarities)):
if polarities[i]['polarity'] == 'positive':
score += 1
elif polarities[i]['polarity'] == 'negative':
score -= 1
averaged_polarity = score/10
return averaged_polarity
The second function calculates the average polarity of the batch of news. It returns a value, which ranges from -1 to 1. The closer the value to 1, the more positive news are in the analyzed batch. If the value is closer to -1, this means that the negative sentiment prevails in this set of articles.The first function is responsible for interacting with the Text Analysis API. It takes as input a list of dictionaries obtained from the previous function. Then, it iterates through all the elements of the list in a loop, extracting the URL of the article from each one. Then, this URL is used to build a request to the API. As a result, we should get an answer, which will contain the polarity (positive, neutral, or negative) and the value of confidence in this sentiment. At the output, the function again generates a list of dictionaries with the polarity and polarity confidence values.
In our application, after the user enters a search query, we want to display the list of found news, polarity, and confidence score for each item, the average polarity for the entire batch, and also a brief summary of each article. To perform summarization, we will use the corresponding endpoint of the Text Analysis API:
def summarization(articles, sentences_number=5):
summaries = []
for i in range(len(articles)):
url = articles[i]['url']
apicall = "https://aylien-text.p.rapidapi.com/summarize?sentences_number=" + str(sentences_number) + "&url=" + url
rapidapi_host = "aylien-text.p.rapidapi.com"
rapidapi_key = "251fdxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
headers = {
"X-RapidAPI-Host": rapidapi_host,
"X-RapidAPI-Key": rapidapi_key
}
r = requests.get(apicall, headers=headers)
body = json.loads(r.text)
summaries.append({'sentences': body['sentences']})
return summaries
After all the things that we mentioned earlier will be displayed by our app, the user can go over the results and click on some articles if these articles attract his or her attention. Then, our idea is to open the next page, where the results of the entity extraction and hashtag suggestion for this article will be presented to the user. The functions entity_extraction() and hashtag_suggestion() are pretty simple and similar. They both take URL of an article as input, make a call to the corresponding endpoint of the Text Analysis API and return results provided by the endpoint. You can find the code for these functions below:The approach here is similar to the approach used in the sentiment_analysis() function. The summarization() function loops through the list of dictionaries, extracts news URLs from these dictionaries, creates an API calls using URLs and sends the requests to the corresponding endpoint of the Text Analysis API. This function returns a list of dictionaries called summaries. Each dictionary contains only one key-value pair, which includes the most important sentences of the corresponding article. By default, 5 sentences should be returned for each article, but there is support for tuning this parameter. So, for example, if you will decide to use a different number of summary sentences somewhere in your application, you will need only to pass a different value to the parameter called sentences_number in this function.
def entity_extraction(url):
apicall = "https://aylien-text.p.rapidapi.com/entities?url=" + url
rapidapi_host = "aylien-text.p.rapidapi.com"
rapidapi_key = "251fdxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
headers = {
"X-RapidAPI-Host": rapidapi_host,
"X-RapidAPI-Key": rapidapi_key
}
r = requests.get(apicall, headers=headers)
body = json.loads(r.text)
entities = body['entities']
return entities
def hashtags_suggestion(url):
apicall = "https://aylien-text.p.rapidapi.com/hashtags?url=" + url
rapidapi_host = "aylien-text.p.rapidapi.com"
rapidapi_key = "251fdxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
headers = {
"X-RapidAPI-Host": rapidapi_host,
"X-RapidAPI-Key": rapidapi_key
}
r = requests.get(apicall, headers=headers)
body = json.loads(r.text)
hashtags = body['hashtags']
return hashtags
Let’s check how our functions work in an interactive shell. Assume, that the user wants to find the latest news about machine learning. Here you can see how this task was completed by the search_news() function:
Then, our application performs sentiment analysis and shows us various sentiment scores:
Here is an example of how the summarization works. You can see, that for the second item the list of extracted sentences is empty. This is because actually, it is not an article, but a video. It is probably a way of improving our application in the future.
Find below the example of how the hashtags suggestions for this article look like. At the top of the list, you can see the most relevant hashtags, such as #MachineLearning, #ArtificialIntelligence, or #Hollywood.
Conclusion
In this article, we briefly described the basics of the application programming interfaces and how the RapidAPI platform can help developers to build their applications using different APIs. We touched 2 APIs: Bing News Search API and Text Analysis API. The first can be used for news search, and the second is used for the various tasks related to natural language processing.
We showed an example of how the APIs mentioned above could be used to build an application. You should understand that the example application is far from production ready product. In fact, we demonstrated mainly the piece of backend logic which directly works with APIs. There are a lot more things to do for improving the application.
Nevertheless, we hope, that the example shed some light on the process of how to use the APIs in the real-world applications (for news or social media). The general idea should be clear. In our opinion, the level of the APIs usage is going to increase in the future, so, it is worth to become familiar with them and learn how they can help you.
Leave a Reply