Many APIs are publicly available. All APIs on the RapidAPI platform are publicly available. But there are also so-called internal APIs, which are created and used by developers that need APIs for private applications.
When it comes to web and mobile applications, there is a process to follow in order to determine what API the website uses. In some cases, you can also use the internal API without interacting with the website (i.e. programmatically, by issuing requests directly from the code). In this tutorial, we will demonstrate how to find an API of a website.
How websites use APIs
There are two main ways in which websites use APIs. For the sake of keeping things simple, we’ll call them the backend-tied method and frontend-tied method.
The backend-tied approach is the method where the application makes requests to the API on the server’s side. The pipeline looks like this:
- The user wants to visit a page of the website. He/she uses a browser to send an HTTP request to the server where the site is hosted.
- To provide a response to the user, the server needs to use an API.
- The server sends the request(s) to the API and processes the response of the API. The response to the user is prepared using the response of the API.
- The server sends the prepared response (most often it is the HTML page) to the user.
- The user’s browser renders the response and shows the result to the user.
The key peculiarity of this method is that the user’s browser knows nothing about what is going on on the server’s side (how the response was prepared). This means that we (as users) cannot detect whether any API was used by this site. The exception comes in when the information about used APIs are available on the website, or when t is obvious that the website uses an API.
A vivid example of the backend-tied approach of API usage can be found in our How To Build a Sentiment Analysis App tutorial. The Django web application there sends requests to the API on the server’s side.
Another method is the frontend-tied approach.
Most modern web frameworks use client-side rendering. They send a blank HTML file to the browser along with JavaScript that fills it with data. In this case, it takes data from the internal API. And this is handy for us because if it is done with our browser we can find it. Let’s explore how this works in more detail.
How to see whether a website uses an API
Let’s create a little sandbox to demonstrate how it works. We can use Flask to make a simple internal API which will send only one string of text data. The ‘Access-Control-Allow-Origin’ header is added so we can simply open the HTML file in the browser and make this request from it.
from flask import Flask
from flask import jsonify
from flask import after_this_request
app = Flask(__name__)
@app.route('/test_data', methods=['GET'])
def get_data():
@after_this_request
def add_header(response):
response.headers['Access-Control-Allow-Origin'] = '*'
return response
data = 'An internal API is an interface that enables access to a company’s backend information and application functionality for use by the organization’s developers. The new applications created by these developers can then be distributed publicly although the interface itself is not visible to anyone not working directly with the API publisher.'
return jsonify(data)
if __name__ == '__main__':
app.run(debug=True)
Then we will write an HTML page that will request this string and view it.
<!DOCTYPE html>
<html>
<head>
<meta charset="utf-8">
<title>Test page</title>
</head>
<body>
<h1>Hello world</h1>
<p id="text"></p>
<script>
const text_el = document.getElementById('text')
const url = 'http://127.0.0.1:5000/test_data'
fetch(url)
.then((resp) => resp.json())
.then(function(data) {
text_el.innerHTML = data
})
</script>
</body>
</html>
If we open this page in the browser and view the page source code we will not see the data and this is typical for a frontend-tied approach using the API. If it was not a test page, we wouldn’t see the script either.
Let’s go back to our page in the browser and open the Developer Tools.
Now we need to open the Network tab and choose the XHR filter. XHR refers to the XMLHttpRequest which is the JavaScript object that is used to retrieve data from a server. (We actually use the fetch() request but it’s almost the same.) In some cases, we’ll need to repeat an action or refresh the page to see requests here. With real sites, there could also be a lot of requests. We need to choose the ones that look similar to what we expect to find.
If we click on a request we’ll see the full URL, the HTTP method, and any other necessary information. With our page, we see the “http://127.0.0.1:5000/test_data” address and the GET method.
Now we can open the Response tab and suppose that the data is what we are looking for.
All that we still need to do is to simply make a similar request programmatically and check if we get the same answer.
Test your hypothesis from Python code
Python has a Requests library that is ideal for this purpose. For our example, the code will look pretty simple.
import requests url = 'http://127.0.0.1:5000/test_data' response = requests.get(url) print(response.text)
Here is what the code above returns:
"An internal API is an interface that enables access to a companyu2019s backend information and application functionality for use by the organizationu2019s developers. The new applications created by these developers can then be distributed publicly although the interface itself is not visible to anyone not working directly with the API publisher."
But Requests itself has a lot more options. Any other HTTP method can be used with requests the same way as the GET method. We may also add data to the request body.
In most cases, we’ll get our response in JSON or XML format, so we’ll need to make some transformations or parsing.
Conclusion
There are two ways in which websites can use APIs. If we can see the data on the page but can’t find it in the page source, that’s okay. Finding an API with browser tools is not that complicated. Working with it may be easier than writing parsers for the HTML.
But if we can’t see requests from the browser it does not mean that the site doesn’t use API. If it sends server-side requests we may have trouble finding out which API a website uses.
Can I get AP1 codes of energy suppliers from this website or is there another way
i have one futronic device and i want api of that deviceto integrate image in website . how should i get the api of that device
I want an api link like this http://api.open-notify.org/iss-now.json .
I want an api link on natural disaster.
Please could you help me?