Instagram Scraper API
9.9 / 10
2,713ms
100%
N/A
Use Pagination to retrieve User Posts to .CSV
For more help, contact us using the blue “Contact” button on the right, after signing in.
What is pagination?
Pagination divides all the data (e.g. all the posts of a user) into manageable sets or “pages”. You receive a limited amount on each request, and can ask for more results on demand.
For example to retrieve user posts, you will receive 12 posts per API request. If you need all posts of a user, you will need multiple API requests.
How does pagination work?
First request
Say you make some request, for example retrieving posts:
GET {...}/v1/posts?username_or_id_or_url=mrbeast
Expect the response structure to look like:
{
"data": {
"count": 12,
"items": [{post_1}, {post_2}, ..., {post_12}]
},
"pagination_token": "some_string";
}
Subsequent requests
Just repeat the same request, adding pagination_token to your query parameters. Example:
GET {...}/v1/posts?username_or_id_or_url=mrbeast&pagination_token=some_string
You will get 12 more new results.
Last request
Once you get pagination_token: null in your response, it means there is no more data to fetch.
Code Example
Python
This is a simple example to retrieve all posts of an account. You will need to provide your own API key.
import requests
# rapid_api_key = "your-api-key"
def get_user_posts_request(username, pagination_token=None):
# Set the API endpoint URL, request parameters and RapidAPI headers
url = "https://instagram-scraper-api2.p.rapidapi.com/v1/posts"
querystring = {
"username_or_id_or_url" : username,
"pagination_token": pagination_token
}
headers = {
"X-RapidAPI-Key": rapid_api_key,
"X-RapidAPI-Host": "instagram-scraper-api2.p.rapidapi.com"
}
posts = [] # list of posts retrieved
next_token = None # pagination token retrieved for next page
try:
response = requests.get(url, headers=headers, params=querystring)
status_code = response.status_code
if status_code == 200: # success
response_json = response.json()
posts = response_json["data"]["items"]
next_token = response_json["pagination_token"]
elif status_code == 404:
print("Account invalid or private")
else:
print("Error:", response.text)
except Exception as e:
print("Unexpected error:", e)
return posts, next_token
def get_user_posts_all(username):
# List to store all posts
posts = []
# Set no pagination for first request
pagination_token = None
while True:
# Get one "page" of posts
new_posts, next_token = get_user_posts_request(username, pagination_token=pagination_token)
# update the list of posts and the pagination token
posts += new_posts
pagination_token = next_token
print(f"Total posts: {len(posts)}")
# No `pagination_token` = no more posts
if pagination_token is None:
break
return posts
# Let's get the posts of some account
username = "insert_username"
posts_list = get_user_posts_all(username)
print(len(posts_list))
To extend the example, let’s export to a .CSV file some key metrics:
import pandas as pd
posts_dict = {}
for post in posts_list:
post_id = post["id"]
is_video = post["is_video"]
posts_dict[post_id] = {
"is_video": is_video,
"timestamp": post["taken_at_timestamp"],
"num_likes": post["preview_likes"]["count"],
"num_comments": post["comments"]["count"],
"caption": post["captions"]["items"][0]["text"],
"image_url": post["display_url"],
"is_pinned": bool(post["pinned_for_users"]),
"video_url": post.get("video_url", None),
"video_view_count": post.get("video_view_count", None),
}
df_posts = pd.DataFrame(posts_dict.values(), index=posts_dict.keys())
df_posts.index.name = "post_id"
df_posts.to_csv('user_posts.csv')