Extract a article
import requests
url = "https://gnepro.p.rapidapi.com/crawl"
payload = {
"url": "https://news.yahoo.com/montgomery-brawl-mayor-reed-boat-fight-violence-black-white-210932513.html",
"js": False,
"charset": "auto"
}
headers = {
"X-RapidAPI-Key": "your key",
"X-RapidAPI-Host": "gnepro.p.rapidapi.com",
}
response = requests.post(url, json=payload, headers=headers)
print(response.json())
Extract a article page with js render
import requests
url = "https://gnepro.p.rapidapi.com/crawl"
payload = {
"url": "https://xxx.com/yyy?id=123",
"js": True, # ==========Here set True
"charset": "auto"
}
headers = {
"X-RapidAPI-Key": "your key",
"X-RapidAPI-Host": "gnepro.p.rapidapi.com",
}
response = requests.post(url, json=payload, headers=headers)
print(response.json())
Extract a article whose charset is gb18030
import requests
url = "https://gnepro.p.rapidapi.com/crawl"
payload = {
"url": "https://xxx.com/yyy?id=123",
"js": False,
"charset": "gb18030" # =======Here set the target charset. ##NOTE: in most of time, set 'auto' is enough ##
}
headers = {
"X-RapidAPI-Key": "your key",
"X-RapidAPI-Host": "gnepro.p.rapidapi.com",
}
response = requests.post(url, json=payload, headers=headers)
print(response.json())