Hi,
The main site I scrape/pull data from is now having the majority of requests being blocked by cloudflare. It has worked previously since I signed up for the API subscription. However, I am now running into issues if I attempt to query the site at short intervals.
I would say about 20% of requests are getting through and successfully pulling back the page.
However, the majority of requests will return:
{“info”:{},“body”:""}
I am using PHP and curl using the following code:
$curl = curl_init();
curl_setopt_array($curl, [
CURLOPT_URL => "https://scrapeninja.p.rapidapi.com/scrape",
CURLOPT_RETURNTRANSFER => true,
CURLOPT_FOLLOWLOCATION => true,
CURLOPT_ENCODING => "",
CURLOPT_MAXREDIRS => 10,
CURLOPT_TIMEOUT => 30,
CURLOPT_HTTP_VERSION => CURL_HTTP_VERSION_1_1,
CURLOPT_CUSTOMREQUEST => "POST",
CURLOPT_POSTFIELDS => "{\r
\"url\": \"$url\",\r
\"headers\": [\r
\"X-Header: some-random-header\"\r
]\r
}",
CURLOPT_HTTPHEADER => [
"content-type: application/json",
"x-rapidapi-host: scrapeninja.p.rapidapi.com",
"x-rapidapi-key: ........"
],
]);
Any ideas?
URL:
https://www.oddschecker.com/horse-racing
Thanks
아래에 의견을 추가하고 토론에 참여하세요.
Hi @restyler
The problem with CloudFlare is back (or maybe it’s impossible to bypass it?)… Can you check this in PHP? I run this code locally using LocalWP, but it does work from Rapidapi
curl_setopt_array($curl, [
CURLOPT_URL => “https://scrapeninja.p.rapidapi.com/scrape”,
CURLOPT_RETURNTRANSFER => true,
CURLOPT_FOLLOWLOCATION => true,
CURLOPT_ENCODING => “”,
CURLOPT_MAXREDIRS => 10,
CURLOPT_TIMEOUT => 30,
CURLOPT_HTTP_VERSION => CURL_HTTP_VERSION_1_1,
CURLOPT_CUSTOMREQUEST => “POST”,
CURLOPT_POSTFIELDS => “{\r
"url”: “https://www.instant-gaming.com/fr/”\r
}",
CURLOPT_HTTPHEADER => [
“X-RapidAPI-Host: scrapeninja.p.rapidapi.com”,
“X-RapidAPI-Key: xxx”,
“content-type: application/json”
],
CURLOPT_SSL_VERIFYHOST => false,
CURLOPT_SSL_VERIFYPEER => false,
]);
Looks to be working as usual now! Thanks for the quick help
Hey this was a temporary hiccup due to proxies degradation. Should be good now - let me know.