constant request rejections
Hi, I have used your API for a while and recenly I have noticed that more often than not, my requests are being rejected. Has there been any change to your proxies setup? Would you be able to suggest some workaround?
I am scraping the website in Python but here is the same response I get when running through your endpoint tester.
{
“info”: {
“version”: “2”,
“statusCode”: 403,
“statusMessage”: “”,
“headers”: {
“server”: “DataDome”,
“x-datadome”: “protected”,
“accept-ch”: “Sec-CH-UA,Sec-CH-UA-Mobile,Sec-CH-UA-Platform,Sec-CH-UA-Arch,Sec-CH-UA-Full-Version-List,Sec-CH-UA-Model,Sec-CH-Device-Memory, Sec-CH-UA,Sec-CH-UA-Mobile,Sec-CH-UA-Platform,Sec-CH-UA-Arch,Sec-CH-UA-Full-Version-List,Sec-CH-UA-Model,Sec-CH-Device-Memory”,
“content-type”: “text/html;charset=utf-8”,
“charset”: “utf-8”,
“cache-control”: “max-age=0, private, no-cache, no-store, must-revalidate”,
“pragma”: “no-cache”,
“access-control-allow-credentials”: “true”,
“access-control-expose-headers”: “x-dd-b, x-set-cookie”,
“access-control-allow-origin”: “*”,
“x-datadome-cid”: “AHrlqAAAAAMA2nFj7HWCiewAwAOLkw==”,
“set-cookie”: [
“datadome=.1W7F7XJ3TYb9fA1f.lmEghl4CCDg5UD2WR_okC7IdLxeDwOiBrCANVStYntPmSthh-LhNJPo4~0k.JgDG1-QA7hg8N9eN1-fIA04GmrLb8WvgJcsSozgOdyj8wbq_; Max-Age=31536000; Domain=.hermes.com; Path=/; Secure; SameSite=Lax”
],
“accept-ranges”: “bytes”,
“date”: “Tue, 13 Sep 2022 02:08:27 GMT”,
“via”: “1.1 varnish”,
“x-served-by”: “cache-ewr18176-EWR”,
“x-cache”: “MISS”,
“x-cache-hits”: “0”,
“x-timer”: “S1663034908.947287,VS0,VE12”,
“x-cdn-forward”: “Fastly”,
“strict-transport-security”: “max-age=31557600”,
“content-length”: “580”
}
},
“body”: "<html><head><title>hermes.com</title><style>#cmsg{animation: A 1.5s;}@keyframes A{0%{opacity:0;}99%{opacity:0;}100%{opacity:1;}}</style></head><body style=“margin:0”><p id=“cmsg”>Please enable JS and disable any ad blocker</p><script data-cfasync=“false”>var dd={‘cid’:‘AHrlqAAAAAMA2nFj7HWCiewAwAOLkw==’,‘hsh’:‘2211F522B61E269B869FA6EAFFB5E1’,‘t’:‘bv’,‘r’:‘b’,‘s’:13461,‘e’:‘1db226c22e05a7a568e8e293d0a120e1ae6d4eeb1ac3c061b18cae699fcf8c16’,‘host’:‘geo.captcha-delivery.com’}</script><script data-cfasync=“false” src="https://ct.captcha-delivery.com/c.js"></script></body></html>\n"
}
Join in the discussion - add comment below:
Thanks for getting back to me. Telegram doesn’t allow me to send the message. Is there another way to contact?
Hi! please drop me a message: https://t.me/scrapeninja and provide the target website URL you are trying to scrape, and also please list ScrapeNinja options you are using.