ScrapingAnt helps you to solve complex scraping tasks. Our Javascript execution feature is available for everyone without any additional effort.
For your convenience, we have two similar endpoints (POST
and GET
). They use the same backend under the hood but allow our users to be more flexible at the code writing points.
Parameters passed to GET
endpoint have to be encoded. If you don’t want to encode your request parameters, please, consider using POST
endpoint.
Warning : If you use GET
endpoint, please always ensure that the parameters are ENCODED! This is extremely important to avoid mixing your API parameters and the possible HTTP parameters of the URL you want to scrape.
If you need help encoding your URL you can find below how to do it:
in Python
in JS
in PHP
in ruby
For POST
endpoint you don’t need to encode anything,
The only one parameter is required to use our API - url
. So just pass this parameter to start scraping. Pretty easy, isn’t it? 😉
The optional parameters are:
cookies
- String parameter. You can pass custom cookies to the web pages you want to crawl. To do this just passe the cookie string in the cookies parameter. We currently only handle the name and value of custom cookies. If you want to set multiple cookies just separate cookies with ;
Example: cookies = "cookie_name_1=cookie_value1;cookie_name_2=cookie_value_2"
return_text
- Boolean parameter. By enabling this parameter you’ll be able to fetch only the text content from the Web resource without any of HTML tags. Disabled by default!
Warning:
Each request has 60 seconds to return results successfully. So if your page is not loaded completely after 60 seconds, we will return a partially loaded page. Please be aware of this maximum timeout when writing your own code.
Do not hesitate to contact us in case of questions, suggestions, and proposals. We are always trying to provide the best customer support for our users.