{ "info": { "_postman_id": "1b9fd705-9f62-408e-8e5f-757a1c443ca7", "name": "ScrAPI", "description": "ScrAPI is a full-featured and flexible web scraping service with a focus on privacy.\n\n### Features\n\n- Scrape with or without a real browser\n- Browser control commands (clicking, filling forms etc)\n- Automatic captcha solving\n- Automatic IP rotation\n- Data center, residential and Tor proxy options\n- Geo targeting to bypass region restrictions\n- Set custom cookies and headers\n- Advanced bot detection bypass techniques\n- JSON or HTML response options\n- Webhooks/callbacks\n- Unlimited concurrency\n- Unlimited bandwidth\n \n\n### BETA\n\nRequest your own API key from [hello@scrapi.dev](https://mailto:hello@scrapi.dev) with free credits to use the service without test key limitations.\n\nThere will be no charge for usage during the beta although we will be tracking credits and feature usage to determine where we need to focus.\n\nWhen using the test API key (or no API key) you are limited to one concurrent call and fifty free calls per day with minimal queuing capabilities.\n\nIf you prefer to browse the API documentation in Swagger you can do that here: [https://api.scrapi.dev/swagger](https://api.scrapi.dev/swagger)\n\n### Migrating\n\n### Automatic Captcha Solving\n\nWe currently support automatically solving the following captchas:\n\n- reCAPTCHA v2 and v3 (click, invisible and enterprise)\n- hCaptcha (normal and invisible)\n- Cloudflare (turnstile and challenge page)\n \n\nWe will be adding many more based on demand and requests such as DataDome, FunCaptcha, KeyCaptcha etc.\n\n### Pricing / Credits\n\nNo credits are used for any failures that do not result in any HTML.\n\n| Feature | Credits |\n| --- | --- |\n| No Browser | 1 |\n| Use Browser | 5 |\n| Residential Proxy | 10 |\n| Data Center Proxy | 5 |\n| Tor Proxy | 1 |\n| Captcha Solved | 30 (per captcha solved) |\n\nPricing is not available yet as we want to determine what a fair cost model will look like as we test ScrAPI as a public service so the credits table above might change.\n\n### SDK / API Client\n\nWe currently have an official SDK for .NET developers on NuGet. \nFor other languages you can use [Microsoft Kiota](https://learn.microsoft.com/en-us/openapi/kiota/overview) to generate SDKs until we offer a fully supported version:\n\n``` bash\n# GO\nkiota generate --ebc -c ScrapiClient -n Scrapi -d https://api.scrapi.dev/swagger/v1/swagger.json -o ScrapiClientGo -l go\n# Java\nkiota generate --ebc -c ScrapiClient -n Scrapi -d https://api.scrapi.dev/swagger/v1/swagger.json -o ScrapiClientJava -l java\n# PHP\nkiota generate --ebc -c ScrapiClient -n Scrapi -d https://api.scrapi.dev/swagger/v1/swagger.json -o ScrapiClientPhp -l php\n# Python\nkiota generate --ebc -c ScrapiClient -n Scrapi -d https://api.scrapi.dev/swagger/v1/swagger.json -o ScrapiClientPython -l python\n#Ruby\nkiota generate --ebc -c ScrapiClient -n Scrapi -d https://api.scrapi.dev/swagger/v1/swagger.json -o ScrapiClientRuby -l ruby\n# TypeScript\nkiota generate --ebc -c ScrapiClient -n Scrapi -d https://api.scrapi.dev/swagger/v1/swagger.json -o ScrapiClientTypeScript -l typescript\n\n ```\n\n### Limitations\n\nWe currently do not support the concept of \"sessions\" because of how the services are load-balanced. We'll add this functionality soon to make it easier to perform repeated tasks without having to manage your own cookies and headers.\n\nWe do not apply any TLS fingerprinting spoofing on non-browser calls yet so using this option often fails these types of bot detection checks.\n\nWe do not offer an option to take a screenshot (we've never needed to) although this would not be difficult to add if required.\n\n### About Us\n\nWe've been building web scraping and data aggregation solutions for over a decade and we have a lot of experience with the challenges of web scraping at scale and. We started by using existing web scraping services that had the features required but over time we had to create our own custom solutions because of limitations or lack of features in these services. These services often keep the data that you have extracted and we needed to have stronger privacy in place for our clients.\n\nAs we added more features and capabilities into various different web scraping verticals we decided to bundle them all together into a single service that we have been using ourselves internally for a few years. We believe this can be a viable option for the general public so want to put it out there and get feedback that other people have experienced and make web scraping as simple and painless as possible. We also have no interest in what the service gets used for so we do not log your data and will never sell any information you extract through the service.", "schema": "https://schema.getpostman.com/json/collection/v2.1.0/collection.json", "_exporter_id": "2902066" }, "item": [ { "name": "Web Scraping", "item": [ { "name": "Scrape URL", "event": [ { "listen": "test", "script": { "exec": [ "var json = pm.response.json();\r", "\r", "if (pm.response.code === 202) {\r", " pm.collectionVariables.set(\"reference\", json.reference);\r", "}\r", "\r", "if (pm.response.code === 200) {\r", " var html = json.content;\r", " var htmlNoScript = html.replace(/)<[^<]*)*<\\/script>/gi, '');\r", "\r", " pm.visualizer.set(htmlNoScript);\r", "}\r", "\r", "pm.test(\"No Error Messages\", function () {\r", " pm.expect(json).to.not.have.property('errorMessages');\r", "});" ], "type": "text/javascript" } } ], "request": { "method": "POST", "header": [], "body": { "mode": "raw", "raw": "{\r\n \"url\": \"https://deventerprise.com\"\r\n}", "options": { "raw": { "language": "json" } } }, "url": { "raw": "https://api.scrapi.dev/v1/scrape", "host": [ "https://api.scrapi.dev" ], "path": [ "v1", "scrape" ] }, "description": "POST method to perform a web scraping operation using the provided URL.\n\n#### **JSON Parameters**\n\nThe API supports the following parameters.\n\n| Name | Data Type | Description |\n| --- | --- | --- |\n| url | String (URL) | (Required) URL to scrape for content. |\n| useBrowser | Boolean | (Optional) Indicate whether you want to use a full headless browser (will execute JavaScript) or just a regular web client call to retrieve the response (faster but no JavaScript). |\n| solveCaptchas | Boolean | (Optional) Indicate whether you want to attempt to solve any captchas detected on the page. Will automatically enable using a browser. |\n| callbackUrl | String (URL) | (Optional) A URL that the response data will the POSTed to once complete. |\n| cookies | String | (Optional) A key/value pair list of cookies to provide in the scrape request (format: _cookie1=value1;cookie2=value2;_). |\n| headers | String | (Optional) A key/value pair list of headers to provide in the scrape request (format: _header1=value1;header2=value2;_). |\n| proxyType | String or Number | (Optional) The type of the proxy to use for the scrape request. |\n| proxyCountry | String (3) | (Optional) The proxy country to use if you require the scrape request to come from. |\n| responseFormat | String or Number | (Optional) The type of API response. Refer to the HTML response section. |\n| acceptDialogs | Boolean | (Optional) By default all popup dialogs are cancelled, if that it not the expected behavior you can choose to accept any popup dialogs instead. |\n| browserCommands | Array | (Optional) List of browser commands to execute once the web page has loaded. Refer to the browser commands section. |\n\n#### **Browser Commands**\n\nThe following commands can be used to control the browser in order to create a specific state before responding with the content. These commands are all executed with human-like behaviour such as moving the mouse to elements before actioning them, moving the mouse while waiting, variable click and type speeds etc. Any issues with your commands will highlighted in the error message response field.\n\n| **Command** | **Value** | **Example** |\n| --- | --- | --- |\n| click / tap | CSS selector of the element to click. | { \"click\": \"#buttonId\" } |\n| input / fill / type | CSS selector to target and value to enter into input field. | { \"input\": { \"input\\[name='email'\\]\": \"[example@test.com](https://mailto:example@test.com)\" } }, |\n| select / choose / pick | CSS selector to target and value or text to select from the options. | { \"select\": { \"select\\[name='country'\\]\": \"USA\" } }, |\n| wait | Amount of time to wait in milliseconds (maximum of 15000). | { \"wait\": 5000 } |\n| waitfor / wait-for / wait_for | CSS selector of the element to wait for. | { \"waitfor\": \"#newForm\" }, |\n| javascript / js / eval | Custom JavaScript snippet to execute against the page. | { \"javascript\": \"console.log('hello!!')\" } |\n\n#### HTML Response\n\nYou can use the response format parameter to change the result to HTML instead of JSON. This is useful if you want to proxy through ScrAPI using your own browser or automation implementation or simply don't want to parse JSON to get the HTML content. When using this option the other response values will be added to headers.\n\n| **Header** | **Description** |\n| --- | --- |\n| X-ScrAPI-Duration | The amount of time taken to perform the scrape operation. |\n| X-ScrAPI-Attempts | The number of attempts (retries) to perform the scrape operation. |\n| X-ScrAPI-CreditsUsed | The number of credits used for the scrape operation. |\n| X-ScrAPI-Cookies | A semi-colon separated list of cookie key/value pairs that were part of the scrape response. |\n| X-ScrAPI-Headers | A semi-colon separated list of header key/value pairs that were part of the scrape response. |\n| X-ScrAPI-CaptchasSolved | The name of captchas detected and how many times they were solved. |\n| X-ScrAPI-ErrorMessages | A list of error messages that occurred during the scrape operation or while executing browser commands. |" }, "response": [ { "name": "With Browser - No Proxy", "originalRequest": { "method": "POST", "header": [], "body": { "mode": "raw", "raw": "{\r\n \"url\": \"https://deventerprise.com\",\r\n \"useBrowser\": true,\r\n \"proxyType\": \"None\"\r\n}", "options": { "raw": { "language": "json" } } }, "url": { "raw": "https://api.scrapi.dev/v1/scrape", "host": [ "https://api.scrapi.dev" ], "path": [ "v1", "scrape" ] } }, "_postman_previewlanguage": null, "header": null, "cookie": [], "body": null }, { "name": "With Browser - Free Proxy", "originalRequest": { "method": "POST", "header": [], "body": { "mode": "raw", "raw": "{\r\n \"url\": \"https://deventerprise.com\",\r\n \"useBrowser\": true,\r\n \"proxyType\": \"Free\"\r\n}", "options": { "raw": { "language": "json" } } }, "url": { "raw": "https://api.scrapi.dev/v1/scrape", "host": [ "https://api.scrapi.dev" ], "path": [ "v1", "scrape" ] } }, "_postman_previewlanguage": null, "header": null, "cookie": [], "body": null }, { "name": "With Browser - Data Center Proxy", "originalRequest": { "method": "POST", "header": [], "body": { "mode": "raw", "raw": "{\r\n \"url\": \"https://deventerprise.com\",\r\n \"useBrowser\": true,\r\n \"proxyType\": \"DataCenter\"\r\n}", "options": { "raw": { "language": "json" } } }, "url": { "raw": "https://api.scrapi.dev/v1/scrape", "host": [ "https://api.scrapi.dev" ], "path": [ "v1", "scrape" ] } }, "_postman_previewlanguage": null, "header": null, "cookie": [], "body": null }, { "name": "With Browser - Residential Proxy", "originalRequest": { "method": "POST", "header": [], "body": { "mode": "raw", "raw": "{\r\n \"url\": \"https://etherscan.io/contractsVerified\",\r\n \"useBrowser\": true,\r\n \"proxyType\": \"Residential\"\r\n}", "options": { "raw": { "language": "json" } } }, "url": { "raw": "https://api.scrapi.dev/v1/scrape", "host": [ "https://api.scrapi.dev" ], "path": [ "v1", "scrape" ] } }, "_postman_previewlanguage": null, "header": null, "cookie": [], "body": null }, { "name": "With Browser - Tor Proxy", "originalRequest": { "method": "POST", "header": [], "body": { "mode": "raw", "raw": "{\r\n \"url\": \"http://a4ygisnerpgtc5ayerl22pll6cls3oyj54qgpm7qrmb66xrxts6y3lyd.onion/index.html\",\r\n \"useBrowser\": true,\r\n \"proxyType\": \"Tor\"\r\n}", "options": { "raw": { "language": "json" } } }, "url": { "raw": "https://api.scrapi.dev/v1/scrape", "host": [ "https://api.scrapi.dev" ], "path": [ "v1", "scrape" ] } }, "_postman_previewlanguage": null, "header": null, "cookie": [], "body": null }, { "name": "No Browser - No Proxy", "originalRequest": { "method": "POST", "header": [], "body": { "mode": "raw", "raw": "{\r\n \"url\": \"https://deventerprise.com\",\r\n \"useBrowser\": false,\r\n \"proxyType\": \"None\"\r\n}", "options": { "raw": { "language": "json" } } }, "url": { "raw": "https://api.scrapi.dev/v1/scrape", "host": [ "https://api.scrapi.dev" ], "path": [ "v1", "scrape" ] } }, "_postman_previewlanguage": null, "header": null, "cookie": [], "body": null }, { "name": "No Browser - Free Proxy", "originalRequest": { "method": "POST", "header": [], "body": { "mode": "raw", "raw": "{\r\n \"url\": \"https://deventerprise.com\",\r\n \"useBrowser\": false,\r\n \"proxyType\": \"Free\"\r\n}", "options": { "raw": { "language": "json" } } }, "url": { "raw": "https://api.scrapi.dev/v1/scrape", "host": [ "https://api.scrapi.dev" ], "path": [ "v1", "scrape" ] } }, "_postman_previewlanguage": null, "header": null, "cookie": [], "body": null }, { "name": "No Browser - Data Center Proxy", "originalRequest": { "method": "POST", "header": [], "body": { "mode": "raw", "raw": "{\r\n \"url\": \"https://deventerprise.com\",\r\n \"useBrowser\": false,\r\n \"proxyType\": \"DataCenter\"\r\n}", "options": { "raw": { "language": "json" } } }, "url": { "raw": "https://api.scrapi.dev/v1/scrape", "host": [ "https://api.scrapi.dev" ], "path": [ "v1", "scrape" ] } }, "_postman_previewlanguage": null, "header": null, "cookie": [], "body": null }, { "name": "No Browser - Residential Proxy", "originalRequest": { "method": "POST", "header": [], "body": { "mode": "raw", "raw": "{\r\n \"url\": \"https://etherscan.io/contractsVerified\",\r\n \"useBrowser\": false,\r\n \"proxyType\": \"Residential\"\r\n}", "options": { "raw": { "language": "json" } } }, "url": { "raw": "https://api.scrapi.dev/v1/scrape", "host": [ "https://api.scrapi.dev" ], "path": [ "v1", "scrape" ] } }, "_postman_previewlanguage": null, "header": null, "cookie": [], "body": null }, { "name": "No Browser - Tor Proxy", "originalRequest": { "method": "POST", "header": [], "body": { "mode": "raw", "raw": "{\r\n \"url\": \"http://a4ygisnerpgtc5ayerl22pll6cls3oyj54qgpm7qrmb66xrxts6y3lyd.onion/index.html\",\r\n \"useBrowser\": false,\r\n \"proxyType\": \"Tor\"\r\n}", "options": { "raw": { "language": "json" } } }, "url": { "raw": "https://api.scrapi.dev/v1/scrape", "host": [ "https://api.scrapi.dev" ], "path": [ "v1", "scrape" ] } }, "_postman_previewlanguage": null, "header": null, "cookie": [], "body": null }, { "name": "Browser Control - Form Fill", "originalRequest": { "method": "POST", "header": [], "body": { "mode": "raw", "raw": "{\r\n \"url\": \"https://www.roboform.com/filling-test-all-fields\",\r\n \"useBrowser\": true,\r\n \"proxyType\": \"None\",\r\n \"solveCaptchas\": true,\r\n \"browserCommands\": [\r\n { \"input\": { \"input[name='01___title']\": \"Mr\" } },\r\n { \"input\": { \"input[name='02frstname']\": \"Werner\" } },\r\n { \"input\": { \"input[name='04lastname']\": \"van Deventer\" } },\r\n { \"select\": { \"select[name='40cc__type']\": \"Discover\" } },\r\n { \"wait\": 3000 },\r\n { \"waitfor\": \"input[type='reset']\" },\r\n { \"click\": \"input[type='reset']\" },\r\n { \"wait\": 1000 },\r\n { \"javascript\": \"console.log('anything you want?')\" }\r\n ]\r\n}", "options": { "raw": { "language": "json" } } }, "url": { "raw": "https://api.scrapi.dev/v1/scrape", "host": [ "https://api.scrapi.dev" ], "path": [ "v1", "scrape" ] } }, "_postman_previewlanguage": null, "header": null, "cookie": [], "body": null }, { "name": "HTML Response", "originalRequest": { "method": "POST", "header": [], "body": { "mode": "raw", "raw": "{\r\n \"url\": \"https://deventerprise.com\",\r\n \"useBrowser\": false,\r\n \"proxyType\": \"None\",\r\n \"responseFormat\": \"HTML\"\r\n}", "options": { "raw": { "language": "json" } } }, "url": { "raw": "https://api.scrapi.dev/v1/scrape", "host": [ "https://api.scrapi.dev" ], "path": [ "v1", "scrape" ] } }, "_postman_previewlanguage": null, "header": null, "cookie": [], "body": null }, { "name": "Callback / Webhook", "originalRequest": { "method": "POST", "header": [], "body": { "mode": "raw", "raw": "{\r\n \"url\": \"https://deventerprise.com\",\r\n \"useBrowser\": false,\r\n \"proxyType\": \"None\",\r\n \"callbackUrl\": \"https://webhook-test.com/75b9b9bb6a6a7726ead42e4cf8fc6d87\"\r\n}", "options": { "raw": { "language": "json" } } }, "url": { "raw": "https://api.scrapi.dev/v1/scrape", "host": [ "https://api.scrapi.dev" ], "path": [ "v1", "scrape" ] } }, "_postman_previewlanguage": null, "header": null, "cookie": [], "body": null }, { "name": "Cookies / Headers", "originalRequest": { "method": "POST", "header": [], "body": { "mode": "raw", "raw": "{\r\n \"url\": \"https://deventerprise.com\",\r\n \"useBrowser\": false,\r\n \"proxyType\": \"None\",\r\n \"cookies\": {\r\n \"My-Cookie\": \"Test 1\",\r\n \"Session-ID\": \"Test 2\"\r\n },\r\n \"headers\": {\r\n \"My-Header\": \"Test 3\",\r\n \"X-SITE-KEY\": \"Test 4\"\r\n }\r\n}\r\n", "options": { "raw": { "language": "json" } } }, "url": { "raw": "https://api.scrapi.dev/v1/scrape", "host": [ "https://api.scrapi.dev" ], "path": [ "v1", "scrape" ] } }, "_postman_previewlanguage": null, "header": null, "cookie": [], "body": null }, { "name": "Geotargeting", "originalRequest": { "method": "POST", "header": [], "body": { "mode": "raw", "raw": "{\r\n \"url\": \"https://whatismyipaddress.com\",\r\n \"useBrowser\": true,\r\n \"proxyType\": \"DataCenter\",\r\n \"proxyCountry\": \"USA\"\r\n}", "options": { "raw": { "language": "json" } } }, "url": { "raw": "https://api.scrapi.dev/v1/scrape", "host": [ "https://api.scrapi.dev" ], "path": [ "v1", "scrape" ] } }, "_postman_previewlanguage": null, "header": null, "cookie": [], "body": null }, { "name": "Validation Error", "originalRequest": { "method": "POST", "header": [], "body": { "mode": "raw", "raw": "{\r\n \"url\": \"https://deventerprise.com\",\r\n \"useBrowser\": false,\r\n \"proxyType\": \"None\",\r\n \"proxyCountry\": \"USA\"\r\n}", "options": { "raw": { "language": "json" } } }, "url": { "raw": "https://api.scrapi.dev/v1/scrape", "host": [ "https://api.scrapi.dev" ], "path": [ "v1", "scrape" ] } }, "_postman_previewlanguage": "json", "header": [ { "key": "Content-Type", "value": "application/json", "name": "Content-Type", "description": "", "type": "text" } ], "cookie": [], "body": "\"Cannot specify a proxy country when not using proxy (Residential or DataCenter) or when using Tor.\"" } ] }, { "name": "Scrape URL", "event": [ { "listen": "test", "script": { "exec": [ "var json = pm.response.json();\r", "\r", "if (pm.response.code === 202) {\r", " pm.collectionVariables.set(\"reference\", json.reference);\r", "}\r", "\r", "if (pm.response.code === 200) {\r", " var html = json.content;\r", " var htmlNoScript = html.replace(/)<[^<]*)*<\\/script>/gi, '');\r", "\r", " pm.visualizer.set(htmlNoScript);\r", "}\r", "\r", "pm.test(\"No Error Messages\", function () {\r", " pm.expect(json).to.not.have.property('errorMessages');\r", "});" ], "type": "text/javascript" } } ], "request": { "auth": { "type": "noauth" }, "method": "GET", "header": [], "url": { "raw": "https://api.scrapi.dev/v1/scrape?url=https://deventerprise.com&apiKey=00000000-0000-0000-0000-000000000000", "host": [ "https://api.scrapi.dev" ], "path": [ "v1", "scrape" ], "query": [ { "key": "url", "value": "https://deventerprise.com", "description": "(Required) URL to scrape for content." }, { "key": "apiKey", "value": "00000000-0000-0000-0000-000000000000", "description": "(Optional) API Key to avoid free usage limitations." }, { "key": "useBrowser", "value": "false", "description": "(Optional) Indicate whether you want to use a full headless browser (will execute JavaScript) or just a regular web client call to retrieve the response (faster but no JavaScript).", "disabled": true }, { "key": "solveCaptchas", "value": "false", "description": "(Optional) Indicate whether you want to attempt to solve any captchas detected on the page. Will automatically enable using a browser.", "disabled": true }, { "key": "callbackUrl", "value": null, "description": "(Optional) A URL that the response data will the POSTed to once complete.", "disabled": true }, { "key": "cookies", "value": null, "description": "(Optional) A key/value pair list of cookies to provide in the scrape request (format: cookie1=value1;cookie2=value2;).", "disabled": true }, { "key": "headers", "value": null, "description": "(Optional) A key/value pair list of headers to provide in the scrape request (format: header1=value1;header2=value2;).", "disabled": true }, { "key": "proxyType", "value": "None", "description": "(Optional) The type of the proxy to use for the scrape request. | 0 = None, 1 = Free, 2 = Residential, 3 = DataCenter, 4 = Tor |", "disabled": true }, { "key": "proxyCountry", "value": null, "description": "(Optional) The proxy country to use if you require the scrape request to come from.", "disabled": true }, { "key": "responseFormat", "value": "JSON", "description": "(Optional) The type of API response. | 0 = JSON, 1 = HTML |", "disabled": true } ] }, "description": "GET method to perform a web scraping operation using the provided URL. \nBrowser commands are not avaialble using this method." }, "response": [ { "name": "With Browser - No Proxy", "originalRequest": { "method": "GET", "header": [], "url": { "raw": "https://api.scrapi.dev/v1/scrape?url=https://deventerprise.com&useBrowser=true&proxyType=None&apiKey=00000000-0000-0000-0000-000000000000", "host": [ "https://api.scrapi.dev" ], "path": [ "v1", "scrape" ], "query": [ { "key": "url", "value": "https://deventerprise.com", "description": "(Required) URL to scrape for content." }, { "key": "useBrowser", "value": "true", "description": "(Optional) Indicate whether you want to use a full headless browser (will execute JavaScript) or just a regular web client call to retrieve the response (faster but no JavaScript)." }, { "key": "proxyType", "value": "None", "description": "(Optional) The type of the proxy to use for the scrape request. | 0 = None, 1 = Free, 2 = Residential, 3 = DataCenter, 4 = Tor |" }, { "key": "apiKey", "value": "00000000-0000-0000-0000-000000000000", "description": "(Optional) API Key to avoid free usage limitations." } ] } }, "_postman_previewlanguage": null, "header": null, "cookie": [], "body": null }, { "name": "With Browser - Free Proxy", "originalRequest": { "method": "GET", "header": [], "url": { "raw": "https://api.scrapi.dev/v1/scrape?url=https://deventerprise.com&useBrowser=true&proxyType=Free&apiKey=00000000-0000-0000-0000-000000000000", "host": [ "https://api.scrapi.dev" ], "path": [ "v1", "scrape" ], "query": [ { "key": "url", "value": "https://deventerprise.com", "description": "(Required) URL to scrape for content." }, { "key": "useBrowser", "value": "true", "description": "(Optional) Indicate whether you want to use a full headless browser (will execute JavaScript) or just a regular web client call to retrieve the response (faster but no JavaScript)." }, { "key": "proxyType", "value": "Free", "description": "(Optional) The type of the proxy to use for the scrape request. | 0 = None, 1 = Free, 2 = Residential, 3 = DataCenter, 4 = Tor |" }, { "key": "apiKey", "value": "00000000-0000-0000-0000-000000000000", "description": "(Optional) API Key to avoid free usage limitations." } ] } }, "_postman_previewlanguage": null, "header": null, "cookie": [], "body": null }, { "name": "With Browser - Data Center Proxy", "originalRequest": { "method": "GET", "header": [], "url": { "raw": "https://api.scrapi.dev/v1/scrape?url=https://deventerprise.com&useBrowser=true&proxyType=DataCenter&apiKey=00000000-0000-0000-0000-000000000000", "host": [ "https://api.scrapi.dev" ], "path": [ "v1", "scrape" ], "query": [ { "key": "url", "value": "https://deventerprise.com", "description": "(Required) URL to scrape for content." }, { "key": "useBrowser", "value": "true", "description": "(Optional) Indicate whether you want to use a full headless browser (will execute JavaScript) or just a regular web client call to retrieve the response (faster but no JavaScript)." }, { "key": "proxyType", "value": "DataCenter", "description": "(Optional) The type of the proxy to use for the scrape request. | 0 = None, 1 = Free, 2 = Residential, 3 = DataCenter, 4 = Tor |" }, { "key": "apiKey", "value": "00000000-0000-0000-0000-000000000000", "description": "(Optional) API Key to avoid free usage limitations." } ] } }, "_postman_previewlanguage": null, "header": null, "cookie": [], "body": null }, { "name": "With Browser - Residential Proxy", "originalRequest": { "method": "GET", "header": [], "url": { "raw": "https://api.scrapi.dev/v1/scrape?url=https://deventerprise.com&useBrowser=true&proxyType=Residential&apiKey=00000000-0000-0000-0000-000000000000", "host": [ "https://api.scrapi.dev" ], "path": [ "v1", "scrape" ], "query": [ { "key": "url", "value": "https://deventerprise.com", "description": "(Required) URL to scrape for content." }, { "key": "useBrowser", "value": "true", "description": "(Optional) Indicate whether you want to use a full headless browser (will execute JavaScript) or just a regular web client call to retrieve the response (faster but no JavaScript)." }, { "key": "proxyType", "value": "Residential", "description": "(Optional) The type of the proxy to use for the scrape request. | 0 = None, 1 = Free, 2 = Residential, 3 = DataCenter, 4 = Tor |" }, { "key": "apiKey", "value": "00000000-0000-0000-0000-000000000000", "description": "(Optional) API Key to avoid free usage limitations." } ] } }, "_postman_previewlanguage": null, "header": null, "cookie": [], "body": null }, { "name": "With Browser - Tor Proxy", "originalRequest": { "method": "GET", "header": [], "url": { "raw": "https://api.scrapi.dev/v1/scrape?url=http://a4ygisnerpgtc5ayerl22pll6cls3oyj54qgpm7qrmb66xrxts6y3lyd.onion/index.html&useBrowser=true&proxyType=Tor&apiKey=00000000-0000-0000-0000-000000000000", "host": [ "https://api.scrapi.dev" ], "path": [ "v1", "scrape" ], "query": [ { "key": "url", "value": "http://a4ygisnerpgtc5ayerl22pll6cls3oyj54qgpm7qrmb66xrxts6y3lyd.onion/index.html", "description": "(Required) URL to scrape for content." }, { "key": "useBrowser", "value": "true", "description": "(Optional) Indicate whether you want to use a full headless browser (will execute JavaScript) or just a regular web client call to retrieve the response (faster but no JavaScript)." }, { "key": "proxyType", "value": "Tor", "description": "(Optional) The type of the proxy to use for the scrape request. | 0 = None, 1 = Free, 2 = Residential, 3 = DataCenter, 4 = Tor |" }, { "key": "apiKey", "value": "00000000-0000-0000-0000-000000000000", "description": "(Optional) API Key to avoid free usage limitations." } ] } }, "_postman_previewlanguage": null, "header": null, "cookie": [], "body": null }, { "name": "No Browser - No Proxy", "originalRequest": { "method": "GET", "header": [], "url": { "raw": "https://api.scrapi.dev/v1/scrape?url=https://deventerprise.com&useBrowser=false&proxyType=None&apiKey=00000000-0000-0000-0000-000000000000", "host": [ "https://api.scrapi.dev" ], "path": [ "v1", "scrape" ], "query": [ { "key": "url", "value": "https://deventerprise.com", "description": "(Required) URL to scrape for content." }, { "key": "useBrowser", "value": "false", "description": "(Optional) Indicate whether you want to use a full headless browser (will execute JavaScript) or just a regular web client call to retrieve the response (faster but no JavaScript)." }, { "key": "proxyType", "value": "None", "description": "(Optional) The type of the proxy to use for the scrape request. | 0 = None, 1 = Free, 2 = Residential, 3 = DataCenter, 4 = Tor |" }, { "key": "apiKey", "value": "00000000-0000-0000-0000-000000000000", "description": "(Optional) API Key to avoid free usage limitations." } ] } }, "_postman_previewlanguage": null, "header": null, "cookie": [], "body": null }, { "name": "No Browser - Free Proxy", "originalRequest": { "method": "GET", "header": [], "url": { "raw": "https://api.scrapi.dev/v1/scrape?url=https://deventerprise.com&useBrowser=false&proxyType=Free&apiKey=00000000-0000-0000-0000-000000000000", "host": [ "https://api.scrapi.dev" ], "path": [ "v1", "scrape" ], "query": [ { "key": "url", "value": "https://deventerprise.com", "description": "(Required) URL to scrape for content." }, { "key": "useBrowser", "value": "false", "description": "(Optional) Indicate whether you want to use a full headless browser (will execute JavaScript) or just a regular web client call to retrieve the response (faster but no JavaScript)." }, { "key": "proxyType", "value": "Free", "description": "(Optional) The type of the proxy to use for the scrape request. | 0 = None, 1 = Free, 2 = Residential, 3 = DataCenter, 4 = Tor |" }, { "key": "apiKey", "value": "00000000-0000-0000-0000-000000000000", "description": "(Optional) API Key to avoid free usage limitations." } ] } }, "_postman_previewlanguage": null, "header": null, "cookie": [], "body": null }, { "name": "No Browser - Data Center Proxy", "originalRequest": { "method": "GET", "header": [], "url": { "raw": "https://api.scrapi.dev/v1/scrape?url=https://deventerprise.com&useBrowser=false&proxyType=DataCenter&apiKey=00000000-0000-0000-0000-000000000000", "host": [ "https://api.scrapi.dev" ], "path": [ "v1", "scrape" ], "query": [ { "key": "url", "value": "https://deventerprise.com", "description": "(Required) URL to scrape for content." }, { "key": "useBrowser", "value": "false", "description": "(Optional) Indicate whether you want to use a full headless browser (will execute JavaScript) or just a regular web client call to retrieve the response (faster but no JavaScript)." }, { "key": "proxyType", "value": "DataCenter", "description": "(Optional) The type of the proxy to use for the scrape request. | 0 = None, 1 = Free, 2 = Residential, 3 = DataCenter, 4 = Tor |" }, { "key": "apiKey", "value": "00000000-0000-0000-0000-000000000000", "description": "(Optional) API Key to avoid free usage limitations." } ] } }, "_postman_previewlanguage": null, "header": null, "cookie": [], "body": null }, { "name": "No Browser - Residential Proxy", "originalRequest": { "method": "GET", "header": [], "url": { "raw": "https://api.scrapi.dev/v1/scrape?url=https://deventerprise.com&useBrowser=false&proxyType=Residential&apiKey=00000000-0000-0000-0000-000000000000", "host": [ "https://api.scrapi.dev" ], "path": [ "v1", "scrape" ], "query": [ { "key": "url", "value": "https://deventerprise.com", "description": "(Required) URL to scrape for content." }, { "key": "useBrowser", "value": "false", "description": "(Optional) Indicate whether you want to use a full headless browser (will execute JavaScript) or just a regular web client call to retrieve the response (faster but no JavaScript)." }, { "key": "proxyType", "value": "Residential", "description": "(Optional) The type of the proxy to use for the scrape request. | 0 = None, 1 = Free, 2 = Residential, 3 = DataCenter, 4 = Tor |" }, { "key": "apiKey", "value": "00000000-0000-0000-0000-000000000000", "description": "(Optional) API Key to avoid free usage limitations." } ] } }, "_postman_previewlanguage": null, "header": null, "cookie": [], "body": null }, { "name": "No Browser - Tor Proxy", "originalRequest": { "method": "GET", "header": [], "url": { "raw": "https://api.scrapi.dev/v1/scrape?url=http://a4ygisnerpgtc5ayerl22pll6cls3oyj54qgpm7qrmb66xrxts6y3lyd.onion/index.html&useBrowser=false&proxyType=Residential&apiKey=00000000-0000-0000-0000-000000000000", "host": [ "https://api.scrapi.dev" ], "path": [ "v1", "scrape" ], "query": [ { "key": "url", "value": "http://a4ygisnerpgtc5ayerl22pll6cls3oyj54qgpm7qrmb66xrxts6y3lyd.onion/index.html", "description": "(Required) URL to scrape for content." }, { "key": "useBrowser", "value": "false", "description": "(Optional) Indicate whether you want to use a full headless browser (will execute JavaScript) or just a regular web client call to retrieve the response (faster but no JavaScript)." }, { "key": "proxyType", "value": "Residential", "description": "(Optional) The type of the proxy to use for the scrape request. | 0 = None, 1 = Free, 2 = Residential, 3 = DataCenter, 4 = Tor |" }, { "key": "apiKey", "value": "00000000-0000-0000-0000-000000000000", "description": "(Optional) API Key to avoid free usage limitations." } ] } }, "_postman_previewlanguage": null, "header": null, "cookie": [], "body": null }, { "name": "HTML Response", "originalRequest": { "method": "GET", "header": [], "url": { "raw": "https://api.scrapi.dev/v1/scrape?url=https://deventerprise.com&useBrowser=false&proxyType=None&responseFormat=HTML&apiKey=00000000-0000-0000-0000-000000000000", "host": [ "https://api.scrapi.dev" ], "path": [ "v1", "scrape" ], "query": [ { "key": "url", "value": "https://deventerprise.com", "description": "(Required) URL to scrape for content." }, { "key": "useBrowser", "value": "false", "description": "(Optional) Indicate whether you want to use a full headless browser (will execute JavaScript) or just a regular web client call to retrieve the response (faster but no JavaScript)." }, { "key": "proxyType", "value": "None", "description": "(Optional) The type of the proxy to use for the scrape request. | 0 = None, 1 = Free, 2 = Residential, 3 = DataCenter, 4 = Tor |" }, { "key": "responseFormat", "value": "HTML", "description": "(Optional) The type of API response. | 0 = JSON, 1 = HTML |" }, { "key": "apiKey", "value": "00000000-0000-0000-0000-000000000000", "description": "(Optional) API Key to avoid free usage limitations." } ] } }, "_postman_previewlanguage": null, "header": null, "cookie": [], "body": null }, { "name": "Callback / Webhook", "originalRequest": { "method": "GET", "header": [], "url": { "raw": "https://api.scrapi.dev/v1/scrape?url=https://deventerprise.com&useBrowser=false&proxyType=None&callbackUrl=https://webhook-test.com/75b9b9bb6a6a7726ead42e4cf8fc6d87&apiKey=00000000-0000-0000-0000-000000000000", "host": [ "https://api.scrapi.dev" ], "path": [ "v1", "scrape" ], "query": [ { "key": "url", "value": "https://deventerprise.com", "description": "(Required) URL to scrape for content." }, { "key": "useBrowser", "value": "false", "description": "(Optional) Indicate whether you want to use a full headless browser (will execute JavaScript) or just a regular web client call to retrieve the response (faster but no JavaScript)." }, { "key": "proxyType", "value": "None", "description": "(Optional) The type of the proxy to use for the scrape request. | 0 = None, 1 = Free, 2 = Residential, 3 = DataCenter, 4 = Tor |" }, { "key": "callbackUrl", "value": "https://webhook-test.com/75b9b9bb6a6a7726ead42e4cf8fc6d87", "description": "(Optional) A URL that the response data will the POSTed to once complete." }, { "key": "apiKey", "value": "00000000-0000-0000-0000-000000000000", "description": "(Optional) API Key to avoid free usage limitations." } ] } }, "_postman_previewlanguage": null, "header": null, "cookie": [], "body": null }, { "name": "Cookies / Headers", "originalRequest": { "method": "GET", "header": [], "url": { "raw": "https://api.scrapi.dev/v1/scrape?url=https://deventerprise.com&useBrowser=false&proxyType=None&cookies=My-Cookie=Test 1;Session-ID=Test2&headers=My-Header=Test 3;X-SITE-KEY=Test 4&apiKey=00000000-0000-0000-0000-000000000000", "host": [ "https://api.scrapi.dev" ], "path": [ "v1", "scrape" ], "query": [ { "key": "url", "value": "https://deventerprise.com", "description": "(Required) URL to scrape for content." }, { "key": "useBrowser", "value": "false", "description": "(Optional) Indicate whether you want to use a full headless browser (will execute JavaScript) or just a regular web client call to retrieve the response (faster but no JavaScript)." }, { "key": "proxyType", "value": "None", "description": "(Optional) The type of the proxy to use for the scrape request. | 0 = None, 1 = Free, 2 = Residential, 3 = DataCenter, 4 = Tor |" }, { "key": "cookies", "value": "My-Cookie=Test 1;Session-ID=Test2", "description": "(Optional) A key/value pair list of cookies to provide in the scrape request (format: cookie1=value1;cookie2=value2;)." }, { "key": "headers", "value": "My-Header=Test 3;X-SITE-KEY=Test 4", "description": "(Optional) A key/value pair list of headers to provide in the scrape request (format: header1=value1;header2=value2;)." }, { "key": "apiKey", "value": "00000000-0000-0000-0000-000000000000", "description": "(Optional) API Key to avoid free usage limitations." } ] } }, "_postman_previewlanguage": null, "header": null, "cookie": [], "body": null }, { "name": "Geotargeting", "originalRequest": { "method": "GET", "header": [], "url": { "raw": "https://api.scrapi.dev/v1/scrape?url=https://deventerprise.com&useBrowser=false&proxyType=None&proxyCountry=USA&apiKey=00000000-0000-0000-0000-000000000000", "host": [ "https://api.scrapi.dev" ], "path": [ "v1", "scrape" ], "query": [ { "key": "url", "value": "https://deventerprise.com", "description": "(Required) URL to scrape for content." }, { "key": "useBrowser", "value": "false", "description": "(Optional) Indicate whether you want to use a full headless browser (will execute JavaScript) or just a regular web client call to retrieve the response (faster but no JavaScript)." }, { "key": "proxyType", "value": "None", "description": "(Optional) The type of the proxy to use for the scrape request. | 0 = None, 1 = Free, 2 = Residential, 3 = DataCenter, 4 = Tor |" }, { "key": "proxyCountry", "value": "USA", "description": "(Optional) The proxy country to use if you require the scrape request to come from." }, { "key": "apiKey", "value": "00000000-0000-0000-0000-000000000000", "description": "(Optional) API Key to avoid free usage limitations." } ] } }, "_postman_previewlanguage": null, "header": null, "cookie": [], "body": null }, { "name": "Validation Error", "originalRequest": { "method": "GET", "header": [], "url": { "raw": "https://api.scrapi.dev/v1/scrape?url=https://deventerprise.com&useBrowser=false&proxyType=None&proxyCountry=USA&apiKey=00000000-0000-0000-0000-000000000000", "host": [ "https://api.scrapi.dev" ], "path": [ "v1", "scrape" ], "query": [ { "key": "url", "value": "https://deventerprise.com", "description": "(Required) URL to scrape for content." }, { "key": "useBrowser", "value": "false", "description": "(Optional) Indicate whether you want to use a full headless browser (will execute JavaScript) or just a regular web client call to retrieve the response (faster but no JavaScript)." }, { "key": "proxyType", "value": "None", "description": "(Optional) The type of the proxy to use for the scrape request. | 0 = None, 1 = Free, 2 = Residential, 3 = DataCenter, 4 = Tor |" }, { "key": "proxyCountry", "value": "USA", "description": "(Optional) The proxy country to use if you require the scrape request to come from." }, { "key": "apiKey", "value": "00000000-0000-0000-0000-000000000000", "description": "(Optional) API Key to avoid free usage limitations." } ] } }, "_postman_previewlanguage": null, "header": null, "cookie": [], "body": null } ] }, { "name": "Check Callback Status", "request": { "method": "GET", "header": [], "url": { "raw": "https://api.scrapi.dev/v1/scrape/status/{{reference}}", "host": [ "https://api.scrapi.dev" ], "path": [ "v1", "scrape", "status", "{{reference}}" ] }, "description": "Get the status of a scrape request with a callback URL specified. The status is cached for 10 minutes afterwhich it will always return \"Uknown\" for a reference. Nothing about the callback is stored except the status as it's executing.\n\n| Status | Code |\n| --- | --- |\n| Unknown | 0 |\n| Scheduled | 1 |\n| Busy | 2 |\n| Error | 3 |\n| Complete | 4 |" }, "response": [ { "name": "Example Response", "originalRequest": { "method": "GET", "header": [], "url": { "raw": "https://api.scrapi.dev/v1/scrape/status/{{reference}}", "host": [ "https://api.scrapi.dev" ], "path": [ "v1", "scrape", "status", "{{reference}}" ] } }, "_postman_previewlanguage": "json", "header": [ { "key": "Content-Type", "value": "application/json", "name": "Content-Type", "description": "", "type": "text" } ], "cookie": [], "body": "{\n \"status\": \"Complete\",\n \"code\": 4\n}" } ] } ], "description": "ScrAPI provides GET and POST end points for scraping operations. Browser Commands are only available with POST since the parameters are too complex to build into the URL." }, { "name": "Captcha Solving", "item": [ { "name": "Cloudflare - Turnstile", "event": [ { "listen": "test", "script": { "exec": [ "const $ = cheerio.load(pm.response.json().content);\r", "\r", "pm.test(\"Captcha was solved\", function () {\r", " pm.expect($(\"form p\").text()).to.include(\"Captcha is passed successfully!\");\r", "});\r", "\r", "var html = pm.response.json().content;\r", "var htmlNoScript = html.replace(/)<[^<]*)*<\\/script>/gi, '');\r", "\r", "pm.visualizer.set(htmlNoScript);\r", "" ], "type": "text/javascript" } } ], "request": { "method": "POST", "header": [], "body": { "mode": "raw", "raw": "{\r\n \"url\": \"https://2captcha.com/demo/cloudflare-turnstile\",\r\n \"proxyType\": \"None\",\r\n \"solveCaptchas\": true,\r\n \"browserCommands\": [\r\n { \"click\": \"button[data-action='demo_action']\" },\r\n { \"wait\": 1000 }\r\n ]\r\n}\r\n", "options": { "raw": { "language": "json" } } }, "url": { "raw": "https://api.scrapi.dev/v1/scrape", "host": [ "https://api.scrapi.dev" ], "path": [ "v1", "scrape" ] } }, "response": [] }, { "name": "Cloudflare - Challenge", "event": [ { "listen": "test", "script": { "exec": [ "const $ = cheerio.load(pm.response.json().content);\r", "\r", "pm.test(\"Captcha was solved\", function () {\r", " pm.expect($(\".main_info_block\").text()).to.include(\"You have successfully passed the Cloudflare Challenge\");\r", "});\r", "\r", "var html = pm.response.json().content;\r", "var htmlNoScript = html.replace(/)<[^<]*)*<\\/script>/gi, '');\r", "\r", "pm.visualizer.set(htmlNoScript);\r", "" ], "type": "text/javascript" } } ], "request": { "method": "POST", "header": [], "body": { "mode": "raw", "raw": "{\r\n \"url\": \"https://rucaptcha.com/42\",\r\n \"proxyType\": \"None\",\r\n \"solveCaptchas\": true,\r\n \"browserCommands\": [\r\n { \"wait\": 3000 }\r\n ]\r\n}\r\n", "options": { "raw": { "language": "json" } } }, "url": { "raw": "https://api.scrapi.dev/v1/scrape", "host": [ "https://api.scrapi.dev" ], "path": [ "v1", "scrape" ] } }, "response": [] }, { "name": "hCaptcha", "event": [ { "listen": "test", "script": { "exec": [ "const $ = cheerio.load(pm.response.json().content);\r", "\r", "pm.test(\"Captcha was solved\", function () {\r", " pm.expect($(\"form p\").text()).to.include(\"Captcha is passed successfully!\");\r", "});\r", "\r", "var html = pm.response.json().content;\r", "var htmlNoScript = html.replace(/)<[^<]*)*<\\/script>/gi, '');\r", "\r", "pm.visualizer.set(htmlNoScript);\r", "" ], "type": "text/javascript" } } ], "request": { "method": "POST", "header": [], "body": { "mode": "raw", "raw": "{\r\n \"url\": \"https://2captcha.com/demo/hcaptcha\",\r\n \"proxyType\": \"None\",\r\n \"solveCaptchas\": true,\r\n \"browserCommands\": [\r\n { \"click\": \"button[data-action='demo_action']\" },\r\n { \"wait\": 1000 }\r\n ]\r\n}\r\n", "options": { "raw": { "language": "json" } } }, "url": { "raw": "https://api.scrapi.dev/v1/scrape", "host": [ "https://api.scrapi.dev" ], "path": [ "v1", "scrape" ] } }, "response": [] }, { "name": "hCaptcha - Invisible", "event": [ { "listen": "test", "script": { "exec": [ "const $ = cheerio.load(pm.response.json().content);\r", "\r", "pm.test(\"Captcha was solved\", function () {\r", " pm.expect($(\"form p\").text()).to.include(\"Captcha is passed successfully!\");\r", "});\r", "\r", "var html = pm.response.json().content;\r", "var htmlNoScript = html.replace(/)<[^<]*)*<\\/script>/gi, '');\r", "\r", "pm.visualizer.set(htmlNoScript);\r", "" ], "type": "text/javascript" } } ], "request": { "method": "POST", "header": [], "body": { "mode": "raw", "raw": "{\r\n \"url\": \"https://2captcha.com/demo/hcaptcha-invisible\",\r\n \"proxyType\": \"None\",\r\n \"solveCaptchas\": true,\r\n \"browserCommands\": [\r\n { \"wait\": 1000 }\r\n ]\r\n}", "options": { "raw": { "language": "json" } } }, "url": { "raw": "https://api.scrapi.dev/v1/scrape", "host": [ "https://api.scrapi.dev" ], "path": [ "v1", "scrape" ] } }, "response": [] }, { "name": "reCAPTCHA v2", "event": [ { "listen": "test", "script": { "exec": [ "const $ = cheerio.load(pm.response.json().content);\r", "\r", "pm.test(\"Captcha was solved\", function () {\r", " pm.expect($(\"form p\").text()).to.include(\"Captcha is passed successfully!\");\r", "});\r", "\r", "var html = pm.response.json().content;\r", "var htmlNoScript = html.replace(/)<[^<]*)*<\\/script>/gi, '');\r", "\r", "pm.visualizer.set(htmlNoScript);\r", "" ], "type": "text/javascript" } } ], "request": { "method": "POST", "header": [], "body": { "mode": "raw", "raw": "{\r\n \"url\": \"https://2captcha.com/demo/recaptcha-v2\",\r\n \"proxyType\": \"None\",\r\n \"solveCaptchas\": true,\r\n \"browserCommands\": [\r\n { \"click\": \"button[data-action='demo_action']\" },\r\n { \"wait\": 1000 }\r\n ]\r\n}\r\n", "options": { "raw": { "language": "json" } } }, "url": { "raw": "https://api.scrapi.dev/v1/scrape", "host": [ "https://api.scrapi.dev" ], "path": [ "v1", "scrape" ] } }, "response": [] }, { "name": "reCAPTCHA v2 - Callback", "event": [ { "listen": "test", "script": { "exec": [ "const $ = cheerio.load(pm.response.json().content);\r", "\r", "pm.test(\"Captcha was solved\", function () {\r", " pm.expect($(\"form p\").text()).to.include(\"Captcha is passed successfully!\");\r", "});\r", "\r", "var html = pm.response.json().content;\r", "var htmlNoScript = html.replace(/)<[^<]*)*<\\/script>/gi, '');\r", "\r", "pm.visualizer.set(htmlNoScript);\r", "" ], "type": "text/javascript" } } ], "request": { "method": "POST", "header": [], "body": { "mode": "raw", "raw": "{\r\n \"url\": \"https://2captcha.com/demo/recaptcha-v2-callback\",\r\n \"proxyType\": \"None\",\r\n \"solveCaptchas\": true,\r\n \"browserCommands\": [\r\n { \"wait\": 1000 }\r\n ]\r\n}\r\n", "options": { "raw": { "language": "json" } } }, "url": { "raw": "https://api.scrapi.dev/v1/scrape", "host": [ "https://api.scrapi.dev" ], "path": [ "v1", "scrape" ] } }, "response": [] }, { "name": "reCAPTCHA v2 - Invisible", "event": [ { "listen": "test", "script": { "exec": [ "const $ = cheerio.load(pm.response.json().content);\r", "\r", "pm.test(\"Captcha was solved\", function () {\r", " pm.expect($(\"form p\").text()).to.include(\"Captcha is passed successfully!\");\r", "});\r", "\r", "var html = pm.response.json().content;\r", "var htmlNoScript = html.replace(/)<[^<]*)*<\\/script>/gi, '');\r", "\r", "pm.visualizer.set(htmlNoScript);\r", "" ], "type": "text/javascript" } } ], "request": { "method": "POST", "header": [], "body": { "mode": "raw", "raw": "{\r\n \"url\": \"https://2captcha.com/demo/recaptcha-v2-invisible\",\r\n \"proxyType\": \"None\",\r\n \"solveCaptchas\": true,\r\n \"browserCommands\": [\r\n { \"wait\": 1000 }\r\n ]\r\n}\r\n", "options": { "raw": { "language": "json" } } }, "url": { "raw": "https://api.scrapi.dev/v1/scrape", "host": [ "https://api.scrapi.dev" ], "path": [ "v1", "scrape" ] } }, "response": [] }, { "name": "reCAPTCHA v2 - Enterprise", "event": [ { "listen": "test", "script": { "exec": [ "const $ = cheerio.load(pm.response.json().content);\r", "\r", "pm.test(\"Captcha was solved\", function () {\r", " pm.expect($(\"form p\").text()).to.include(\"Captcha is passed successfully!\");\r", "});\r", "\r", "var html = pm.response.json().content;\r", "var htmlNoScript = html.replace(/)<[^<]*)*<\\/script>/gi, '');\r", "\r", "pm.visualizer.set(htmlNoScript);\r", "" ], "type": "text/javascript" } } ], "request": { "method": "POST", "header": [], "body": { "mode": "raw", "raw": "{\r\n \"url\": \"https://2captcha.com/demo/recaptcha-v2-enterprise\",\r\n \"proxyType\": \"None\",\r\n \"solveCaptchas\": true,\r\n \"browserCommands\": [\r\n { \"click\": \"button[data-action='demo_action']\" },\r\n { \"wait\": 1000 }\r\n ]\r\n}\r\n", "options": { "raw": { "language": "json" } } }, "url": { "raw": "https://api.scrapi.dev/v1/scrape", "host": [ "https://api.scrapi.dev" ], "path": [ "v1", "scrape" ] } }, "response": [] }, { "name": "reCAPTCHA v3", "event": [ { "listen": "test", "script": { "exec": [ "const $ = cheerio.load(pm.response.json().content);\r", "\r", "pm.test(\"Captcha was solved\", function () {\r", " pm.expect($(\"form p\").text()).to.include(\"Captcha is passed successfully!\");\r", "});\r", "\r", "var html = pm.response.json().content;\r", "var htmlNoScript = html.replace(/)<[^<]*)*<\\/script>/gi, '');\r", "\r", "pm.visualizer.set(htmlNoScript);\r", "" ], "type": "text/javascript" } } ], "request": { "method": "POST", "header": [], "body": { "mode": "raw", "raw": "{\r\n \"url\": \"https://2captcha.com/demo/recaptcha-v3\",\r\n \"proxyType\": \"None\",\r\n \"solveCaptchas\": true,\r\n \"browserCommands\": [\r\n { \"click\": \"button[data-action='demo_action']\" },\r\n { \"wait\": 1000 }\r\n ]\r\n}\r\n", "options": { "raw": { "language": "json" } } }, "url": { "raw": "https://api.scrapi.dev/v1/scrape", "host": [ "https://api.scrapi.dev" ], "path": [ "v1", "scrape" ] } }, "response": [] }, { "name": "reCAPTCHA v3 - Enterprise", "event": [ { "listen": "test", "script": { "exec": [ "const $ = cheerio.load(pm.response.json().content);\r", "\r", "pm.test(\"Captcha was solved\", function () {\r", " pm.expect($(\"form p\").text()).to.include(\"Captcha is passed successfully!\");\r", "});\r", "\r", "var html = pm.response.json().content;\r", "var htmlNoScript = html.replace(/)<[^<]*)*<\\/script>/gi, '');\r", "\r", "pm.visualizer.set(htmlNoScript);\r", "" ], "type": "text/javascript" } } ], "request": { "method": "POST", "header": [], "body": { "mode": "raw", "raw": "{\r\n \"url\": \"https://2captcha.com/demo/recaptcha-v3-enterprise\",\r\n \"proxyType\": \"None\",\r\n \"solveCaptchas\": true,\r\n \"browserCommands\": [\r\n { \"click\": \"button[data-action='demo_action']\" },\r\n { \"wait\": 1000 }\r\n ]\r\n}\r\n", "options": { "raw": { "language": "json" } } }, "url": { "raw": "https://api.scrapi.dev/v1/scrape", "host": [ "https://api.scrapi.dev" ], "path": [ "v1", "scrape" ] } }, "response": [] } ], "description": "Captcha solving examples." }, { "name": "Checks", "item": [ { "name": "Cookies Check", "event": [ { "listen": "test", "script": { "exec": [ "var html = pm.response.json().content;\r", "\r", "pm.visualizer.set(html);\r", "\r", "pm.test(\"Contains Custom Cookies\", function () {\r", " pm.expect(html).to.include(\"My-Cookie\");\r", " pm.expect(html).to.include(\"Session-ID\");\r", "});\r", "" ], "type": "text/javascript" } } ], "request": { "method": "POST", "header": [], "body": { "mode": "raw", "raw": "{\r\n \"url\": \"http://httpbin.org/cookies\",\r\n \"useBrowser\": false,\r\n \"proxyType\": \"Residential\",\r\n \"cookies\": {\r\n \"My-Cookie\": \"Test 1\",\r\n \"Session-ID\": \"Test 2\"\r\n }\r\n}\r\n", "options": { "raw": { "language": "json" } } }, "url": { "raw": "https://api.scrapi.dev/v1/scrape", "host": [ "https://api.scrapi.dev" ], "path": [ "v1", "scrape" ] } }, "response": [] }, { "name": "Headers Check", "event": [ { "listen": "test", "script": { "exec": [ "var html = pm.response.json().content;\r", "\r", "pm.visualizer.set(html);\r", "\r", "pm.test(\"Contains Custom Headers\", function () {\r", " pm.expect(html).to.include(\"My-Header\");\r", " pm.expect(html).to.include(\"X-Site-Key\");\r", "});\r", "" ], "type": "text/javascript" } } ], "request": { "method": "POST", "header": [], "body": { "mode": "raw", "raw": "{\r\n \"url\": \"http://httpbin.org/headers\",\r\n \"useBrowser\": false,\r\n \"proxyType\": \"Residential\",\r\n \"headers\": {\r\n \"My-Header\": \"Test Header 1\",\r\n \"X-SITE-KEY\": \"Test Header 2\"\r\n }\r\n}\r\n", "options": { "raw": { "language": "json" } } }, "url": { "raw": "https://api.scrapi.dev/v1/scrape", "host": [ "https://api.scrapi.dev" ], "path": [ "v1", "scrape" ] } }, "response": [] }, { "name": "User Agent Check", "event": [ { "listen": "test", "script": { "exec": [ "var html = pm.response.json().content;\r", "\r", "pm.visualizer.set(html);" ], "type": "text/javascript" } } ], "request": { "method": "POST", "header": [], "body": { "mode": "raw", "raw": "{\r\n \"url\": \"https://httpbin.io/user-agent\",\r\n \"useBrowser\": true,\r\n \"proxyType\": \"None\"\r\n}", "options": { "raw": { "language": "json" } } }, "url": { "raw": "https://api.scrapi.dev/v1/scrape", "host": [ "https://api.scrapi.dev" ], "path": [ "v1", "scrape" ] } }, "response": [] }, { "name": "IP Address Check", "event": [ { "listen": "test", "script": { "exec": [ "var json = pm.response.json();\r", "var html = json.content;\r", "var htmlNoScript = html.replace(/)<[^<]*)*<\\/script>/gi, '');\r", "\r", "pm.visualizer.set(htmlNoScript);\r", " " ], "type": "text/javascript" } } ], "request": { "method": "POST", "header": [], "body": { "mode": "raw", "raw": "{\r\n \"url\": \"https://whatismyipaddress.com\",\r\n \"useBrowser\": true,\r\n \"proxyType\": \"DataCenter\",\r\n \"proxyCountry\": \"USA\"\r\n}", "options": { "raw": { "language": "json" } } }, "url": { "raw": "https://api.scrapi.dev/v1/scrape", "host": [ "https://api.scrapi.dev" ], "path": [ "v1", "scrape" ] } }, "response": [] }, { "name": "Sannysoft Bot Check", "event": [ { "listen": "test", "script": { "exec": [ "var html = pm.response.json().content;\r", "var htmlNoScript = html.replace(/)<[^<]*)*<\\/script>/gi, '');\r", "\r", "pm.visualizer.set(htmlNoScript);" ], "type": "text/javascript" } } ], "request": { "method": "POST", "header": [], "body": { "mode": "raw", "raw": "{\r\n \"url\": \"https://bot.sannysoft.com\",\r\n \"useBrowser\": true,\r\n \"proxyType\": \"None\"\r\n}", "options": { "raw": { "language": "json" } } }, "url": { "raw": "https://api.scrapi.dev/v1/scrape", "host": [ "https://api.scrapi.dev" ], "path": [ "v1", "scrape" ] } }, "response": [] }, { "name": "Antoine Vastel Bot Check", "event": [ { "listen": "test", "script": { "exec": [ "var html = pm.response.json().content;\r", "var htmlNoScript = html.replace(/)<[^<]*)*<\\/script>/gi, '');\r", "\r", "pm.visualizer.set(htmlNoScript);" ], "type": "text/javascript" } } ], "request": { "method": "POST", "header": [], "body": { "mode": "raw", "raw": "{\r\n \"url\": \"https://antoinevastel.com/bots/\",\r\n \"useBrowser\": true,\r\n \"proxyType\": \"None\"\r\n}", "options": { "raw": { "language": "json" } } }, "url": { "raw": "https://api.scrapi.dev/v1/scrape", "host": [ "https://api.scrapi.dev" ], "path": [ "v1", "scrape" ] } }, "response": [] }, { "name": "Incolumitas Bot Test", "event": [ { "listen": "test", "script": { "exec": [ "var html = pm.response.json().content;\r", "var htmlNoScript = html.replace(/)<[^<]*)*<\\/script>/gi, '');\r", "\r", "pm.visualizer.set(htmlNoScript);\r", "\r", "const $ = cheerio.load(html);\r", "\r", "const score = $('#behavioralScore').text();\r", "const scoreNumber = parseFloat(score.replace(/(\\d+)\\.(\\d+)\\.(\\d+)/, \"$1.$2$3\"));\r", "\r", "console.log('Behavioral Score: ' + scoreNumber);\r", "\r", "pm.test(\"Behavioral Score > 0.5\", function () {\r", " pm.expect(scoreNumber).to.be.at.least(0.5);\r", "});\r", "" ], "type": "text/javascript" } } ], "request": { "method": "POST", "header": [], "body": { "mode": "raw", "raw": "{\r\n \"url\": \"https://bot.incolumitas.com/\",\r\n \"useBrowser\": true,\r\n \"proxyType\": \"None\",\r\n \"acceptDialogs\": \"true\",\r\n \"browserCommands\": [\r\n { \"waitfor\": \"#formStuff\" },\r\n { \"input\": { \"[name='userName']\": \"bot3000\" } },\r\n { \"input\": { \"[name='eMail']\": \"bot3000@gmail.com\" } },\r\n { \"select\": { \"[name='cookies']\": \"I want all the Cookies\" } },\r\n { \"click\": \"#smolCat\" },\r\n { \"click\": \"#bigCat\" },\r\n { \"click\": \"#submit\" },\r\n { \"waitfor\": \"#tableStuff tbody tr .url\" },\r\n { \"wait\": 100 },\r\n { \"click\": \"#updatePrice0\" },\r\n { \"click\": \"#updatePrice1\" },\r\n { \"wait\": 2000 }\r\n ]\r\n}", "options": { "raw": { "language": "json" } } }, "url": { "raw": "https://api.scrapi.dev/v1/scrape", "host": [ "https://api.scrapi.dev" ], "path": [ "v1", "scrape" ] } }, "response": [] }, { "name": "Pixelscan Fingerprint Test", "event": [ { "listen": "test", "script": { "exec": [ "var html = pm.response.json().content;\r", "var htmlNoScript = html.replace(/)<[^<]*)*<\\/script>/gi, '');\r", "\r", "pm.visualizer.set(htmlNoScript);" ], "type": "text/javascript" } } ], "request": { "method": "POST", "header": [], "body": { "mode": "raw", "raw": "{\r\n \"url\": \"https://pixelscan.net/\",\r\n \"useBrowser\": true,\r\n \"proxyType\": \"None\",\r\n \"browserCommands\": [\r\n { \"wait\": 10000 },\r\n { \"waitfor\": \".consistency-status-text\"}\r\n ]\r\n}", "options": { "raw": { "language": "json" } } }, "url": { "raw": "https://api.scrapi.dev/v1/scrape", "host": [ "https://api.scrapi.dev" ], "path": [ "v1", "scrape" ] } }, "response": [] }, { "name": "CreepJS Tests", "event": [ { "listen": "test", "script": { "exec": [ "var html = pm.response.json().content;\r", "var htmlNoScript = html.replace(/)<[^<]*)*<\\/script>/gi, '');\r", "\r", "pm.visualizer.set(htmlNoScript);\r", "\r", "const $ = cheerio.load(html);\r", "\r", "const score = $('#fingerprint-data .unblurred').text();\r", "const scoreNumber = parseFloat(score.replace(/(\\d+)\\.(\\d+)\\.(\\d+)/, \"$1.$2$3\"));\r", "\r", "console.log(score);\r", "console.log('Trust Score: ' + scoreNumber);\r", "\r", "//pm.test(\"Trust Score > 60%\", function () {\r", "// pm.expect(scoreNumber).to.be.at.least(60);\r", "//});\r", "" ], "type": "text/javascript" } } ], "request": { "method": "POST", "header": [], "body": { "mode": "raw", "raw": "{\r\n \"url\": \"https://abrahamjuliot.github.io/creepjs/\",\r\n \"useBrowser\": true,\r\n \"proxyType\": \"None\",\r\n \"browserCommands\": [\r\n { \"wait\": 15000 },\r\n { \"waitfor\": \"#fingerprint-data .unblurred\"}\r\n ]\r\n}", "options": { "raw": { "language": "json" } } }, "url": { "raw": "https://api.scrapi.dev/v1/scrape", "host": [ "https://api.scrapi.dev" ], "path": [ "v1", "scrape" ] } }, "response": [] } ], "description": "Various checks to validate scraping functionality and demonstrate bot automation." }, { "name": "Credit Balance", "request": { "method": "GET", "header": [], "url": { "raw": "https://api.scrapi.dev/v1/balance", "host": [ "https://api.scrapi.dev" ], "path": [ "v1", "balance" ] }, "description": "Get the current balance for your API key. When usinf the test API key (or no API key) this just returns a random number of credits." }, "response": [] }, { "name": "Available Country List", "request": { "method": "GET", "header": [], "url": { "raw": "https://api.scrapi.dev/v1/countries", "host": [ "https://api.scrapi.dev" ], "path": [ "v1", "countries" ] }, "description": "Get a list of available country codes that can be used as the proxy country parameter for geotargeting." }, "response": [] } ], "auth": { "type": "apikey", "apikey": [ { "key": "value", "value": "00000000-0000-0000-0000-000000000000", "type": "string" }, { "key": "key", "value": "X-API-KEY", "type": "string" } ] }, "event": [ { "listen": "prerequest", "script": { "type": "text/javascript", "exec": [ "" ] } }, { "listen": "test", "script": { "type": "text/javascript", "exec": [ "" ] } } ], "variable": [ { "key": "reference", "value": "" } ] }