openapi: 3.0.3 info: title: Web Scraper API description: | # Web Scraper API A powerful web scraping API with anti-bot bypass capabilities using Chrome-like fingerprints. ## Features - **Anti-Bot Bypass**: Real Chrome TLS fingerprints to bypass detection - **JavaScript Rendering**: Full browser rendering with smart browser pooling - **Browser Automation**: Execute click, fill, scroll, wait steps before capture - **Proxy Support**: Multi-geo proxies (US, EU, Residential, country-specific) - **Content Extraction**: Readability, JSON-LD, Next.js data extraction - **Custom Extractors**: Execute sandboxed JavaScript for custom parsing ## Endpoints | Endpoint | Description | Billing | |----------|-------------|---------| | `/scrape` | Fast HTTP scraping with anti-detection | 1 point | | `/scrapeJs` | Full browser rendering with JavaScript | 5 points | | `/extract` | Standalone content extraction | 1 point | | `/proxies/status` | Proxy health status | Free | ## Quick Start ```bash curl -X POST "https://api.scraperex.com/scrape" \ -H "X-RapidAPI-Key: YOUR_API_KEY" \ -H "X-RapidAPI-Host: api.scraperex.com" \ -H "Content-Type: application/json" \ -d '{"url": "https://httpbin.org/html"}' ``` ## When to Use Each Endpoint - **Use `/scrape`** for: Static sites, APIs, news articles, most websites - **Use `/scrapeJs`** for: SPAs (React/Vue/Angular), sites requiring JS, Cloudflare-protected sites - **Use `/scrapeJs` with `steps`** for: Login flows, pagination, interactive elements ## Proxy Geos | Geo | Description | Plans | |-----|-------------|-------| | `default` | Auto-select best proxy | All | | `us` | US datacenter/residential | All | | `eu` | European proxies | All | | `residential` | Residential IPs (anti-bot) | All | | `br`, `de`, `fr`, `es`, `au`, `ca` | Country-specific | All | | `4g-eu` | Premium 4G mobile proxies | **MEGA/ULTRA only** | > **Note**: Premium geos (`4g-eu`) require MEGA or ULTRA subscription plans. version: 1.2.0 contact: name: API Support email: support@apivex.io termsOfService: https://apivex.io/terms license: name: Commercial url: https://apivex.io/license servers: - url: https://api.scraperex.com description: RapidAPI Production tags: - name: Scraping description: | Web scraping endpoints for fetching content from any URL. - **`/scrape`** - Fast HTTP scraping with anti-detection (1 point) - **`/scrapeJs`** - Full browser rendering with JavaScript support (5 points) Choose based on whether the target site requires JavaScript rendering. - name: Extraction description: | Content extraction and processing endpoints. Convert raw HTML into clean, structured formats like markdown or extract embedded JSON data from script tags. - name: Utilities description: | Utility endpoints for monitoring and status checks. Check proxy health, API status, and other operational information. paths: /scrape: post: tags: - Scraping summary: Scrape a URL (Fast HTTP) description: | High-performance HTTP scraper with authentic Chrome TLS fingerprints for anti-bot bypass. **Billing**: 1 point per request --- ## When to Use This Endpoint ✅ **Use `/scrape` for:** - Static websites and server-rendered pages - REST APIs and JSON endpoints - News articles, blog posts, product pages - Sites that don't require JavaScript to display content - High-volume scraping where speed matters ❌ **Use `/scrapeJs` instead for:** - Single Page Applications (React, Vue, Angular) - Pages with infinite scroll or lazy loading - Sites protected by Cloudflare, PerimeterX, or DataDome - Content loaded via AJAX after page load --- ## Features | Feature | Description | |---------|-------------| | **Chrome TLS Fingerprint** | Mimics real Chrome browser at the TLS/SSL level, bypassing many bot detection systems | | **Custom Headers** | Add any HTTP headers (cookies, authorization, user-agent) | | **HTTP Methods** | Supports GET, POST, and PUT requests | | **Automatic Retries** | Configurable retry logic with different proxies on each attempt | | **Geo-targeted Proxies** | Route requests through specific countries (US, EU, residential, etc.) | | **Content Extraction** | Built-in extractors to convert HTML to clean markdown or extract JSON | | **Sticky Sessions** | Maintain the same IP across multiple requests using `stickyIpKey` | --- ## Proxy Geo Options | Geo Code | Location | Plan Required | |----------|----------|---------------| | `default` | Mixed global | All | | `us` | United States | All | | `eu` | Europe | All | | `ca` | Canada | All | | `residential` | Residential IPs | All | | `4g-eu` | European 4G Mobile | MEGA/ULTRA only | --- ## Best Practices 1. **Start without proxy geo** - Many sites work fine without geo-targeting 2. **Add retries for unreliable sites** - Set `retryNum: 2` or `3` 3. **Use extractors** - Set `extractorPreset: "markdown"` to get clean content 4. **Handle rate limits** - Check for 429 status in `textNotExpected` operationId: scrape requestBody: required: true description: | ## Fast HTTP Scraping High-speed scraping that mimics a real browser. Fast and efficient for most websites. ### Key Parameters | Parameter | Description | |-----------|-------------| | `url` | **(Required)** The URL to scrape | | `geo` | Proxy location: `us`, `eu`, `ca`, `residential`, `4g-eu` | | `headers` | Custom HTTP headers as array: `["Header: Value"]` | | `retryNum` | Number of retry attempts (default: 0) | | `extractorPreset` | Extract content: `markdown`, `json_ld`, `nextjs_data` | ### When to Use ✅ Static websites, APIs, news articles, product pages ❌ For JavaScript-heavy sites (React, Vue), use `/scrapeJs` instead --- 💡 **Check the "Example Name" dropdown above for ready-to-use examples!** content: application/json: schema: $ref: '#/components/schemas/ScrapeRequest' examples: Basic Request: summary: Simple GET request to fetch a page value: url: "https://httpbin.org/html" With Custom Headers: summary: Add custom HTTP headers to the request value: url: "https://httpbin.org/headers" headers: - "X-Custom-Header: MyValue" - "Accept: application/json" POST Request: summary: Send POST request with form data value: url: "https://httpbin.org/post" method: "POST" data: "username=test&password=demo" With Retry and Extraction: summary: Auto-retry failed requests and extract content value: url: "https://news.ycombinator.com/" retryNum: 2 extractorPreset: "markdown" Disable Redirects: summary: Don't follow HTTP redirects value: url: "https://httpbin.org/redirect/1" followRedirects: false With Geo Proxy: summary: Route through US proxy with sticky session value: url: "https://httpbin.org/ip" geo: "us" stickyIpKey: "session123" Custom Extractor: summary: Use custom JavaScript to extract data value: url: "https://quotes.toscrape.com/" extractor: "function(input, cheerio) { let $ = cheerio.load(input); return { quotes: $('.quote .text').map((i, el) => $(el).text()).get().slice(0, 5) }; }" responses: '200': description: Successful scrape headers: X-Apivex-Billing-Points: description: Number of billing points consumed schema: type: integer example: 1 content: application/json: schema: $ref: '#/components/schemas/ScrapeResponse' examples: Successful Response: summary: Standard successful scrape response value: body: "ExampleHello World" info: statusCode: 200 latency: 523 isBinary: false billingPoints: 1 headers: content-type: "text/html; charset=utf-8" date: "Sun, 08 Feb 2026 10:00:00 GMT" With Content Extraction: summary: Response with extracted markdown content value: body: "..." info: statusCode: 200 latency: 856 billingPoints: 2 extractor: result: markdown: "# Article Title\n\nThis is the extracted content..." html: "

Article Title

This is the extracted content...

" meta: title: "Article Title" author: "John Doe" '403': description: Premium geo requires plan upgrade content: application/json: schema: $ref: '#/components/schemas/ErrorResponse' example: status: "fail" message: "Premium geo code, please upgrade to ULTRA or MEGA to use it!" errors: - geo: "Premium geo code, please upgrade to ULTRA or MEGA to use it!" '422': description: Validation error content: application/json: schema: $ref: '#/components/schemas/ValidationError' '503': description: Scrape failed after retries content: application/json: schema: $ref: '#/components/schemas/ErrorResponse' /scrapeJs: post: tags: - Scraping summary: Scrape with JavaScript Rendering (Browser) description: | Full headless browser scraping with JavaScript rendering. Executes page scripts and captures the final DOM state exactly as a real user would see it. **Billing**: 5 points per request --- ## When to Use This Endpoint ✅ **Use `/scrapeJs` for:** - Single Page Applications (React, Vue, Angular, Next.js) - Sites with infinite scroll or lazy-loaded content - Cloudflare, PerimeterX, or DataDome protected sites - Pages requiring login or session handling - Content loaded dynamically via JavaScript/AJAX - Sites that detect and block traditional HTTP scrapers - When you need screenshots of rendered pages ❌ **Use `/scrape` instead for:** - Static HTML pages (faster and cheaper) - REST APIs returning JSON - Sites that work without JavaScript --- ## Features | Feature | Description | |---------|-------------| | **Full Browser Rendering** | Real browser executes all JavaScript | | **Wait for Elements** | Wait for specific CSS selectors before capturing | | **Screenshots** | Automatic PNG screenshot of the rendered page | | **Device Emulation** | Emulate iPhone, iPad, Pixel, and other devices | | **Cookie Injection** | Pre-set cookies for authenticated sessions | | **Resource Blocking** | Block ads, images, media to speed up loading | | **AJAX Interception** | Capture background API requests made by the page | | **Browser Automation** | Execute click, fill, scroll, and navigation steps | | **Anti-Detection** | Stealth mode to avoid headless browser detection | --- ## Browser Automation Steps The `steps` parameter allows you to automate browser interactions before capturing the page: | Step Type | Description | Example | |-----------|-------------|---------| | `click` | Click an element | `{"type": "click", "selector": "#button"}` | | `fill` | Type text into input | `{"type": "fill", "selector": "#email", "value": "test@example.com"}` | | `wait` | Wait milliseconds | `{"type": "wait", "value": 2000}` | | `waitForSelector` | Wait for element | `{"type": "waitForSelector", "selector": ".loaded"}` | | `scroll` | Scroll to element | `{"type": "scroll", "selector": "#footer"}` | | `select` | Select dropdown option | `{"type": "select", "selector": "#country", "value": "US"}` | | `hover` | Hover over element | `{"type": "hover", "selector": ".menu"}` | --- ## Resource Blocking Speed up scraping by blocking unnecessary resources: | Parameter | What it Blocks | |-----------|----------------| | `blockAds: true` | Ad networks, trackers, analytics (default: true) | | `blockImages: true` | All images (PNG, JPG, GIF, WebP, SVG) | | `blockMedia: true` | Videos, audio, fonts, stylesheets | **Tip**: Enable all blocking for fastest scraping when you only need text content. --- ## Device Emulation Emulate mobile devices to scrape mobile-specific content: - `iPhone X`, `iPhone 12`, `iPhone 13 Pro` - `Pixel 5`, `Galaxy S21` - `iPad`, `iPad Pro` --- ## Best Practices 1. **Always use `waitForSelector`** - Ensure dynamic content has loaded 2. **Set appropriate `timeout`** - Default 30s, increase for slow sites 3. **Block resources when possible** - Faster scraping, lower costs 4. **Use `steps` for interactive content** - Login forms, pagination, modals 5. **Check screenshots** - Debug issues by viewing the captured screenshot operationId: scrapeJs requestBody: required: true description: | ## Browser Scraping with JavaScript Rendering Full browser that executes JavaScript and renders pages exactly like a real user would see them. ### Key Parameters | Parameter | Description | |-----------|-------------| | `url` | **(Required)** The URL to scrape | | `waitForSelector` | CSS selector to wait for before capturing | | `timeout` | Max wait time in seconds (default: 30) | | `screenshot` | Take screenshot (default: true) | | `blockImages` | Block image loading for faster scraping | | `blockAds` | Block ads and trackers (default: true) | | `emulateDevice` | Mobile emulation: `iPhone X`, `Pixel 5`, etc. | | `steps` | Browser automation: click, fill, scroll, wait | | `cookies` | Pre-set cookies for authenticated sessions | ### Browser Automation Steps Use `steps` array to interact with the page before capturing: ```json "steps": [ {"type": "fill", "selector": "#email", "value": "user@example.com"}, {"type": "click", "selector": "#submit", "waitForNavigation": true} ] ``` **Step types**: `click`, `fill`, `wait`, `waitForSelector`, `scroll`, `select`, `hover` ### When to Use ✅ SPAs (React, Vue, Angular), Cloudflare-protected sites, login flows, infinite scroll --- 💡 **Check the "Example Name" dropdown above for ready-to-use examples!** content: application/json: schema: $ref: '#/components/schemas/ScrapeJsRequest' examples: Basic Browser Scrape: summary: Simple page scrape with full JavaScript rendering value: url: "https://books.toscrape.com/" Wait for Dynamic Content: summary: Wait for elements to load before capturing value: url: "https://quotes.toscrape.com/js/" waitForSelector: ".quote" timeout: 30 Mobile Device Emulation: summary: Scrape as mobile device (iPhone X) value: url: "https://news.ycombinator.com/" emulateDevice: "iPhone X" Block Resources: summary: Block ads, images, and media for faster loading value: url: "https://www.bbc.com/news" blockAds: true blockImages: true blockMedia: true Intercept AJAX Requests: summary: Capture background API calls made by the page value: url: "https://quotes.toscrape.com/scroll" catchAjaxHeadersUrlMask: "/api/" waitForSelector: ".quote" With Session Cookies: summary: Pre-set cookies for authenticated sessions value: url: "https://httpbin.org/cookies" cookies: - name: "session" value: "abc123" domain: "httpbin.org" path: "/" Login Form Automation: summary: Fill form and click submit button value: url: "https://quotes.toscrape.com/login" steps: - type: "fill" selector: "#username" value: "testuser" - type: "fill" selector: "#password" value: "secret123" - type: "click" selector: "input[type='submit']" waitForNavigation: true waitForSelector: ".header-box" responses: '200': description: Successful browser scrape headers: X-Apivex-Billing-Points: description: Number of billing points consumed schema: type: integer example: 5 content: application/json: schema: $ref: '#/components/schemas/ScrapeJsResponse' example: info: js: true attemptsNum: 1 statusCode: 200 screenshot: "https://cdn.example.com/screenshots/2026-02-08T10-00-00-abc123.png" finalUrl: "https://books.toscrape.com/" headers: - "content-type: text/html; charset=utf-8" pageCookies: "session=abc123; theme=dark" billingPoints: 5 body: "..." '403': description: Premium geo requires plan upgrade content: application/json: schema: $ref: '#/components/schemas/ErrorResponse' example: status: "fail" message: "Premium geo code, please upgrade to ULTRA or MEGA to use it!" '503': description: Browser scrape failed content: application/json: schema: $ref: '#/components/schemas/ErrorResponse' /extract: post: tags: - Extraction summary: Extract content from HTML description: | Standalone content extraction service. Process HTML you already have and extract clean, structured content without making additional HTTP requests. **Billing**: 1 point per request --- ## When to Use This Endpoint ✅ **Use `/extract` when:** - You've already scraped HTML and want to extract content - Processing cached or stored HTML documents - Building a pipeline: scrape first, extract later - You need to re-extract with different presets 💡 **Tip**: For one-step scraping + extraction, use `/scrape` with `extractorPreset` parameter instead. --- ## Extraction Presets | Preset | Output | Best For | |--------|--------|----------| | `markdown` | Full page as markdown | Complete document conversion | | `markdown_content` | Main article as markdown | Blog posts, news articles | | `content` | Clean HTML of main content | Further HTML processing | | `json` | All JSON from script tags | SPAs, data extraction | | `json_ld` | JSON-LD structured data | SEO data, product info, recipes | | `nextjs_data` | Next.js `__NEXT_DATA__` | Next.js applications | --- ## Features | Feature | Description | |---------|-------------| | **Readability Algorithm** | Mozilla's Readability extracts main article content, removing navigation, ads, and sidebars | | **Markdown Conversion** | Clean markdown output with proper headings, lists, links, and code blocks | | **JSON Extraction** | Parse and extract JSON data embedded in `" preset: json_ld Extract Next.js Data: summary: Get __NEXT_DATA__ from Next.js applications value: html: "" preset: nextjs_data responses: '200': description: Extraction successful content: application/json: schema: type: object properties: markdown: type: string description: Extracted content as markdown html: type: string description: Cleaned HTML meta: type: object description: Extracted metadata properties: title: type: string author: type: string description: type: string json: type: array description: Extracted JSON objects (for json preset) items: type: object json_ld: type: array description: JSON-LD structured data items: type: object nextjs_data: type: object description: Next.js page data /proxies/status: get: tags: - Utilities summary: Get proxy status description: | Check the status of proxy pools for each geo location. Shows total and healthy proxy counts. operationId: proxyStatus responses: '200': description: Proxy status content: application/json: schema: type: object properties: proxies: type: object additionalProperties: type: object properties: total: type: integer healthy: type: integer example: proxies: default: total: 10 healthy: 8 us: total: 5 healthy: 5 eu: total: 5 healthy: 4 /: get: tags: - Utilities summary: Health check description: Check if the API is running operationId: healthCheck responses: '200': description: API is healthy content: application/json: schema: type: object properties: status: type: string example: "ok" message: type: string example: "Scraper API is running" components: schemas: ScrapeRequest: type: object required: - url properties: url: type: string format: uri description: URL to scrape (must be http:// or https://) example: "https://example.com" headers: type: array description: "Custom headers in Header: Value format" example: ["User-Agent: MyBot/1.0", "Accept: text/html"] items: type: string method: type: string enum: [GET, POST, PUT] default: GET description: HTTP method data: type: string description: Request body for POST/PUT example: "key=value&foo=bar" multipart: type: boolean default: false description: Send data as multipart form geo: type: string enum: [default, us, eu, residential, br, de, fr, es, au, ca, 4g-eu] default: default description: | Proxy geo location. Use 'residential' for highly protected sites. Note: '4g-eu' requires MEGA or ULTRA plan. proxy: type: string description: Custom proxy URL (overrides geo) example: "http://user:pass@proxy.example.com:8080" stickyIpKey: type: string description: Key for sticky IP sessions (same key = same IP) example: "session123" timeout: type: integer minimum: 4 maximum: 30 default: 10 description: Request timeout in seconds retryNum: type: integer minimum: 0 maximum: 4 default: 0 description: Number of retry attempts followRedirects: type: boolean default: true description: Follow HTTP redirects textNotExpected: type: array description: Retry if response contains these strings example: ["Access Denied", "Please verify"] items: type: string statusNotExpected: type: array description: Retry if response has these status codes example: [403, 429, 503] items: type: integer extractorPreset: type: string enum: [markdown, markdown_content, content] description: Content extraction preset extractor: type: string description: Custom extractor JavaScript code outputBody: type: boolean default: true description: Include response body in output outputHeaders: type: boolean default: true description: Include response headers in output ScrapeJsRequest: type: object required: - url properties: url: type: string format: uri description: URL to scrape example: "https://example.com" headers: type: array description: Custom headers items: type: string cookies: type: array items: type: object properties: name: type: string value: type: string domain: type: string description: Cookies to set before navigation geo: type: string enum: [default, us, eu, residential, br, de, fr, es, au, ca, 4g-eu] default: default description: | Proxy geo location. Note: '4g-eu' requires MEGA or ULTRA plan. proxy: type: string description: Custom proxy URL stickyIpKey: type: string description: Sticky IP session key timeout: type: integer minimum: 10 maximum: 40 default: 30 description: Page load timeout in seconds retryNum: type: integer minimum: 0 maximum: 2 default: 1 description: Number of retry attempts userAgent: type: string description: Custom user agent emulateDevice: type: string enum: [iPhone X, iPhone 12, Pixel 5, iPad] description: Emulate a mobile device viewport: type: object properties: width: type: integer example: 1920 height: type: integer example: 1080 description: Custom viewport size blockAds: type: boolean default: true description: Block ad networks blockImages: type: boolean default: false description: Block image loading blockMedia: type: boolean default: false description: Block CSS, fonts, media waitForSelector: type: string description: CSS selector to wait for before returning example: ".main-content" postWaitTime: type: integer minimum: 1 maximum: 12 description: Additional seconds to wait after page load catchAjaxHeadersUrlMask: type: string description: URL pattern to intercept XHR requests example: "/api/" textNotExpected: type: array description: Retry if page contains these strings items: type: string dumpIframe: type: string description: Extract iframe content (use "all" for all iframes) screenshot: type: boolean default: true description: Take a screenshot extractorPreset: type: string enum: [markdown, markdown_content, content] description: Content extraction preset extractor: type: string description: Custom extractor code steps: type: array description: Browser automation steps to execute before capturing content items: type: object required: - type properties: type: type: string enum: [click, fill, type, select, wait, scroll, hover, press, navigate, keydown, keyup] description: Step action type selector: type: string description: CSS selector for element actions value: type: string description: Value for fill/type/select/press actions url: type: string description: URL for navigate action waitForNavigation: type: boolean description: Wait for page navigation after action timeout: type: integer description: Timeout for this step in milliseconds example: - type: fill selector: "#username" value: "testuser" - type: fill selector: "#password" value: "testpass" - type: click selector: "#login-btn" waitForNavigation: true ScrapeResponse: type: object properties: body: type: string description: Response body (HTML or text) info: type: object properties: statusCode: type: integer description: HTTP status code example: 200 latency: type: integer description: Request latency in milliseconds example: 523 isBinary: type: boolean description: Whether response is binary billingPoints: type: integer description: Points consumed example: 1 headers: type: object description: Response headers extractor: type: object nullable: true properties: result: type: object properties: markdown: type: string html: type: string meta: type: object err: type: string nullable: true stderr: type: string nullable: true description: Error message if any ScrapeJsResponse: type: object properties: body: type: string description: Rendered page HTML info: type: object properties: js: type: boolean description: Always true for /scrapeJs attemptsNum: type: integer description: Number of attempts made statusCode: type: integer description: HTTP status code screenshot: type: string nullable: true description: Screenshot URL finalUrl: type: string description: Final URL after redirects headers: type: array description: Response headers items: type: string pageCookies: type: string description: Cookies as string catchedAjax: type: object nullable: true description: Intercepted XHR data billingPoints: type: integer example: 5 bodyIframe: type: array nullable: true items: type: object properties: url: type: string content: type: string description: Iframe contents if requested extractor: type: object nullable: true description: Extraction results if requested ErrorResponse: type: object properties: status: type: string example: "fail" message: type: string example: "Security check failed: Blocked hostname" errors: type: array items: type: object ValidationError: type: object properties: detail: type: array items: type: object properties: loc: type: array items: type: string msg: type: string type: type: string