Crawler Extractor API Documentation

Welcome to the e-crawler API. This document will guide you on how to use the API endpoints.

Base URL

The base URL for the API is https://crawler-extractor.p.rapidapi.com.

Endpoints

1. Get Status

• Endpoint: /
• Method: GET
• Description: This endpoint returns the status of the API.
• Response: A JSON object containing status, date, environment, and AWS details.

Example:

curl -X GET https://crawler-extractor.p.rapidapi.com/

2. Extract Data

• Endpoint: /extract
• Method: POST
• Description: This endpoint is used to extract data from a given URL using the provided selectors.
• Request Body: A JSON object containing the URL and selectors.

Example:

curl -X POST https://crawler-extractor.p.rapidapi.com/extract \
-H 'Content-Type: application/json' \
-d '{
  "url": "http://example.com",
  "selectors": [
    {
      "key": "exampleKey",
      "selector": "exampleSelector",
      "type": "text"
    }
  ]
}'

The selectors array contains objects with the following properties:

• key: The name you want to give to the extracted data.
• selector: The CSS selector to target the data on the webpage.
• type: The type of data to extract. It can be ‘text’, ‘attr’, or ‘html’.

Note: If the type is ‘attr’, the attribute to be extracted is ‘href’. If the type is not provided, it defaults to ‘text’.

Error Handling

In case of an error, the API will return a status code of 500 along with a JSON object containing an error message.

Example:

{
  "error": "Failed to extract data"
}

Please ensure that your requests are correctly formatted to avoid errors.