ScrapeNinja

FREEMIUM
Par Anthony | Mise à jour 4 days ago | Data
Popularité

9.9 / 10

Latence

2,900ms

Niveau de service

98%

Health Check

N/A

Retour à toutes les discussions

List of "types" in JS Steps

Rapid account: Listebbins 5
listebbins5
a year ago

I am trying to scrape some pages that are very similar to the sandbox https://apiroad.net/ajax-test.html example and get the API responses.
In the first, the main difference is that there is no search criteria you need to enter. Upon HTML page load, the backend API is called and gets some json data. So I don’t need this stuff:
{
“type”: “change”,
“selector”: “#q”,
“value”: “toyota”
}
because there is nothing to type. So I deleted it and that leaves this part:
{
“type”: “click”,
“selector”: “#search-button”,
“waitForXhrRequest”: “the_url_called_from_my_initial_page.php”
}
But I don’t need a click event. I probably want some other “type” then the waitForXhrRequest. But I cannot find anything in the tutorials or documentation that lists what else can be used.

In the other page I am scraping, I need to press “enter” after typing my search criteria. There is no “search” button to click. So I need to figure out a “type” for that too. Like “type”: “keyboard_press”, “value”: “enter” or something similar.

Rapid account: Restyler
restyler Commented a year ago

You are right! I have just added an example in /scrape-js which uses catchAjaxHeadersUrlMask param. Thanks!

Rapid account: Listebbins 5
listebbins5 Commented a year ago

Ok, thanks. Yes I made some progress with catchAjaxHeadersUrlMask and now think I can do what I need. FYI, it would have been helpful to see an example using catchAjaxHeadersUrlMask or in the documentation. I kept experimenting until it worked because I didn’t know what type of value it needed. I originally passed it the entire url like “catchAjaxHeadersUrlMask”: “https://domain.com/xyz.php?param1=xyz&param2=abc” and it didn’t work. Then I thought I may need wildcards in the url but looks like that isn’t the case either.

Rapid account: Restyler
restyler Commented a year ago

When you are finished debugging, I recommend you to use these params to cut loading time:

blockImages: true,
screenshot: 0,
blockMedia: true

These will prevent images and css from loading, and disable screenshot feature which also takes time and resources.

Rapid account: Restyler
restyler Commented a year ago

Hi! Sorry for not getting back to you yesterday. Looking into logs I see you are now using catchAjaxHeadersUrlMask param of /scrape-js which is a right thing to do in your case, as far as I understood it, as this one sets up ajax handler when no interaction is needed with the page. Let me know if you are still struggling.

Rapid account: Listebbins 5
listebbins5 Commented a year ago

Does someone have an idea on this? I am at a dead end until I can figure out how to scrape using something besides a click event.

Participez à la discussion - ajoutez un commentaire ci-dessous:

Connectez-vous / Inscrivez-vous pour publier de nouveaux commentaires