Durch Anthony | Aktualisiert 2 months ago | Data

9.9 / 10



Service Level


Zurück zu allen Diskussionen

List of "types" in JS Steps

Rapid account: Listebbins 5
5 months ago

I am trying to scrape some pages that are very similar to the sandbox example and get the API responses.
In the first, the main difference is that there is no search criteria you need to enter. Upon HTML page load, the backend API is called and gets some json data. So I don’t need this stuff:
“type”: “change”,
“selector”: “#q”,
“value”: “toyota”
because there is nothing to type. So I deleted it and that leaves this part:
“type”: “click”,
“selector”: “#search-button”,
“waitForXhrRequest”: “the_url_called_from_my_initial_page.php”
But I don’t need a click event. I probably want some other “type” then the waitForXhrRequest. But I cannot find anything in the tutorials or documentation that lists what else can be used.

In the other page I am scraping, I need to press “enter” after typing my search criteria. There is no “search” button to click. So I need to figure out a “type” for that too. Like “type”: “keyboard_press”, “value”: “enter” or something similar.

Rapid account: Restyler
restyler Commented 5 months ago

You are right! I have just added an example in /scrape-js which uses catchAjaxHeadersUrlMask param. Thanks!

Rapid account: Listebbins 5
listebbins5 Commented 5 months ago

Ok, thanks. Yes I made some progress with catchAjaxHeadersUrlMask and now think I can do what I need. FYI, it would have been helpful to see an example using catchAjaxHeadersUrlMask or in the documentation. I kept experimenting until it worked because I didn’t know what type of value it needed. I originally passed it the entire url like “catchAjaxHeadersUrlMask”: “” and it didn’t work. Then I thought I may need wildcards in the url but looks like that isn’t the case either.

Rapid account: Restyler
restyler Commented 5 months ago

When you are finished debugging, I recommend you to use these params to cut loading time:

blockImages: true,
screenshot: 0,
blockMedia: true

These will prevent images and css from loading, and disable screenshot feature which also takes time and resources.

Rapid account: Restyler
restyler Commented 5 months ago

Hi! Sorry for not getting back to you yesterday. Looking into logs I see you are now using catchAjaxHeadersUrlMask param of /scrape-js which is a right thing to do in your case, as far as I understood it, as this one sets up ajax handler when no interaction is needed with the page. Let me know if you are still struggling.

Rapid account: Listebbins 5
listebbins5 Commented 5 months ago

Does someone have an idea on this? I am at a dead end until I can figure out how to scrape using something besides a click event.

Nehmen Sie an der Diskussion teil - fügen Sie unten einen Kommentar hinzu:

Anmelden / Registrieren, um neue Kommentare zu veröffentlichen