List of "types" in JS Steps
I am trying to scrape some pages that are very similar to the sandbox https://apiroad.net/ajax-test.html example and get the API responses.
In the first, the main difference is that there is no search criteria you need to enter. Upon HTML page load, the backend API is called and gets some json data. So I don’t need this stuff:
{
“type”: “change”,
“selector”: “#q”,
“value”: “toyota”
}
because there is nothing to type. So I deleted it and that leaves this part:
{
“type”: “click”,
“selector”: “#search-button”,
“waitForXhrRequest”: “the_url_called_from_my_initial_page.php”
}
But I don’t need a click event. I probably want some other “type” then the waitForXhrRequest. But I cannot find anything in the tutorials or documentation that lists what else can be used.
In the other page I am scraping, I need to press “enter” after typing my search criteria. There is no “search” button to click. So I need to figure out a “type” for that too. Like “type”: “keyboard_press”, “value”: “enter” or something similar.
Junte-se à discussão - adicione o comentário abaixo:
You are right! I have just added an example in /scrape-js which uses catchAjaxHeadersUrlMask param. Thanks!
Ok, thanks. Yes I made some progress with catchAjaxHeadersUrlMask and now think I can do what I need. FYI, it would have been helpful to see an example using catchAjaxHeadersUrlMask or in the documentation. I kept experimenting until it worked because I didn’t know what type of value it needed. I originally passed it the entire url like “catchAjaxHeadersUrlMask”: “https://domain.com/xyz.php?param1=xyz¶m2=abc” and it didn’t work. Then I thought I may need wildcards in the url but looks like that isn’t the case either.
When you are finished debugging, I recommend you to use these params to cut loading time:
blockImages: true,
screenshot: 0,
blockMedia: true
These will prevent images and css from loading, and disable screenshot feature which also takes time and resources.
Hi! Sorry for not getting back to you yesterday. Looking into logs I see you are now using catchAjaxHeadersUrlMask param of /scrape-js which is a right thing to do in your case, as far as I understood it, as this one sets up ajax handler when no interaction is needed with the page. Let me know if you are still struggling.
Does someone have an idea on this? I am at a dead end until I can figure out how to scrape using something besides a click event.