The same issue as occurs https://rapidapi.com/apidojo/api/idealista2/discussions/31192
I’d like to know what can be done to allow consuming district by district without getting the weird pagination.
The list that i’m following is:
[‘0-EU-PT-18’,‘0-EU-PT-17’, ‘0-EU-PT-16’, and so on…]
The base URL ('https://idealista2.p.rapidapi.com/properties/list")
Join in the discussion - add comment below:
@apidojo What about the other data? That does not make any sense not having an accurate feedback on why it is getting random on a few pages, 56 is not that big of a number. Also we’ve bough the highest license to actually consume the most data but if this ain’t the speed just let us know
To keep it simple, just apply numPage between [1-50], it will work normally for you.
@apidojo The way we setup was reading from the autocomplete a variety of Id’s and matching names, we’ve also observed that the locationName has no impact or what so ever, only locationId.
Again in order to replicate the screenshow demonstrates you Lisbon as an example, you could try to fetch another district with a high volume of data.
Wrapping up, the behaviour that we’ve noticed is that the pagination is not acting correct once there is a high volume of data. If as you’ve described, the scrapper starts throwing off random pages/elements, i believe its apidojo responsability to provide either a fallback or not allowing 800 pages as i’ve seen where only the first 50 come correct
If this doesn’t help i can send a full document over email with more details but i think that the above screenshot already shows a location as a use case when it starts becoming problematic
I am tracing back your requests. How did you get the values of “locationId”, and “locationName” to pass in …/properties/list. Please provide some samples step by step so I have some thing to forward to the IT team for review. You may send it to our email at firstname.lastname@example.org.
@apidojo Thanks for the quick response. We have tried a low amount of data, a small zone as you can see in the screenshot and i’ve noticed that even on 20 pages the randomless has appeared, it isn’t about high volume but the way the pagination is built is not stable enough. Could you research a little bit what’s going on in your end and get back to me on something more concrete?
P.S: We are consuming the API through RapidAPI, not doing any “scrapping work” therefore your answer could only apply if the API that you provide uses a scrapping mechanism but for the “api consumer” it shouldn’t matter, should only work
Hello, You may read this :
All of our APIs reproduce public data and features of the official site/application. They are intended to work with application having actual user interfaces. No users would go through that large range of pages in real world, they should apply filter instead. It seems like you are trying to scrape the API for data, don’t you? I think that is how Idealista deals with scrapers to protect their data. Our API may help you to get the data easier but we don’t have full control or access to the server side of Idealista to change the behavior for you. You may try again with smaller area, and loop through smaller area.
Although is there any API-wide recommendation to be able to consume the district? The API takes around 45min to resolve all the data with duplicates which isn’t ideal so narrowing further is not an option.
Here’s a screenshot of pratical example, going even further to a zone…
Yes, i’m able to open the page but i’m not sure what that has to do with the fact that i’m retrieving through the API a district, for example Porto which code is: 0-EU-PT-13 (it does have around 600 pages) and when i start consuming 1 by 1, after >= 50 it will get my random pages.
I believe i need a clean plan to execute for consuming the data without getting duplicated pages, a sequential consumption.
May you confirm that you can open the page on idealista.com by sharing with me some links or screenshot as reference?