Stock, Ticker, Security and Company Search database

FREEMIUM
By API | Updated 4 months ago | Data
Popularity

8.6 / 10

Latency

398ms

Service Level

100%

Back to All Discussions

HTTP 500s

avatar
gobbagpete
il y a 2 ans

Hi. I’m calling your stock-ticker-security-and-company-search-database api on a list of stock names. Somewhere in the region of one in ten calls get a 500 error, with the message

The server encountered an internal error and was unable to complete your request. Either the server is overloaded or there is an error in the application

I am using the paid version, with 1M calls/month and 100 calls/s. I have tried throttling the application to slow down the rate of calls, to no effect. For the most part, the same query strings are affected each time, but with some variation from run to run. Typically, if I then call the api on an individual query string that failed, it will fail again. If I call it on a query that succeeded, it will succeed again.

Please help

avatar
logicione commented il y a 2 ans

Hi,
I am happy to report that:

  1. Some extra server capacity added, you should able to do more requests without getting 500 errors.
  2. Pushed a newer version of /all_search by Security Term, which will get you more results than before.

The trick to do the security search is to send the partial name of the company instead of sending “Apple Inc.” or “Apple Corp” send “Apple” only.

Please report if you see any more issues.

Thanks,
LogiciOne

avatar
gobbagpete commented il y a 2 ans

Hi - thanks for your response and apologies for my slow reply - I’ve been busy. I did try to to build in some retry logic but it seemed that once a stock had returned a 500 it would continue to do so. I’ll try some more options. At least I wasn’t doing anything obviously wrong 😉

avatar
logicione commented il y a 2 ans

Hi Pete,
Thanks for the details and sorry for the inconvienance.

By looking at the 500 errors, we can confirm that you didn’t send any special char in the request recently.
We re-ran the 500 errored requests, and it ran fine, so it seems there was a temporary load issue. We suggest having some re-try logic for such failed requests.

In the future, we will buy more servers and increase the through-put to handle the more concurrent requests. For now, we like to keep the cost low for our customers.

It might interest you to know that we are also adding more company names to our database, which is not necessarily listed on some stock exchanges currently.

Thanks,
LogiciOne

avatar
gobbagpete commented il y a 2 ans

I don’t know how I typed sleep(500) - I meant sleep(1) for the slower throttle - about one call per second-and-a-bit 😉

avatar
gobbagpete commented il y a 2 ans

Hi, thanks for getting back to me, and thanks for your diligent investigation. The reason for the url encoding was that I found, using the Python http.client package, I got a 500 for multi-word search terms unless I percent-encoded them. Today I switched to the Requests package, which I know does a lot more hand-holding under the covers and found that if I url-encoded my search term it was doubly encoding them and causing problems. It seems that Requests plus-encodes spaces by default, which seems to work ok. I think you’ll see that today (i.e. in the last few hours, I’m on UTC) I’ve not sent any percent-encoded strings (or not many - I may have gone back and crosschecked the http.client code)

I’ve found that, using the Requests package, calling sleep(0.1) still results in a 500 on about 1 in 10 calls. Slowing the throttle to sleep(500), though, does at least improve matters when using Requests, to about one 500 per 200 calls. Better, but far from ideal 😕

cheers,
Pete

avatar
logicione commented il y a 2 ans

Thanks for providing details on throttling.

Upon some more investigation, we found that the some of 500 errors happened because there were special characters in the Search Term.
Can you please make sure that your search term is alpha-numeric only
For example, the error is happening for a search term like:
“111%20inc”

If possible, before calling the API, clean the search term to “111 inc”

In the next release, we can handle such special characters gracefully.
Hope this helps

avatar
gobbagpete commented il y a 2 ans

I throttle it with a sleep(0.1) by default, i.e. the max rate would be 10 per second with zero latency (the UI is currently reporting 411 ms). I have tried upping that to a sleep(1), i.e. a max of 1 per sec, with no noticeable difference.

avatar
logicione commented il y a 2 ans

Hi,
Thanks for reporting this issue and providing the details.

We will investigate it and get back to you. Meanwhile, it seems the issue is happening due to exceeding requests per second. Can you please try to throttle the request further by adding some sleep time between your requests and see if the issue still exists?

Or, let’s know what is your request-per-second when you are seeing this issue. So that we can try to recreate it.

Thanks,
LogiciOne

Join in the discussion - add comment below:

Login / Signup to post new comments
Rating: 5 - Votes: 1