REST Call Throttling/Rate Limiting

[Kevin B.36]

With APIV1 you could make a single REST call and get all of a user's shared folder data and all of the members of each of those shared folders. Now with the new and Improved APIV2 you need to make a separate REST call for every shared folder to get that shared folder's members. So if a user has 100 shared folders the overhead went from One (1) REST call to One Hundred and One (101) REST calls - that's right, two orders of magnitude more REST calls. That is a ridiculously large overhead.

Anyway I have put threaded the calls and that saves some time. But now I'm bumping into this little problem (I think), from the Dropbox APIV2 documentation:

https://www.dropbox.com/developers/reference/data-ingress-guide

Rate Limits

Rate limit errors occur when your app is trying to make more requests in a specific window of time than is allowed. There are many technical and non-technical reasons why these rate limits exist but they are set high enough such that legitimate uses of the API should rarely encounter them.

Dropbox does not/will not tell me when these limits kick in. My application checks shared folder membership every minute (remember it used to be 1 REST call a minute, NOW it's 82 REST calls a minute in this instance). So with 81 shared folders the following is consistently observed:

1. 79 Folders get members: between .6 and 1.2 seconds (averaging around .7 seconds)
2. 80th folder around 5.8 seconds
3. 81st folder around 122 seconds

Clearly this is ridiculous, the throttling of the REST response is, for one folder,  twice the interval I have available to me to get ALL folders' members' data. I am hoping there is some other reason why this is happening that I can fix, but:

Can someone confirm that this is within the bounds of reality that Rest response throttling may be occurring in this situation?

Certainly the empirical evidence suggests so.

[Greg-DB · Accepted solution]

Thanks for the detailed post and feedback. API v2 does separate out some things like this, where API v1 would return everything in one call. This was done partly because the API v1 calls didn't scale well, and sometimes failed when there were too many items to return in a single call.

Also, the Dropbox API does have a rate limiting system, but it doesn't operate by slowing down the HTTPS response itself. If an API call gets rate limited, it will just return a specific error response with a 429 error code, per:

https://www.dropbox.com/developers/documentation/http/documentation#error-handling
https://www.dropbox.com/developers/reference/data-ingress-guide

How long it actually takes to retrieve the requested information will depend on the specific shared folder and members though. In your sample, it sounds like the 80th folder, and especially the 81st folder, may have large member lists that take longer to retrieve than the others.