Take Your Search Game to the Next Level with Dropbox Dash 🚀✨ Curious how it works? Ask us here!
Forum Discussion
Solved
Rate limiting when uploading files with rclone
I've received complaints from rclone users that file uploads are progressing really slowly.
Digging into it what I see is this:
2020-09-03 11:44:33 DEBUG : too_many_requests/: Too many re...
Thanks for the detailed feedback! I'm sharing this with the team.
Greg-DB
Dropbox Staff
Dropbox StaffThat rate limit with the longer Retry-After should just be a result of making a very large number of API calls for a particular user in a short period of time (as opposed to lock contention). If you're sending thousands of requests very quickly like that, you can run in to it. (It sounds like you would have normally run in to lock contention before you hit this limit, but that's no longer an issue with the batch commit so you can run much faster.) The only way to avoid this then is to just call at a slower rate. It sounds like you already tried that, but if it didn't help you'll need to limit it further.
> The only way to avoid this then is to just call at a slower rate. It sounds like you already tried that, but if it didn't help you'll need to limit it further.
I tried limiting it to 10 per second.
I'll do some experiments to see what I think the right number is.
Or even better do you have a figure you could tell me?
- Greg-DB
Unfortunately I don't have any specific rate limit numbers to share.
- Greg-DB
I'm glad to hear you got this working well. I can't share any guidance on the rate limiting parameters though, as we try not to encourage apps to rely on hard-coded limits, as the system can change over time. So, make sure your app is still able to always check and use the returned Retry-After value.
Sure thing, we obey Retry-After.
I'll just note that Google Drive specifies the queries per second per user publically which is 1000 queries per 100 seconds and it is very helpful for app developers to know. You see that number in your app console and you can set it to fewer queries per second if you want. Rclone could deliver API hits at 1000/s so having some published guidance is useful here! Maybe you could pass that on to the dev team?
The only Retry-After values I've seen are 15s and 300s, both of which are too long to use as an effective rate limiter - users get annoyed that rclone has crashed and stop it if it waits for 15 seconds or 300 seconds. I'd much rather have a 5s Retry-After more often.
Thanks for all your help
- Greg-DB
Thanks for the detailed feedback! I'm sharing this with the team.
Very glad to see dropbox collaborating with rclone! Both are great tools/services. I really appreciate the colloration here!
About Dropbox API Support & Feedback
Find help with the Dropbox API from other developers.
