cancel
Showing results for 
Show  only  | Search instead for 
Did you mean: 
Announcements
Want to learn some quick and useful tips to make your day easier? Check out how Calvin uses Replay to get feedback from other teams at Dropbox here.

Dropbox API Support & Feedback

Find help with the Dropbox API from other developers.

cancel
Showing results for 
Show  only  | Search instead for 
Did you mean: 

Re: Rate limiting when uploading files with rclone

Rate limiting when uploading files with rclone

ncw
Collaborator | Level 8
Go to solution

I've received complaints from rclone users that file uploads are progressing really slowly.

 

Digging into it what I see is this:

 

2020-09-03 11:44:33 DEBUG : too_many_requests/: Too many requests or write operations. Trying again in 15 seconds.

What happened there is rclone received the `upload failed: too_many_requests` error and backed off for 15 seconds according to the Retry-After header.

 

Dropbox didn't used to do this with rclone so I suspect we've hit some sort of rate limiting.

 

Note that rclone has 35,000 users at Dropbox generating 60M API calls per day!

 

I'm not sure this is the right channel for this kind of support - maybe rclone should be a tech partner?

 

Any help much appreciated - thanks!

 

Nick Craig-Wood

Rclone lead developer

[email removed as per Community Guidelines]

15 Replies 15

ncw
Collaborator | Level 8
Go to solution

A bit of experimenting makes us think that if we can at or below 12 transactions per second then we don't get rate limited.

 

I know you can't confirm or deny that @Greg-DB but maybe you can give us the secret wink or something like that 😉

Greg-DB
Dropbox Staff
Go to solution

I'm glad to hear you got this working well. I can't share any guidance on the rate limiting parameters though, as we try not to encourage apps to rely on hard-coded limits, as the system can change over time. So, make sure your app is still able to always check and use the returned Retry-After value.

ncw
Collaborator | Level 8
Go to solution

Sure thing, we obey Retry-After.

 

I'll just note that Google Drive specifies the queries per second per user publically which is 1000 queries per 100 seconds and it is very helpful for app developers to know. You see that number in your app console and you can set it to fewer queries per second if you want. Rclone could deliver API hits at 1000/s so having some published guidance is useful here! Maybe you could pass that on to the dev team?

 

The only Retry-After values I've seen are 15s and 300s, both of which are too long to use as an effective rate limiter - users get annoyed that rclone has crashed and stop it if it waits for 15 seconds or 300 seconds. I'd much rather have a 5s Retry-After more often.

 

Thanks for all your help

 

 

Greg-DB
Dropbox Staff
Go to solution

Thanks for the detailed feedback! I'm sharing this with the team.

Trevor K Smith
New member | Level 2
Go to solution

Very glad to see dropbox collaborating with rclone! Both are great tools/services. I really appreciate the colloration here!

capturemomentsphotography209
New member | Level 2
Go to solution

@ncw How do you implement the batching? I am getting this error on a regular basis and its quite annoying.

Need more support?