Need to see if your shared folder is taking up space on your dropbox 👨‍💻? Find out how to check here.

Forum Discussion

ncw's avatar
ncw
Collaborator | Level 8
6 years ago
Solved

Rate limiting when uploading files with rclone

I've received complaints from rclone users that file uploads are progressing really slowly.   Digging into it what I see is this:   2020-09-03 11:44:33 DEBUG : too_many_requests/: Too many re...
API
  • Greg-DB's avatar
    Greg-DB
    6 years ago

    Thanks for the detailed feedback! I'm sharing this with the team.

ncw's avatar
ncw
Collaborator | Level 8
6 years ago

> The only way to avoid this then is to just call at a slower rate. It sounds like you already tried that, but if it didn't help you'll need to limit it further.

 

I tried limiting it to 10 per second.

 

I'll do some experiments to see what I think the right number is.

 

Or even better do you have a figure you could tell me?

Greg-DB's avatar
Greg-DB
Icon for Dropbox Community Moderator rankDropbox Community Moderator
6 years ago

Unfortunately I don't have any specific rate limit numbers to share.

  • ncw's avatar
    ncw
    Collaborator | Level 8
    6 years ago

    A bit of experimenting makes us think that if we can at or below 12 transactions per second then we don't get rate limited.

     

    I know you can't confirm or deny that Greg-DB but maybe you can give us the secret wink or something like that ;-)

  • Greg-DB's avatar
    Greg-DB
    Icon for Dropbox Community Moderator rankDropbox Community Moderator
    6 years ago

    I'm glad to hear you got this working well. I can't share any guidance on the rate limiting parameters though, as we try not to encourage apps to rely on hard-coded limits, as the system can change over time. So, make sure your app is still able to always check and use the returned Retry-After value.

  • ncw's avatar
    ncw
    Collaborator | Level 8
    6 years ago

    Sure thing, we obey Retry-After.

     

    I'll just note that Google Drive specifies the queries per second per user publically which is 1000 queries per 100 seconds and it is very helpful for app developers to know. You see that number in your app console and you can set it to fewer queries per second if you want. Rclone could deliver API hits at 1000/s so having some published guidance is useful here! Maybe you could pass that on to the dev team?

     

    The only Retry-After values I've seen are 15s and 300s, both of which are too long to use as an effective rate limiter - users get annoyed that rclone has crashed and stop it if it waits for 15 seconds or 300 seconds. I'd much rather have a 5s Retry-After more often.

     

    Thanks for all your help

     

     
  • Greg-DB's avatar
    Greg-DB
    Icon for Dropbox Community Moderator rankDropbox Community Moderator
    6 years ago

    Thanks for the detailed feedback! I'm sharing this with the team.

  • Trevor K Smith's avatar
    Trevor K Smith
    New member | Level 2
    5 years ago

    Very glad to see dropbox collaborating with rclone! Both are great tools/services. I really appreciate the colloration here!

About Dropbox API Support & Feedback

Node avatar for Dropbox API Support & Feedback
Find help with the Dropbox API from other developers.

The Dropbox Community team is active from Monday to Friday. We try to respond to you as soon as we can, usually within 2 hours.

If you need more help you can view your support options (expected response time for an email or ticket is 24 hours), or contact us on X, Facebook or Instagram.

For more info on available support options for your Dropbox plan, see this article.

If you found the answer to your question in this Community thread, please 'like' the post to say thanks and to let us know it was useful!