Need to see if your shared folder is taking up space on your dropbox 👨💻? Find out how to check here.
Forum Discussion
Solved
Rate limiting when uploading files with rclone
I've received complaints from rclone users that file uploads are progressing really slowly.
Digging into it what I see is this:
2020-09-03 11:44:33 DEBUG : too_many_requests/: Too many re...
Thanks for the detailed feedback! I'm sharing this with the team.
My Test user is uploading 10TB of images. So lots of 2-3MB files.
Using batching the upload is proceeding very quickly (at about 35MB/s) which is great. However every 30 minutes or so they get this error
too_many_requests/.: Too many requests or write operations.
Along with a 300 second Retry-After which rclone obeys.
Rclone is now using batching of 1000 files at a time. It only sends one batch at a time and waits for it to complete. I think that is all the ingredients for successful batching.
If I upload lots of very small (10 byte) files, I can provoke this message after uploading about 5000 of them.
Any ideas on how I can avoid this 300 second lockout? I tried pacing the uploads but it didn't seem to help.
Thanks
@ncw How do you implement the batching? I am getting this error on a regular basis and its quite annoying.
About Dropbox API Support & Feedback
Find help with the Dropbox API from other developers.
