Dropbox CLI Error: ng

[VastDatalytics]

Hi,

I am uploading a 101 GiB file to my dropbox and got an Error: ng 99GiB/101GiB.

I used the command ./dbxcli put [src]/myfile [dest]/myfile

Is uploading a 100gb with the CLI not supported? I assume it is using sessions in the background.

[tahsini]

Is there a more detailed error response that you're receiving from the cli when uploading? And is this occurring only for files 100gb or larger?

[VastDatalytics]

That is all the cli gave me. I've seen it once at 99GiB/101GiB. It takes me 12 hours to get to that point. I tried the second time but it threw an error "too many requests" around 4GiB/101GiB. 

I assume the too many requests error is because the dropbox cli chunks are defaulted to 4mb? So it is hitting the API too many times?

[kylea]

We've recently updated our upload sessions to enable parallel upload of chunks.  The team just updated the cli to incorporate this functionality after your report.  Parallel uploads of chunks should substantially improve upload time and reduce error for these larger files.

Would you be able to patch your cli with this and let it know if it resolves?  https://github.com/dropbox/dbxcli/pull/170.  We expect to update the main release with it.

[VastDatalytics]

Thanks for looking into this!

I ended up writing my own uploader with retry + chunksize setting. I was able to successfully upload with 16mb chunks.

What is the largest chunk possible on a 1000mbps up bandwidth for concurrently uploads?

[tahsini]

The largest chunk-size possible with concurrent uploads is 148MB (the largest multiple of 4MB below 150MB) but we recommend 16MB or 32MB and more parallel workers to optimize the speed.

If you do use our latest patch to upload your content, we would love to know how long it now takes you to complete your upload! It is always helpful for us to know how these upload cases perform with our users.

[VastDatalytics]

Got it. Thank you!