We’re Still Here to Help (Even Over the Holidays!) - find out more here.
Forum Discussion
SamiSan
9 years agoExplorer | Level 3
delays between chunks using cURL
I'm trying to built a server script using bash scripting and cURL for upload/download/archive operations through the HTTP API. However, after(!) every append (v2) there is a varying delay betwee...
SamiSan
9 years agoExplorer | Level 3
Just for clarification once again: delay happens after the actual transmission of the data and waiting for the HTTP response (HTTP/1.1 200 OK etc.). I measure using 'time' from GNU coreutils alongside other linux specific tools and/or on our hardware firewall. Additionaly I use a paket filter to determine when the "HTTP/1.1 200 OK" request come back from Dropbox, and when it does, cURL exits. So the delay comes from waiting for this response.
Both, during simple file uploads and chunk uploads - the bigger the file/chunk, the longer the delay it seems. In the meantime I uploaded a file using python for testing and the delay seems to be present too.
I also checked in with a friend to check it out and he can confirm this (he uses the JAVA API), but he didn't pay attention before. Maybe other people experience the same but no one notices the delay, because they don't monitor traffic during the execution of their scripts and thus don't correlate the runtime of a script with the actual data transmission time? At least this is my guess.
Here are a few metrics for uploading into the same file into Dropbox but with different chunksizes (I've throttled the upload to constant ~50Mbit/s for testing purposes):
File Size Chunksize Total Transmission Time
100MB 5MB 2:03 Minutes
100MB 20MB 55 Seconds
100MB 40MB 51 Seconds
But since the maximum possible chunk size is 150MB including a considerable delay between the chunks, the total time of delays during big file uploads is a HUGE hit. Here is another example including other providers offering HTTP API and using cURL and a fixed upload speed of ~50Mbit/s :
To Filesize Chunksize Total Transmission Time
Dropbox 1000M 150M 9:29 Minutes
Cloud Provider "X" 1000M 5M 3:55 Minutes
Cloud Provider "Y" 1000M 222M 3:49 Minutes
So while the transmission speed is the same during upload to different providers, the delays between chunks uploading to drop box almost adds to a double amount of time needed to finish the whole file.
Needless to say, this HUGE time overhead is not acceptable. The only thing I can explain this with is dropbox calculating a hash for the file server side and then returning a HTTP OK, which also explains the longer delays for bigger chunk sizes.
Greg-DB
Dropbox Community Moderator
9 years agoThanks! That's helpful. We'll look into it.
- SamiSan9 years agoExplorer | Level 3
Any progress on this matter?
- Greg-DB9 years ago
Dropbox Community Moderator
Thanks for checking in. There are actually two things happening here:
1) We had a recent performance regression that slowed down processing on these calls. We're working on that.
2) Separate from that regression, there is some delay expected with these calls, while the servers process the uploaded data. That is expected, though not ideal. We are looking into ways to avoid this, but I can't make any promises or offer a timeline right now.- SamiSan9 years agoExplorer | Level 3Thank you for the update!
I'd be glad if you would keep us posted on this matter whenever these issues get resolved.
About Dropbox API Support & Feedback
Find help with the Dropbox API from other developers.
The Dropbox Community team is active from Monday to Friday. We try to respond to you as soon as we can, usually within 2 hours.
If you need more help you can view your support options (expected response time for an email or ticket is 24 hours), or contact us on X, Facebook or Instagram.
For more info on available support options for your Dropbox plan, see this article.
If you found the answer to your question in this Community thread, please 'like' the post to say thanks and to let us know it was useful!