cancel
Showing results for 
Show  only  | Search instead for 
Did you mean: 
Announcements
Back again with another edition of 'How we use Dropbox', so find out how Emma uses to-do lists to get it all done here!

Dropbox API Support & Feedback

Find help with the Dropbox API from other developers.

cancel
Showing results for 
Show  only  | Search instead for 
Did you mean: 

Concurrent uploads completely broken since V2 api.

Concurrent uploads completely broken since V2 api.

Stefan D.6
New member | Level 1

It appears that uploading multiple different files to the same dropbox at the same time, is completely broken in the V2 api. Even if 2 different computers are uploading a file. Please see the following testcase:

 

 

#!/bin/env python2
import tempfile
 
import dropbox
 
FILE = ""
ACCESSTOKEN = ""
 
client = dropbox.Dropbox(ACCESSTOKEN)
 
for i in range(10000):
    with tempfile.NamedTemporaryFile() as fp:
        metadata, restdata = client.files_download(FILE)
        fp.write(restdata.content)
        fp.write("{}\n".format(i))
        fp.seek(0)
        client.files_upload(fp, FILE, dropbox.files.WriteMode('overwrite', None))
 
 
 
 
Steps to reproduce:
fill in your accesstoken.
fill in in the variables FILE, to a file that exists ('/foo/test.txt'). Start the program.
Change the variable FILE to another file that exists ('/foo/test2.txt'). Start another instance of the program
 
 
What should happen is that both test.txt and test2.txt are continuously appended with an new integer followed by a newline. But in practice, the files occasionally end up completely empty after an update cycle. The download of the file does not fail. client.files_upload simply creates an completely empty file.
 
This behavior is not reproduceable when running only one instance of this testcase. Running the testcase on one computer or on different computers/ip's with different/same accesstokens/accounts does not make a difference in my testing as long as the same dropbox (or dropbox folder?) is accessed.
 
Time to reproduce: about 10 updates. Might depend on your internet connection and proximity to dropbox' servers. (not tested)
 

Tested with dropbox 6.8.0 on windows 10 and dropbox 6.5.0 on debian. Only tested with python 2.7

6 Replies 6

Re: Concurrent uploads completely broken since V2 api.

Greg-DB
Dropboxer

Thanks for the report Stefan! I just tried running two of these though and I'm not seeing the same behavior, using the same version of Python and the Dropbox library.

How are you checking the file size to see the unexpected empty file? Can you add some logging and share the output showing the issue? In particular, it might be good to check the size in three places: in the metadata returned by files_download, of fp itself, and in the metadata returned by files_upload. That way we can see exactly when/where the empty file originates.

Thanks in advance! 

Re: Concurrent uploads completely broken since V2 api.

Stefan D.6
New member | Level 1

files_download always returns an error code 200 and the correct file size. Calling tell() on the file pointer after writing also always returns the correct file size. The metadata returned by files_upload (occasionally) claims to have resulted in a zero bytes upload.

Based on the behavior i see in my app, i am quite confident that is is the upload that fails.

I also ran a few experiments with the files in different folders / in shared folders, but that doesn't seem to make a difference.

Is there any internel dropbox lib logging i can enable?

Re: Concurrent uploads completely broken since V2 api.

Greg-DB
Dropboxer

You can add the following to enable some debugging output that would show the HTTP requests:

import httplib
httplib.HTTPConnection.debuglevel = 1

Re: Concurrent uploads completely broken since V2 api.

Stefan D.6
New member | Level 1

Good idea. I modified the inner loop to this:

 

for i in range(10000):
    with tempfile.NamedTemporaryFile() as fp:
        metadata, restdata = client.files_download(FILE)
        fp.write(restdata.content)
        fp.write("{}\r".format(i))

        fp.seek(0)

        print '=================================='
        httplib.HTTPConnection.debuglevel = 1
        md2 = client.files_upload(fp, FILE, dropbox.files.WriteMode('overwrite', None))
        httplib.HTTPConnection.debuglevel = 0
        assert md2.size

 

 

 

Output:

 

 

==================================
send: 'POST /2/files/upload HTTP/1.1\r\nHost: content.dropboxapi.com\r\nConnection: keep-alive\r\nAccept-Encoding: gzip, deflate\r\nAccept: */*\r\nUser-Agent: OfficialDropboxPythonSDKv2/6.8.0\r\nContent-Type: application/octet-stream\r\nDropbox-API-Arg: {"path": "/foo/test.txt", "mode": {".tag": "overwrite"}, "autorename": false, "mute": false}\r\nAuthorization: Bearer CENSORED\r\nContent-Length: 38\r\n\r\n'
send: <open file '<fdopen>', mode 'w+b' at 0x0000000003A57390>
sendIng a read()able
reply: 'HTTP/1.1 200 OK\r\n'
header: Server: nginx
header: Date: Tue, 20 Sep 2016 17:28:13 GMT
header: Content-Type: application/json
header: Transfer-Encoding: chunked
header: Connection: keep-alive
header: pragma: no-cache
header: cache-control: no-cache
header: X-Server-Response-Time: 1122
header: X-Dropbox-Request-Id: 754c6605f013cd1dcad17ad97551c672
header: X-Robots-Tag: noindex, nofollow, noimageindex
header: Content-Encoding: gzip
==================================
send: 'POST /2/files/upload HTTP/1.1\r\nHost: content.dropboxapi.com\r\nConnection: keep-alive\r\nAccept-Encoding: gzip, deflate\r\nAccept: */*\r\nUser-Agent: OfficialDropboxPythonSDKv2/6.8.0\r\nContent-Type: application/octet-stream\r\nDropbox-API-Arg: {"path": "/foo/test.txt", "mode": {".tag": "overwrite"}, "autorename": false, "mute": false}\r\nAuthorization: Bearer CENSORED\r\nContent-Length: 41\r\n\r\n'
send: <open file '<fdopen>', mode 'w+b' at 0x0000000003771E40>
sendIng a read()able
reply: 'HTTP/1.1 200 OK\r\n'
header: Server: nginx
header: Date: Tue, 20 Sep 2016 17:28:14 GMT
header: Content-Type: application/json
header: Transfer-Encoding: chunked
header: Connection: keep-alive
header: pragma: no-cache
header: cache-control: no-cache
header: X-Server-Response-Time: 837
header: X-Dropbox-Request-Id: d716609d9fdfd3cb17001967b6f82009
header: X-Robots-Tag: noindex, nofollow, noimageindex
header: Content-Encoding: gzip
==================================
send: 'POST /2/files/upload HTTP/1.1\r\nHost: content.dropboxapi.com\r\nConnection: keep-alive\r\nAccept-Encoding: gzip, deflate\r\nAccept: */*\r\nUser-Agent: OfficialDropboxPythonSDKv2/6.8.0\r\nContent-Type: application/octet-stream\r\nDropbox-API-Arg: {"path": "/foo/test.txt", "mode": {".tag": "overwrite"}, "autorename": false, "mute": false}\r\nAuthorization: Bearer CENSORED\r\nContent-Length: 44\r\n\r\n'
send: <open file '<fdopen>', mode 'w+b' at 0x0000000003A57390>
sendIng a read()able
reply: 'HTTP/1.1 429 too_many_write_operations/..\r\n'
header: Server: nginx
header: Date: Tue, 20 Sep 2016 17:28:16 GMT
header: Content-Type: application/json
header: Transfer-Encoding: chunked
header: Connection: keep-alive
header: Retry-After: 1
header: X-Dropbox-Request-Id: 99633340ed675344e6a7d07bad3ddb4b
header: X-Robots-Tag: noindex, nofollow, noimageindex
send: 'POST /2/files/upload HTTP/1.1\r\nHost: content.dropboxapi.com\r\nConnection: keep-alive\r\nAccept-Encoding: gzip, deflate\r\nAccept: */*\r\nUser-Agent: OfficialDropboxPythonSDKv2/6.8.0\r\nContent-Type: application/octet-stream\r\nDropbox-API-Arg: {"path": "/foo/test.txt", "mode": {".tag": "overwrite"}, "autorename": false, "mute": false}\r\nAuthorization: Bearer CENSORED\r\nTransfer-Encoding: chunked\r\n\r\n'
send: '0\r\n\r\n'
reply: 'HTTP/1.1 200 OK\r\n'
header: Server: nginx
header: Date: Tue, 20 Sep 2016 17:28:18 GMT
header: Content-Type: application/json
header: Transfer-Encoding: chunked
header: Connection: keep-alive
header: pragma: no-cache
header: cache-control: no-cache
header: X-Server-Response-Time: 598
header: X-Dropbox-Request-Id: 71ff38bee72b24a9afd7172830e85671
header: X-Robots-Tag: noindex, nofollow, noimageindex
header: Content-Encoding: gzip
Traceback (most recent call last):
  File "C:/Users/Stefan/git/dropbox_test.py", line 28, in <module>
    assert md2.size
AssertionError

 

Seems that the upload fails for some reason (reply: 'HTTP/1.1 429 too_many_write_operations/..\r\n') and when it retries, it sends an "0\r\n\r\n" instead of "<open file '<fdopen>', mode 'w+b' at 0x0000000003A57390>" ?

Re: Concurrent uploads completely broken since V2 api.

Greg-DB
Dropboxer

Thanks! That's very helpful. We're looking into it.

Re: Concurrent uploads completely broken since V2 api.

Greg-DB
Dropboxer
Poll
We love to learn from the educators who use Dropbox. Whether you teach kids, teens, adults or a combination of all three, we want to know what apps and integrations you use with Dropbox to help with teaching. Which of the ones below is your favorite, or most used tool?
Who's talking

Top contributors to this post

  • User avatar
    Greg-DB Dropboxer
  • User avatar
    Stefan D.6 New member | Level 1
What do Dropbox user levels mean?
Need more support?