Announcements
Known issues updated. Learn more

socket.timeout issue even with session upload

Level 3

Hello I have quite a large amount of files on my machine and I have given up using the dropbox app as it just will not sync.

I am using the dropbox python sdk but I am having socket.timeout issues.

I am using upload_session for the large files but this affects both large (files_upload_session) and small (files_upload).

I have tried the following:

  1. change the chunk sizes in the session upload from 8mb to 90mb. I am still getting upload_session.
  2. I tried adding time.sleep() to delay api calls but I am still getting the same problem.

 

Here is the gist on github.

https://gist.github.com/bramburn/224743c9e58f375c3790af8472c6d8b9

I have 200mbps up and down on a business line.

 

Can someone please shed some light as to how I can fix this?

1 Accepted solution

Accepted Solutions

Re: socket.timeout issue even with session upload

Dropboxer

@bramburn Please try replacing this line:

dbx = dropbox.Dropbox(ACCESS_TOKEN)

with this instead:

dbx = dropbox.Dropbox(ACCESS_TOKEN, timeout=None)

Then try again and let me know if that does or doesn't help.

Also, you don't need those time.sleep calls, so you can remove those.

Reply
Loading...
10 Replies

Re: socket.timeout issue even with session upload

Dropboxer

Generally speaking, nearly everything below 150 mb should work fine with the files/upload endpoint. 

Could you please post a copy of the timeout error you're getting in both scenarios? 

This could indicate some server or network issues. Is there anything that could be interfering with your connection to Dropbox such as a firewall or proxy?

Reply
Loading...

Re: socket.timeout issue even with session upload

Level 3

Here is the error. I've tried it on several machine and even on my Mac at different locations (coffee shop, client's office and the home office) i still get the socket issue

Traceback (most recent call last):
  File "C:\cygwin64\home\drop\envy2\lib\site-packages\urllib3\connectionpool.py", line 387, in _make_request
    six.raise_from(e, None)
  File "<string>", line 2, in raise_from
  File "C:\cygwin64\home\drop\envy2\lib\site-packages\urllib3\connectionpool.py", line 383, in _make_request
    httplib_response = conn.getresponse()
  File "C:\Python32\lib\http\client.py", line 1321, in getresponse
    response.begin()
  File "C:\Python32\lib\http\client.py", line 296, in begin
    version, status, reason = self._read_status()
  File "C:\Python32\lib\http\client.py", line 257, in _read_status
    line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
  File "C:\Python32\lib\socket.py", line 589, in readinto
    return self._sock.recv_into(b)
  File "C:\Python32\lib\ssl.py", line 1052, in recv_into
    return self.read(nbytes, buffer)
  File "C:\Python32\lib\ssl.py", line 911, in read
    return self._sslobj.read(len, buffer)
socket.timeout: The read operation timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\cygwin64\home\drop\envy2\lib\site-packages\requests\adapters.py", line 449, in send
    timeout=timeout
  File "C:\cygwin64\home\drop\envy2\lib\site-packages\urllib3\connectionpool.py", line 641, in urlopen
    _stacktrace=sys.exc_info()[2])
  File "C:\cygwin64\home\drop\envy2\lib\site-packages\urllib3\util\retry.py", line 368, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "C:\cygwin64\home\drop\envy2\lib\site-packages\urllib3\packages\six.py", line 686, in reraise
    raise value
  File "C:\cygwin64\home\drop\envy2\lib\site-packages\urllib3\connectionpool.py", line 603, in urlopen
    chunked=chunked)
  File "C:\cygwin64\home\drop\envy2\lib\site-packages\urllib3\connectionpool.py", line 389, in _make_request
    self._raise_timeout(err=e, url=url, timeout_value=read_timeout)
  File "C:\cygwin64\home\drop\envy2\lib\site-packages\urllib3\connectionpool.py", line 307, in _raise_timeout
    raise ReadTimeoutError(self, url, "Read timed out. (read timeout=%s)" % timeout_value)
urllib3.exceptions.ReadTimeoutError: HTTPSConnectionPool(host='content.dropboxapi.com', port=443): Read timed out. (read timeout=30)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "upload.py", line 150, in <module>
    ProcessFolder()
  File "upload.py", line 42, in ProcessFolder
    uploadLarge(local_path, dropbox_path)
  File "upload.py", line 77, in uploadLarge
    for r in readTheLines(f, fSize, destination):
  File "upload.py", line 114, in readTheLines
    dbx.files_upload_session_append_v2(chunk, CURSOR, False)
  File "C:\cygwin64\home\drop\envy2\lib\site-packages\dropbox\base.py", line 2495, in files_upload_session_append_v2
    f,
  File "C:\cygwin64\home\drop\envy2\lib\site-packages\dropbox\dropbox.py", line 274, in request
    timeout=timeout)
  File "C:\cygwin64\home\drop\envy2\lib\site-packages\dropbox\dropbox.py", line 365, in request_json_string_with_retry
    timeout=timeout)
  File "C:\cygwin64\home\drop\envy2\lib\site-packages\dropbox\dropbox.py", line 449, in request_json_string
    timeout=timeout,
  File "C:\cygwin64\home\drop\envy2\lib\site-packages\requests\sessions.py", line 581, in post
    return self.request('POST', url, data=data, json=json, **kwargs)
  File "C:\cygwin64\home\drop\envy2\lib\site-packages\requests\sessions.py", line 533, in request
    resp = self.send(prep, **send_kwargs)
  File "C:\cygwin64\home\drop\envy2\lib\site-packages\requests\sessions.py", line 646, in send
    r = adapter.send(request, **kwargs)
  File "C:\cygwin64\home\drop\envy2\lib\site-packages\requests\adapters.py", line 529, in send
    raise ReadTimeout(e, request=request)
requests.exceptions.ReadTimeout: HTTPSConnectionPool(host='content.dropboxapi.com', port=443): Read timed out. (read timeout=30)
Reply
Loading...

Re: socket.timeout issue even with session upload

Level 3

i'll post the files_upload error when it happens next. but the above is the file upload session

Reply
Loading...

Re: socket.timeout issue even with session upload

Level 3

hi Taylor,

Some investigations I am getting the following tracert for dropbox's IP

 

I am getting really bad latency to 162.125.64.8 at around 600-800ms on the python call. Something weird is going on here. Is this an SDK or python issue?

https://www.dropbox.com/s/yacp7lt72xv78qg/issue.png?dl=0

 

The tracert and ping looks awesome.

tracert on original machine:

Tracing route to 162.125.64.8 over a maximum of 30 hops

  1    <1 ms    <1 ms    <1 ms  READYSHARE [192.168.1.1]
  2     1 ms    <1 ms    <1 ms  host-80-195-102-233.static.cable.virginmedia.com [80.195.102.233]
  3     1 ms     1 ms     1 ms  brnt-metnet-3a-pw-281.network.virginmedia.net [80.194.148.83]
  4     1 ms     1 ms     1 ms  brnt-core-2a-ae2-3006.network.virginmedia.net [86.28.68.97]
  5     *        *        *     Request timed out.
  6     2 ms     1 ms     1 ms  m686-mp2.cvx1-b.lis.dial.ntli.net [62.254.42.174]
  7     1 ms     1 ms     1 ms  185.45.9.10
  8     2 ms     2 ms     2 ms  185.45.10.155
  9     1 ms     1 ms     1 ms  162.125.64.8

Ping is:

Pinging 162.125.64.8 with 32 bytes of data:
Reply from 162.125.64.8: bytes=32 time=2ms TTL=54
Reply from 162.125.64.8: bytes=32 time=3ms TTL=54
Reply from 162.125.64.8: bytes=32 time=2ms TTL=54
Reply from 162.125.64.8: bytes=32 time=2ms TTL=54

Ping statistics for 162.125.64.8:
    Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
    Minimum = 2ms, Maximum = 3ms, Average = 2ms

 

tracert on another machine outside the business in my client's office

Tracing route to 162.125.64.8 over a maximum of 30 hops

  1     7 ms     3 ms     1 ms  172.30.200.2
  2     6 ms     2 ms    17 ms  10.200.219.85
  3     5 ms     5 ms     5 ms  10.200.217.9
  4     7 ms     8 ms     6 ms  10.200.217.10
  5     8 ms     7 ms    12 ms  10.10.209.4
  6     *        *        *     Request timed out.
  7     *        *        *     Request timed out.
  8     *        *        *     Request timed out.
  9     *        *        *     Request timed out.
 10     *        *        *     Request timed out.
 11     *        *        *     Request timed out.
 12     *        *        *     Request timed out.
 13     *        *        *     Request timed out.
 14     *        *        *     Request timed out.
 15     *        *        *     Request timed out.
 16    15 ms    18 ms    15 ms  162.125.64.8

Trace complete.

ping in the client's office

Pinging 162.125.64.8 with 32 bytes of data:
Reply from 162.125.64.8: bytes=32 time=64ms TTL=49
Reply from 162.125.64.8: bytes=32 time=25ms TTL=49
Reply from 162.125.64.8: bytes=32 time=27ms TTL=49
Reply from 162.125.64.8: bytes=32 time=23ms TTL=49

Ping statistics for 162.125.64.8:
    Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
    Minimum = 23ms, Maximum = 64ms, Average = 34ms

 

Reply
Loading...

Re: socket.timeout issue even with session upload

Dropboxer

Thanks for the additional information! To try to isolate whether this is a general networking issue, or if this is related to the Python environment/code in particular, can you try making an upload call outside of Python?

You can use the API v2 Explorer to make a call directly, or to get the curl code for running it bash. Let me know what the result is.

Reply
Loading...

Re: socket.timeout issue even with session upload

Level 3

Hi Greg,

Here is the result for a 150mb

 

	

{
  "name": "test.mp4",
  "path_lower": "/cloud/test.mp4",
  "path_display": "/Cloud/test.mp4",
  "id": "id:aEq7bJXKe2AAAAAAAEBAmg",
  "client_modified": "2019-07-10T15:15:11Z",
  "server_modified": "2019-07-10T15:15:11Z",
  "rev": "4dd3bf4bf85ef6",
  "size": 154077723,
  "is_downloadable": true,
  "content_hash": "30f29202b00488a7900fd77a22f03ed62243d86b252064a8f3069109892badb0"
}

 

I am using a different (non supported) package using bash and the latency is very low.

https://www.dropbox.com/s/qnfnhmf716ial32/Untitled.png?dl=0

 

 

Reply
Loading...

Re: socket.timeout issue even with session upload

Level 3

this is the package I am using

Reply
Loading...

Re: socket.timeout issue even with session upload

Dropboxer

Thanks for trying that. That output indicates the upload was succesful, so the issue does seem to be specific to your Python environment/code. We'll take a closer look at the code you shared.

Reply
Loading...

Re: socket.timeout issue even with session upload

Dropboxer

@bramburn Please try replacing this line:

dbx = dropbox.Dropbox(ACCESS_TOKEN)

with this instead:

dbx = dropbox.Dropbox(ACCESS_TOKEN, timeout=None)

Then try again and let me know if that does or doesn't help.

Also, you don't need those time.sleep calls, so you can remove those.

Reply
Loading...
API Support

Have a question? Our Dropbox Community is here to help you find an answer!
You won’t have to wait too long for a reply, but why not learn more about the Community while you’re here.

Not the right discussion for you? Find out what else our Community has to say about API Support.

Or, search our Help Center for even more.

Who's talking
Top contributors to this post