upload API only works for very small files

[ags65]

Hello,

I try this command from my raspberry pi 2 with raspbian buster:

curl -v -X POST https://content.dropboxapi.com/2/files/upload --header "Authorization: Bearer xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx" --header "Dropbox-API-Arg: {\"path\": \"/README.md\",\"mode\": \"add\",\"autorename\": true,\"mute\": false,\"strict_conflict\": false}" --header "Content-Type: application/octet-stream" --data-binary @/home/pi/Dropbox-Uploader/README.md

and I get the error 52 Empty reply from server:

> POST /2/files/upload HTTP/1.1
> Host: content.dropboxapi.com
> User-Agent: curl/7.64.0
> Accept: */*
> Authorization: Bearer xxxxxxxxx
> Dropbox-API-Arg: {"path": "/README.md","mode": "add","autorename": true,"mute": false,"strict_conflict": false}
> Content-Type: application/octet-stream
> Content-Length: 8900
> Expect: 100-continue
>
* Expire in 1000 ms for 0 (transfer 0x1679880)
< HTTP/1.1 100 Continue
* We are completely uploaded and fine
* Empty reply from server
* Connection #0 to host content.dropboxapi.com left intact
curl: (52) Empty reply from server

but the same command with a very small file works:

> POST /2/files/upload HTTP/1.1
> Host: content.dropboxapi.com
> User-Agent: curl/7.64.0
> Accept: */*
> Authorization: Bearer xxxxxxxxxxxx
> Dropbox-API-Arg: {"path": "/borrar.txt","mode": "add","autorename": true,"mute": false,"strict_conflict": false}
> Content-Type: application/octet-stream
> Content-Length: 5
>
* upload completely sent off: 5 out of 5 bytes
< HTTP/1.1 200 OK
< Server: nginx
< Date: Sun, 01 Mar 2020 18:54:34 GMT
< Content-Type: application/json
< Transfer-Encoding: chunked
< Connection: keep-alive
< Vary: Accept-Encoding
< cache-control: no-cache
< pragma: no-cache
< X-Server-Response-Time: 823
< X-Dropbox-Request-Id: 6a837c20651b6d6f2f6498f45e297b3f
< X-Robots-Tag: noindex, nofollow, noimageindex
<

Thanks for any clue you can give me to help me.

[Здравко]

Hi @ags65,

That's a old curl feature currently almost deprecated. The idea is when relatively big file have to be passed, credentials to be check in advance, so prevents get notification for invalid credentials after passing everything. Generally something useful. But seems your curl version doesn't behave as expected. Actual files data have to be passed after the confirmation, but seems passed before that. Probably your network connection is very slow and that's why response delay is more than 1 second (the default threshold). You can try set this threshold to something appropriate in your case or try disable this check by explicitly set "Expect" header with empty value, for example.

Hope this gives the right direction.

[ags65]

Hi,

Thank you very much for your help, but the problem persists:

my network connection is fast, the raspberry pi is connected by ethernet cable to the router and my internet connection works with optic fibre cable.

cURL version is the last in raspbian: 7.64.0

the cURL command with option -H 'Expect:' does not send the "Expect: 100-continue" to dropbox but directly send the file. But the error is the same:

> POST /2/files/upload HTTP/1.1
> Host: content.dropboxapi.com
> User-Agent: curl/7.64.0
> Accept: */*
> Authorization: Bearer xxxxxxx
> Dropbox-API-Arg: {"path": "/README.md","mode": "add","autorename": true,"mute": false,"strict_conflict": false}
> Content-Type: application/octet-stream
> Content-Length: 8900
>
* upload completely sent off: 8900 out of 8900 bytes
* TLSv1.2 (IN), TLS alert, close notify (256):
* Empty reply from server
* Connection #0 to host content.dropboxapi.com left intact
curl: (52) Empty reply from server

If there is a problem in my system, why it does not affect with small files to upload ?

From dropbox, is there a way to see the log of connection attempts?

Thank you for your help

[Здравко]

Wow... That's strange.

Make your own investigation, what's going on. Try crosscompile working desktop version and see if the behaviour changes. If I'm in your shoes, this would be the first check. Try if same file transfer gonna completes on desktop version.

[Greg-DB]

We're not currently experiencing any known service disruption that would cause empty replies from the Dropbox API. Might there be something on your network connection preventing your app from connecting to the Dropbox servers? For example, might there be some firewall, proxy, or caching layer that is either blocking or failing on larger requests?

[ags65]

Hi,

I sent the very same command from Ubuntu to upload a file of similar size and it works perfectly.

The problem have to be in the raspberry pi system or cURL command version.

I will report any progress I make on this subject.

Thank you very much.

[ags65]

Hi,

Following with the subject, the problem arises when the file size is greater than the mtu (1500 bytes) and cURL send the more than one data packet.

I'm not un expert but what I see in Wireshark is that dropbox answer back the first packet with the ACK but inform parameters SLE and SRE with packets that I dont know what they are.

After that the raspberry try several times the retransmission of the first packet and finally dropbox send the reset that ends the connection

I dont know if someone used to deal with network problems can explain what's happening...

[Здравко]

That what I can see is you had send 4 bytes less than Dropbox expects and the confirmations given are for packets that the server doesn't signal for... Bug in the software!

[ags65]

I've compiled in the raspberry pi the same version of cURL that works in my ubuntu (7.65.3)  and the problem remains:

* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-CHACHA20-POLY1305
* ALPN, server accepted to use http/1.1
* Server certificate:
* subject: C=US; ST=California; L=San Francisco; O=Dropbox, Inc; CN=content.dropboxapi.com
* start date: Jan 29 00:00:00 2020 GMT
* expire date: Mar 2 12:00:00 2022 GMT
* subjectAltName: host "content.dropboxapi.com" matched cert's "content.dropboxapi.com"
* issuer: C=US; O=DigiCert Inc; OU=www.digicert.com; CN=DigiCert SHA2 High Assurance Server CA
* SSL certificate verify ok.
> POST /2/files/upload HTTP/1.1
> Host: content.dropboxapi.com
> User-Agent: curl/7.65.3
> Accept: */*
> Authorization: Bearer xxxxxxxxxxx
> Dropbox-API-Arg: {"path": "/Cromosoma.cc","mode": "add","autorename": true,"mute": false,"strict_conflict": false}
> Content-Type: application/octet-stream
> Content-Length: 2863
> Expect: 100-continue
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 100 Continue
* We are completely uploaded and fine
* Empty reply from server
* Connection #0 to host content.dropboxapi.com left intact
curl: (52) Empty reply from server

[Здравко]

Hmm... You can try another diagnose way. Check if the transaction completes successfully using something like:

curl -v -X POST https://httpbin.org/post -H "Authorization: Bearer xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx" -H "Dropbox-API-Arg: {\"path\": \"/Cromosoma.cc\",\"mode\": \"add\",\"autorename\": true,\"mute\": false,\"strict_conflict\": false}" -H "Content-Type: application/octet-stream" -H "Expect:" --data-binary @/home/pi/Dropbox-Uploader/Cromosoma.cc

The response would show what actually got send. Evaluate it.

Let's hope result will drive you to the right conclusion.