Error 400 using download link from embedded code

terryz · ‎05-18-2023

I have an embedded piece of code that once upon a time worked before TLS v1 was deprecated. Note the URL download link in the subject line above was modified to protect the file. The unmodified link works fine with Chrome and Edge as the client. I have been trying to modify the code in my embedded application to use TLS v1.2 to make it work again. Below is the code snippet that I am trying to modify. I am guessing my header {} settings are incorrect. Below the code is the html response I get back from Dropbox - basically error 400. Do I need to create my own agent to do this or did I not put in the correct header {} information? I am using the typical Edge and Chrome header for 'Agent'...

Code:

local https = require("ssl.https")

local ltn12 = require("ltn12")

local options = {}

local fileMode = isbinary and "wb" or "w"

f = io.open(fulllocalpath, fileMode)

if not f then

resultstr = "Failed to open local file for writing:" .. fulllocalpath .."\r"

return false, nil, nil, resultstr

end

options = {

url = rawname,

sink = ltn12.sink.file(f),

protocol = "tlsv1_2",

headers = { --IS THERE SOMETHING WRONG HERE???

["Host"] = "www.dropbox.com",

["User-Agent"] = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.190 Safari/537.36",

["Accept"] = "*/*",

["Referer"] = "https://www.dropbox.com/",

["Accept-Encoding"] = "gzip, deflate, br",

["Connection"] = "keep-alive"

}

Here is the html response back from Dropbox:

<.DOCTYPE html> <html> <head><meta http-equiv="Content-Type" content="text/html; charset=utf-8"> <meta name="viewport" content="width=device-width, initial-scale=1" /> <title>Dropbox - 400</title> <link href="https://cfl.dropboxstatic.com/static/metaserver/static/css/error.css" rel="stylesheet" type="text/css"/> <link rel="shortcut icon" href="https://cfl.dropboxstatic.com/static/images/favicon.ico"/> </head> <body> <div class="figure"> <img src="https://assets.dropbox.com/www/en-us/illustrations/spot/target-miss.svg" alt="Error: 400"/> </div> <div id="errorbox"> <h1>Error (400)</h1>Something went wrong. Don't worry, your files are still safe and the Dropbox team has been notified. Check out our <a href="https://status.dropbox.com">Status Page</a> to see if there is a known incident, our <a href="https://www.dropbox.com/help">Help Center</a> and <a href="https://forums.dropbox.com">forums</a> for help, or head back to <a href="https://www.dropbox.com/home">home</a>. </div> </body> </html>

DB-Des · ‎05-19-2023

Hi there!

Since you are receiving an HTML response, we can definitely rule out issues with TLS.

I have been unable to replicate the issue you have described. Is it possible you could enable a more verbose output in your network client? It would be helpful if we could get a better view into the raw request that the client is making.

terryz · ‎05-19-2023

I tired adding code to get the https request, without luck. Instead, A gathered a WireShark trace, which you can access here: https://www.dropbox.com/s/3v10bjapy7evnm6/iplogging.pcap?dl=0

Please review and let me know your thoughts.

The trace was captured using the following header (same as original, but with the addition of the 'GET' method.

-- Set the options for the HTTPS request

local options = {

url = rawname,

protocol = "tlsv1_2",

sink = ltn12.sink.file(f),

method = "GET", -- Add the method field to specify the HTTP method

headers = {

["Host"] = "www.dropbox.com",

["User-Agent"] = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36",

["Accept"] = "*/*",

["Referer"] = "https://www.dropbox.com/",

["Accept-Encoding"] = "gzip, deflate, br",

["Connection"] = "keep-alive"

}

Greg-DB · ‎05-19-2023

@terryz Thanks for sharing that, though unfortunately I don't think that we'd be able to determine the issue from that; a 400 like this should generally indicate that there was something incorrect about the HTTP request, but the request data would be encrypted in that capture so we couldn't read it anyway.

Perhaps instead we can try to approach this another way. Can you reproduce the issue with another client where you can inspect the actual requests that get sent? For example, I just tried to replicate the request you're sending using curl with a test shared link of my own, but it worked successfully:

curl -v --tlsv1.2 --http1.1 -X GET -L "https://www.dropbox.com/s/9kkiqvfe457hrs0/test.csv?dl=1" -o out.csv \
	-H "Host: www.dropbox.com" \
	-H "User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36" \
	-H "Accept: */*" \
	-H "Referer: https://www.dropbox.com/" \
	-H "Accept-Encoding: gzip, deflate, br" \
	-H "Connection: keep-alive"

(I'm guessing you're using HTTP 1.1, but that --http1.1 option can be changed to --http2 to match if you are using HTTP 2.)

Also, you could try your code with my test shared link to see if your code works with that. If it does, it would indicate an issue with the particular shared link you're using and not the code itself.

By the way, I see you're mimicking a web browser by sending those "User-Agent" and "Referer" headers; that may not necessarily cause issues, but I can't guarantee it. Further, given that those aren't necessary, it may be good to remove them.

terryz · ‎05-19-2023

I did add method = "GET" in my last test, does that need to be there?

Greg-DB · ‎05-19-2023

@terryz The HTTP GET method is correct for this scenario. It's often the default for HTTP clients, so even when you don't specify it, clients will usually use GET automatically.

terryz · ‎05-22-2023

HI Greg:

Can you send me an exact example of what the HTTP request should look like?

I may be able to approach the problem in a different way.

Thanks!

Greg-DB · ‎05-22-2023

@terryz You can run the curl example I shared in my earlier comment to see how the requests are formatted. (There are some redirects involved, but curl will handle that automatically since "-L" is specified.)

terryz · ‎05-22-2023

I ran he following example in curl, note it fails using the original dropbox link ?I sent to FAULTMSG.CSV - why:

C:\Users\tzarnowski>curl -v --tlsv1.2 --http1.1 -X GET -L "https://www.dropbox.com/s/y5jlwu5nfr3s8xe/FAULTMSG.CSV?dl=1" -o out.csv -H "Host: www.dropbox.com" -H "User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36" -H "Accept: */*" -H "Referer: https://www.dropbox.com/" -H "Accept-Encoding: gzip, deflate, br" -H "Connection: keep-alive"
Note: Unnecessary use of -X or --request, GET is already inferred.
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0* Trying 162.125.4.18:443...
* Connected to www.dropbox.com (162.125.4.18) port 443 (#0)
* schannel: disabled automatic use of client certificate
* ALPN: offers http/1.1
* ALPN: server accepted http/1.1
* using HTTP/1.1
> GET /s/y5jlwu5nfr3s8xe/FAULTMSG.CSV?dl=1 HTTP/1.1
> Host: www.dropbox.com
> User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36
> Accept: */*
> Referer: https://www.dropbox.com/
> Accept-Encoding: gzip, deflate, br
> Connection: keep-alive
>
< HTTP/1.1 302 Found
< Content-Security-Policy: style-src https://* 'unsafe-inline' 'unsafe-eval' ; form-action 'self' https://www.dropbox.com/ https://dl-web.dropbox.com/ https://photos.dropbox.com/ https://paper.dropbox.com/ https://showcase.dropbox.com/ https://www.hellofax.com/ https://app.hellofax.com/ https://www.hellosign.com/ https://app.hellosign.com/ https://docsend.com/ https://www.docsend.com/ https://help.dropbox.com/ https://navi.dropbox.jp/ https://a.sprig.com/ https://selfguidedlearning.dropboxbusiness.com/ https://instructorledlearning.dropboxbusiness.com/ https://sales.dropboxbusiness.com/ https://accounts.google.com/ https://api.login.yahoo.com/ https://login.yahoo.com/ https://experience.dropbox.com/ https://pal-test.adyen.com https://2e83413d8036243b-Dropbox-pal-live.adyenpayments.com/ ; report-uri https://www.dropbox.com/csp_log?policy_name=metaserver-whitelist ; default-src https://www.dropbox.com/playlist/ https://www.dropbox.com/v/s/playlist/ https://*.dropboxusercontent.com/p/hls_master_playlist/ https://*.dropboxusercontent.com/p/hls_playlist/ ; connect-src https://* ws://127.0.0.1:*/ws ; worker-src https://www.dropbox.com/static/serviceworker/ blob: ; script-src 'unsafe-eval' https://www.dropbox.com/static/api/ https://www.dropbox.com/page_success/ https://cfl.dropboxstatic.com/static/ https://www.dropboxstatic.com/static/ https://accounts.google.com/gsi/client https://canny.io/sdk.js 'nonce-LxuDkSxDrHQIhSAbz8B7DzKtWeY=' ; base-uri 'self' ; child-src https://www.dropbox.com/static/serviceworker/ blob: ; object-src 'self' https://cfl.dropboxstatic.com/static/ https://www.dropboxstatic.com/static/ ; frame-ancestors 'self' ; frame-src https://* carousel: dbapi-6: dbapi-7: dbapi-8: dropbox-client: itms-apps: itms-appss: ; img-src https://* data: blob: ; media-src https://* blob: ; font-src https://* data:
< Content-Security-Policy: report-uri https://www.dropbox.com/csp_log?policy_name=metaserver-dynamic ; script-src 'unsafe-eval' 'strict-dynamic' 'nonce-LxuDkSxDrHQIhSAbz8B7DzKtWeY=' 'nonce-zBujYfSPIVul0nCMhmjaCTh2Jt8='
< Content-Type: text/html; charset=utf-8
< Location: /s/dl/y5jlwu5nfr3s8xe/FAULTMSG.CSV
< Pragma: no-cache
< Referrer-Policy: strict-origin-when-cross-origin
< Set-Cookie: gvc=MjU5MjkxMzYzOTcwMzYwNjQ3NzU2MTA4Mzc2NDQ5MzQ0NDQwMTgx; Path=/; Expires=Sat, 20 May 2028 13:02:20 GMT; HttpOnly; Secure; SameSite=None
< Set-Cookie: t=DAyXLOMSKcjbZX3ChslTIs4P; Path=/; Domain=dropbox.com; Expires=Thu, 21 May 2026 13:02:20 GMT; HttpOnly; Secure; SameSite=None
< Set-Cookie: __Host-js_csrf=DAyXLOMSKcjbZX3ChslTIs4P; Path=/; Expires=Thu, 21 May 2026 13:02:20 GMT; Secure; SameSite=None
< Set-Cookie: __Host-ss=j8axqReYhY; Path=/; Expires=Thu, 21 May 2026 13:02:20 GMT; HttpOnly; Secure; SameSite=Strict
< Set-Cookie: locale=en; Path=/; Domain=dropbox.com; Expires=Sat, 20 May 2028 13:02:20 GMT
< X-Content-Type-Options: nosniff
< X-Frame-Options: SAMEORIGIN
< X-Permitted-Cross-Domain-Policies: none
< X-Robots-Tag: noindex, nofollow, noimageindex
< X-Xss-Protection: 1; mode=block
< Date: Mon, 22 May 2023 13:02:20 GMT
< Strict-Transport-Security: max-age=31536000; includeSubDomains
< Server: envoy
< Cache-Control: no-cache, no-store
< Content-Encoding: gzip
< Vary: Accept-Encoding
< X-Dropbox-Response-Origin: far_remote
< X-Dropbox-Request-Id: 94922619ce9e4dd790b3857b3ece7272
< Transfer-Encoding: chunked
<
* Ignoring the response-body
{ [48 bytes data]
100 37 0 37 0 0 128 0 --:--:-- --:--:-- --:--:-- 128
* Connection #0 to host www.dropbox.com left intact
* Issue another request to this URL: 'https://www.dropbox.com/s/dl/y5jlwu5nfr3s8xe/FAULTMSG.CSV'
* Found bundle for host: 0x1e652216ac0 [serially]
* Re-using existing connection #0 with host www.dropbox.com
> GET /s/dl/y5jlwu5nfr3s8xe/FAULTMSG.CSV HTTP/1.1
> Host: www.dropbox.com
> User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36
> Accept: */*
> Referer: https://www.dropbox.com/
> Accept-Encoding: gzip, deflate, br
> Connection: keep-alive
>
< HTTP/1.1 302 Found
< Content-Security-Policy: sandbox
< Location: https://ucf9fab352ae7acf48b9a3bdcf92.dl.dropboxusercontent.com/cd/0/get/B8h0yjnJ6oYV-TIlmsyGYsN8Xr66...
< Referrer-Policy: strict-origin-when-cross-origin
< Set-Cookie: gvc=NTkyNzA4NDA3MDk1OTQ2OTY0NjIxNDQxNjEyODQyMDM1NDYyMjk%3D; expires=Sat, 20 May 2028 13:02:20 GMT; HttpOnly; Path=/; SameSite=None; Secure
< Set-Cookie: t=32mN4AJJyZphNSOFUWMKOpOS; Domain=dropbox.com; expires=Thu, 21 May 2026 13:02:20 GMT; HttpOnly; Path=/; SameSite=None; Secure
< Set-Cookie: __Host-js_csrf=32mN4AJJyZphNSOFUWMKOpOS; expires=Thu, 21 May 2026 13:02:20 GMT; Path=/; SameSite=None; Secure
< Set-Cookie: __Host-ss=pBbPy8XPV4; expires=Thu, 21 May 2026 13:02:20 GMT; HttpOnly; Path=/; SameSite=Strict; Secure
< Set-Cookie: locale=en; Domain=dropbox.com; expires=Sat, 20 May 2028 13:02:20 GMT; Path=/; SameSite=None; Secure
< X-Content-Type-Options: nosniff
< X-Frame-Options: DENY
< X-Permitted-Cross-Domain-Policies: none
< X-Xss-Protection: 1; mode=block
< Content-Type: text/html; charset=utf-8
< Accept-Encoding: identity,gzip
< Date: Mon, 22 May 2023 13:02:20 GMT
< Server: envoy
< Strict-Transport-Security: max-age=31536000; includeSubDomains
< Strict-Transport-Security: max-age=31536000; includeSubDomains
< Cache-Control: no-cache, no-store
< Content-Encoding: gzip
< Vary: Accept-Encoding
< X-Dropbox-Response-Origin: far_remote
< X-Dropbox-Request-Id: 8497494043f64814996dae65b7a5ce95
< Transfer-Encoding: chunked
<
* Ignoring the response-body
{ [350 bytes data]
100 333 0 333 0 0 553 0 --:--:-- --:--:-- --:--:-- 553
* Connection #0 to host www.dropbox.com left intact
* Issue another request to this URL: 'https://ucf9fab352ae7acf48b9a3bdcf92.dl.dropboxusercontent.com/cd/0/get/B8h0yjnJ6oYV-TIlmsyGYsN8Xr66...'
* Trying 162.125.4.15:443...
* Connected to ucf9fab352ae7acf48b9a3bdcf92.dl.dropboxusercontent.com (162.125.4.15) port 443 (#1)
* schannel: disabled automatic use of client certificate
* ALPN: offers http/1.1
* ALPN: server accepted http/1.1
* using HTTP/1.1
> GET /cd/0/get/B8h0yjnJ6oYV-TIlmsyGYsN8Xr66RaWCguHFlKjzPZ8fc7h7wQmUOZumls40XqPLIulsyHieE2J5iQIMiUf6Rt1-JtaZ2NrbpQAwUg_79GIsikZE5nRKWcVO4xCs2-iKD54Dae1Xyph_7vttjFVIaa2pYUcerpbEJ4HDMA3cW3x5IWZOj1ynpxoboeJGFOogISA/file?dl=1 HTTP/1.1
> Host: ucf9fab352ae7acf48b9a3bdcf92.dl.dropboxusercontent.com
> User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36
> Accept: */*
> Referer: https://www.dropbox.com/
> Accept-Encoding: gzip, deflate, br
> Connection: keep-alive
>
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0* schannel: failed to decrypt data, need more data
< HTTP/1.1 200 OK
< Accept-Ranges: bytes
< Cache-Control: max-age=60
< Content-Disposition: attachment; filename="FAULTMSG.CSV"; filename*=UTF-8''FAULTMSG.CSV
< Content-Security-Policy: sandbox
< Etag: 1681308133112497d
< Pragma: public
< Referrer-Policy: no-referrer
< Vary: Origin
< X-Content-Security-Policy: sandbox
< X-Content-Type-Options: nosniff
< X-Robots-Tag: noindex, nofollow, noimageindex
< X-Server-Response-Time: 192
< X-Webkit-Csp: sandbox
< Content-Type: application/binary
< Accept-Encoding: identity,gzip
< Date: Mon, 22 May 2023 13:02:20 GMT
< Server: envoy
< Strict-Transport-Security: max-age=31536000; includeSubDomains; preload
< Content-Length: 6848
< X-Dropbox-Response-Origin: far_remote
< X-Dropbox-Request-Id: 1dfba269edcd4785a07aff50ce3ea9ba
<
{ [6848 bytes data]
100 6848 100 6848 0 0 6830 0 0:00:01 0:00:01 --:--:-- 44467
* Connection #1 to host ucf9fab352ae7acf48b9a3bdcf92.dl.dropboxusercontent.com left intact

Greg-DB · ‎05-22-2023

@terryz I see you are getting the two expected 302 responses, and then the final successful 200 response. When you say it fails, are you referring to the "schannel: failed to decrypt data, need more data" message? That doesn't occur for me, and I do get the expected 6848 bytes of data saved out when I try. Does the file get downloaded properly for you with this command?

In any case, it isn't resulting in a 400, so it appears this doesn't replicate the issue with your original code; it seems like the client in your original code is doing something differently that may be corrupting the request there.

Error 400 using download link from embedded code

Error 400 using download link from embedded code

Top contributors to this post