downloading a large file using python v2 API

[Olaf B.2]

Dear Dropboxers,

would it be possible to see an example for large file download, equivalent to https://www.dropboxforum.com/hc/en-us/community/posts/205544836-python-upload-big-file-example for the upload?

Thanks.

[Greg-DB]

Hi Olaf, for downloading, you generally just need a simple call to the files_download method. There's a sample here:

https://stackoverflow.com/documentation/dropbox-api/408/downloading-a-file/1350/downloading-a-file-u...

Are you running into issues downloading large files?

[Olaf B.2]

Hi Gregory,

thanks. I am working with very large files, up to several tens of GB, and it would be nice to be able to parcel them in chunks, as we do for the upload. The motivation is the same: to be able to monitor progress and retry in the case of failure. I cannot find the the right tools for that in the API. Are they not provided, or am I looking in the wrong place?

[Greg-DB]

I see, thanks for the additional context! The API itself does actually support Range Retrieval Requests which can be used to download files in pieces, but this functionality unfortunately isn't currently exposed in the API v2 Python SDK. We'll consider this a feature request for that though.

[Olaf B.2]

Is it available in some other programing language?

[Greg-DB · Accepted solution]

It is already implemented in the Java SDK. 

(It is also implemented in the API v1 Python client, but I can't recommend using that as it's deprecated.)

If you wanted to implement it manually, or modify the Python SDK, here's a sample of what it would look like in curl for reference:

curl -X POST https://content.dropboxapi.com/2/files/download \
 --header "Authorization: Bearer ACCESS_TOKEN" \
 --header "Dropbox-API-Arg: {\"path\": \"/test.txt\"}" \
 --header "Range:bytes=0-2"

That would download just the first 3 bytes of the file at /test.txt.

[Olaf B.2]

OK, thank you, this helps.

[MattFahrner]

I have the same problem - want to download a very large file in a streaming format in the v2 API and used to use v1 "get_file()" functionality. The idea is to use it for a "tar" restore from backup, and pulling it all into memory ala "files_download()" would get ugly fast.

Any hope that this will make it back into the exposed Python API?

Thanks!

[Greg-DB]

Hi Matt, it sounds like your request may actually be slightly different than what was being discussed on this thread. We were talking about downloading files in distinct chunks (similar to the chunked upload), but it sounds like you want to be able to stream the download as desired, like you currently do with the get_file method.

I believe the files_download method does already work the same way as that though. It returns a requests.models.Response object on which you can call iter_content to iterate over the content, streaming it off the connection. 

That would look something like:

metadata, res = dbx.files_download(path)
for data in res.iter_content(10):
 print(data)

Hope this helps! 

[MattFahrner]

Excellent! I will give that a try - thanks!