cancel
Showing results for 
Show  only  | Search instead for 
Did you mean: 
Announcements
Back again with another edition of 'How we use Dropbox', so find out how Emma uses to-do lists to get it all done here!

Dropbox API Support & Feedback

Find help with the Dropbox API from other developers.

cancel
Showing results for 
Show  only  | Search instead for 
Did you mean: 

Downloading zip files from Dropbox via API returns size zero

Downloading zip files from Dropbox via API returns size zero

lekn
Explorer | Level 3

Hi,

 

I was trying to download zip files within a folder from my dropbox.

For example, there are 7z files named 0.7z, 1.7z, ... 5.7z under folder name 'extra'.

I could download the 7z files via API, but the size of each 7z file turned out zero although they are around 11GB each.

Other text files under the same folder are downloaded correctly.

 

Could you let me know how I can download the zip files correctly?

Here is the Python 3 code that I am using:

 

import dropbox

 

def process_folder_entries(current_state, entries):
for entry in entries:
if isinstance(entry, dropbox.files.FileMetadata):
current_state[entry.path_lower] = entry
elif isinstance(entry, dropbox.files.DeletedMetadata):
current_state.pop(entry.path_lower, None) # ignore KeyError if missing
return current_state

# Initializing Dropbox API
dbx = dropbox.Dropbox("token")

# Scanning for files
path = r"/download"

result = dbx.files_list_folder(folder_name)
files = process_folder_entries({}, result.entries)

# check for and collect any additional entries
while result.has_more:
result = dbx.files_list_folder_continue(result.cursor)
files = process_folder_entries(files, result.entries)

 

# define the download location.
directory = directoryname

 

for entry in files.values():

new_file_path = directory + entry.name

# download the file, returns the File Metadata and File Content.
#file_metadata, file_content = dbx.files_download(entry.path_lower)
file_metadata, res = dbx.files_download(entry.path_lower)

try:
# write the file to the directory
with open(new_file_path,'w') as new_file:
new_file.write(res.content)
except:
continue

1 Accepted Solution

Accepted Solutions

Re: Downloading zip files from Dropbox via API returns size zero

Greg-DB
Dropboxer

Apologies, I had the wrong link in that post. I just fixed it. It should be this post.

 

Anyway, it looks like the issue is that you're running out of memory when downloading the file, since you're accessing the whole thing, and the file is very large.

 

You can read off the download result in pieces like this:

with open(new_file_path, 'wb') as new_file:
    for chunk in res.iter_content(chunk_size=4*1024*1024):  # or whatever chunk size you want
        new_file.write(chunk)

(Note I also changed the file open mode to binary mode.)

View solution in original post

7 Replies 7

Re: Downloading zip files from Dropbox via API returns size zero

Greg-DB
Dropboxer

It looks like you may be silently dropping exceptions, based on this code:

except:
    continue

Add some error handling there to see what may be going wrong with writing the file. At least something like:

except Exception as e:
    print(e)
    continue

Re: Downloading zip files from Dropbox via API returns size zero

lekn
Explorer | Level 3

Hi Greg,

 

Thank you for your reply. 

 

Do you have any idea about the following issue?

 

 self._content = b''.join(self.iter_content(CONTENT_CHUNK_SIZE)) or b''
MemoryError

 

 

Re: Downloading zip files from Dropbox via API returns size zero

Greg-DB
Dropboxer

In Python, a MemoryError is "Raised when an operation runs out of memory". That certainly seems possible here given the file sizes you mentioned. 

 

This new line of code seems to reference several things not included in the original code you shared, and doesn't seem to mention Dropbox. Is this new code of yours, or is it from something else?

 

Is something still not working as expected on the Dropbox side of things? If so, please share the latest relevant code and full error output so we can take a look.

 

If you need to download the file in pieces due to memory constraints, you may want to try using res.iter_content directly. I have some information on that in this post.

Re: Downloading zip files from Dropbox via API returns size zero

lekn
Explorer | Level 3
Hi,
 
1. The new code that I run was just without try and except.
I just changed the latter part as the following:
 
for entry in files.values():
new_file_path = directory + entry.name
# download the file, returns the File Metadata and File Content.
#file_metadata, file_content = dbx.files_download(entry.path_lower)
file_metadata, res = dbx.files_download(entry.path_lower)

# write the file to the directory
with open(new_file_path,'w') as new_file:
new_file.write(res.content)
 
2. The full error is here:
 
Traceback (most recent call last):
File "./dropbox_load.py", line 50, in <MODULE>
new_file.write(res.content)
File "/opt/apps/apps/binapps/anaconda3/2019.03/lib/python3.7/site-packages/requests/models.py", line 828, in content
self._content = b''.join(self.iter_content(CONTENT_CHUNK_SIZE)) or b''
MemoryError
&nbsp;

When I tried the try except with print (e) you advised, I also got Memory Error.

3. Could you elaborate more on how to download the compressed files in pieces? Also, the link that you shared does not work. Could you share the address instead?
&nbsp;
Thank you.</MODULE>

Re: Downloading zip files from Dropbox via API returns size zero

Greg-DB
Dropboxer

Apologies, I had the wrong link in that post. I just fixed it. It should be this post.

 

Anyway, it looks like the issue is that you're running out of memory when downloading the file, since you're accessing the whole thing, and the file is very large.

 

You can read off the download result in pieces like this:

with open(new_file_path, 'wb') as new_file:
    for chunk in res.iter_content(chunk_size=4*1024*1024):  # or whatever chunk size you want
        new_file.write(chunk)

(Note I also changed the file open mode to binary mode.)

View solution in original post

Re: Downloading zip files from Dropbox via API returns size zero

lekn
Explorer | Level 3

This works well. Thank you.

 

The link still doesn't work. Could you upload the link again?

I was wondering what the chunk size means and how to choose the size. 

Re: Downloading zip files from Dropbox via API returns size zero

Greg-DB
Dropboxer

I'm glad to hear that works well. 

 

The chunk size is how much data to read from the Dropbox API and then write to the local file at a time. I used 4 MB in this sample, but you can change that as desired, with the constraint being how much memory you can use.

 

I'm not sure why that link got broken again, but I fixed it again and confirmed it's working now.

Poll
We love to learn from the educators who use Dropbox. Whether you teach kids, teens, adults or a combination of all three, we want to know what apps and integrations you use with Dropbox to help with teaching. Which of the ones below is your favorite, or most used tool?
Who's talking

Top contributors to this post

  • User avatar
    Greg-DB Dropboxer
  • User avatar
    lekn Explorer | Level 3
What do Dropbox user levels mean?
Need more support?