We’re Still Here to Help (Even Over the Holidays!) - find out more here.
Forum Discussion
ivanhigueram
7 years agoExplorer | Level 4
[Python V2] How to do batch upload?
Hi! I am trying to upload several files using the sessions and batch operators in the Dropbox SDK for Python. I'm trying to do something like this: dbx = dropbox.Dropbox(<API KEY>)
commit_info...
Greg-DB
Dropbox Community Moderator
5 years ago9krausec I posted a sample here that may be useful. Hope this helps!
9krausec
5 years agoExplorer | Level 4
Thank you for the reply Greg. I did stumble on that post.
I should of been more specific with my question -
If I have 20 files that I want uploaded in a single session, using "dbx.files_upload_session_finish_batch()" and "dbx.files_upload_session_append_v2()" for files exceeding the set chunk size, how would I go about doing so?
The example you posted here works great without using "files_upload_session_append_v2" as the files are assumed to be small.
The example you linked works great, if uploading a single large file and wanted to chunk it out.
I'm having difficulties figuring out how to approach combining the two examples with successful results.
Thank you for any help.
- Greg-DB5 years ago
Dropbox Community Moderator
9krausec Thanks for following up with the clarification. First, note that one "upload session" is required per file to be uploaded. So, you'd need to call files_upload_session_start once per file (so, 20 total in this case), and files_upload_session_append_v2 zero or more times per file (depending on each file's size).
Then, to commit them together to avoid contention, you'd call files_upload_session_finish_batch once with the information for all 20 upload sessions. (That's instead of calling files_upload_session_finish once per file.)
So, you will need to combine portions of each of the examples for your use case. The example in this thread shows how to start multiple upload sessions and then commit them all together, while the example in the other thread shows how to append to any given upload session repeatedly as needed.
- 9krausec5 years agoExplorer | Level 4
Greg-DB - Thank you for following this and providing input. I think I'm close to a solution and will continue tinkering. In the meantime, please review my method below and provide any further suggestions.
For output, I am getting an "UploadSessionFinishBatchLaunch" return at the end with an async_job_id. However, nothing is uploaded to dropbox after completion. Thank you.from os import listdir from os.path import isfile, join import dropbox def batchUpload(srcFiles, dstDir, dbx=None): if not dbx: dbx = DropboxTeam(DROPBOX_TOKEN).as_admin(ADMIN_ID) chunkSize = 2 * 1024 * 1024 uploadEntryList = [] for src in srcFiles: baseName = os.path.basename(src) dstPath = dstDir + "/" + baseName fileSize = os.path.getsize(src) print("{0} ==> {1}".format(src, dstPath)) with open(src, "rb") as fp: if fileSize <= chunkSize: # file smaller than chunkSize, no files_upload_session_append_v2 uploadSessionStartResult = dbx.files_upload_session_start(fp.read(), dstPath, close=True) cursor = dropbox.files.UploadSessionCursor(session_id=uploadSessionStartResult.session_id, offset=fp.tell()) commit = dropbox.files.CommitInfo(path=dstPath) else: # file larger than chunkSize. Use files_upload_session_append_v2 uploadSessionStartResult = dbx.files_upload_session_start(fp.read(chunkSize)) cursor = dropbox.files.UploadSessionCursor(session_id=uploadSessionStartResult.session_id, offset=fp.tell()) commit = dropbox.files.CommitInfo(path=dstPath) while fp.tell() <= fileSize: if ((fileSize - fp.tell()) <= chunkSize): # file chunk append complete print("COMPLETE CHUNKS") uploadEntryList.append(files.UploadSessionFinishArg(cursor=cursor, commit=commit)) break else: print(fp.tell()) dbx.files_upload_session_append_v2(fp.read(chunkSize), cursor) cursor.offset = fp.tell() uploadEntryList.append(files.UploadSessionFinishArg(cursor=cursor, commit=commit)) print( dbx.files_upload_session_finish_batch(uploadEntryList) ) srcDir = "C:/Users/alpha/Documents/scratch/LIGHTS - Copy" srcFiles = [join(srcDir, f).replace('\\', '/') for f in listdir(srcDir) if isfile(join(srcDir, f))] dstDir = "/scratch/poopOne" dbx = DropboxTeam(DROPBOX_TOKEN).as_admin(ADMIN_ID) batchUpload(srcFiles, dstDir, dbx)- Greg-DB5 years ago
Dropbox Community Moderator
9krausec You'll need to poll files_upload_session_finish_batch_check to check on the result of the job. If you're not seeing anything uploaded, something must have failed, and the result from files_upload_session_finish_batch_check should tell you what it was.
About Dropbox API Support & Feedback
Find help with the Dropbox API from other developers.
The Dropbox Community team is active from Monday to Friday. We try to respond to you as soon as we can, usually within 2 hours.
If you need more help you can view your support options (expected response time for an email or ticket is 24 hours), or contact us on X, Facebook or Instagram.
For more info on available support options for your Dropbox plan, see this article.
If you found the answer to your question in this Community thread, please 'like' the post to say thanks and to let us know it was useful!