We’re Still Here to Help (Even Over the Holidays!) - find out more here.
Forum Discussion
ivanhigueram
7 years agoExplorer | Level 4
[Python V2] How to do batch upload?
Hi! I am trying to upload several files using the sessions and batch operators in the Dropbox SDK for Python. I'm trying to do something like this: dbx = dropbox.Dropbox(<API KEY>)
commit_info...
Greg-DB
Dropbox Community Moderator
7 years ago[Cross-linking for reference: https://stackoverflow.com/questions/54758978/dropbox-python-api-upload-multiple-files ]
Apologies for the confusion! The Python SDK documentation unfortunately doesn't do a good job identifying the types expected in certain parameters like this; I'll ask the team to work on improving that in the future.
The `files_upload_session_finish_batch` method does work differently than the `files_upload_session_finish` method. The `files_upload_session_finish_batch` method expects a list of `UploadSessionFinishArg`, where each one encapsulates the cursor and commit info together.
Here's a basic working example that shows how to do this:
import dropbox
ACCESS_TOKEN = "..."
dbx = dropbox.Dropbox(ACCESS_TOKEN)
local_file_path = "..."
upload_entry_list = []
for i in range(5):
f = open(local_file_path)
upload_session_start_result = dbx.files_upload_session_start(f.read(), close=True) # assuming small files
cursor = dropbox.files.UploadSessionCursor(session_id=upload_session_start_result.session_id,
offset=f.tell())
commit = dropbox.files.CommitInfo(path="/test_329517/%s" % i)
upload_entry_list.append(dropbox.files.UploadSessionFinishArg(cursor=cursor, commit=commit))
print(dbx.files_upload_session_finish_batch(upload_entry_list))
# then use files_upload_session_finish_batch_check to check on the job
- ivanhigueram7 years agoExplorer | Level 4
Hi Greg!
Again thanks for replying back so promptly. I have some follow up questions. If I am looping through different Python objects, not files, that's why I'm first converting the pd.DataFrame to a string, and then pointing it to the dbx.files_upload_session_start() function.
Since my file is complete, and I am not passing a files in a context manager or a StringIO, I did not specify any offset in the cursor. Now that I'm trying to run the loop, I received the following error:Traceback (most recent call last): File "/Users/ivan/.pyenv/versions/weather_data/lib/python3.6/site-packages/dropbox/stone_serializers.py", line 337, in encode_struct field_value = getattr(value, field_name) File "/Users/ivan/.pyenv/versions/weather_data/lib/python3.6/site-packages/dropbox/files.py", line 10278, in offset raise AttributeError("missing required field 'offset'") AttributeError: missing required field 'offset'What are the real advantages of using the batch operations to upload files? Seems a convulted use case for objects in memory, rather than files.
Thanks again for your help.- Greg-DB7 years ago
Dropbox Community Moderator
Regardless of where the data is coming from, the `UploadSessionCursor` object does require an `offset`, in order "to make sure upload data isn’t lost or duplicated in the event of a network error". It sounds like in your case the `offset` value would be the length of the string.
The main advantage of using `files_upload_session_finish_batch` is to minimize the number of "locks" needed when uploading multiple files. The Data Ingress Guide covers this in more detail. This applies to uploading from memory or files.
The main advantage of using "upload sessions" to begin with is to enable apps to upload large files. If you're just uploading small files, you can certainly do so just using `files_upload`, but you'd need to do so serially to avoid lock contention. Based on the code you provided, you're already uploading serially though, so it may not make much of a difference if you wish to switch to `files_upload` for simplicity.
- 9krausec5 years agoExplorer | Level 4
Hey Greg. Would you mind going over an approach assuming the files being batch uploaded aren't small? As in "files_upload_session_append_v2" would be best to read in each file by a chunk size? Thank you.
- Greg-DB5 years ago
Dropbox Community Moderator
9krausec I posted a sample here that may be useful. Hope this helps!
- 9krausec5 years agoExplorer | Level 4
Thank you for the reply Greg. I did stumble on that post.
I should of been more specific with my question -
If I have 20 files that I want uploaded in a single session, using "dbx.files_upload_session_finish_batch()" and "dbx.files_upload_session_append_v2()" for files exceeding the set chunk size, how would I go about doing so?
The example you posted here works great without using "files_upload_session_append_v2" as the files are assumed to be small.
The example you linked works great, if uploading a single large file and wanted to chunk it out.
I'm having difficulties figuring out how to approach combining the two examples with successful results.
Thank you for any help.
About Dropbox API Support & Feedback
Find help with the Dropbox API from other developers.
The Dropbox Community team is active from Monday to Friday. We try to respond to you as soon as we can, usually within 2 hours.
If you need more help you can view your support options (expected response time for an email or ticket is 24 hours), or contact us on X, Facebook or Instagram.
For more info on available support options for your Dropbox plan, see this article.
If you found the answer to your question in this Community thread, please 'like' the post to say thanks and to let us know it was useful!