We’re Still Here to Help (Even Over the Holidays!) - find out more here.
Forum Discussion
too many write operations when uploading files
I am using Elixir wrapper to DropboxAPI to upload files, which is doing simple upload using `files/upload` route. def upload(client, path, file, mode \\ "add", autorename \\ true, mute \\ false) d...
TaylorKrusen
Dropbox Staff
Dropbox StaffHello,
The data ingress guide should be of some help here. Specifically, there are some instructions towards the bottom that explain how to handle many files in parallel.
I'll paraphrase a bit here:
- Organize files into batches of under 1,000
- Call the files/upload_session/start endpoint once per file
- Call the files/upload_session/append_v2 endpoint as needed for each file
- The last call to files/upload_session/append_v2 for each file must contain a `close: true` parameter
- Call the files/upload_session/finish_batch endpoint to commit the files.
- Note: you may need to call this multiple times if you're doing more than 1,000 files or you'll run into the lock contention issue again.
- This will return an `async_job_id`. You can use that to check the status of your upload with the files/upload_session/finish_batch/check endpoint.
Hope that helps!
Okay I have few questions:
How to organize files into batches of under 1000? like is there any structure for it? which is not written in documentation?
and how you can pas that batch of files to the endpoint?
First I made a new client with the access token
iex(3)> client = ElixirDropbox.Client.new("PWK08cePo_kAAAAAAAB2oRsQSyG")
%ElixirDropbox.Client{
access_token: "PWK08cePo_kAAAAAAAB2oRsQSyG"
}and uploaded a file
iex(4)> ElixirDropbox.Files.UploadSession.start(client, false, "mix.lock")
%{"session_id" => "AAAAAAAYclutwUwG5odRew"}with close: false,
iex(5)> ElixirDropbox.Files.UploadSession.append(client, "AAAAAAAYclutwUwG5odRew", false, "mix.exs")
{{:status_code, 409},
{:ok,
%{
"error" => %{".tag" => "incorrect_offset", "correct_offset" => 18597},
"error_summary" => "incorrect_offset/."
}}}I uploaded another file now, with the session id, and it gave me error as above?
What is the proper way of sending the batches of files?
https://www.dropbox.com/developers/documentation/http/documentation#files-upload_session-start I don't see any such information here?
- Greg-DB
Dropbox Community Moderator
To clarify a bit, you need one "upload session" per file. Each upload session gets its own session ID. When adding data to a single upload session, you need to identify how much you've already added to that upload session by using the "offset" value. This helps make sure you're uploading the right pieces. If you specify the wrong current offset for an upload session, you'll get that "incorrect_offset" error.
You can then "finish" up to 1000 files together (i.e., up to 1000 upload sessions) using /2/files/upload_session/finish_batch.
Check the /2/files/upload_session/finish_batch documentation for information on how to format the parameters for it. Be sure to click on "UploadSessionFinishArg", and so on, to expand the documentation for the nested fields.
I also recommend using the API v2 Explorer to help guide you in understanding how to build the parameters. Click the "Add" button to add an entry and fill in the fields, for instance, and then click "Show Code" to see what the resulting code would be for the call.
I don't believe we have a sample for using upload sessions with /2/files/upload_session/finish_batch, but there are some examples of just using upload sessions (with some of the SDKs) here, which should be a good starting point for the logic:
So at first, I ran this in the command line
curl -X POST https://content.dropboxapi.com/2/files/upload_session/start \ --header "Authorization: Bearer PWK08cePo_kAAAAAAAB2oRsQSyGQbQ2qy-SC69zC7G" \ --header "Dropbox-API-Arg: {\"close\": false}" \ --header "Content-Type: application/octet-stream" \ --data-binary @local_file.txtIt returned:
{"session_id": "AAAAAAAbQhipmPJOosGS-Q"}I don't see any offset values here but let's take a look at next request of append.
curl -X POST https://content.dropboxapi.com/2/files/upload_session/append_v2 \ --header "Authorization: Bearer C69zip6zi9bwHgqhYHNYQbrpZ5C7G" \ --header "Dropbox-API-Arg: {\"cursor\": {\"session_id\": \"AAAAAAAbQhipmPJOosGS-Q\",\"offset\": 0},\"close\": false}" \ --header "Content-Type: application/octet-stream" \ --data-binary @local_file.txtIt returned:
{"error_summary": "incorrect_offset/.", "error": {".tag": "incorrect_offset", "correct_offset": 37}}Now I sent the offset value 37 and it gave me null as
curl -X POST https://content.dropboxapi.com/2/files/upload_session/append_v2 --header "Authorization: Bearer SC69zip6zi9bwHgqhYHNYQbrpZ5C7G" --header "Dropbox-API-Arg: {\"cursor\": {\"session_id\": \"AAAAAAAbQhipmPJOosGS-Q\",\"offset\": 37},\"close\": false}" --header "Content-Type: application/octet-stream" --data-binary @local_file.txtit returned: `null`
Now I did finish_batch,
curl -X POST https://api.dropboxapi.com/2/files/upload_session/finish_batch \ > --header "Authorization: Bearer SC69zip6zi9bwHgqhYHNYQbrpZ5C7G" \ > --header "Content-Type: application/json" \ > --data "{\"entries\": [{\"cursor\": {\"session_id\": \"AAAAAAAbQhipmPJOosGS-Q\",\"offset\": 0},\"commit\": {\"path\": \"/Homework/math/Matrices.txt\",\"mode\": \"add\",\"autorename\": true,\"mute\": false,\"strict_conflict\": false}}]}"It returned to me with async_job_id,
{".tag": "async_job_id", "async_job_id": "dbjid:AACkTPVL8VvaGyrUdEPui0uHg-mzJuroOoZ5yQvP3HB-4JxhpBGNlRrt8XOH6XiOJIpUKlNjUbHtJgUPGWKhQnXm"}on the batch check,
curl -X POST https://api.dropboxapi.com/2/files/upload_session/finish_batch/check --header "Authorization: Bearer SC69zip6zi9bwHgqhYHNYQbrpZ5C7G" --header "Content-Type: application/json" --data "{\"async_job_id\": \"dbjid:AACkTPVL8VvaGyrUdEPui0uHg-mzJuroOoZ5yQvP3HB-4JxhpBGNlRrt8XOH6XiOJIpUKlNjUbHtJgUPGWKhQnXm\"}It returned as
{".tag": "complete", "entries": [{".tag": "failure", "failure": {".tag": "lookup_failed", "lookup_failed": {".tag": "incorrect_offset", "correct_offset": 74}}}]}My question is, from where I can have the offset value? its nowhere from the very start?
I followed the complete documentation guide, and I am not able to even upload one file with upload session.- Greg-DB
The 'offset' value is "The amount of data that has been uploaded so far. We use this to make sure upload data isn't lost or duplicated in the event of a network error.".
That is to say, it should be the number of bytes that you have already uploaded so far for that particular upload session. You should keep track of this client-side as you send the file data up to Dropbox.
About Dropbox API Support & Feedback
Find help with the Dropbox API from other developers.
