One month down in 2025: How are your resolutions coming along? Check out how to get back on track here.
Forum Discussion
ijunaid8989
6 years agoExplorer | Level 3
too many write operations when uploading files
I am using Elixir wrapper to DropboxAPI to upload files, which is doing simple upload using `files/upload` route.
def upload(client, path, file, mode \\ "add", autorename \\ true, mute \\ false) do dropbox_headers = %{ :path => path, :mode => mode, :autorename => autorename, :mute => mute } headers = %{ "Dropbox-API-Arg" => Poison.encode!(dropbox_headers), "Content-Type" => "application/octet-stream" } upload_request( client, Application.get_env(:elixir_dropbox, :upload_url), "files/upload", file, headers ) end
But I am having an issue lately while uploading so many files parallelly, I am getting an error as `too many write operations`
I have been reading and looking into https://www.dropbox.com/developers/documentation/http/documentation#files-upload_session-start
But I am not following it totally, What I read from it, I can upload multiple files in one session, When I start a session while uploading one file, I think it works for sending chunks of a single file not, different files.
says, This route helps you commit many files at once into a user's Dropbox.
But how?? How I can send multiple files to Dropbox account not
- TaylorKrusen
Dropbox Staff
Hello,
The data ingress guide should be of some help here. Specifically, there are some instructions towards the bottom that explain how to handle many files in parallel.
I'll paraphrase a bit here:
- Organize files into batches of under 1,000
- Call the files/upload_session/start endpoint once per file
- Call the files/upload_session/append_v2 endpoint as needed for each file
- The last call to files/upload_session/append_v2 for each file must contain a `close: true` parameter
- Call the files/upload_session/finish_batch endpoint to commit the files.
- Note: you may need to call this multiple times if you're doing more than 1,000 files or you'll run into the lock contention issue again.
- This will return an `async_job_id`. You can use that to check the status of your upload with the files/upload_session/finish_batch/check endpoint.
Hope that helps!
- ijunaid8989Explorer | Level 3
Okay I have few questions:
How to organize files into batches of under 1000? like is there any structure for it? which is not written in documentation?
and how you can pas that batch of files to the endpoint?
First I made a new client with the access token
iex(3)> client = ElixirDropbox.Client.new("PWK08cePo_kAAAAAAAB2oRsQSyG") %ElixirDropbox.Client{ access_token: "PWK08cePo_kAAAAAAAB2oRsQSyG" }
and uploaded a file
iex(4)> ElixirDropbox.Files.UploadSession.start(client, false, "mix.lock") %{"session_id" => "AAAAAAAYclutwUwG5odRew"}
with close: false,
iex(5)> ElixirDropbox.Files.UploadSession.append(client, "AAAAAAAYclutwUwG5odRew", false, "mix.exs") {{:status_code, 409}, {:ok, %{ "error" => %{".tag" => "incorrect_offset", "correct_offset" => 18597}, "error_summary" => "incorrect_offset/." }}}
I uploaded another file now, with the session id, and it gave me error as above?
What is the proper way of sending the batches of files?
https://www.dropbox.com/developers/documentation/http/documentation#files-upload_session-start I don't see any such information here?
- Greg-DB
Dropbox Staff
To clarify a bit, you need one "upload session" per file. Each upload session gets its own session ID. When adding data to a single upload session, you need to identify how much you've already added to that upload session by using the "offset" value. This helps make sure you're uploading the right pieces. If you specify the wrong current offset for an upload session, you'll get that "incorrect_offset" error.
You can then "finish" up to 1000 files together (i.e., up to 1000 upload sessions) using /2/files/upload_session/finish_batch.
Check the /2/files/upload_session/finish_batch documentation for information on how to format the parameters for it. Be sure to click on "UploadSessionFinishArg", and so on, to expand the documentation for the nested fields.
I also recommend using the API v2 Explorer to help guide you in understanding how to build the parameters. Click the "Add" button to add an entry and fill in the fields, for instance, and then click "Show Code" to see what the resulting code would be for the call.
I don't believe we have a sample for using upload sessions with /2/files/upload_session/finish_batch, but there are some examples of just using upload sessions (with some of the SDKs) here, which should be a good starting point for the logic:
About Dropbox API Support & Feedback
Find help with the Dropbox API from other developers.5,950 PostsLatest Activity: 14 hours ago
If you need more help you can view your support options (expected response time for an email or ticket is 24 hours), or contact us on X or Facebook.
For more info on available support options for your Dropbox plan, see this article.
If you found the answer to your question in this Community thread, please 'like' the post to say thanks and to let us know it was useful!