cancel
Showing results for 
Show  only  | Search instead for 
Did you mean: 
Announcements
Want to learn some quick and useful tips to make your day easier? Check out how Calvin uses Replay to get feedback from other teams at Dropbox here.

Dropbox API Support & Feedback

Find help with the Dropbox API from other developers.

cancel
Showing results for 
Show  only  | Search instead for 
Did you mean: 

Re: questRe: too many write operations when uploading files

too many write operations when uploading files

ijunaid8989
Explorer | Level 3

I am using Elixir wrapper to DropboxAPI to upload files, which is doing simple upload using `files/upload` route.

  def upload(client, path, file, mode \\ "add", autorename \\ true, mute \\ false) do
    dropbox_headers = %{
      :path => path,
      :mode => mode,
      :autorename => autorename,
      :mute => mute
    }

    headers = %{
      "Dropbox-API-Arg" => Poison.encode!(dropbox_headers),
      "Content-Type" => "application/octet-stream"
    }

    upload_request(
      client,
      Application.get_env(:elixir_dropbox, :upload_url),
      "files/upload",
      file,
      headers
    )
  end

But I am having an issue lately while uploading so many files parallelly, I am getting an error as `too many write operations`

 

I have been reading and looking into https://www.dropbox.com/developers/documentation/http/documentation#files-upload_session-start

 

But I am not following it totally, What I read from it, I can upload multiple files in one session, When I start a session while uploading one file, I think it works for sending chunks of a single file not, different files.

 

But https://www.dropbox.com/developers/documentation/http/documentation#files-upload_session-finish_batc...

 

says, This route helps you commit many files at once into a user's Dropbox.

 

But how?? How I can send multiple files to Dropbox account not 

17 Replies 17

ijunaid8989
Explorer | Level 3

Also I can send data only through upload session start and finish..

 

Finish batch never worked for me to upload all data at once.

Greg-DB
Dropbox Staff

Each of the /start, /append, and /finish endpoints, but not /finish_batch, can accept file data. 

You would need to use /append if there is more data than you can send in the /start and /finish requests.

Upload sessions work by having you send only a portion of the file's data per request. 

Exactly how much data you send per request is up to you. For example, a maximum of 8 MB per request is reasonable, but it can vary by use case. So, if you were using a maximum of 8 MB per request, and the file is only 2 MB, you would only need a single /start request to send all of the file data (and wouldn't need to call /append for that particular file). You could then use /finish without sending any more data to commit that file, or use /finish_batch to commit that file (as well as others).

ijunaid8989
Explorer | Level 3

Okay thanks for all your help and replies, right now what I am doing is:

 

    client = ElixirDropbox.Client.new(System.get_env["DROP_BOX_TOKEN"])
    {:ok, file_size} = get_file_size(image_path)
    %{"session_id" => session_id} = ElixirDropbox.Files.UploadSession.start(client, false, image_path)
    ElixirDropbox.Files.UploadSession.finish(client, session_id, upload_image_path, image_path, file_size) |> handle_upload_response

I still got one error as

 

"error_summary\": \"too_many_write_operations/\", \"error\": {\"reason\": {\".tag\": \"too_many_write_operations\"

and 5 errors are API timed out. 

 

I have no idea why still there is too many write operations error? where as I am right now first starting the session, then finishing it in a proper way? 

but these requests almost goes in parallel, should I need to change something as well? more..

Greg-DB
Dropbox Staff

In this code I see you're using the 'finish' method, presumably once per file, to finish the uploads. If you send off multiple of these at the same time, you will cause lock contention. 

You should use a single 'finish_batch' to finish multiple file uploads in a single call to avoid this. (Or, only send the multiple 'finish' calls one at a time.)

ijunaid8989
Explorer | Level 3

I uploaded using session start and I got an session_id. 

AAAAAAAb0vwx3ypjoH836Q

and then instead of doing append I just finished the batch of this file 

 

curl -X POST https://api.dropboxapi.com/2/files/upload_session/finish_batch \
    --header "Authorization: Bearer -SC69zip6zi9bwHgqhYHNYQbrpZ5C7G" \
    --header "Content-Type: application/json" \
    --data "{\"entries\": [{\"cursor\": {\"session_id\": \"AAAAAAAb0vwx3ypjoH836Q\",\"offset\": 34520},\"commit\": {\"path\": \"/REGULAR/licence\",\"mode\": \"add\",\"autorename\": true,\"mute\": false,\"strict_conflict\": false}}]}"

This gave me results as 

 

{".tag": "async_job_id", "async_job_id": "dbjid:AACg_hehcOljTPYxh6wQhigGQhtc3-eCFneDKK--vzA5V9wBLNi74iNeNqWUQDFOnx1_k-sViaZJpJvXRG0UTlLw"}

and whereas on normal session start and finish I just get the file uploaded moreover  when I check batch  as

curl -X POST https://api.dropboxapi.com/2/files/upload_session/finish_batch/check \
>     --header "Authorization: Bearer -SC69zip6zi9bwHgqhYHNYQbrpZ5C7G" \
>     --header "Content-Type: application/json" \
>     --data "{\"async_job_id\": \"dbjid:AACg_hehcOljTPYxh6wQhigGQhtc3-eCFneDKK--vzA5V9wBLNi74iNeNqWUQDFOnx1_k-sViaZJpJvXRG0UTlLw\"}"

it returns as 

 

{".tag": "complete", "entries": [{".tag": "failure", "failure": {".tag": "lookup_failed", "lookup_failed": {".tag": "not_closed"}}}]}

So now where I am wrong? the same file is getting uploaded through session start and finish but giving the error in this way? 

 

Greg-DB
Dropbox Staff

@ijunaid8989 All of the upload sessions that you wish to finish using /finish_batch need to be "closed" before you call /finish_batch. That message is indicating that the upload session was not closed.

You should close the upload session with the last call you make to upload the final data for that upload session, whether that's on /start or /append, using the "close" parameter.

ijunaid8989
Explorer | Level 3

Thanks for all your replies and I have fixed this issue, my only question is right now, what could be the reason for timeouts from dropbox api? 

 

I have timeout errors a few, while upload session start.

Greg-DB
Dropbox Staff

Can you share the code and error output you're getting for these timeouts so we can take a look? Thanks!

Need more support?