cancel
Showing results for 
Show  only  | Search instead for 
Did you mean: 
Announcements
Want to learn some quick and useful tips to make your day easier? Check out how Calvin uses Replay to get feedback from other teams at Dropbox here.

Dropbox API Support & Feedback

Find help with the Dropbox API from other developers.

cancel
Showing results for 
Show  only  | Search instead for 
Did you mean: 

Do I have to wait until /upload_session/finish_batch/check finishes before starting the next batch?

Do I have to wait until /upload_session/finish_batch/check finishes before starting the next batch?

ncw
Collaborator | Level 8
Go to solution

When reading the docs for upload_session/finish_batch it says

 

For the same account, this route should be executed serially. That means you should not start the next job before current job finishes

 

I'm unsure whether that means I have to wait for the batch to finish with upload_session/finish_batch/check or not before starting the next batch?

 

It seems to work if I call upload_session/finish_batch/check in concurrently with opening a new batch.

 

However it also seems to drop files from the batch sometimes returning "too_many_write_operations" so maybe it isn't supported.

 

2021/03/19 09:21:03 DEBUG : HTTP REQUEST (req 0xc000f76000)
2021/03/19 09:21:03 DEBUG : POST /2/files/upload_session/finish_batch/check HTTP/1.1
Host: api.dropboxapi.com
User-Agent: rclone/v1.55.0-DEV
Content-Length: 113
Authorization: XXXX
Content-Type: application/json
Accept-Encoding: gzip

 

{"async_job_id":"dbjid:AABy2FH-VQkY-9lq_BMdfJVJI2_C-vRjliacnw5tIIKeKFitZxUgGsnWlDim84xSqANIbZr2bmqWWWRIPl7Dw_69"}

 

 

2021/03/19 09:21:04 DEBUG : HTTP RESPONSE (req 0xc000f76000)
2021/03/19 09:21:04 DEBUG : HTTP/2.0 200 OK
Accept-Encoding: identity,gzip
Cache-Control: no-cache
Content-Type: application/json
Date: Fri, 19 Mar 2021 09:21:03 GMT
Server: envoy
Vary: Accept-Encoding
X-Content-Type-Options: nosniff
X-Dropbox-Request-Id: f1ed7f49b82e4b08b96a6aa47c13b688
X-Dropbox-Response-Origin: far_remote
X-Frame-Options: SAMEORIGIN
X-Server-Response-Time: 63

 

{".tag": "complete", "entries": [{".tag": "failure", "failure": {".tag": "too_many_write_operations"}}, {".tag": "failure", "failure": {".tag": "too_many_write_operations"}}, {".tag": "failure", "failure": {".tag": "too_many_write_operations"}}, {".tag": "failure", "failure": {".tag": "too_many_write_operations"}}, {".tag": "failure", "failure": {".tag": "too_many_write_operations"}}, {".tag": "failure", "failure": {".tag": "too_many_write_operations"}}, {".tag": "failure", "failure": {".tag": "too_many_write_operations"}}, {".tag": "failure", "failure": {".tag": "too_many_write_operations"}}, {".tag": "failure", "failure": {".tag": "too_many_write_operations"}}, {".tag": "failure", "failure": {".tag": "too_many_write_operations"}}, {".tag": "failure", "failure": {".tag": "too_many_write_operations"}}]}

 

It is, of course, a lot faster waiting for multiple batches to complete concurrently!

 

1 Accepted Solution

Accepted Solutions

Greg-DB
Dropbox Staff
Go to solution

The documentation and this error are referring to "lock contention" which is an inability to make multiple changes in the same account (or more accurately, namespace) at the same time. I recommend reading the Performance Guide for more information for context, if you haven't already.

 

So, you should only run one /2/files/upload_session/finish_batch job per namespace at a time. That means waiting until /2/files/upload_session/finish_batch/check returns "complete" before initiating another job via another call to /2/files/upload_session/finish_batch. 

 

You can however start and upload to other upload sessions, even for the same namespace, using /2/files/upload_session/start and /2/files/upload_session/append_v2, at the same time.

 

This is because the actual change is committed by the job started by /2/files/upload_session/finish_batch. The other calls, /2/files/upload_session/start and /2/files/upload_session/append_v2, don't commit changes; they just upload the data in preparation of being committed later.

View solution in original post

2 Replies 2

Greg-DB
Dropbox Staff
Go to solution

The documentation and this error are referring to "lock contention" which is an inability to make multiple changes in the same account (or more accurately, namespace) at the same time. I recommend reading the Performance Guide for more information for context, if you haven't already.

 

So, you should only run one /2/files/upload_session/finish_batch job per namespace at a time. That means waiting until /2/files/upload_session/finish_batch/check returns "complete" before initiating another job via another call to /2/files/upload_session/finish_batch. 

 

You can however start and upload to other upload sessions, even for the same namespace, using /2/files/upload_session/start and /2/files/upload_session/append_v2, at the same time.

 

This is because the actual change is committed by the job started by /2/files/upload_session/finish_batch. The other calls, /2/files/upload_session/start and /2/files/upload_session/append_v2, don't commit changes; they just upload the data in preparation of being committed later.

ncw
Collaborator | Level 8
Go to solution

Thank you Greg that was extremely clear.

 

I understand exactly now what I am doing 🙂

Need more support?
Who's talking

Top contributors to this post

  • User avatar
    ncw Collaborator | Level 8
  • User avatar
    Greg-DB Dropbox Staff
What do Dropbox user levels mean?