Do I have to wait until /upload_session/finish_batch/check finishes before starting the next batch?

[ncw]

When reading the docs for upload_session/finish_batch it says

For the same account, this route should be executed serially. That means you should not start the next job before current job finishes

I'm unsure whether that means I have to wait for the batch to finish with upload_session/finish_batch/check or not before starting the next batch?

It seems to work if I call upload_session/finish_batch/check in concurrently with opening a new batch.

However it also seems to drop files from the batch sometimes returning "too_many_write_operations" so maybe it isn't supported.

2021/03/19 09:21:03 DEBUG : HTTP REQUEST (req 0xc000f76000)
2021/03/19 09:21:03 DEBUG : POST /2/files/upload_session/finish_batch/check HTTP/1.1
Host: api.dropboxapi.com
User-Agent: rclone/v1.55.0-DEV
Content-Length: 113
Authorization: XXXX
Content-Type: application/json
Accept-Encoding: gzip

{"async_job_id":"dbjid:AABy2FH-VQkY-9lq_BMdfJVJI2_C-vRjliacnw5tIIKeKFitZxUgGsnWlDim84xSqANIbZr2bmqWWWRIPl7Dw_69"}

2021/03/19 09:21:04 DEBUG : HTTP RESPONSE (req 0xc000f76000)
2021/03/19 09:21:04 DEBUG : HTTP/2.0 200 OK
Accept-Encoding: identity,gzip
Cache-Control: no-cache
Content-Type: application/json
Date: Fri, 19 Mar 2021 09:21:03 GMT
Server: envoy
Vary: Accept-Encoding
X-Content-Type-Options: nosniff
X-Dropbox-Request-Id: f1ed7f49b82e4b08b96a6aa47c13b688
X-Dropbox-Response-Origin: far_remote
X-Frame-Options: SAMEORIGIN
X-Server-Response-Time: 63

{".tag": "complete", "entries": [{".tag": "failure", "failure": {".tag": "too_many_write_operations"}}, {".tag": "failure", "failure": {".tag": "too_many_write_operations"}}, {".tag": "failure", "failure": {".tag": "too_many_write_operations"}}, {".tag": "failure", "failure": {".tag": "too_many_write_operations"}}, {".tag": "failure", "failure": {".tag": "too_many_write_operations"}}, {".tag": "failure", "failure": {".tag": "too_many_write_operations"}}, {".tag": "failure", "failure": {".tag": "too_many_write_operations"}}, {".tag": "failure", "failure": {".tag": "too_many_write_operations"}}, {".tag": "failure", "failure": {".tag": "too_many_write_operations"}}, {".tag": "failure", "failure": {".tag": "too_many_write_operations"}}, {".tag": "failure", "failure": {".tag": "too_many_write_operations"}}]}

It is, of course, a lot faster waiting for multiple batches to complete concurrently!

[Greg-DB · Accepted solution]

The documentation and this error are referring to "lock contention" which is an inability to make multiple changes in the same account (or more accurately, namespace) at the same time. I recommend reading the Performance Guide for more information for context, if you haven't already.

So, you should only run one /2/files/upload_session/finish_batch job per namespace at a time. That means waiting until /2/files/upload_session/finish_batch/check returns "complete" before initiating another job via another call to /2/files/upload_session/finish_batch. 

You can however start and upload to other upload sessions, even for the same namespace, using /2/files/upload_session/start and /2/files/upload_session/append_v2, at the same time.

This is because the actual change is committed by the job started by /2/files/upload_session/finish_batch. The other calls, /2/files/upload_session/start and /2/files/upload_session/append_v2, don't commit changes; they just upload the data in preparation of being committed later.

[ncw]

Thank you Greg that was extremely clear.

I understand exactly now what I am doing 🙂