cancel
Showing results for 
Show  only  | Search instead for 
Did you mean: 
Announcements
Want to learn some quick and useful tips to make your day easier? Check out how Calvin uses Replay to get feedback from other teams at Dropbox here.

Dropbox API Support & Feedback

Find help with the Dropbox API from other developers.

cancel
Showing results for 
Show  only  | Search instead for 
Did you mean: 

Issue UploadSession and UploadSessionBatch with big files

Issue UploadSession and UploadSessionBatch with big files

DavidM27
Explorer | Level 4
Go to solution

Hello, 

 

I'm working with the dropbox API in TypeScript. I do want to be able to able all kind of files (small, large, multiple files...)

For that I have made a method to uplaod a single file, and a method to uploadMultiple file. The fisrt one is using the uploadSessionFinish() method provided by the sdk, the other one is using uploadSessionFinishBatch(). To handle small and large file I also have made a method that handle the file and return a filesUploadSessionFinishArg so I can use it in my uploadFile and uploadMultipleFiles method. 
This is working well for a single file either small or large. But as soon as I upload multiple files containing at least one big file it fails... 

Here is what the code looks like : 

Upload single file ->

  async uploadFile( file: ArrayBuffer, path: string): Promise<MyFile> {
    const finishSessionArg : DropboxTypes.files.UploadSessionFinishArg = await 
    this.prepareUploadFile({ file, path, })
    const _file: DropboxTypes.files.FileMetadata = await 
    this.dropBox.filesUploadSessionFinish(finishSessionArg)
    return dropBoxMapper.convertDropboxFileToFile(_file)
  }

Upload multiple file -> 

async uploadMultipleFiles(files: Array<ArrayBuffer>, filesPath: Array<string>) : Promise<Array<MyFile>> {
    const finishUploadBatchArg: 
     DropboxTypes.files.UploadSessionFinishBatchArg = {
      entries: [],
    }
    // For each file I get the finishArg
    for (const [index, file,] of files.entries()) {
      const finishSessionArg : DropboxTypes.files.UploadSessionFinishArg = 
      await this.prepareUploadFile({ file:file, path:filesPath[index], })
      finishUploadBatchArg.entries.push(finishSessionArg)
    }

    const response: DropboxTypes.files.UploadSessionFinishBatchLaunch = 
    await this.dropBox.filesUploadSessionFinishBatch(finishUploadBatchArg)

    //Check the batch status from here until it is complete
    const checkArg : DropboxTypes.async.PollArg = {
      async_job_id: (<DropboxTypes.async.LaunchResultBaseAsyncJobId> 
       response).async_job_id,
    }

    return this.checkUploadSessionBatch(checkArg)
  }

prepareUpload method -> 

  async prepareUploadFile(file: ArrayBuffer, path: string) : Promise<DropboxTypes.files.UploadSessionFinishArg> {
    const UPLOAD_FILE_SIZE_LIMIT: number = 8 * 1024 * 1024 //For testing purpose max size for a single upload is 8Mb
    const CHUNK_SIZE: number = 4 * 1024 * 1024 // Each chunk is 4Mb 
   // If smaller than 8Mb single upload
    if (UPLOAD_FILE_SIZE_LIMIT > file.byteLength) {
      const uploadSessionStartArg : DropboxTypes.files.UploadSessionStartArg = {
        contents: file,
        close: true,
      }
      const response: DropboxTypes.files.UploadSessionStartResult = await this.dropBox.filesUploadSessionStart(uploadSessionStartArg)
      const sessionId: string = response.session_id
      const cursor: DropboxTypes.files.UploadSessionCursor = {
        contents: file,
        offset:file.byteLength,
        session_id: sessionId,
      }
      const commitInfo: DropboxTypes.files.CommitInfo = {
        contents: file,
        path: path,
        autorename: true,
      }

      const finishSessionArg : DropboxTypes.files.UploadSessionFinishArg= {
        contents : file,
        cursor: cursor,
        commit: commitInfo,
      }
      // Workaround to fix issues explained here : https://www.dropboxforum.com/t5/API-Support-Feedback/Uploading-multiple-files-with-UploadSession-with-TypeScript/m-p/399797/thread-id/21878#M21888
      delete finishSessionArg.cursor.contents
      delete finishSessionArg.commit.contents
      delete finishSessionArg.contents

      return finishSessionArg
    } else { //If bigger than 8Mb divide the file in multiple 4Mb chunks
      const fileChunks: Array<ArrayBuffer> = []
      let offset: number = 0

      while (offset < file.byteLength) {
        var chunkSize = Math.min(CHUNK_SIZE, file.byteLength - offset)
        fileChunks.push(file.slice(offset, offset + chunkSize))
        offset += chunkSize
      }

      let sessionId: string = ''
      const finishSessionArg : DropboxTypes.files.UploadSessionFinishArg= {
        cursor: { contents: {}, session_id: sessionId, offset: 0, },
        commit: { contents: {}, path: path, },
        contents: {},
      }
      for (const [index,chunk,] of fileChunks.entries()) {
        if (0 === index) {
          const uploadSessionStartArg : DropboxTypes.files.UploadSessionStartArg = {
            contents: chunk,
            close: false,
          }
          const response = await this.dropBox.filesUploadSessionStart(uploadSessionStartArg)
          sessionId = response.session_id
        } else if (index < fileChunks.length-1) {
          const cursor: DropboxTypes.files.UploadSessionCursor = {
            contents: chunk,
            offset: index * CHUNK_SIZE,
            session_id: sessionId,
          }
          const uploadSessionAppendArg : DropboxTypes.files.UploadSessionAppendArg = {
            cursor: cursor,
            contents: chunk,
            close:false,
          }
          delete cursor.contents

          await this.dropBox.filesUploadSessionAppendV2(uploadSessionAppendArg)
        } else {
          const cursor: DropboxTypes.files.UploadSessionCursor = {
            contents: chunk,
            offset: index * CHUNK_SIZE,
            session_id: sessionId,
          }
          const commitInfo: DropboxTypes.files.CommitInfo = {
            contents: chunk,
            path: path,
            autorename: true,
          }
          finishSessionArg.commit = commitInfo
          finishSessionArg.cursor = cursor
          finishSessionArg.contents = chunk
        }
      }

      // Workaround to fix issues explained here : https://www.dropboxforum.com/t5/API-Support-Feedback/Uploading-multiple-files-with-UploadSession-with-TypeScript/m-p/399797/thread-id/21878#M21888
      delete finishSessionArg.cursor.contents
      delete finishSessionArg.commit.contents
      delete finishSessionArg.contents

      return finishSessionArg
    }
  }

So to explained what is wrong when I use that code, it tries to upload every file properly, each chunks seems to be uplad with the appendV2 method. But once it is done all the small files are properly uploaded but the large ones are getting this status : 

{".tag": "failure", "failure": {".tag": "lookup_failed", "lookup_failed": {".tag": "not_closed"}}}

Do you see any reason for this to happen ? 

Thank you for your help.

1 Accepted Solution

Accepted Solutions

Greg-DB
Dropbox Staff
Go to solution

When using filesUploadSessionFinishBatch, "UploadSessionStartArg.close or UploadSessionAppendArg.close needs to be true for the last upload_session/start or upload_session/append_v2 call". That is to say, since filesUploadSessionFinishBatch can't itself take any further file data, you need to make sure you've already finished sending all of the file data for each upload session, and have indicated that to the Dropbox API on the previous call for that upload session, by setting 'close' to 'true'.

 

The 'not_closed' error you're getting is indicating that you didn't close the upload session before calling filesUploadSessionFinishBatch. So, you'll need to update your code to set 'close:true' on UploadSessionAppendArg for the the last call to filesUploadSessionAppendV2 for each upload session that you're then sending to filesUploadSessionFinishBatch.

View solution in original post

3 Replies 3

Greg-DB
Dropbox Staff
Go to solution

When using filesUploadSessionFinishBatch, "UploadSessionStartArg.close or UploadSessionAppendArg.close needs to be true for the last upload_session/start or upload_session/append_v2 call". That is to say, since filesUploadSessionFinishBatch can't itself take any further file data, you need to make sure you've already finished sending all of the file data for each upload session, and have indicated that to the Dropbox API on the previous call for that upload session, by setting 'close' to 'true'.

 

The 'not_closed' error you're getting is indicating that you didn't close the upload session before calling filesUploadSessionFinishBatch. So, you'll need to update your code to set 'close:true' on UploadSessionAppendArg for the the last call to filesUploadSessionAppendV2 for each upload session that you're then sending to filesUploadSessionFinishBatch.

DavidM27
Explorer | Level 4
Go to solution

Greg, thank you for your reply !

So if I understand properly, correct me if I'm wrong, in case of uplaoding a big files within a batch session, I should not stop the AppendArgV2() at the chunkList-1 but until all the chunk are uploaded and then I build my finishArg. But in the other case I should stick with what I've done.

What if I always append all chunks and close on the last chunk ? Does it sound possible to you ?

DavidM27
Explorer | Level 4
Go to solution

I'm answering my own question, it seems to work when I always close the fileUploadSession with the appendArg on the last chunk event when I'm not using the batch.

Thank you for your help ! If you have any comments or suggestion on the code I sent you above please let me know I would be glad to have your feedback on it.

Need more support?
Who's talking

Top contributors to this post

  • User avatar
    DavidM27 Explorer | Level 4
  • User avatar
    Greg-DB Dropbox Staff
What do Dropbox user levels mean?