cancel
Showing results for 
Show  only  | Search instead for 
Did you mean: 
Announcements
Tell us what you want to see on the Community here!
Close

Dropbox API Support & Feedback

Find help with the Dropbox API from other developers.

cancel
Showing results for 
Show  only  | Search instead for 
Did you mean: 

Issue UploadSession and UploadSessionBatch with big files

Highlighted

Issue UploadSession and UploadSessionBatch with big files

Explorer | Level 4

Hello, 

 

I'm working with the dropbox API in TypeScript. I do want to be able to able all kind of files (small, large, multiple files...)

For that I have made a method to uplaod a single file, and a method to uploadMultiple file. The fisrt one is using the uploadSessionFinish() method provided by the sdk, the other one is using uploadSessionFinishBatch(). To handle small and large file I also have made a method that handle the file and return a filesUploadSessionFinishArg so I can use it in my uploadFile and uploadMultipleFiles method. 
This is working well for a single file either small or large. But as soon as I upload multiple files containing at least one big file it fails... 

Here is what the code looks like : 

Upload single file ->

  async uploadFile( file: ArrayBuffer, path: string): Promise<MyFile> {
    const finishSessionArg : DropboxTypes.files.UploadSessionFinishArg = await 
    this.prepareUploadFile({ file, path, })
    const _file: DropboxTypes.files.FileMetadata = await 
    this.dropBox.filesUploadSessionFinish(finishSessionArg)
    return dropBoxMapper.convertDropboxFileToFile(_file)
  }

Upload multiple file -> 

async uploadMultipleFiles(files: Array<ArrayBuffer>, filesPath: Array<string>) : Promise<Array<MyFile>> {
    const finishUploadBatchArg: 
     DropboxTypes.files.UploadSessionFinishBatchArg = {
      entries: [],
    }
    // For each file I get the finishArg
    for (const [index, file,] of files.entries()) {
      const finishSessionArg : DropboxTypes.files.UploadSessionFinishArg = 
      await this.prepareUploadFile({ file:file, path:filesPath[index], })
      finishUploadBatchArg.entries.push(finishSessionArg)
    }

    const response: DropboxTypes.files.UploadSessionFinishBatchLaunch = 
    await this.dropBox.filesUploadSessionFinishBatch(finishUploadBatchArg)

    //Check the batch status from here until it is complete
    const checkArg : DropboxTypes.async.PollArg = {
      async_job_id: (<DropboxTypes.async.LaunchResultBaseAsyncJobId> 
       response).async_job_id,
    }

    return this.checkUploadSessionBatch(checkArg)
  }

prepareUpload method -> 

  async prepareUploadFile(file: ArrayBuffer, path: string) : Promise<DropboxTypes.files.UploadSessionFinishArg> {
    const UPLOAD_FILE_SIZE_LIMIT: number = 8 * 1024 * 1024 //For testing purpose max size for a single upload is 8Mb
    const CHUNK_SIZE: number = 4 * 1024 * 1024 // Each chunk is 4Mb 
   // If smaller than 8Mb single upload
    if (UPLOAD_FILE_SIZE_LIMIT > file.byteLength) {
      const uploadSessionStartArg : DropboxTypes.files.UploadSessionStartArg = {
        contents: file,
        close: true,
      }
      const response: DropboxTypes.files.UploadSessionStartResult = await this.dropBox.filesUploadSessionStart(uploadSessionStartArg)
      const sessionId: string = response.session_id
      const cursor: DropboxTypes.files.UploadSessionCursor = {
        contents: file,
        offset:file.byteLength,
        session_id: sessionId,
      }
      const commitInfo: DropboxTypes.files.CommitInfo = {
        contents: file,
        path: path,
        autorename: true,
      }

      const finishSessionArg : DropboxTypes.files.UploadSessionFinishArg= {
        contents : file,
        cursor: cursor,
        commit: commitInfo,
      }
      // Workaround to fix issues explained here : https://www.dropboxforum.com/t5/API-Support-Feedback/Uploading-multiple-files-with-UploadSession-with-TypeScript/m-p/399797/thread-id/21878#M21888
      delete finishSessionArg.cursor.contents
      delete finishSessionArg.commit.contents
      delete finishSessionArg.contents

      return finishSessionArg
    } else { //If bigger than 8Mb divide the file in multiple 4Mb chunks
      const fileChunks: Array<ArrayBuffer> = []
      let offset: number = 0

      while (offset < file.byteLength) {
        var chunkSize = Math.min(CHUNK_SIZE, file.byteLength - offset)
        fileChunks.push(file.slice(offset, offset + chunkSize))
        offset += chunkSize
      }

      let sessionId: string = ''
      const finishSessionArg : DropboxTypes.files.UploadSessionFinishArg= {
        cursor: { contents: {}, session_id: sessionId, offset: 0, },
        commit: { contents: {}, path: path, },
        contents: {},
      }
      for (const [index,chunk,] of fileChunks.entries()) {
        if (0 === index) {
          const uploadSessionStartArg : DropboxTypes.files.UploadSessionStartArg = {
            contents: chunk,
            close: false,
          }
          const response = await this.dropBox.filesUploadSessionStart(uploadSessionStartArg)
          sessionId = response.session_id
        } else if (index < fileChunks.length-1) {
          const cursor: DropboxTypes.files.UploadSessionCursor = {
            contents: chunk,
            offset: index * CHUNK_SIZE,
            session_id: sessionId,
          }
          const uploadSessionAppendArg : DropboxTypes.files.UploadSessionAppendArg = {
            cursor: cursor,
            contents: chunk,
            close:false,
          }
          delete cursor.contents

          await this.dropBox.filesUploadSessionAppendV2(uploadSessionAppendArg)
        } else {
          const cursor: DropboxTypes.files.UploadSessionCursor = {
            contents: chunk,
            offset: index * CHUNK_SIZE,
            session_id: sessionId,
          }
          const commitInfo: DropboxTypes.files.CommitInfo = {
            contents: chunk,
            path: path,
            autorename: true,
          }
          finishSessionArg.commit = commitInfo
          finishSessionArg.cursor = cursor
          finishSessionArg.contents = chunk
        }
      }

      // Workaround to fix issues explained here : https://www.dropboxforum.com/t5/API-Support-Feedback/Uploading-multiple-files-with-UploadSession-with-TypeScript/m-p/399797/thread-id/21878#M21888
      delete finishSessionArg.cursor.contents
      delete finishSessionArg.commit.contents
      delete finishSessionArg.contents

      return finishSessionArg
    }
  }

So to explained what is wrong when I use that code, it tries to upload every file properly, each chunks seems to be uplad with the appendV2 method. But once it is done all the small files are properly uploaded but the large ones are getting this status : 

{".tag": "failure", "failure": {".tag": "lookup_failed", "lookup_failed": {".tag": "not_closed"}}}

Do you see any reason for this to happen ? 

Thank you for your help.

1 Accepted Solution

Accepted Solutions
Highlighted

Re: Issue UploadSession and UploadSessionBatch with big files

Dropboxer

When using filesUploadSessionFinishBatch, "UploadSessionStartArg.close or UploadSessionAppendArg.close needs to be true for the last upload_session/start or upload_session/append_v2 call". That is to say, since filesUploadSessionFinishBatch can't itself take any further file data, you need to make sure you've already finished sending all of the file data for each upload session, and have indicated that to the Dropbox API on the previous call for that upload session, by setting 'close' to 'true'.

 

The 'not_closed' error you're getting is indicating that you didn't close the upload session before calling filesUploadSessionFinishBatch. So, you'll need to update your code to set 'close:true' on UploadSessionAppendArg for the the last call to filesUploadSessionAppendV2 for each upload session that you're then sending to filesUploadSessionFinishBatch.

View solution in original post

3 Replies 3
Highlighted

Re: Issue UploadSession and UploadSessionBatch with big files

Dropboxer

When using filesUploadSessionFinishBatch, "UploadSessionStartArg.close or UploadSessionAppendArg.close needs to be true for the last upload_session/start or upload_session/append_v2 call". That is to say, since filesUploadSessionFinishBatch can't itself take any further file data, you need to make sure you've already finished sending all of the file data for each upload session, and have indicated that to the Dropbox API on the previous call for that upload session, by setting 'close' to 'true'.

 

The 'not_closed' error you're getting is indicating that you didn't close the upload session before calling filesUploadSessionFinishBatch. So, you'll need to update your code to set 'close:true' on UploadSessionAppendArg for the the last call to filesUploadSessionAppendV2 for each upload session that you're then sending to filesUploadSessionFinishBatch.

View solution in original post

Highlighted

Re: Issue UploadSession and UploadSessionBatch with big files

Explorer | Level 4

Greg, thank you for your reply !

So if I understand properly, correct me if I'm wrong, in case of uplaoding a big files within a batch session, I should not stop the AppendArgV2() at the chunkList-1 but until all the chunk are uploaded and then I build my finishArg. But in the other case I should stick with what I've done.

What if I always append all chunks and close on the last chunk ? Does it sound possible to you ?

Highlighted

Re: Issue UploadSession and UploadSessionBatch with big files

Explorer | Level 4

I'm answering my own question, it seems to work when I always close the fileUploadSession with the appendArg on the last chunk event when I'm not using the batch.

Thank you for your help ! If you have any comments or suggestion on the code I sent you above please let me know I would be glad to have your feedback on it.

Work Smarter with Dropbox

The way we work is changing. Share and discover new ways to work smarter with Dropbox in our community.

Sound good? Let's get started.
Who's talking

Top contributors to this post

What do Dropbox user levels mean?
Need more support?