API v2 upload_session/append_v2 doc help

[David W.152]

Hi dropbox team,

I have no idea how to upload a larger file (>= 1GB) via upload_session/* API.

Can not find any full example code with that.

APIs:

/upload_session/start
/upload_session/append_v2
/upload_session/finish

Logs:

---- first try ---------------------------------

Step 1:
Request:
curl "/upload_session/start" file="" \
--header "Dropbox-API-Arg: {}"
Response:
status code: 200
JSON:
{
"session_id" = AAA123456;
}

Step 2:
Request:
curl "/upload_session/append_v2" file="test_file_5MB_for_upload.bin" \
--header {\"cursor\": {\"session_id\": \"AAA123456\",\"offset\": 0},\"close\": false}"
Response:
status code: 200
TEXT:
<null>
Step 3:
Request:
curl "/upload_session/finish" file="" \
--header "Dropbox-API-Arg" = "{\"commit\":{\"path\":\"\\/test_file_5MB_for_upload.bin\",\"autorename\":false,\"mute\":true,\"mode\":\"overwrite\"},\"cursor\":{\"offset\":5242880,\"session_id\":\"AAA123456\"}}"
Response:
status code: 200
JSON:
{
"client_modified" = "2016-xx-xxTxx:xx:xxZ";
id = "id:uNFAAA123456";
name = "test_file_5MB_for_upload.bin";
"path_display" = "/test_file_5MB_for_upload.bin";
"path_lower" = "/test_file_5mb_for_upload.bin";
rev = 325399abc05;
"server_modified" = "2016-xx-xxTxx:xx:xxZ";
size = 5242880;
}

---- second try ---------------------------------

Step 1:
Request:
curl "/upload_session/start" file="test_file_5MB_for_upload.bin" \
--header "Dropbox-API-Arg: {}"
Response:
status code: 200
JSON:
{
"session_id" = AAA123456;
}

Step 2:
Request:
curl "/upload_session/finish" file="" \
--header "Dropbox-API-Arg" = "{\"commit\":{\"path\":\"\\/test_file_5MB_for_upload.bin\",\"autorename\":false,\"mute\":true,\"mode\":\"overwrite\"},\"cursor\":{\"offset\":5242880,\"session_id\":\"AAA123456\"}}"
Response:
status code: 200
JSON:
{
"client_modified" = "2016-xx-xxTxx:xx:xxZ";
id = "id:uNFAAA123456";
name = "test_file_5MB_for_upload.bin";
"path_display" = "/test_file_5MB_for_upload.bin";
"path_lower" = "/test_file_5mb_for_upload.bin";
rev = 325399abc05;
"server_modified" = "2016-xx-xxTxx:xx:xxZ";
size = 5242880;
}

---- third try ---------------------------------

Step 1:
Request:
curl "/upload_session/start" file="" \
--header "Dropbox-API-Arg: {}"
Response:
status code: 200
JSON:
{
"session_id" = AAA12345678;
}

Step 2:
Request:
curl "/upload_session/append_v2" file="test_file_1GB_for_upload.bin" \
--header {\"cursor\": {\"session_id\": \"AAA12345678\",\"offset\": 0},\"close\": false}"
Response:
Response:
status code: 413
TEXT:
<DOCTYPE html>
<title>4xx</title>
...
</html>

----------------------------------------------------

Would be very grateful for any suggestion.

Thank you!

[Greg-DB]

Hi David, I unfortunately don't have a sample of using upload sessions in bash/curl directly, but here are some examples using the Python and .NET SDKs, which should serve as a useful reference for the necessary logic:

# This uses the Dropbox Python SDK https://github.com/dropbox/dropbox-sdk-python to upload a file to the Dropbox API from the local file as specified by `file_path` to the remote path as specified by `dest_path`. It also chooses whether or not to use an upload session based on the size of the file:

    f = open(file_path)
    file_size = os.path.getsize(file_path)

    CHUNK_SIZE = 4 * 1024 * 1024

    if file_size <= CHUNK_SIZE:

        print dbx.files_upload(f.read(), dest_path)

    else:

        upload_session_start_result = dbx.files_upload_session_start(f.read(CHUNK_SIZE))
        cursor = dropbox.files.UploadSessionCursor(session_id=upload_session_start_result.session_id,
                                                   offset=f.tell())
        commit = dropbox.files.CommitInfo(path=dest_path)

        while f.tell() < file_size:
            if ((file_size - f.tell()) <= CHUNK_SIZE):
                print dbx.files_upload_session_finish(f.read(CHUNK_SIZE),
                                                cursor,
                                                commit)
            else:
                dbx.files_upload_session_append_v2(f.read(CHUNK_SIZE),
                                                cursor)
                cursor.offset = f.tell()

    f.close()

// This example uses the Dropbox .NET library https://github.com/dropbox/dropbox-sdk-dotnet to upload a file to a Dropbox account, using upload sessions for larger files:

    private async Task Upload(string localPath, string remotePath)
    {
        const int ChunkSize = 4096 * 1024;
        using (var fileStream = File.Open(localPath, FileMode.Open))
        {
            if (fileStream.Length <= ChunkSize)
            {
                await this.client.Files.UploadAsync(remotePath, body: fileStream);
            }
            else
            {
                await this.ChunkUpload(remotePath, fileStream, (int)ChunkSize);
            }
        }
    }

    private async Task ChunkUpload(String path, FileStream stream, int chunkSize)
    {
        ulong numChunks = (ulong)Math.Ceiling((double)stream.Length / chunkSize);
        byte[] buffer = new byte[chunkSize];
        string sessionId = null;
        for (ulong idx = 0; idx < numChunks; idx++)
        {
            var byteRead = stream.Read(buffer, 0, chunkSize);

            using (var memStream = new MemoryStream(buffer, 0, byteRead))
            {
                if (idx == 0)
                {
                    var result = await this.client.Files.UploadSessionStartAsync(false, memStream);
                    sessionId = result.SessionId;
                }
                else
                {
                    var cursor = new UploadSessionCursor(sessionId, (ulong)chunkSize * idx);

                    if (idx == numChunks - 1)
                    {
                        FileMetadata fileMetadata = await this.client.Files.UploadSessionFinishAsync(cursor, new CommitInfo(path), memStream);
                        Console.WriteLine (fileMetadata.PathDisplay);
                    }
                    else
                    {
                        await this.client.Files.UploadSessionAppendV2Async(cursor, false, memStream);
                    }
                }
            }
        }
    }

[David W.152]

Thanks Gregory, you save my life. I'll check it later. Thank you!

[ivanhigueram]

Hi Greg! Is possible to update the SO Documentation links, they're broken now 😞 

[Greg-DB]

@ivanhigueram I've updated my earlier post to embed the code snippets from the now-defunct StackOverflow Documentation pages directly. Hope this helps! 

[ivanhigueram]

Hi @Greg-DB Thanks for replying back and point me to the code links. I recently opened another issue in the forum (https://www.dropboxforum.com/t5/API-Support-Feedback/Python-V2-How-to-do-batch-upload/m-p/329517) with my specific problem using batch upload. 

Thanks! 

[Greg-DB]

@ivanhigueram We'll review your other post and follow up there as soon we can. Thanks in advance for your patience.