cancel
Showing results for 
Show  only  | Search instead for 
Did you mean: 
Announcements
Back again with another edition of 'How we use Dropbox', so find out how Emma uses to-do lists to get it all done here!

Dropbox API Support & Feedback

Find help with the Dropbox API from other developers.

cancel
Showing results for 
Show  only  | Search instead for 
Did you mean: 

Issues about Chunkupload

Issues about Chunkupload

0ylanmorisson
Explorer | Level 3

I'm writing a very simple console uploading program using most of the sample code from Dropbox.net Api. But I have two issues now.

 

When the network drops and reconnects, I get an error saying "lookup_failed/incorrect_offset/". Where I go wrong?

 

Another related question is,  say I have a 5120kb file to upload and the chunk is 1024kb. So when the third chunk is uploading,  suppose the network suddently drops after uploading 500kb, then where should I set the offset to continue uploading, from offset=3072 or from offset=3072+500= 3572?

 

Here is my ChunkUpload code:

 

       private async Task ChunkUpload(DropboxClient client, string srcfile, string dstfile)
        {
            if (File.Exists(srcfile))
            {
                Console.WriteLine("\nUploading file {0} to Dropbox path: {1}\n", srcfile, dstfile);
                // Chunk size is 1024KB.
                const int chunkSize = 1024 * 1024;

                FileStream fs = new FileStream(srcfile, FileMode.Open);
                byte[] data = new byte[fs.Length];
                fs.Read(data, 0, data.Length);
                fs.Close();

                using (var stream = new MemoryStream(data))
                {
                    int numChunks = (int)Math.Ceiling((double)stream.Length / chunkSize);

                    byte[] buffer = new byte[chunkSize];
                    string sessionId = null;

                    var idx = 0;
                   do
                    {
                        var byteRead = stream.Read(buffer, 0, chunkSize);
                        var percentage = 0;
                        int row = Console.CursorTop - 1;  //Set cursor position
                        Console.SetCursorPosition(0, row);
                        if ( (idx+1) == numChunks)  //Last chunk, 99% done!
                        {
                            percentage =  99;
                            Console.WriteLine("Uploaded {0}: {1}% of total {2}kb", idx, percentage, (int)Math.Ceiling((double)stream.Length));
                        }
                        else
                        {
                            percentage = 100 * (idx + 1) * chunkSize / (int)Math.Ceiling((double)stream.Length);
                            Console.WriteLine("Uploaded {0}: {1}% of total {2}kb", idx, percentage, (int)Math.Ceiling((double)stream.Length));
                        }
                        try
                        {
                            using (MemoryStream memStream = new MemoryStream(buffer, 0, byteRead))
                            {
                                if (idx == 0)
                                {
                                    var result = await client.Files.UploadSessionStartAsync(body: memStream);
                                    sessionId = result.SessionId;
                                }

                                else
                                {
                                    UploadSessionCursor cursor = new UploadSessionCursor(sessionId, (ulong)(chunkSize * idx));

                                    if (idx == numChunks - 1)
                                    {
                                        await client.Files.UploadSessionFinishAsync(cursor, new CommitInfo(dstfile), memStream);
                                        row = Console.CursorTop - 1;
                                        Console.SetCursorPosition(0, row);
                                        Console.WriteLine("Uploaded {0}: {1}% of total {2}kb", idx, 100, (int)Math.Ceiling((double)stream.Length));
                                    }
                                    else
                                    {
                                        await client.Files.UploadSessionAppendV2Async(cursor, body: memStream);
                                    }
                                }
                            }
                            idx++;
                        }
                        catch (Exception e)
                        {
                            Console.WriteLine("Error uploading:{0}", e.Message);
                            Console.WriteLine("SessionID: {0}", sessionId);
                            Console.WriteLine("\nMake sure everything is OK then press enter. \n");
                            Console.ReadLine();
                        }
                    } while (idx < numChunks);
                }
                Console.WriteLine("Upload OK! Press enter to exit...");
                Console.ReadLine();
            }
            else
            {
                Console.WriteLine("Wrong path:{0}", srcfile);
            }
        }
1 Accepted Solution

Accepted Solutions

Re: Issues about Chunkupload

Greg-DB
Dropboxer

The documentation for UploadSessionLookupError.IncorrectOffset covers an example of how this can occur:

 

"The specified offset was incorrect. See the value for the correct offset. This error may occur when a previous request was received and processed successfully but the client did not receive the response, e.g. due to a network error."

 

To get the correct offset, you can catch and inspect the exception like this:

try {
    await client.Files.UploadSessionAppendV2Async(cursor, body: memStream);
} catch (ApiException<UploadSessionLookupError> e) {

    if (e.ErrorResponse.IsIncorrectOffset) {
        Console.WriteLine("Expected offset: " + e.ErrorResponse.AsIncorrectOffset.Value.CorrectOffset);
    }
    // todo: other error handling
    throw;
}

In your 5120kb file scenario, I believe the offset would still be 3072 at that point, since the connection failed. However that may depend on the exact nature of the failure, so you should always check the returned value anyway.

 

 

 

View solution in original post

4 Replies 4

Re: Issues about Chunkupload

Greg-DB
Dropboxer

The documentation for UploadSessionLookupError.IncorrectOffset covers an example of how this can occur:

 

"The specified offset was incorrect. See the value for the correct offset. This error may occur when a previous request was received and processed successfully but the client did not receive the response, e.g. due to a network error."

 

To get the correct offset, you can catch and inspect the exception like this:

try {
    await client.Files.UploadSessionAppendV2Async(cursor, body: memStream);
} catch (ApiException<UploadSessionLookupError> e) {

    if (e.ErrorResponse.IsIncorrectOffset) {
        Console.WriteLine("Expected offset: " + e.ErrorResponse.AsIncorrectOffset.Value.CorrectOffset);
    }
    // todo: other error handling
    throw;
}

In your 5120kb file scenario, I believe the offset would still be 3072 at that point, since the connection failed. However that may depend on the exact nature of the failure, so you should always check the returned value anyway.

 

 

 

View solution in original post

Re: Issues about Chunkupload

0ylanmorisson
Explorer | Level 3

I'm getting another related problem here. If the program accendentally exits, then how can I resume upload when it restarts?

 

I tried a solution which is,

1. record the sessionID and cursor.offset to a txt after each chunk upload;

2. every time the program starts, it will check if "record.txt" exists. If txt exists, then it means this is a resume-upload. The program should upload from last offset.

 

But here is the problem, how can I get the right MemoryStream from last offset position?

The following code 

 

 

                    if (File.Exists("record.txt"))
                    {
                        foreach (string line in File.ReadLines("record.txt"))
                        {
                            if (line.Contains("AAAA"))
                            {
                                sessionId = line;
                            }
                            else
                            {
                                currentPosition = long.Parse(line);
                            }
                        }
                        File.Delete("record.txt");
                    }

                    if (isResume)
                    {
                        byteRead = stream.Read(buffer, 0, chunkSize);   //this line doesn't work.
                    }

Re: Issues about Chunkupload

Greg-DB
Dropboxer
It sounds like you already have the right idea, in that you should store upload session ID and resume the upload from where you left off. I don't recommend relying on the session ID having "AAAA" though, as that's not guaranteed. You should store it in some format you can rely on, and use whatever session ID string the API gave you.

As far as how you interact with the local file, I'm afraid that's a bit outside the scope of Dropbox API support, as that has to do with the local file system and not Dropbox. You'll probably want to refer to the documentation for FileStream and MemoryStream to determine how to start reading from the file at a particular offset in the file.

Re: Issues about Chunkupload

0ylanmorisson
Explorer | Level 3

Alright. Thank you!

Who's talking

Top contributors to this post

  • User avatar
    0ylanmorisson Explorer | Level 3
  • User avatar
    Greg-DB Dropboxer
What do Dropbox user levels mean?
Need more support?