Issues about Chunkupload

[0ylanmorisson]

I'm writing a very simple console uploading program using most of the sample code from Dropbox.net Api. But I have two issues now.

When the network drops and reconnects, I get an error saying "lookup_failed/incorrect_offset/". Where I go wrong?

Another related question is,  say I have a 5120kb file to upload and the chunk is 1024kb. So when the third chunk is uploading,  suppose the network suddently drops after uploading 500kb, then where should I set the offset to continue uploading, from offset=3072 or from offset=3072+500= 3572?

Here is my ChunkUpload code:

       private async Task ChunkUpload(DropboxClient client, string srcfile, string dstfile)
        {
            if (File.Exists(srcfile))
            {
                Console.WriteLine("\nUploading file {0} to Dropbox path: {1}\n", srcfile, dstfile);
                // Chunk size is 1024KB.
                const int chunkSize = 1024 * 1024;

                FileStream fs = new FileStream(srcfile, FileMode.Open);
                byte[] data = new byte[fs.Length];
                fs.Read(data, 0, data.Length);
                fs.Close();

                using (var stream = new MemoryStream(data))
                {
                    int numChunks = (int)Math.Ceiling((double)stream.Length / chunkSize);

                    byte[] buffer = new byte[chunkSize];
                    string sessionId = null;

                    var idx = 0;
                   do
                    {
                        var byteRead = stream.Read(buffer, 0, chunkSize);
                        var percentage = 0;
                        int row = Console.CursorTop - 1;  //Set cursor position
                        Console.SetCursorPosition(0, row);
                        if ( (idx+1) == numChunks)  //Last chunk, 99% done!
                        {
                            percentage =  99;
                            Console.WriteLine("Uploaded {0}: {1}% of total {2}kb", idx, percentage, (int)Math.Ceiling((double)stream.Length));
                        }
                        else
                        {
                            percentage = 100 * (idx + 1) * chunkSize / (int)Math.Ceiling((double)stream.Length);
                            Console.WriteLine("Uploaded {0}: {1}% of total {2}kb", idx, percentage, (int)Math.Ceiling((double)stream.Length));
                        }
                        try
                        {
                            using (MemoryStream memStream = new MemoryStream(buffer, 0, byteRead))
                            {
                                if (idx == 0)
                                {
                                    var result = await client.Files.UploadSessionStartAsync(body: memStream);
                                    sessionId = result.SessionId;
                                }

                                else
                                {
                                    UploadSessionCursor cursor = new UploadSessionCursor(sessionId, (ulong)(chunkSize * idx));

                                    if (idx == numChunks - 1)
                                    {
                                        await client.Files.UploadSessionFinishAsync(cursor, new CommitInfo(dstfile), memStream);
                                        row = Console.CursorTop - 1;
                                        Console.SetCursorPosition(0, row);
                                        Console.WriteLine("Uploaded {0}: {1}% of total {2}kb", idx, 100, (int)Math.Ceiling((double)stream.Length));
                                    }
                                    else
                                    {
                                        await client.Files.UploadSessionAppendV2Async(cursor, body: memStream);
                                    }
                                }
                            }
                            idx++;
                        }
                        catch (Exception e)
                        {
                            Console.WriteLine("Error uploading:{0}", e.Message);
                            Console.WriteLine("SessionID: {0}", sessionId);
                            Console.WriteLine("\nMake sure everything is OK then press enter. \n");
                            Console.ReadLine();
                        }
                    } while (idx < numChunks);
                }
                Console.WriteLine("Upload OK! Press enter to exit...");
                Console.ReadLine();
            }
            else
            {
                Console.WriteLine("Wrong path:{0}", srcfile);
            }
        }

[Greg-DB · Accepted solution]

The documentation for UploadSessionLookupError.IncorrectOffset covers an example of how this can occur:

"The specified offset was incorrect. See the value for the correct offset. This error may occur when a previous request was received and processed successfully but the client did not receive the response, e.g. due to a network error."

To get the correct offset, you can catch and inspect the exception like this:

try {
    await client.Files.UploadSessionAppendV2Async(cursor, body: memStream);
} catch (ApiException<UploadSessionLookupError> e) {

    if (e.ErrorResponse.IsIncorrectOffset) {
        Console.WriteLine("Expected offset: " + e.ErrorResponse.AsIncorrectOffset.Value.CorrectOffset);
    }
    // todo: other error handling
    throw;
}

In your 5120kb file scenario, I believe the offset would still be 3072 at that point, since the connection failed. However that may depend on the exact nature of the failure, so you should always check the returned value anyway.

[0ylanmorisson]

I'm getting another related problem here. If the program accendentally exits, then how can I resume upload when it restarts?

I tried a solution which is,

1. record the sessionID and cursor.offset to a txt after each chunk upload;

2. every time the program starts, it will check if "record.txt" exists. If txt exists, then it means this is a resume-upload. The program should upload from last offset.

But here is the problem, how can I get the right MemoryStream from last offset position?

The following code 

                    if (File.Exists("record.txt"))
                    {
                        foreach (string line in File.ReadLines("record.txt"))
                        {
                            if (line.Contains("AAAA"))
                            {
                                sessionId = line;
                            }
                            else
                            {
                                currentPosition = long.Parse(line);
                            }
                        }
                        File.Delete("record.txt");
                    }

                    if (isResume)
                    {
                        byteRead = stream.Read(buffer, 0, chunkSize);   //this line doesn't work.
                    }

[Greg-DB]

It sounds like you already have the right idea, in that you should store upload session ID and resume the upload from where you left off. I don't recommend relying on the session ID having "AAAA" though, as that's not guaranteed. You should store it in some format you can rely on, and use whatever session ID string the API gave you.

As far as how you interact with the local file, I'm afraid that's a bit outside the scope of Dropbox API support, as that has to do with the local file system and not Dropbox. You'll probably want to refer to the documentation for FileStream and MemoryStream to determine how to start reading from the file at a particular offset in the file.

[0ylanmorisson]

Alright. Thank you!