Trying to get dotnet batch upload working (to solve "too many operations" issue)

[donaldp]

Hi,

   I'm saving a lot of log files from parallel process and running into "too many operations" exceptions occasionally (I already rate-limited them to 1 per second start rate to deal with a rate limit issue) and saw mention of using batch upload to address this, so I'm trying to do so but I'm missing something somewhere after reading all the doco.

   My code builds, runs, but at the end I have no files. At first it was crashing if the file didn't already exist, which I thought was odd, and my code ran when I pre-made the files, but then they were still empty at the end. I deleted them again and this time the code ran, but no files. I thought at first maybe I had something wrong with the offset - e.g. maybe wasn't uploading files because I'd prematurely set the offset to the end, so I set it to 0 instead (you'll see that in the following code), but still not working. Wondering if you could look at my code and see if you can spot what I'm doing wrong?

First my custom class for this

public class FileToBeSaved
{
public string Content {get;set;}
public string Path {get;set;}
}

This is the method I've written...

public async Task<string> WriteBatchOfFilesAsync(List<FileToBeSaved> Files)
{
string asyncJobId=string.Empty;
string statusString=string.Empty;
List<UploadSessionFinishArg> entries=new List<UploadSessionFinishArg>();
foreach (FileToBeSaved file in Files) {
    string sessionId=(await DxClient.Files.UploadSessionStartAsync(close:true,body:StringToStream(file.Content))).SessionId;
    UploadSessionFinishArg finishArg=new UploadSessionFinishArg();
    //ulong offset=(await DxClient.Files.GetMetadataAsync(new GetMetadataArg(file.Path))).AsFile.Size;
    ulong offset=0;
    UploadSessionCursor cursor=new UploadSessionCursor(sessionId,offset);
    CommitInfo commit=new CommitInfo(file.Path,WriteMode.Overwrite.Instance,false,DateTime.Now);
    entries.Add(new UploadSessionFinishArg(cursor,commit));
    }
UploadSessionFinishBatchLaunch result=null;
UploadSessionFinishBatchJobStatus status=null;
int delay=5_000;
int i=0;
bool complete=false;
try {
    result=await DxClient.Files.UploadSessionFinishBatchAsync(entries);
    while (!complete) {
        if (result.IsComplete) {
            statusString="IsComplete";
            complete=true;
        } else {
// Is either is progress or "IsOther"
            if (result.IsOther) {
                statusString="IsOther";
                complete=true;
            } else {
                asyncJobId=result.AsAsyncJobId.Value;
                statusString="InProgress";
                while (statusString=="InProgress") {
                    await Task.Delay(delay);
                    status=await DxClient.Files.UploadSessionFinishBatchCheckAsync(new PollArg(asyncJobId));
                    if (status.IsComplete) {
                        statusString="IsComplete";
                        complete=true;
                        }
                    }
                }
            }
        }
    }
catch (Exception ex) {
    statusString=$"EXCEPTION FOR {nameof(WriteBatchOfFilesAsync)} is {ex.Message}\r\n";
    DebugUtil.ConsoleWrite(debugOn,$"EXCEPTION IN SAVEFROMURLASYNC IS {ex.Message}");
    }
return statusString;
}

And this is the code I'm using to call it to test it...

List<FileToBeSaved> filesToBeSaved=new List<FileToBeSaved>();
FileToBeSaved file1=new FileToBeSaved();
file1.Content="This is file 1";
file1.Path=$"{FolderPath}File1.txt";
filesToBeSaved.Add(file1);
FileToBeSaved file2=new FileToBeSaved();
file2.Content="This is file 2";
file2.Path=$"{FolderPath}File2.txt";
filesToBeSaved.Add(file2);
FileToBeSaved file3=new FileToBeSaved();
file3.Content="This is file 3";
file3.Path=$"{FolderPath}File3.txt";
filesToBeSaved.Add(file3);
FileToBeSaved file4=new FileToBeSaved();
file4.Content="This is file 4";
file4.Path=$"{FolderPath}File4.txt";
filesToBeSaved.Add(file4);
string result=await DataService.WriteBatchOfFilesAsync(filesToBeSaved);

Any help would be appreciated.

thanks,

   Donald.

[Greg-DB]

First, for reference, if there are multiple changes at the same time in the same account or shared folder, you can run in to the 'too_many_write_operations' error, which is "lock contention". That's not explicit rate limiting, but rather a result of how Dropbox works on the back-end. This is a technical inability to make a modification in the account or shared folder at the time of the API call. This error indicates that there was simultaneous activity in the account or shared/team folder preventing your app from making the state-modifying call (e.g., adding, editing, moving, copying, sharing, or deleting files/folders) it is attempting. The simultaneous activity could be coming from your app itself, or elsewhere, e.g., from the user's desktop client. It can come from the same user, or another member of a shared folder. You can find more information about lock contention here. 

In short, to avoid this error, you should avoid making multiple concurrent state modifications and use batch endpoints where possible. That won't guarantee that you won't run in to this error though, as contention can still come from other sources, so you should also implement error handling and automatic retrying as needed.
 
I also recommend referring to the error documentation and Error Handling Guide for more information.

Looking at your code, I see you are using UploadSessionFinishBatchAsync to commit multiple files in a batch, which is good and can help avoid lock contention. Likewise though, make sure you're only running one WriteBatchOfFilesAsync at a time though (that is, effectively one UploadSessionFinishBatchAsync job at a time), otherwise they may conflict with each other.

Also, the offset for any particular upload session (that is, for any particular file that you're uploading), should be however much data you've uploaded to that upload session for that file so far. So, unless you are intending to exclusively upload empty files, you should not be setting your offset to 0 as you have in this code, nor should you be setting it to the current size of the file on the Dropbox servers, if any, as you have in the commented out version of the code. Since you're just using one UploadSessionStartAsync call per file, it would be the size of the file data you're passing to that, which you have as StringToStream(file.Content). (Also, make sure StringToStream(file.Content) is returning the data you expect it to.)

Once you do have UploadSessionFinishBatchJobStatus.Complete, you should check each UploadSessionFinishBatchResultEntry in UploadSessionFinishBatchResult.Entries, to see whether each one succeeded or failed, and why.

[donaldp]

Hi Greg,

   There are no concurrent operations - I'm running this standalone (I'm still just testing it), so it's definitely not that.

> Also, make sure StringToStream(file.Content) is returning the data you expect it to

   No, I hadn't checked that actually, so that's one thing I can look at.

> you should check each UploadSessionFinishBatchResultEntry in UploadSessionFinishBatchResult.Entries, to see whether each one succeeded or failed, and why

   Ah, yes I had missed that (a lot to take in with making this run in batches)! I thought "IsComplete" meant it worked, but that's an incorrect assumption. I'll look at those 2 things and hopefully it'll work. Thanks! 🙂

[donaldp]

Hi Greg,

> check each UploadSessionFinishBatchResultEntry in UploadSessionFinishBatchResult.Entries, to see whether each one succeeded or failed, and why

Where do I get the UploadSessionFinishBatchResult from? What do I call? I don't see it being returned by anything I'm calling, and can't see any mention of what to call to get it. https://dropbox.github.io/dropbox-sdk-dotnet/html/T_Dropbox_Api_Files_UploadSessionFinishBatchResult... doesn't say what it's called/returned by

[Greg-DB]

The UploadSessionFinishBatchCheckAsync method returns UploadSessionFinishBatchJobStatus, and once UploadSessionFinishBatchJobStatus.IsComplete is true, you can use UploadSessionFinishBatchJobStatus.Complete to access  UploadSessionFinishBatchJobStatus.Complete.Value which is UploadSessionFinishBatchResult, to access UploadSessionFinishBatchResult.Entries which is a list of UploadSessionFinishBatchResultEntry.

[donaldp]

Oh ok. I didn't realise it went deeper than "IsComplete"! Thanks! Have a good Easter Greg.

[donaldp]

Hey Greg,

   I've ALMOST got this one licked. I got down to "too many write operations", and I single-streamed UploadAsync and my batch writes through a lock file, and was working, working, working, then bam, all of a sudden got another couple of "too many write operations", and I'm like "Huh? How can that be when I'm single-streaming my writes?", so I'm guessing something else is counting as a write.

   So, could you let me know what operations count as writes? I've got UploadAsync and batch uploads already, and obviously ReadFileAsync isn't a write, but what about, for example, CreateFolderAsync? Or SaveUrlAsync? I need to know what all such "write" methods would be so that I can pipe all of them through my write lock.

thanks,

  Donald.

[Greg-DB]

Yes, any change at all will count as a write, such as adding, editing, moving, copying, sharing, or deleting files or folders, by any means (such as the Dropbox API, web site, client, etc.). That will include both CreateFolderAsync and SaveUrlAsync, but not Download.