One month down in 2025: How are your resolutions coming along? Check out how to get back on track here.
Forum Discussion
donaldp
3 years agoCollaborator | Level 9
Trying to get dotnet batch upload working (to solve "too many operations" issue)
Hi,
I'm saving a lot of log files from parallel process and running into "too many operations" exceptions occasionally (I already rate-limited them to 1 per second start rate to deal with a rate limit issue) and saw mention of using batch upload to address this, so I'm trying to do so but I'm missing something somewhere after reading all the doco.
My code builds, runs, but at the end I have no files. At first it was crashing if the file didn't already exist, which I thought was odd, and my code ran when I pre-made the files, but then they were still empty at the end. I deleted them again and this time the code ran, but no files. I thought at first maybe I had something wrong with the offset - e.g. maybe wasn't uploading files because I'd prematurely set the offset to the end, so I set it to 0 instead (you'll see that in the following code), but still not working. Wondering if you could look at my code and see if you can spot what I'm doing wrong?
First my custom class for this
public class FileToBeSaved
{
public string Content {get;set;}
public string Path {get;set;}
}
This is the method I've written...
public async Task<string> WriteBatchOfFilesAsync(List<FileToBeSaved> Files)
{
string asyncJobId=string.Empty;
string statusString=string.Empty;
List<UploadSessionFinishArg> entries=new List<UploadSessionFinishArg>();
foreach (FileToBeSaved file in Files) {
string sessionId=(await DxClient.Files.UploadSessionStartAsync(close:true,body:StringToStream(file.Content))).SessionId;
UploadSessionFinishArg finishArg=new UploadSessionFinishArg();
//ulong offset=(await DxClient.Files.GetMetadataAsync(new GetMetadataArg(file.Path))).AsFile.Size;
ulong offset=0;
UploadSessionCursor cursor=new UploadSessionCursor(sessionId,offset);
CommitInfo commit=new CommitInfo(file.Path,WriteMode.Overwrite.Instance,false,DateTime.Now);
entries.Add(new UploadSessionFinishArg(cursor,commit));
}
UploadSessionFinishBatchLaunch result=null;
UploadSessionFinishBatchJobStatus status=null;
int delay=5_000;
int i=0;
bool complete=false;
try {
result=await DxClient.Files.UploadSessionFinishBatchAsync(entries);
while (!complete) {
if (result.IsComplete) {
statusString="IsComplete";
complete=true;
} else {
// Is either is progress or "IsOther"
if (result.IsOther) {
statusString="IsOther";
complete=true;
} else {
asyncJobId=result.AsAsyncJobId.Value;
statusString="InProgress";
while (statusString=="InProgress") {
await Task.Delay(delay);
status=await DxClient.Files.UploadSessionFinishBatchCheckAsync(new PollArg(asyncJobId));
if (status.IsComplete) {
statusString="IsComplete";
complete=true;
}
}
}
}
}
}
catch (Exception ex) {
statusString=$"EXCEPTION FOR {nameof(WriteBatchOfFilesAsync)} is {ex.Message}\r\n";
DebugUtil.ConsoleWrite(debugOn,$"EXCEPTION IN SAVEFROMURLASYNC IS {ex.Message}");
}
return statusString;
}
And this is the code I'm using to call it to test it...
List<FileToBeSaved> filesToBeSaved=new List<FileToBeSaved>();
FileToBeSaved file1=new FileToBeSaved();
file1.Content="This is file 1";
file1.Path=$"{FolderPath}File1.txt";
filesToBeSaved.Add(file1);
FileToBeSaved file2=new FileToBeSaved();
file2.Content="This is file 2";
file2.Path=$"{FolderPath}File2.txt";
filesToBeSaved.Add(file2);
FileToBeSaved file3=new FileToBeSaved();
file3.Content="This is file 3";
file3.Path=$"{FolderPath}File3.txt";
filesToBeSaved.Add(file3);
FileToBeSaved file4=new FileToBeSaved();
file4.Content="This is file 4";
file4.Path=$"{FolderPath}File4.txt";
filesToBeSaved.Add(file4);
string result=await DataService.WriteBatchOfFilesAsync(filesToBeSaved);
Any help would be appreciated.
thanks,
Donald.
- Greg-DB
Dropbox Staff
First, for reference, if there are multiple changes at the same time in the same account or shared folder, you can run in to the 'too_many_write_operations' error, which is "lock contention". That's not explicit rate limiting, but rather a result of how Dropbox works on the back-end. This is a technical inability to make a modification in the account or shared folder at the time of the API call. This error indicates that there was simultaneous activity in the account or shared/team folder preventing your app from making the state-modifying call (e.g., adding, editing, moving, copying, sharing, or deleting files/folders) it is attempting. The simultaneous activity could be coming from your app itself, or elsewhere, e.g., from the user's desktop client. It can come from the same user, or another member of a shared folder. You can find more information about lock contention here.
In short, to avoid this error, you should avoid making multiple concurrent state modifications and use batch endpoints where possible. That won't guarantee that you won't run in to this error though, as contention can still come from other sources, so you should also implement error handling and automatic retrying as needed.
I also recommend referring to the error documentation and Error Handling Guide for more information.Looking at your code, I see you are using UploadSessionFinishBatchAsync to commit multiple files in a batch, which is good and can help avoid lock contention. Likewise though, make sure you're only running one WriteBatchOfFilesAsync at a time though (that is, effectively one UploadSessionFinishBatchAsync job at a time), otherwise they may conflict with each other.
Also, the offset for any particular upload session (that is, for any particular file that you're uploading), should be however much data you've uploaded to that upload session for that file so far. So, unless you are intending to exclusively upload empty files, you should not be setting your offset to 0 as you have in this code, nor should you be setting it to the current size of the file on the Dropbox servers, if any, as you have in the commented out version of the code. Since you're just using one UploadSessionStartAsync call per file, it would be the size of the file data you're passing to that, which you have as StringToStream(file.Content). (Also, make sure StringToStream(file.Content) is returning the data you expect it to.)
Once you do have UploadSessionFinishBatchJobStatus.Complete, you should check each UploadSessionFinishBatchResultEntry in UploadSessionFinishBatchResult.Entries, to see whether each one succeeded or failed, and why.
- donaldpCollaborator | Level 9
Hi Greg,
There are no concurrent operations - I'm running this standalone (I'm still just testing it), so it's definitely not that.
> Also, make sure StringToStream(file.Content) is returning the data you expect it to
No, I hadn't checked that actually, so that's one thing I can look at.
> you should check each UploadSessionFinishBatchResultEntry in UploadSessionFinishBatchResult.Entries, to see whether each one succeeded or failed, and why
Ah, yes I had missed that (a lot to take in with making this run in batches)! I thought "IsComplete" meant it worked, but that's an incorrect assumption. I'll look at those 2 things and hopefully it'll work. Thanks! 🙂 - donaldpCollaborator | Level 9
Hi Greg,
> check each UploadSessionFinishBatchResultEntry in UploadSessionFinishBatchResult.Entries, to see whether each one succeeded or failed, and why
Where do I get the UploadSessionFinishBatchResult from? What do I call? I don't see it being returned by anything I'm calling, and can't see any mention of what to call to get it. https://dropbox.github.io/dropbox-sdk-dotnet/html/T_Dropbox_Api_Files_UploadSessionFinishBatchResultEntry.htm doesn't say what it's called/returned by
- Greg-DB
Dropbox Staff
The UploadSessionFinishBatchCheckAsync method returns UploadSessionFinishBatchJobStatus, and once UploadSessionFinishBatchJobStatus.IsComplete is true, you can use UploadSessionFinishBatchJobStatus.Complete to access UploadSessionFinishBatchJobStatus.Complete.Value which is UploadSessionFinishBatchResult, to access UploadSessionFinishBatchResult.Entries which is a list of UploadSessionFinishBatchResultEntry.
About Dropbox API Support & Feedback
Find help with the Dropbox API from other developers.5,942 PostsLatest Activity: 55 minutes ago
If you need more help you can view your support options (expected response time for an email or ticket is 24 hours), or contact us on X or Facebook.
For more info on available support options for your Dropbox plan, see this article.
If you found the answer to your question in this Community thread, please 'like' the post to say thanks and to let us know it was useful!