Need to see if your shared folder is taking up space on your dropbox 👨💻? Find out how to check here.
Forum Discussion
Engg
2 years agoExplorer | Level 3
System.Threading.Tasks.TaskCanceledException: A task was canceled.
We have 500 machine which upload file at same time. We are getting RateLimitException. We are doing a retry after waiting rateLimitException.ErrorResponse.RetryAfter seconds. to test this. I came ...
- 2 years ago
In general modifying the folder should not affect probability for exception appearing, but cannot prevent it. Some decrement may happen if different files/folders appear in different namespaces (namespaces are locked independently, so less likely compete), but better don't rely on.
Again, try to schedule the uploads in non overlap way and handle the possible exception in proper way. Example of such exception handling (again just example and far from optimal) follows:
private readonly int retryLimit = 20; ... var backoff = 0; var rand = new Random(); var commitInfo = new CommitInfo(filePath.Substring(filePath.LastIndexOf('/')), WriteMode.Overwrite.Instance, false); for (var i = 0; i < retryLimit && !string.IsNullOrEmpty(sessionId); ++i) { if (backoff != 0) { Console.WriteLine("Ограничение при завършване на файл {0}. Нов опит след {1} милисекунди...", commitInfo.Path, backoff); Thread.Sleep(backoff); } lastCursor = new UploadSessionCursor(sessionId, (ulong)fileStream.Length); memStream = new MemoryStream(buffer, 0, 0); try { await dbx.Files.UploadSessionFinishAsync(lastCursor, commitInfo, body: memStream); sessionId = string.Empty; } catch (RateLimitException rateLimit) { backoff = (int)(backoff * 1.1); backoff = Math.Max(backoff, rateLimit.RetryAfter * 1000); backoff = Math.Max(backoff, 1000); backoff += rand.Next(500); } } if (string.IsNullOrEmpty(sessionId)) { Console.WriteLine("Качването на {0} приключи \ud83d\ude09.", commitInfo.Path); } else { Console.WriteLine("Неуспешно завършване на файл {0} след {1} опита \ud83d\ude15!", commitInfo.Path, retryLimit); }The above just shows how my advices (from previous page) can be implemented. Here sessionId represents a session ready for finishing (i.e. content uploaded already).
Здравко
2 years agoLegendary | Level 20
Engg wrote:... Because in all the Call we need Stream how to get the remaining Stream without uploading it again.
Hm..🤔🤷 What kind of "remaining Stream" do you need at all and why? Honestly, I cannot understand you.
Probably you don't understand how Dropbox upload works in general and that's what confuses you. In all cases (either using explicitly upload session or using simplified upload) the actual upload is performed as from your stream (either provided in one step for small files or in many steps for large files) to a POST HTTP request body gets formed, which fills (at once or step by step) the upload session body (residing on Dropbox server and represented with session id, that you may have or not). Once upload session gets filled upload may be finished (gets associated with file name and path) to users account. 😉 That's it. This flow happens even when you just use simple upload - then it just happens in one call and you don't have access to session id (you don't need it).
No one file can be uploaded directly to particular place in users account (even though such your use of 'UploadAsync' builds such an illusion)! Upload always happens to something like temporary file - upload session! At the end, this session' content can be copied to desired place in users account - upload finishing. About expected streams - all this is option and such a stream always can be empty.
Some example, showing all explained, follows:
private readonly long CHUNK_SIZE = 4 * 1024 * 1024;
private async Task UploadSessionAppenV2Async(DropboxClient dbx, UploadSessionCursor cursor, byte[] buffer, int bytesRead, bool close = false)
{
try
{
var memStream = new MemoryStream(buffer, 0, bytesRead);
await dbx.Files.UploadSessionAppendV2Async(cursor, close, null, memStream);
}
catch(Exception ex)
{
Console.WriteLine("Error while append at offset {0}: {1}", cursor.Offset, ex);
}
}
async Task TestBigUpload(DropboxClient dbx, string filePath)
{
Console.WriteLine("Започва сесия за качване на {0} като {1}", filePath, filePath.Substring(filePath.LastIndexOf('/')));
var fileStream = File.OpenRead(filePath);
Console.WriteLine("Размерът на файла е {0} байта.", fileStream.Length);
var numChunks = (int)Math.Ceiling(((double)fileStream.Length) / CHUNK_SIZE);
Console.WriteLine("Ще се качва на {0} части.", numChunks);
var buffer = new byte[CHUNK_SIZE];
var memStream = new MemoryStream(buffer, 0, 0);
var sessionId = (await dbx.Files.UploadSessionStartAsync(body: memStream)).SessionId;
Console.WriteLine("Отвори се сессия с идентификатор: {0}", sessionId);
for (var i = 0; i < numChunks - 1; ++i)
{
var bytesRead = fileStream.Read(buffer, 0, (int)CHUNK_SIZE);
var cursor = new UploadSessionCursor(sessionId, (ulong)(i * CHUNK_SIZE));
Console.WriteLine("Качва се блок с размер {0} на позиция {1}", bytesRead, i * CHUNK_SIZE);
await UploadSessionAppenV2Async(dbx, cursor, buffer, bytesRead);
}
var bytesReaded = fileStream.Read(buffer, 0, (int)CHUNK_SIZE);
Console.WriteLine("Последият прочетен блок е с размер {0}", bytesReaded);
var lastCursor = new UploadSessionCursor(sessionId, (ulong)((numChunks - 1) * CHUNK_SIZE));
await UploadSessionAppenV2Async(dbx, lastCursor, buffer, bytesReaded, true);
lastCursor = new UploadSessionCursor(sessionId, (ulong)fileStream.Length);
var commitInfo = new CommitInfo(filePath.Substring(filePath.LastIndexOf('/')), WriteMode.Overwrite.Instance, false);
memStream = new MemoryStream(buffer, 0, 0);
await dbx.Files.UploadSessionFinishAsync(lastCursor, commitInfo, body: memStream);
Console.WriteLine("Качването на {0} приключи.", filePath.Substring(filePath.LastIndexOf('/')));
Console.WriteLine("Започва приключване и на неговото копие {0} \ud83d\ude09...", "/testSubfolder" + filePath.Substring(filePath.LastIndexOf('/')));
lastCursor = new UploadSessionCursor(sessionId, (ulong)fileStream.Length);
commitInfo = new CommitInfo("/testSubfolder" + filePath.Substring(filePath.LastIndexOf('/')), WriteMode.Overwrite.Instance, false);
memStream = new MemoryStream(buffer, 0, 0);
await dbx.Files.UploadSessionFinishAsync(lastCursor, commitInfo, body: memStream);
}This code can upload file of any size. In some places empty streams are used. Just call 'TestBigUpload' from your code to see the result. The temporary file will appear as 2 copies - showing how the session can be used later again without real stream (including in exception handler - not used here). 🙂
Hope this sheds some more light.
PS: The code is just example and is far from optimal - implement it in your code better.
Engg
2 years agoExplorer | Level 3
Hey Здравко ,
Thank you for sharing the code to upload large files. I appreciate your help.
I've been utilizing the provided code for uploading large files, and it's been working smoothly. However, I have a question regarding potential exceptions.
Suppose I attempt to upload files with the same parent folder but different subfolders,
like the structure below:
machine1
- testfolder/machine1
machine2
- testfolder/machine2
This pattern extends to 500 machines.
Could modifying the subfolder of the same parent folder lead to exceptions, specifically something like Dropbox.Api.RateLimitException: too_many_requests/...?
Note : These uploads are initiated concurrently from different machine.
Your insights would be greatly appreciated.
- Здравко2 years agoLegendary | Level 20
In general modifying the folder should not affect probability for exception appearing, but cannot prevent it. Some decrement may happen if different files/folders appear in different namespaces (namespaces are locked independently, so less likely compete), but better don't rely on.
Again, try to schedule the uploads in non overlap way and handle the possible exception in proper way. Example of such exception handling (again just example and far from optimal) follows:
private readonly int retryLimit = 20; ... var backoff = 0; var rand = new Random(); var commitInfo = new CommitInfo(filePath.Substring(filePath.LastIndexOf('/')), WriteMode.Overwrite.Instance, false); for (var i = 0; i < retryLimit && !string.IsNullOrEmpty(sessionId); ++i) { if (backoff != 0) { Console.WriteLine("Ограничение при завършване на файл {0}. Нов опит след {1} милисекунди...", commitInfo.Path, backoff); Thread.Sleep(backoff); } lastCursor = new UploadSessionCursor(sessionId, (ulong)fileStream.Length); memStream = new MemoryStream(buffer, 0, 0); try { await dbx.Files.UploadSessionFinishAsync(lastCursor, commitInfo, body: memStream); sessionId = string.Empty; } catch (RateLimitException rateLimit) { backoff = (int)(backoff * 1.1); backoff = Math.Max(backoff, rateLimit.RetryAfter * 1000); backoff = Math.Max(backoff, 1000); backoff += rand.Next(500); } } if (string.IsNullOrEmpty(sessionId)) { Console.WriteLine("Качването на {0} приключи \ud83d\ude09.", commitInfo.Path); } else { Console.WriteLine("Неуспешно завършване на файл {0} след {1} опита \ud83d\ude15!", commitInfo.Path, retryLimit); }The above just shows how my advices (from previous page) can be implemented. Here sessionId represents a session ready for finishing (i.e. content uploaded already).
About Dropbox API Support & Feedback
Find help with the Dropbox API from other developers.
The Dropbox Community team is active from Monday to Friday. We try to respond to you as soon as we can, usually within 2 hours.
If you need more help you can view your support options (expected response time for an email or ticket is 24 hours), or contact us on X, Facebook or Instagram.
For more info on available support options for your Dropbox plan, see this article.
If you found the answer to your question in this Community thread, please 'like' the post to say thanks and to let us know it was useful!