Forum Discussion

DS6's avatar
DS6
Explorer | Level 4
8 years ago

Dropbox chunks upload slow & unpredictable

I am trying to upload files larger than 150 MB using Dropbox API v2 on iOS. Problem is upload is sometimes slow, sometimes it hangs in the middle for several seconds, sometimes it stalls in the middle forever without failing or invoking the failure block. Using chunk sizes less than 50 MB makes it more reliable but chunk size like 100 MB throws all the issues. This happens during tests so I believe in production users might see more issues.

 

Here is the sample code I use for uploading. I am also looking for a way to continue these uploads in background. Please help.

 

 

#define DB_CHUNK_SIZE (50*1024.0*1024.0)

//Dropbox specific

@property DBUploadTask<DBFILESUploadSessionStartResult *, DBNilObject *> * uploadTask;

@property DBUploadTask<DBNilObject *, DBFILESUploadSessionLookupError *> * continueTask;

@property DBUploadTask<DBFILESFileMetadata *, DBFILESUploadSessionFinishError *> *finishTask;

@property unsigned long long offset;

@property NSFileHandle *fileHandle;

@property NSNumber *fileSize;

@property NSString *sessionId;

 DBUserClient *client = [DBClientsManager authorizedClient];
        _fileHandle = [NSFileHandle fileHandleForReadingAtPath:[url path]];
        Float64 chunkSize = DB_CHUNK_SIZE;//50 MB chunk size
        
        NSNumber *fileSizeNumber = [fileAttributes objectForKey:NSFileSize];
        _fileSize = fileSizeNumber;
        NSData *data = [_fileHandle readDataOfLength:chunkSize];
        
        LibraryViewController *__weak wSelf = self;
        _uploadTask = [client.filesRoutes uploadSessionStartData:data];
        
        [_uploadTask setResponseBlock:^(DBFILESUploadSessionStartResult * _Nullable result, DBNilObject * _Nullable routeError, DBRequestError * _Nullable networkError) {
            if (result) {
                NSLog(@"%@\n", result);
                wSelf.sessionId = result.sessionId;
                DBFILESUploadSessionCursor *cursor = [[DBFILESUploadSessionCursor alloc] initWithSessionId:result.sessionId offset:[NSNumber numberWithUnsignedLongLong:wSelf.fileHandle.offsetInFile]];
                [wSelf uploadNextChunk:cursor atURL:url];
            } else {
                NSLog(@"%@\n%@\n", routeError, networkError);
                [wSelf.fileHandle closeFile];
                dispatch_async(dispatch_get_main_queue(), ^{
                    [wSelf cancelUpload:nil];
                });
                
            }
        }];
        
        [_uploadTask setProgressBlock:^(int64_t bytesWritten, int64_t totalBytesWritten, int64_t totalBytesExpectedToWrite) {
         //    NSLog(@"\n%lld\n%lld\n%lld\n", bytesWritten, totalBytesWritten, totalBytesExpectedToWrite);
            wSelf.progress.progress = totalBytesWritten*1.f/wSelf.fileSize.longLongValue;
        }];


- (void)uploadNextChunk:(DBFILESUploadSessionCursor *)cursor atURL:(NSURL *)url
{
    NSData *data;
    
    DBUserClient *client = [DBClientsManager authorizedClient];
    unsigned long long offset = _fileHandle.offsetInFile;
    
    if (_fileSize.longLongValue - _fileHandle.offsetInFile > DB_CHUNK_SIZE) {
        data = [_fileHandle readDataOfLength:DB_CHUNK_SIZE];
    } else {
        data = [_fileHandle readDataOfLength:(_fileSize.longLongValue - _fileHandle.offsetInFile)];
    }
    
    LibraryViewController *__weak weakSelf = self;
    
    NSString *path = [NSString stringWithFormat:@"/%@", [[url path] lastPathComponent]];
    
    if (data.length < DB_CHUNK_SIZE) {
        DBFILESCommitInfo *info = [[DBFILESCommitInfo alloc] initWithPath:path];
        
        _finishTask = [client.filesRoutes uploadSessionFinishData:cursor commit:info inputData:data];
        [_finishTask setResponseBlock:^(DBFILESFileMetadata * _Nullable result, DBFILESUploadSessionFinishError * _Nullable routeError, DBRequestError * _Nullable networkError) {
            if (result) {
              NSLog(@"Finished uploading: %@", result);
            } else {
                NSLog(@"finishError last chunk: %@", routeError);
                NSLog(@"error: %@", networkError);
            }
            
            dispatch_async(dispatch_get_main_queue(), ^{
                [weakSelf cancelUpload:nil];
            });
        }];
        
        return;
    }
    
    _continueTask = [client.filesRoutes uploadSessionAppendV2Data:cursor inputData:data];
    [_continueTask setResponseBlock:^(DBNilObject * _Nullable result, DBFILESUploadSessionLookupError * _Nullable routeError, DBRequestError * _Nullable networkError) {
        if (result) {
            NSLog(@"Result, uploading chunk %@", result);
            DBFILESUploadSessionCursor *cursor = [[DBFILESUploadSessionCursor alloc] initWithSessionId:weakSelf.sessionId offset:[NSNumber numberWithUnsignedLongLong:weakSelf.fileHandle.offsetInFile]];
            [weakSelf uploadNextChunk:cursor atURL:url];
        } else {
            NSLog(@"finishError: %@", routeError);
            NSLog(@"error: %@", networkError);
            [weakSelf.fileHandle closeFile];
            weakSelf.fileHandle = nil;
            [weakSelf cancelUpload:nil];
            
        }
    }];
    
    [_continueTask setProgressBlock:^(int64_t bytesWritten, int64_t totalBytesWritten, int64_t totalBytesExpectedToWrite) {
       // NSLog(@"Continue: bytesWritten = %lld, total bytes written %lld", bytesWritten, totalBytesWritten);
        dispatch_async(dispatch_get_main_queue(), ^{
            weakSelf.progress.progress = ((totalBytesWritten + offset)*1.f/weakSelf.fileSize.longLongValue);
        });
    }];
}

 

  • Greg-DB's avatar
    Greg-DB
    Icon for Dropbox Staff rankDropbox Staff
    It sounds like you're running in to general network reliability issues. While the API endpoints can technically accept up to 150 MB per request, we generally don't reccomend doing that much, as the connections can become unreliable due to various other factors.

    Using a much smaller chunk size is usually preferable, e.g., 8 MB. I recommend trying a few different sizes to see what's optimal for your scenario.
    • DS6's avatar
      DS6
      Explorer | Level 4

      I will try to conduct test with 8 MB chunks, but I see the issue with chunked approach is it doesn't work in background on iOS. Is there any approach to get the upload working in background?

About Dropbox API Support & Feedback

Node avatar for Dropbox API Support & Feedback
Find help with the Dropbox API from other developers.5,941 PostsLatest Activity: 20 hours ago
351 Following

If you need more help you can view your support options (expected response time for an email or ticket is 24 hours), or contact us on X or Facebook.

For more info on available support options for your Dropbox plan, see this article.

If you found the answer to your question in this Community thread, please 'like' the post to say thanks and to let us know it was useful!