cancel
Showing results for 
Show  only  | Search instead for 
Did you mean: 
Announcements
Back again with another edition of 'How we use Dropbox', so find out how Emma uses to-do lists to get it all done here!

Dropbox API Support & Feedback

Find help with the Dropbox API from other developers.

cancel
Showing results for 
Show  only  | Search instead for 
Did you mean: 

Dropbox chunks upload slow & unpredictable

Dropbox chunks upload slow & unpredictable

DS6
Explorer | Level 4

I am trying to upload files larger than 150 MB using Dropbox API v2 on iOS. Problem is upload is sometimes slow, sometimes it hangs in the middle for several seconds, sometimes it stalls in the middle forever without failing or invoking the failure block. Using chunk sizes less than 50 MB makes it more reliable but chunk size like 100 MB throws all the issues. This happens during tests so I believe in production users might see more issues.

 

Here is the sample code I use for uploading. I am also looking for a way to continue these uploads in background. Please help.

 

 

#define DB_CHUNK_SIZE (50*1024.0*1024.0)

//Dropbox specific

@property DBUploadTask<DBFILESUploadSessionStartResult *, DBNilObject *> * uploadTask;

@property DBUploadTask<DBNilObject *, DBFILESUploadSessionLookupError *> * continueTask;

@property DBUploadTask<DBFILESFileMetadata *, DBFILESUploadSessionFinishError *> *finishTask;

@property unsigned long long offset;

@property NSFileHandle *fileHandle;

@property NSNumber *fileSize;

@property NSString *sessionId;

 DBUserClient *client = [DBClientsManager authorizedClient];
        _fileHandle = [NSFileHandle fileHandleForReadingAtPath:[url path]];
        Float64 chunkSize = DB_CHUNK_SIZE;//50 MB chunk size
        
        NSNumber *fileSizeNumber = [fileAttributes objectForKey:NSFileSize];
        _fileSize = fileSizeNumber;
        NSData *data = [_fileHandle readDataOfLength:chunkSize];
        
        LibraryViewController *__weak wSelf = self;
        _uploadTask = [client.filesRoutes uploadSessionStartData:data];
        
        [_uploadTask setResponseBlock:^(DBFILESUploadSessionStartResult * _Nullable result, DBNilObject * _Nullable routeError, DBRequestError * _Nullable networkError) {
            if (result) {
                NSLog(@"%@\n", result);
                wSelf.sessionId = result.sessionId;
                DBFILESUploadSessionCursor *cursor = [[DBFILESUploadSessionCursor alloc] initWithSessionId:result.sessionId offset:[NSNumber numberWithUnsignedLongLong:wSelf.fileHandle.offsetInFile]];
                [wSelf uploadNextChunk:cursor atURL:url];
            } else {
                NSLog(@"%@\n%@\n", routeError, networkError);
                [wSelf.fileHandle closeFile];
                dispatch_async(dispatch_get_main_queue(), ^{
                    [wSelf cancelUpload:nil];
                });
                
            }
        }];
        
        [_uploadTask setProgressBlock:^(int64_t bytesWritten, int64_t totalBytesWritten, int64_t totalBytesExpectedToWrite) {
         //    NSLog(@"\n%lld\n%lld\n%lld\n", bytesWritten, totalBytesWritten, totalBytesExpectedToWrite);
            wSelf.progress.progress = totalBytesWritten*1.f/wSelf.fileSize.longLongValue;
        }];


- (void)uploadNextChunk:(DBFILESUploadSessionCursor *)cursor atURL:(NSURL *)url
{
    NSData *data;
    
    DBUserClient *client = [DBClientsManager authorizedClient];
    unsigned long long offset = _fileHandle.offsetInFile;
    
    if (_fileSize.longLongValue - _fileHandle.offsetInFile > DB_CHUNK_SIZE) {
        data = [_fileHandle readDataOfLength:DB_CHUNK_SIZE];
    } else {
        data = [_fileHandle readDataOfLength:(_fileSize.longLongValue - _fileHandle.offsetInFile)];
    }
    
    LibraryViewController *__weak weakSelf = self;
    
    NSString *path = [NSString stringWithFormat:@"/%@", [[url path] lastPathComponent]];
    
    if (data.length < DB_CHUNK_SIZE) {
        DBFILESCommitInfo *info = [[DBFILESCommitInfo alloc] initWithPath:path];
        
        _finishTask = [client.filesRoutes uploadSessionFinishData:cursor commit:info inputData:data];
        [_finishTask setResponseBlock:^(DBFILESFileMetadata * _Nullable result, DBFILESUploadSessionFinishError * _Nullable routeError, DBRequestError * _Nullable networkError) {
            if (result) {
              NSLog(@"Finished uploading: %@", result);
            } else {
                NSLog(@"finishError last chunk: %@", routeError);
                NSLog(@"error: %@", networkError);
            }
            
            dispatch_async(dispatch_get_main_queue(), ^{
                [weakSelf cancelUpload:nil];
            });
        }];
        
        return;
    }
    
    _continueTask = [client.filesRoutes uploadSessionAppendV2Data:cursor inputData:data];
    [_continueTask setResponseBlock:^(DBNilObject * _Nullable result, DBFILESUploadSessionLookupError * _Nullable routeError, DBRequestError * _Nullable networkError) {
        if (result) {
            NSLog(@"Result, uploading chunk %@", result);
            DBFILESUploadSessionCursor *cursor = [[DBFILESUploadSessionCursor alloc] initWithSessionId:weakSelf.sessionId offset:[NSNumber numberWithUnsignedLongLong:weakSelf.fileHandle.offsetInFile]];
            [weakSelf uploadNextChunk:cursor atURL:url];
        } else {
            NSLog(@"finishError: %@", routeError);
            NSLog(@"error: %@", networkError);
            [weakSelf.fileHandle closeFile];
            weakSelf.fileHandle = nil;
            [weakSelf cancelUpload:nil];
            
        }
    }];
    
    [_continueTask setProgressBlock:^(int64_t bytesWritten, int64_t totalBytesWritten, int64_t totalBytesExpectedToWrite) {
       // NSLog(@"Continue: bytesWritten = %lld, total bytes written %lld", bytesWritten, totalBytesWritten);
        dispatch_async(dispatch_get_main_queue(), ^{
            weakSelf.progress.progress = ((totalBytesWritten + offset)*1.f/weakSelf.fileSize.longLongValue);
        });
    }];
}

 

4 Replies 4

Re: Dropbox chunks upload slow & unpredictable

Greg-DB
Dropboxer
It sounds like you're running in to general network reliability issues. While the API endpoints can technically accept up to 150 MB per request, we generally don't reccomend doing that much, as the connections can become unreliable due to various other factors.

Using a much smaller chunk size is usually preferable, e.g., 8 MB. I recommend trying a few different sizes to see what's optimal for your scenario.

Re: Dropbox chunks upload slow & unpredictable

DS6
Explorer | Level 4

I will try to conduct test with 8 MB chunks, but I see the issue with chunked approach is it doesn't work in background on iOS. Is there any approach to get the upload working in background?

Re: Dropbox chunks upload slow & unpredictable

Greg-DB
Dropboxer
There's some information about background tasks here:

https://github.com/dropbox/dropbox-sdk-obj-c#note-about-background-sessions

I don't believe there's a great solution though, as that applies to individual upload requests. Wrapping the upload session calls in a background task will get you a couple of minutes, but it won't fail gracefully.

Re: Dropbox chunks upload slow & unpredictable

tarasheth007gma
Explorer | Level 4

@Greg-DB wrote:
There's some information about background tasks here:

https://github.com/dropbox/dropbox-sdk-obj-c#note-about-background-sessions

I don't believe there's a great solution though, as that applies to individual upload requests. Wrapping the upload session calls in a background task will get you a couple of minutes, but it won't fail gracefully.

 

Poll
We love to learn from the educators who use Dropbox. Whether you teach kids, teens, adults or a combination of all three, we want to know what apps and integrations you use with Dropbox to help with teaching. Which of the ones below is your favorite, or most used tool?
Who's talking

Top contributors to this post

  • User avatar
    tarasheth007gma Explorer | Level 4
  • User avatar
    Greg-DB Dropboxer
  • User avatar
    DS6 Explorer | Level 4
What do Dropbox user levels mean?
Need more support?