Uploading big file using Objective-C API V2 very SLOW...

Hi Logan, are you using a chunk size of 32768 bytes? You should probably use something much larger than that, at least 4 MB, or even significantly larger than that.

The optimal size depends on a variety of factors, so you may want to try a few different ones and see what works best for your app. Using a small chunk size can be useful, as it minimizes how much data you need to re-upload for failed requests, and reduces the chance of any particular request of failing, but it causes more overhead. A larger chunk size can reduce the overhead, which can improve overall transfer speed.

1. The chunk size should just be the size of the data you supply for any given call, and that's what should get reported back in the progress callback, so it's unexpected that you'd still get 32768. Can you share the rest of the relevant code, including the definition of length, the progresc callback definitifion, etc.?

2. That also doesn't sound like the expected behavior. If you share the rest of the code and the steps to reproduce it we'll be happy to look into it.

// My code can be summarized as follows:

// sorry for the indentation, it seems the code indentation is gone.

const NSUInteger DB_CHUNK_SIZE = 5*1024*1024;

DropboxClient *client = [DropboxClientsManager authorizedClient];

// create dropboxObj to pass parameters
TGDropboxObject *dropboxObj = [[TGDropboxObject alloc] init];
dropboxObj.progressBlock = progress;
dropboxObj.completionBlock = completion;
dropboxObj.fileSize = [self getFileSize:localPath fromInternal:fromInternal];
dropboxObj.chunkSize = DB_CHUNK_SIZE;
dropboxObj.curTotalBytes = 0;
dropboxObj.srcPath = localPath;
dropboxObj.dstPath = remotePath;
dropboxObj.fromInternal = fromInternal;
if (fromInternal) {
dropboxObj.fileHandle = [NSFileHandle fileHandleForReadingAtPath:localPath];
}
// skip……

NSData *data = [self readDataOfLength:dropboxObj.chunkSize fromObj:dropboxObj];// read one chunk of data

DBUploadTask<DBFILESUploadSessionStartResult *, DBNilObject *> *uploadTask;

uploadTask = [client.filesRoutes uploadSessionStartData:data];
[uploadTask response:^(DBFILESUploadSessionStartResult * _Nullable result, DBNilObject * _Nullable obj, DBRequestError * _Nullable requestError) {

if (result) { // success for the first chunk
dropboxObj.sessionId = result.sessionId;

DBFILESUploadSessionCursor *cursor = [[DBFILESUploadSessionCursor alloc] initWithSessionId:result.sessionId offset:[NSNumber numberWithUnsignedLongLong:[self getOffsetFromObj:dropboxObj]]];

[self uploadNextFromObj:dropboxObj cursor:cursor];// do upload for subsequent chunks
}
else { // failure
[self closeFileFromObj:dropboxObj];

// skip…

dropboxObj.completionBlock(nil, error);
}
}];

// progress block for uploadTask
[uploadTask progress:^(int64_t bytesWritten, int64_t totalBytesWritten, int64_t totalBytesExpectedToWrite) {
dropboxObj.curTotalBytes += bytesWritten;
CGFloat progress = (CGFloat)dropboxObj.curTotalBytes / (CGFloat)dropboxObj.fileSize;
NSLog(@"start: bytesWritten = %lld progress = %.4f", bytesWritten, progress);
dropboxObj.progressBlock(progress);
}];

// The method for uploading subsequent chunks:

//-------------------------------------------------------
- (void)uploadNextFromObj:(TGDropboxObject *)dropboxObj cursor:(DBFILESUploadSessionCursor *)cursor {

DropboxClient *client = [DropboxClientsManager authorizedClient];
NSData *const chunk = [self readDataOfLength:dropboxObj.chunkSize fromObj:dropboxObj];
const BOOL isLastChunk = chunk.length < dropboxObj.chunkSize;

if (isLastChunk == NO) {
DBUploadTask<DBNilObject *, DBFILESUploadSessionLookupError *> *continueTask;

continueTask = [client.filesRoutes uploadSessionAppendV2Data:cursor inputData:chunk];
[continueTask response:^(DBNilObject * _Nullable obj, DBFILESUploadSessionLookupError * _Nullable lookupError, DBRequestError * _Nullable requestError) {

if (lookupError == nil) { // success
DBFILESUploadSessionCursor *cursor = [[DBFILESUploadSessionCursor alloc] initWithSessionId:dropboxObj.sessionId offset:[NSNumber numberWithUnsignedLongLong:[self getOffsetFromObj:dropboxObj]]];

[self uploadNextFromObj:dropboxObj cursor:cursor];

} else { // failure
[self closeFileFromObj:dropboxObj];

// skip…

dropboxObj.completionBlock(nil, error);
}
}];

// progress block for continueTask
[continueTask progress:^(int64_t bytesWritten, int64_t totalBytesWritten, int64_t totalBytesExpectedToWrite) {
dropboxObj.curTotalBytes += bytesWritten;
CGFloat progress = (CGFloat)dropboxObj.curTotalBytes / (CGFloat)dropboxObj.fileSize;
NSLog(@"continue: bytesWritten = %lld progress = %.4f", bytesWritten, progress);
dropboxObj.progressBlock(progress);
}];
}
else { // last chunk
DBUploadTask<DBFILESFileMetadata *, DBFILESUploadSessionFinishError *> *finishTask;
DBFILESCommitInfo *info = [[DBFILESCommitInfo alloc] initWithPath:dropboxObj.dstPath];

finishTask = [client.filesRoutes uploadSessionFinishData:cursor commit:info inputData:chunk];
[finishTask response:^(DBFILESFileMetadata * _Nullable metaData, DBFILESUploadSessionFinishError * _Nullable finishError, DBRequestError * _Nullable requestError) {
DLog(@"metaData: %@", metaData);
[self closeFileFromObj:dropboxObj];

// skip…

dropboxObj.completionBlock(metaData, error);
}];
[finishTask progress:^(int64_t bytesWritten, int64_t totalBytesWritten, int64_t totalBytesExpectedToWrite) {
dropboxObj.curTotalBytes += bytesWritten;
CGFloat progress = (CGFloat)dropboxObj.curTotalBytes / (CGFloat)dropboxObj.fileSize;
NSLog(@"finish: bytesWritten = %lld progress = %.4f", bytesWritten, progress);
dropboxObj.progressBlock(progress);
}];
}
}

// If there is anything unclear, just tell me, I can provide further infomation.

You cannot run the code without our external device. But the following code does the same thing and produces the the same result.

You need to provide the source file to be uploaded. The destination file is hard coded (/temp/test.mp4), you can modify it to another file path. The unused parameters in the API uploadInternal:files:index:fromDir:toDropboxDir:opType:resArray: can be ignored.

If there is anything unclear, feel free to point out.

const NSUInteger DB_BLOCK_SIZE = 5*1024*1024; // 102400000;

@interface DestVC ()

@property DBUploadTask<DBFILESUploadSessionStartResult *, DBNilObject *> * uploadTask;

@property DBUploadTask<DBNilObject *, DBFILESUploadSessionLookupError *> * continueTask;

@property DBUploadTask<DBFILESFileMetadata *, DBFILESUploadSessionFinishError *> *finishTask;

@property unsigned long long offset;

@property NSFileHandle *file;

@property NSData *data;

@property NSNumber *fileSize;

@property DropboxClient *client;

@property NSString *sessionId;

@end

@implementation DestVC

- (void)uploadInternal:(NSArray *)items files:(NSArray *)files index:(int)index fromDir:(NSString *)fromDir toDropboxDir:(NSString *)toDir opType:(OperationType)opType resArray:(NSMutableArray *)resArray {

    NSString *file = [files objectAtIndex:index];

    NSString *srcPath = [fromDir stringByAppendingPathComponent:file];// define your own source file to be uploaded

//    NSString *dstPath = [toDir stringByAppendingPathComponent:file];// define your own destination file in Dropbox

    _client = [DropboxClientsManager authorizedClient];

    if (!_client) {

        [DropboxClientsManager authorizeFromController:[UIApplication sharedApplication]

                                            controller:self

                                               openURL:^(NSURL *url) {

                                                   [[UIApplication sharedApplication] openURL:url];

                                               }

                                           browserAuth:YES];

    } else {

        _fileSize = [FCFileManager sizeOfFileAtPath:srcPath];// get file size

        _file = [NSFileHandle fileHandleForReadingAtPath:srcPath];

        _data = [_file readDataOfLength:DB_BLOCK_SIZE];

        _uploadTask = [_client.filesRoutes uploadSessionStartData:_data];

        [_uploadTask response:^(DBFILESUploadSessionStartResult * _Nullable result, DBNilObject * _Nullable obj, DBRequestError * _Nullable error) {

            DLog(@"result: %@", result);

            DLog(@"routeError: %@", obj);

            DLog(@"error: %@", error);

            if (error == nil) {

                _sessionId = result.sessionId;

                DBFILESUploadSessionCursor *cursor = [[DBFILESUploadSessionCursor alloc] initWithSessionId:result.sessionId offset:[NSNumber numberWithUnsignedLongLong:_file.offsetInFile]];

                [self uploadNext:cursor];

            } else {

                [_file closeFile];

            }

        }];

        [_uploadTask progress:^(int64_t bytesWritten, int64_t totalBytesWritten, int64_t totalBytesExpectedToWrite) {

            NSLog(@"Start: bytesWritten = %lld", bytesWritten);

        }];

    }

    return;

}

- (void)uploadNext:(DBFILESUploadSessionCursor *)cursor {

    if ([_fileSize unsignedLongLongValue] - _file.offsetInFile > DB_BLOCK_SIZE) {

        _data = [_file readDataOfLength:DB_BLOCK_SIZE];

    } else {

        unsigned long long rest = [_fileSize unsignedLongLongValue] - _file.offsetInFile;

        _data = [_file readDataOfLength:rest];

    }

    if (_data.length < DB_BLOCK_SIZE) {

        DBFILESCommitInfo *info = [[DBFILESCommitInfo alloc] initWithPath:@"/temp/test.mp4"];// define your destination file

        _finishTask = [_client.filesRoutes uploadSessionFinishData:cursor commit:info inputData:_data];

        [_finishTask response:^(DBFILESFileMetadata * _Nullable meta, DBFILESUploadSessionFinishError * _Nullable finishError, DBRequestError * _Nullable error) {

            DLog(@"meta: %@", meta);

            DLog(@"finishError: %@", finishError);

            DLog(@"error: %@", error);

        }];

        return;

    }

    _continueTask = [_client.filesRoutes uploadSessionAppendV2Data:cursor inputData:_data];

    [_continueTask response:^(DBNilObject * _Nullable obj, DBFILESUploadSessionLookupError * _Nullable lookupError, DBRequestError * _Nullable error) {

        DLog(@"obj: %@", obj);

        DLog(@"lookupError: %@", lookupError);

        DLog(@"error: %@", error);

        if (error == nil) {

            DLog(@"_file.offsetInFile: %lld", _file.offsetInFile);

            DBFILESUploadSessionCursor *cursor = [[DBFILESUploadSessionCursor alloc] initWithSessionId:_sessionId offset:[NSNumber numberWithUnsignedLongLong:_file.offsetInFile]];

            [self uploadNext:cursor];

        } else {

            [_file closeFile];

        }

    }];

    [_continueTask progress:^(int64_t bytesWritten, int64_t totalBytesWritten, int64_t totalBytesExpectedToWrite) {

        NSLog(@"Continue: bytesWritten = %lld", bytesWritten);

    }];

}

@end

Remark: File size can also be obtained through following code instead of using FCFileManager:

NSDictionary *attributes = [[NSFileManager defaultManager] attributesOfItemAtPath:path error:NULL];

fileSize = [attributes fileSize];

Thanks! That's helpful

A few notes:

- I misunderstood what was being logged in your original post, and testing the code I can confirm the issue isn't the chunk size being 32768. That's the progress callback is showing incremental progress within each single chunk upload call. Your chunk size is working at 5 MB.

- In your latest code though, note that the comment is wrong. The actual code sets a 5 MB chunk size, but the comment indicates 100 MB.

- Testing this code, I have a better idea of where the overhead is. We're looking into it and I'll follow up here once I have more information.

Thanks for the reply!

The comment is just for testing different value of chunk size. I have tested 5 MB and 100 MB, but the result is the same.

If you have tested the upload using the code I provided, did you get the same result? I mean, is the upload blocked/stopped or canceled after a few seconds?

It would be great if you could help us out from the problem described above.

Hope to hear from you soon when you have more information about how to correctly use the API V2 and get rid of the problem.