cancel
Showing results for 
Show  only  | Search instead for 
Did you mean: 
Announcements
What’s new: end-to-end encryption, Replay and Dash updates. Find out more about these updates, new features and more here.

Dropbox API Support & Feedback

Find help with the Dropbox API from other developers.

cancel
Showing results for 
Show  only  | Search instead for 
Did you mean: 

API V2 Can't get large file Upload working

API V2 Can't get large file Upload working

RTS S.
Helpful | Level 6

Some context for the following:
1) DBAPIMetadata is just my wrapper for Files.Metadata that I can pass to Objective-C
2) The BGHelper at the end (when the call is not on the MainThread) just call the function
On the main thread and blocks the background thread until it completes.
Basically any calls on the main thread are async and those on the background thread are
synchronous for my use.

What succeeds ..
1) I pass an empty NDData to filesUploadSessionStart and get a session.
2) I pass the NSData with the real data to filesUploadSessionAppend
Data was smaller than MaxLength so everything was transfer in the first call to ProcessNextBlock
3) On successfull completion it tried to call filesUploadSessionAppend
Here I get:

DropboxAPI:UploadFile:SessionFinish:[request-id 9dc97cd33ec9680b0ea58decc87cc4dd] API route error - handle programmatically

Previously I tried to NOT use filesUploadSessionAppend when a single transfer would work and send the data in filesUploadSessionAppend ... that failed as well with a different error.


Sorry the following really gets mangled on this forum ... it's just a single Swift function that handles the transfer.


func UploadFile(dbPath:String, toPath:String, withParentRev:String, fromPath:String, completion: (DBAPIMetadata?) -> Void) -> Void {
if (NSThread.isMainThread()) {
// Context for Process Next
let ins = NSInputStream(fileAtPath: fromPath)
ins!.open()
let MaxLength = 1024 * 1024 * 2
var transferCnt:UInt64 = 0
var buffer = Array<UInt8>(count:MaxLength, repeatedValue:0)
var data = NSData.init(bytesNoCopy: &buffer, length:MaxLength, freeWhenDone:false)
var sessionID = ""
var retry = 0

// Initially blank ... so it can call itself
var ProcessNextBlock:() -> Void = { () -> Void in }
ProcessNextBlock = {
() in
let readCnt = ins!.read(&buffer, maxLength: MaxLength)
if (readCnt > 0) {
if (readCnt != MaxLength) {
data = NSData.init(bytesNoCopy: &buffer, length:(readCnt >= 0 ? readCnt: 0), freeWhenDone:false)
}
// Initiall blank ... so it can call itself
var compClosure:((Void)?, CallError<(Files.UploadSessionLookupError)>?) -> Void = { (_, _) -> Void in }
compClosure = {
(response, errorType) in
if errorType != nil {
// Lets Rety
retry++
if (retry < 4) {
self.mClient.filesUploadSessionAppend(sessionId:sessionID, offset:transferCnt, body:data).response(compClosure)
} else {
PersistLog.e(self.TAG, andMsg: "UploadFile:SessionAppend:" + fromPath)
PersistLog.e(self.TAG, andMsg: "UploadFile:SessionAppend:" + errorType!.description)
completion(nil)
}
} else {
transferCnt = transferCnt + UInt64(readCnt)
retry = 0
ProcessNextBlock()
}
}
self.mClient.filesUploadSessionAppend(sessionId:sessionID, offset:transferCnt, body:data).response(compClosure)
} else {
// Send the last non full block with filesUploadSession Finish
if (transferCnt > 0) {
let cursor = Files.UploadSessionCursor(sessionId: sessionID, offset: transferCnt)
let info = Files.CommitInfo(path: dbPath)
self.mClient.filesUploadSessionFinish(cursor: cursor, commit: info, body: NSData()).response( {
(response, errorType) in
if let (meta) = response {
completion(DBAPIMetadata(meta:meta))
} else {
PersistLog.e(self.TAG, andMsg: "UploadFile:SessionFinish:" + fromPath)
PersistLog.e(self.TAG, andMsg: "UploadFile:SessionFinish:" + errorType!.description)
completion(nil)
}
})
} else {
PersistLog.e(self.TAG, andMsg: "UploadFile:NoData:" + fromPath)
completion(nil)
}
}
}

// Start the Process - Use empty data just to get a Session ID
let tmp = NSData()
mClient.filesUploadSessionStart(body:tmp).response( {
(response, errorType) -> Void in
if let (startResult) = response {
sessionID = startResult.sessionId
ProcessNextBlock()
} else {
PersistLog.e(self.TAG, andMsg: "UploadFile:SessionStart:" + fromPath)
PersistLog.e(self.TAG, andMsg: "UploadFile:SessionStart:" + errorType!.description)
completion(nil)
}
})
} else {
PersistLog.i(TAG, andMsg: "UploadFile")
let BG = BGHelper<DBAPIMetadata>(completion:completion)
BG.CallerStart( {
self.UploadFile(dbPath, toPath:toPath, withParentRev:withParentRev, fromPath: fromPath, completion: BG.completion)
})
BG.CallerWait()
}
}

9 Replies 9

Greg-DB
Dropbox Staff

Thanks for the detailed write-up. We'll be happy to help. First, a few things:

- Just to make sure, in step 3, you meant "filesUploadSessionFinish" not "filesUploadSessionAppend" right?

- Can you share the log output for reference?

- Can you share the value of dbPath?

Thanks in advance! 

RTS S.
Helpful | Level 6

Oops ... I meant that the filesUploadSessionFinish failed.

In two cases ... the one I just reported on was with an empty  NSDATA()  as the body.

Previously I also had a problem when I passed and NSDATA object with data ... It failed with a different error.  

The dbPath was  /AccountInfo.json   ... logged in as my APP context.

Where were no errors in the console .. just the info I sent you.

 

Greg-DB
Dropbox Staff

Thanks. We're looking into it. In the meantime, please make sure you're using the latest version of the SDK. (Currently 1.0 on GitHub)

RTS S.
Helpful | Level 6

I upgraded to API Version 1.0

Here is the debugger output of the input variables before calling filesUploadSessionFinish and after getting the response:

Printing description of cursor:
{
offset = 82995;
"session_id" = AAAAAAAAEHSrFUSNImgWDg;
}
Printing description of info:
{
autorename = 0;
mode = {
".tag" = add;
};
mute = 0;
path = "/AccountInfo.json";
}
Printing description of errorType:
(SwiftyDropbox.CallError<SwiftyDropbox.Files.UploadSessionFinishError>?) errorType = RouteError {
RouteError = {
0 = 0x000000012cf15130 {
unboxed = LookupFailed {
LookupFailed = IncorrectOffset {
IncorrectOffset = 0x0000000103980030 (correctOffset = 6869401272)
}
}
}
1 = "b5f5e7b4401cc6c0ac522b26c6aafee3"
}
}
UploadFile:SessionFinish:[request-id b5f5e7b4401cc6c0ac522b26c6aafee3] API route error - handle programmatically

Greg-DB
Dropbox Staff

Thanks! This error should just indicate an issue with the path you're supplying, but your path ("/AccountInfo.json") looks fine, so we're currently investigating if/what the issue might be on our side.

RTS S.
Helpful | Level 6

Some more  problems with uploads (Version 1.0) ...

let ins = NSInputStream(fileAtPath: fromPath)

client.files.upload(path:"/AccountInfo.json", ins!)

I also get the following response error:

Printing description of errorType:
(SwiftyDropbox.CallError<SwiftyDropbox.Files.UploadError>?) errorType = RouteError {
RouteError = {
0 = 0x000000015cedf080 {
unboxed = Path {
Path = 0x0000000104594030 {
reason = MalformedPath {
MalformedPath = "
}
uploadSessionId = unable to read data
}
}
}
1 = "4c780c956d3e116ae75fdeee8c7435f3"
}
}

Also if I read the stream into an NSData object and pass the NSData object to the files.upload call instead of the NSInputStream ... I get an exception ... like its trying to read a stream.

A note on this issue and previous issue ... the "/AccountInfo.json" file already exists on dropbox ... I am trying to update it.

 

I have not been able to use any Variation of your API's to upload this file!!!

 

 

Greg-DB
Dropbox Staff

Thanks! This version of the code simplifies things. This version does work for me, and I believe the error you're getting is a "file" WriteConflictError, meaning there's already a file at the path you're trying to upload to:

https://www.dropbox.com/developers/documentation/http#documentation-files-upload

You can specify a WriteMode to change how this scenario is handled.

We're working on getting the SwiftyDropbox specific documentation ready, as well as improving the error reporting in SwiftyDropbox itself. 

RTS S.
Helpful | Level 6

Yep, my upload (with an input stream) and Session upload with chunks of NSData  worked after settings the WriteMode (In the latter case the upload always worked, it was the finishing the session and saving the filed that failed)

The error messages are terrible and I would have never guessed from the error what the problem was. Even now looking back I would not have been able to interpret the error.

The last problem  was mine... upload with an NSData object (I did not show the code)  was a memory management problem. I used an NSData object that did not copy the buffer and the buffer was deleted before it was processed. 

All is good ... with multiple techniques to upload a file.

Greg-DB
Dropbox Staff

Thanks for the feedback on the error messages! We've made some changes to how this is reported, so if you pod update (latest is 1.0.2) issues like this should yield better messages.

Need more support?
Who's talking

Top contributors to this post

  • User avatar
    Greg-DB Dropbox Staff
  • User avatar
    RTS S. Helpful | Level 6
What do Dropbox user levels mean?