Need to see if your shared folder is taking up space on your dropbox 👨💻? Find out how to check here.
Forum Discussion
michal_k
10 years agoExplorer | Level 4
[Java SDK V2] How to do batch upload?
Hello, I am trying to make Batch upload work by using JAVA SDK v2. I read this documentation and tried to perform a batch upload. There is a general algorithm, which should use direct HTTP ca...
- 10 years ago
That looks like a bug that was fixed in version 2.1.2. Please make sure you're using the latest version of the SDK, and let me know if you're still seeing that.
michal_k
10 years agoExplorer | Level 4
I modified the sample into an Unit Test, did the changes you advised, but now I keep getting this error as a result of uploadSessionFinishBatchCheck.
uploadSessionFinishBatchJobStatus={
".tag" : "complete",
"entries" : [ {
".tag" : "failure",
"failure" : {
".tag" : "lookup_failed",
"lookup_failed" : "not_closed"
}
} ]
}
Here is the code, sorry for the messy state:
@Test
public void chunkedUploadFiles() throws Exception {
// Adjust the chunk size based on your network speed and reliability. Larger chunk sizes will
// result in fewer network requests, which will be faster. But if an error occurs, the entire
// chunk will be lost and have to be re-uploaded. Use a multiple of 4MiB for your chunk size.
// final long CHUNKED_UPLOAD_CHUNK_SIZE = 8L << 20; // 8MiB
final long CHUNKED_UPLOAD_CHUNK_SIZE = 15; //in Bytes
final int CHUNKED_UPLOAD_MAX_ATTEMPTS = 5;
final String FILE_UPLOADED_BY_PARTS = "FileUploadedByParts.txt";
String dropboxPath = getDropboxPath(FILE_UPLOADED_BY_PARTS);
File localFile = createTempFileWithContent(FILE_UPLOADED_BY_PARTS, "The content of the file uploaded by parts 01"); //44B of text
long size = localFile.length();
long uploaded = 0L;
DbxException thrown = null;
// Chunked uploads have 3 phases, each of which can accept uploaded bytes:
//
// (1) Start: initiate the upload and get an upload session ID
// (2) Append: upload chunks of the file to append to our session
// (3) Finish: commit the upload and close the session
//
// We track how many bytes we uploaded to determine which phase we should be in.
String sessionId = null;
for (int i = 0; i < CHUNKED_UPLOAD_MAX_ATTEMPTS; ++i) {
if (i > 0) {
System.out.printf("Retrying chunked upload (%d / %d attempts)\n", i + 1, CHUNKED_UPLOAD_MAX_ATTEMPTS);
}
try (InputStream in = new FileInputStream(localFile)) {
// if this is a retry, make sure seek to the correct offset
in.skip(uploaded);
// (1) Start
if (sessionId == null) {
sessionId = tested.files().uploadSessionStart()
.uploadAndFinish(in, CHUNKED_UPLOAD_CHUNK_SIZE)
.getSessionId();
uploaded += CHUNKED_UPLOAD_CHUNK_SIZE;
printProgress(uploaded, size);
}
UploadSessionCursor cursor = new UploadSessionCursor(sessionId, uploaded);
// (2) Append
while ((size - uploaded) > CHUNKED_UPLOAD_CHUNK_SIZE) {
tested.files().uploadSessionAppendV2(cursor)
.uploadAndFinish(in, CHUNKED_UPLOAD_CHUNK_SIZE);
uploaded += CHUNKED_UPLOAD_CHUNK_SIZE;
printProgress(uploaded, size);
cursor = new UploadSessionCursor(sessionId, uploaded);
}
final long remaining = size - uploaded;
if ((remaining) > 0) {
tested.files().uploadSessionAppendV2(cursor)
.uploadAndFinish(in, remaining);
uploaded += remaining;
printProgress(uploaded, size);
cursor = new UploadSessionCursor(sessionId, uploaded);
}
// (3) Finish
CommitInfo commitInfo = CommitInfo.newBuilder(dropboxPath)
.withMode(WriteMode.ADD)
.withClientModified(new Date(localFile.lastModified()))
.build();
UploadSessionFinishArg entry = new UploadSessionFinishArg(cursor, commitInfo);
final String asyncJobIdValue = tested.files().uploadSessionFinishBatch(newArrayList(entry)).getAsyncJobIdValue();
while (tested.files().uploadSessionFinishBatchCheck(asyncJobIdValue).isInProgress()) {
Thread.sleep(1000);
}
final UploadSessionFinishBatchJobStatus uploadSessionFinishBatchJobStatus = tested.files().uploadSessionFinishBatchCheck(
asyncJobIdValue);
System.out.println("uploadSessionFinishBatchJobStatus=" + uploadSessionFinishBatchJobStatus.toStringMultiline());
return;
} catch (RetryException ex) {
thrown = ex;
// RetryExceptions are never automatically retried by the client for uploads. Must
// catch this exception even if DbxRequestConfig.getMaxRetries() > 0.
sleepQuietly(ex.getBackoffMillis());
continue;
} catch (NetworkIOException ex) {
thrown = ex;
// network issue with Dropbox (maybe a timeout?) try again
continue;
} catch (UploadSessionLookupErrorException ex) {
if (ex.errorValue.isIncorrectOffset()) {
thrown = ex;
// server offset into the stream doesn't match our offset (uploaded). Seek to
// the expected offset according to the server and try again.
uploaded = ex.errorValue
.getIncorrectOffsetValue()
.getCorrectOffset();
continue;
} else {
// Some other error occurred, give up.
System.err.println("Error uploading to Dropbox: " + ex.getMessage());
System.exit(1);
return;
}
} catch (UploadSessionFinishErrorException ex) {
if (ex.errorValue.isLookupFailed() && ex.errorValue.getLookupFailedValue().isIncorrectOffset()) {
thrown = ex;
// server offset into the stream doesn't match our offset (uploaded). Seek to
// the expected offset according to the server and try again.
uploaded = ex.errorValue
.getLookupFailedValue()
.getIncorrectOffsetValue()
.getCorrectOffset();
continue;
} else {
// some other error occurred, give up.
System.err.println("Error uploading to Dropbox: " + ex.getMessage());
System.exit(1);
return;
}
} catch (DbxException ex) {
System.err.println("Error uploading to Dropbox: " + ex.getMessage());
System.exit(1);
return;
} catch (IOException ex) {
System.err.println("Error reading from file \"" + localFile + "\": " + ex.getMessage());
System.exit(1);
return;
}
}
// if we made it here, then we must have run out of attempts
System.err.println("Maxed out upload attempts to Dropbox. Most recent error: " + thrown.getMessage());
System.exit(1);
}
private static void printProgress(long uploaded, long size) {
System.out.printf("Uploaded %12d / %12d bytes (%5.2f%%)\n", uploaded, size, 100 * (uploaded / (double) size));
}
private static void sleepQuietly(long millis) {
try {
Thread.sleep(millis);
} catch (InterruptedException ex) {
// just exit
System.err.println("Error uploading to Dropbox: interrupted during backoff.");
System.exit(1);
}
}Greg-DB
Dropbox Community Moderator
10 years agoThe not_closed error is documented as:
"The session must be closed before calling upload_session/finish_batch."
That is, make sure you close each session, e.g., when using /2/files/upload_session/append_v2, before calling /2/files/upload_session/finish_batch.
- michal_k10 years agoExplorer | Level 4
For some reason my posts are disappearing from this thread.
I closed the session like this:
tested.files().uploadSessionAppendV2(cursor, true).uploadAndFinish(in, remaining);
Now the file is being uploaded to Dropbox well, however following exception is raised from following statement:while (tested.files().uploadSessionFinishBatchCheck(asyncJobIdValue).isInProgress()) { Thread.sleep(1000); }com.dropbox.core.BadResponseException: Bad JSON: expected field 'success', but was: 'name'
at [Source: sun.net.www.protocol.http.HttpURLConnection$HttpInputStream@315f43d5; line: 1, column: 63]
at com.dropbox.core.v2.DbxRawClientV2$1.execute(DbxRawClientV2.java:112)
at com.dropbox.core.v2.DbxRawClientV2.executeRetriable(DbxRawClientV2.java:256)
at com.dropbox.core.v2.DbxRawClientV2.rpcStyle(DbxRawClientV2.java:97)
at com.dropbox.core.v2.files.DbxUserFilesRequests.uploadSessionFinishBatchCheck(DbxUserFilesRequests.java:1565)
at com.dropbox.core.v2.files.DbxUserFilesRequests.uploadSessionFinishBatchCheck(DbxUserFilesRequests.java:1592)
... nothing interesting here
Caused by: com.fasterxml.jackson.core.JsonParseException: expected field 'success', but was: 'name'
at [Source: sun.net.www.protocol.http.HttpURLConnection$HttpInputStream@315f43d5; line: 1, column: 63]
at com.dropbox.core.stone.StoneSerializer.expectField(StoneSerializer.java:84)
at com.dropbox.core.v2.files.UploadSessionFinishBatchResultEntry$Serializer.deserialize(UploadSessionFinishBatchResultEntry.java:253)
at com.dropbox.core.v2.files.UploadSessionFinishBatchResultEntry$Serializer.deserialize(UploadSessionFinishBatchResultEntry.java:205)
at com.dropbox.core.stone.StoneSerializers$ListSerializer.deserialize(StoneSerializers.java:265)
at com.dropbox.core.stone.StoneSerializers$ListSerializer.deserialize(StoneSerializers.java:244)
at com.dropbox.core.v2.files.UploadSessionFinishBatchResult$Serializer.deserialize(UploadSessionFinishBatchResult.java:124)
at com.dropbox.core.v2.files.UploadSessionFinishBatchJobStatus$Serializer.deserialize(UploadSessionFinishBatchJobStatus.java:236)
at com.dropbox.core.v2.files.UploadSessionFinishBatchJobStatus$Serializer.deserialize(UploadSessionFinishBatchJobStatus.java:190)
at com.dropbox.core.stone.StoneSerializer.deserialize(StoneSerializer.java:66)
at com.dropbox.core.v2.DbxRawClientV2$1.execute(DbxRawClientV2.java:104)
... 36 common frames omittedI think there are acually two problems. I seems it seems, that there are two colliding versions of the com.fasterxml.jackson library (we use the Spring Boot framework).
Nevertheless the main problem is, that the StoneSerializer.java is not able to parse the message and throws exception where the intended error message should be this:
expected field 'success', but was: 'name'
- Greg-DB10 years ago
Dropbox Community Moderator
That looks like a bug that was fixed in version 2.1.2. Please make sure you're using the latest version of the SDK, and let me know if you're still seeing that.
- michal_k10 years agoExplorer | Level 4
My bad, I just followed the installation guide and didn't checke the MVN Repo.
Thank you for your help.
About Dropbox API Support & Feedback
Find help with the Dropbox API from other developers.
The Dropbox Community team is active from Monday to Friday. We try to respond to you as soon as we can, usually within 2 hours.
If you need more help you can view your support options (expected response time for an email or ticket is 24 hours), or contact us on X, Facebook or Instagram.
For more info on available support options for your Dropbox plan, see this article.
If you found the answer to your question in this Community thread, please 'like' the post to say thanks and to let us know it was useful!