Re: [Java SDK V2] How to do batch upload?

michal_k · ‎11-30-2016

Hello,

I am trying to make Batch upload work by using JAVA SDK v2. I read this documentation and tried to perform a batch upload.

There is a general algorithm, which should use direct HTTP calls.

1. First point sais "Group files that are being uploaded to into one or more batches....". I don't know, what exactly is Group to batches, I assume that it is some logical splitting.

2. Then there is this instruction "For each file in a batch, in parallel, call /files/upload_session/start"

So I tried this code:

UploadSessionStartUploader uploadSessionStartUploader = dbxClientV2.files().uploadSessionStart();

and I got an instance of the UploadSessionStartUploader.

But now I don't know, how to use it, because it does not seem to provide any functionality for batch upload.

Could you please give me some example, how to do the batch upload?

Greg-DB · ‎12-06-2016

That looks like a bug that was fixed in version 2.1.2. Please make sure you're using the latest version of the SDK, and let me know if you're still seeing that.

View solution in original post

Greg-DB · ‎11-30-2016

1. Yes, regarding the grouping, the data ingress guide elaborates:

"Group files that are being uploaded to into one or more batches. A batch can contain up to 1,000 entries. Entries in a batch don’t all have to target the same namespace, but the fewer namespaces involved in a batch, the more efficient committing the batch will be."

So, you can group files into batches of up to 1,000 files, preferably across as few namespaces as possible.

2. There's a sample of using upload sessions with the Java SDK that should be useful here:

https://github.com/dropbox/dropbox-sdk-java/blob/master/examples/upload-file/src/main/java/com/dropb...

That doesn't use the batch functionality, but it should be a good starting point.

The batch functionality is that instead of calling uploadSessionFinish oncer per file, you can instead call uploadSessionFinishBatch once for up to 1000 files each time. You would then use uploadSessionFinishBatchCheck to check on the status of that operation.

michal_k · ‎12-05-2016

I modified the sample into an Unit Test, did the changes you advised, but now I keep getting this error as a result of uploadSessionFinishBatchCheck.

 uploadSessionFinishBatchJobStatus={
".tag" : "complete",
"entries" : [ {
".tag" : "failure",
"failure" : {
".tag" : "lookup_failed",
"lookup_failed" : "not_closed"
}
} ]
}

Here is the code, sorry for the messy state:

    @Test
    public void chunkedUploadFiles() throws Exception {
        // Adjust the chunk size based on your network speed and reliability. Larger chunk sizes will
        // result in fewer network requests, which will be faster. But if an error occurs, the entire
        // chunk will be lost and have to be re-uploaded. Use a multiple of 4MiB for your chunk size.
//        final long CHUNKED_UPLOAD_CHUNK_SIZE = 8L << 20; // 8MiB
        final long CHUNKED_UPLOAD_CHUNK_SIZE = 15; //in Bytes
        final int CHUNKED_UPLOAD_MAX_ATTEMPTS = 5;

        final String FILE_UPLOADED_BY_PARTS = "FileUploadedByParts.txt";
        String dropboxPath = getDropboxPath(FILE_UPLOADED_BY_PARTS);
        File localFile = createTempFileWithContent(FILE_UPLOADED_BY_PARTS, "The content of the file uploaded by parts 01"); //44B of text


        long size = localFile.length();

        long uploaded = 0L;
        DbxException thrown = null;

        // Chunked uploads have 3 phases, each of which can accept uploaded bytes:
        //
        //    (1)  Start: initiate the upload and get an upload session ID
        //    (2) Append: upload chunks of the file to append to our session
        //    (3) Finish: commit the upload and close the session
        //
        // We track how many bytes we uploaded to determine which phase we should be in.
        String sessionId = null;
        for (int i = 0; i < CHUNKED_UPLOAD_MAX_ATTEMPTS; ++i) {
            if (i > 0) {
                System.out.printf("Retrying chunked upload (%d / %d attempts)\n", i + 1, CHUNKED_UPLOAD_MAX_ATTEMPTS);
            }

            try (InputStream in = new FileInputStream(localFile)) {
                // if this is a retry, make sure seek to the correct offset
                in.skip(uploaded);

                // (1) Start
                if (sessionId == null) {
                    sessionId = tested.files().uploadSessionStart()
                            .uploadAndFinish(in, CHUNKED_UPLOAD_CHUNK_SIZE)
                            .getSessionId();
                    uploaded += CHUNKED_UPLOAD_CHUNK_SIZE;
                    printProgress(uploaded, size);
                }

                UploadSessionCursor cursor = new UploadSessionCursor(sessionId, uploaded);

                // (2) Append
                while ((size - uploaded) > CHUNKED_UPLOAD_CHUNK_SIZE) {
                    tested.files().uploadSessionAppendV2(cursor)
                            .uploadAndFinish(in, CHUNKED_UPLOAD_CHUNK_SIZE);
                    uploaded += CHUNKED_UPLOAD_CHUNK_SIZE;
                    printProgress(uploaded, size);
                    cursor = new UploadSessionCursor(sessionId, uploaded);
                }

                final long remaining = size - uploaded;
                if ((remaining) > 0) {
                    tested.files().uploadSessionAppendV2(cursor)
                            .uploadAndFinish(in, remaining);
                    uploaded += remaining;
                    printProgress(uploaded, size);
                    cursor = new UploadSessionCursor(sessionId, uploaded);
                }


                // (3) Finish
                CommitInfo commitInfo = CommitInfo.newBuilder(dropboxPath)
                        .withMode(WriteMode.ADD)
                        .withClientModified(new Date(localFile.lastModified()))
                        .build();
                UploadSessionFinishArg entry = new UploadSessionFinishArg(cursor, commitInfo);
                final String asyncJobIdValue = tested.files().uploadSessionFinishBatch(newArrayList(entry)).getAsyncJobIdValue();

                while (tested.files().uploadSessionFinishBatchCheck(asyncJobIdValue).isInProgress()) {
                    Thread.sleep(1000);
                }

                final UploadSessionFinishBatchJobStatus uploadSessionFinishBatchJobStatus = tested.files().uploadSessionFinishBatchCheck(
                        asyncJobIdValue);
                System.out.println("uploadSessionFinishBatchJobStatus=" + uploadSessionFinishBatchJobStatus.toStringMultiline());

                return;
            } catch (RetryException ex) {
                thrown = ex;
                // RetryExceptions are never automatically retried by the client for uploads. Must
                // catch this exception even if DbxRequestConfig.getMaxRetries() > 0.
                sleepQuietly(ex.getBackoffMillis());
                continue;
            } catch (NetworkIOException ex) {
                thrown = ex;
                // network issue with Dropbox (maybe a timeout?) try again
                continue;
            } catch (UploadSessionLookupErrorException ex) {
                if (ex.errorValue.isIncorrectOffset()) {
                    thrown = ex;
                    // server offset into the stream doesn't match our offset (uploaded). Seek to
                    // the expected offset according to the server and try again.
                    uploaded = ex.errorValue
                            .getIncorrectOffsetValue()
                            .getCorrectOffset();
                    continue;
                } else {
                    // Some other error occurred, give up.
                    System.err.println("Error uploading to Dropbox: " + ex.getMessage());
                    System.exit(1);
                    return;
                }
            } catch (UploadSessionFinishErrorException ex) {
                if (ex.errorValue.isLookupFailed() && ex.errorValue.getLookupFailedValue().isIncorrectOffset()) {
                    thrown = ex;
                    // server offset into the stream doesn't match our offset (uploaded). Seek to
                    // the expected offset according to the server and try again.
                    uploaded = ex.errorValue
                            .getLookupFailedValue()
                            .getIncorrectOffsetValue()
                            .getCorrectOffset();
                    continue;
                } else {
                    // some other error occurred, give up.
                    System.err.println("Error uploading to Dropbox: " + ex.getMessage());
                    System.exit(1);
                    return;
                }
            } catch (DbxException ex) {
                System.err.println("Error uploading to Dropbox: " + ex.getMessage());
                System.exit(1);
                return;
            } catch (IOException ex) {
                System.err.println("Error reading from file \"" + localFile + "\": " + ex.getMessage());
                System.exit(1);
                return;
            }
        }

        // if we made it here, then we must have run out of attempts
        System.err.println("Maxed out upload attempts to Dropbox. Most recent error: " + thrown.getMessage());
        System.exit(1);
    }

    private static void printProgress(long uploaded, long size) {
        System.out.printf("Uploaded %12d / %12d bytes (%5.2f%%)\n", uploaded, size, 100 * (uploaded / (double) size));
    }

    private static void sleepQuietly(long millis) {
        try {
            Thread.sleep(millis);
        } catch (InterruptedException ex) {
            // just exit
            System.err.println("Error uploading to Dropbox: interrupted during backoff.");
            System.exit(1);
        }
    }

michal_k · ‎12-05-2016

I checked the Javadoc and I think, that the problem was in this statement:

tested.files().uploadSessionAppendV2(cursor).uploadAndFinish(in, remaining);

There was the close argument missing, so I think, that correct statement should be this:

tested.files().uploadSessionAppendV2(cursor, true).uploadAndFinish(in, remaining);

The file is now uploaded to Dropbox, however I have a problem with following statement:

while (tested.files().uploadSessionFinishBatchCheck(asyncJobIdValue).isInProgress()) {
   Thread.sleep(1000);
}

I get this exception:

java.lang.NoSuchMethodError: com.fasterxml.jackson.core.JsonParseException.<init>(Lcom/fasterxml/jackson/core/JsonParser;Ljava/lang/String;)V
	at com.dropbox.core.stone.StoneSerializer.expectField(StoneSerializer.java:84)
	at com.dropbox.core.v2.files.UploadSessionFinishBatchResultEntry$Serializer.deserialize(UploadSessionFinishBatchResultEntry.java:253)
	at com.dropbox.core.v2.files.UploadSessionFinishBatchResultEntry$Serializer.deserialize(UploadSessionFinishBatchResultEntry.java:205)
	at com.dropbox.core.stone.StoneSerializers$ListSerializer.deserialize(StoneSerializers.java:265)
	at com.dropbox.core.stone.StoneSerializers$ListSerializer.deserialize(StoneSerializers.java:244)
	at com.dropbox.core.v2.files.UploadSessionFinishBatchResult$Serializer.deserialize(UploadSessionFinishBatchResult.java:124)
	at com.dropbox.core.v2.files.UploadSessionFinishBatchJobStatus$Serializer.deserialize(UploadSessionFinishBatchJobStatus.java:236)
	at com.dropbox.core.v2.files.UploadSessionFinishBatchJobStatus$Serializer.deserialize(UploadSessionFinishBatchJobStatus.java:190)
	at com.dropbox.core.stone.StoneSerializer.deserialize(StoneSerializer.java:66)
	at com.dropbox.core.v2.DbxRawClientV2$1.execute(DbxRawClientV2.java:104)
	at com.dropbox.core.v2.DbxRawClientV2.executeRetriable(DbxRawClientV2.java:256)
	at com.dropbox.core.v2.DbxRawClientV2.rpcStyle(DbxRawClientV2.java:97)
	at com.dropbox.core.v2.files.DbxUserFilesRequests.uploadSessionFinishBatchCheck(DbxUserFilesRequests.java:1565)
	at com.dropbox.core.v2.files.DbxUserFilesRequests.uploadSessionFinishBatchCheck(DbxUserFilesRequests.java:1592)
	...nothing interesting here

When I put breakpoint on the line StoneSerializer.java:84, I found out, that there are two problems, probably bugs.

First is, that StoneSerializer is not able to parse the response and second problem is the usage of new JsonParseException, where first argument should be message, not an instance of the JsonParser.

The actual exception message, that is suppressed by this incorrect usage in my case should be:

expected field 'success', but was: 'name'

michal_k · ‎12-05-2016

For some reason I can't see my last reply here, so I'll write it again.

I noticed in the Javadoc, that I am probably using the functionality wrong and there is a problem in following statement:

tested.files().uploadSessionAppendV2(cursor).uploadAndFinish(in, remaining);

It should be used with close argument true like this:

tested.files().uploadSessionAppendV2(cursor, true).uploadAndFinish(in, remaining);

Now the file is being uploaded to Dropbox well, however following exception is raised from following statement:

while (tested.files().uploadSessionFinishBatchCheck(asyncJobIdValue).isInProgress()) {
    Thread.sleep(1000);
}

java.lang.NoSuchMethodError: com.fasterxml.jackson.core.JsonParseException.<init>(Lcom/fasterxml/jackson/core/JsonParser;Ljava/lang/String;)V
	at com.dropbox.core.stone.StoneSerializer.expectField(StoneSerializer.java:84)
	at com.dropbox.core.v2.files.UploadSessionFinishBatchResultEntry$Serializer.deserialize(UploadSessionFinishBatchResultEntry.java:253)
	at com.dropbox.core.v2.files.UploadSessionFinishBatchResultEntry$Serializer.deserialize(UploadSessionFinishBatchResultEntry.java:205)
	at com.dropbox.core.stone.StoneSerializers$ListSerializer.deserialize(StoneSerializers.java:265)
	at com.dropbox.core.stone.StoneSerializers$ListSerializer.deserialize(StoneSerializers.java:244)
	at com.dropbox.core.v2.files.UploadSessionFinishBatchResult$Serializer.deserialize(UploadSessionFinishBatchResult.java:124)
	at com.dropbox.core.v2.files.UploadSessionFinishBatchJobStatus$Serializer.deserialize(UploadSessionFinishBatchJobStatus.java:236)
	at com.dropbox.core.v2.files.UploadSessionFinishBatchJobStatus$Serializer.deserialize(UploadSessionFinishBatchJobStatus.java:190)
	at com.dropbox.core.stone.StoneSerializer.deserialize(StoneSerializer.java:66)
	at com.dropbox.core.v2.DbxRawClientV2$1.execute(DbxRawClientV2.java:104)
	at com.dropbox.core.v2.DbxRawClientV2.executeRetriable(DbxRawClientV2.java:256)
	at com.dropbox.core.v2.DbxRawClientV2.rpcStyle(DbxRawClientV2.java:97)
	at com.dropbox.core.v2.files.DbxUserFilesRequests.uploadSessionFinishBatchCheck(DbxUserFilesRequests.java:1565)
	at com.dropbox.core.v2.files.DbxUserFilesRequests.uploadSessionFinishBatchCheck(DbxUserFilesRequests.java:1592)
... nothing interesting here

I think there are acually two problems. First of according to my debuger the new JsonParseException should be used in a completely different way, first argument should be an error message. I wonder, how the code is able to compile, it's probably some jackson library missmatch.

Nevertheless the main problem is, that the StoneSerializer.java is not able to parse the message and throws exception where the intended error message should be this:

expected field 'success', but was: 'name'

michal_k · ‎12-05-2016

For some reason I can't see my last reply here, so I'll write it again.

I noticed in the Javadoc, that I am probably using the functionality wrong and there is a problem in following statement:

tested.files().uploadSessionAppendV2(cursor).uploadAndFinish(in, remaining);

It should be used with close argument true like this:

tested.files().uploadSessionAppendV2(cursor, true).uploadAndFinish(in, remaining);

Now the file is being uploaded to Dropbox well, however following exception is raised from following statement:

while (tested.files().uploadSessionFinishBatchCheck(asyncJobIdValue).isInProgress()) {
    Thread.sleep(1000);
}

java.lang.NoSuchMethodError: com.fasterxml.jackson.core.JsonParseException.<init>(Lcom/fasterxml/jackson/core/JsonParser;Ljava/lang/String;)V
at com.dropbox.core.stone.StoneSerializer.expectField(StoneSerializer.java:84)
at com.dropbox.core.v2.files.UploadSessionFinishBatchResultEntry$Serializer.deserialize(UploadSessionFinishBatchResultEntry.java:253)
at com.dropbox.core.v2.files.UploadSessionFinishBatchResultEntry$Serializer.deserialize(UploadSessionFinishBatchResultEntry.java:205)
at com.dropbox.core.stone.StoneSerializers$ListSerializer.deserialize(StoneSerializers.java:265)
at com.dropbox.core.stone.StoneSerializers$ListSerializer.deserialize(StoneSerializers.java:244)
at com.dropbox.core.v2.files.UploadSessionFinishBatchResult$Serializer.deserialize(UploadSessionFinishBatchResult.java:124)
at com.dropbox.core.v2.files.UploadSessionFinishBatchJobStatus$Serializer.deserialize(UploadSessionFinishBatchJobStatus.java:236)
at com.dropbox.core.v2.files.UploadSessionFinishBatchJobStatus$Serializer.deserialize(UploadSessionFinishBatchJobStatus.java:190)
at com.dropbox.core.stone.StoneSerializer.deserialize(StoneSerializer.java:66)
at com.dropbox.core.v2.DbxRawClientV2$1.execute(DbxRawClientV2.java:104)
at com.dropbox.core.v2.DbxRawClientV2.executeRetriable(DbxRawClientV2.java:256)
at com.dropbox.core.v2.DbxRawClientV2.rpcStyle(DbxRawClientV2.java:97)
at com.dropbox.core.v2.files.DbxUserFilesRequests.uploadSessionFinishBatchCheck(DbxUserFilesRequests.java:1565)
at com.dropbox.core.v2.files.DbxUserFilesRequests.uploadSessionFinishBatchCheck(DbxUserFilesRequests.java:1592)
... nothing interesting here

I think there are acually two problems. First all, it seems, that there are two colliding versions of the com.fasterxml.jackson library (we use the Spring Boot framework).

Nevertheless the main problem is, that the StoneSerializer.java is not able to parse the message and throws exception where the intended error message should be this:

expected field 'success', but was: 'name'

Greg-DB · ‎12-05-2016

The not_closed error is documented as:

"The session must be closed before calling upload_session/finish_batch."

That is, make sure you close each session, e.g., when using /2/files/upload_session/append_v2, before calling /2/files/upload_session/finish_batch.

michal_k · ‎12-06-2016

For some reason my posts are disappearing from this thread.

I closed the session like this:

tested.files().uploadSessionAppendV2(cursor, true).uploadAndFinish(in, remaining);

Now the file is being uploaded to Dropbox well, however following exception is raised from following statement:

while (tested.files().uploadSessionFinishBatchCheck(asyncJobIdValue).isInProgress()) {
    Thread.sleep(1000);
}

com.dropbox.core.BadResponseException: Bad JSON: expected field 'success', but was: 'name'
 at [Source: sun.net.www.protocol.http.HttpURLConnection$HttpInputStream@315f43d5; line: 1, column: 63]
 at com.dropbox.core.v2.DbxRawClientV2$1.execute(DbxRawClientV2.java:112)
 at com.dropbox.core.v2.DbxRawClientV2.executeRetriable(DbxRawClientV2.java:256)
 at com.dropbox.core.v2.DbxRawClientV2.rpcStyle(DbxRawClientV2.java:97)
 at com.dropbox.core.v2.files.DbxUserFilesRequests.uploadSessionFinishBatchCheck(DbxUserFilesRequests.java:1565)
 at com.dropbox.core.v2.files.DbxUserFilesRequests.uploadSessionFinishBatchCheck(DbxUserFilesRequests.java:1592)
... nothing interesting here
Caused by: com.fasterxml.jackson.core.JsonParseException: expected field 'success', but was: 'name'
 at [Source: sun.net.www.protocol.http.HttpURLConnection$HttpInputStream@315f43d5; line: 1, column: 63]
 at com.dropbox.core.stone.StoneSerializer.expectField(StoneSerializer.java:84)
 at com.dropbox.core.v2.files.UploadSessionFinishBatchResultEntry$Serializer.deserialize(UploadSessionFinishBatchResultEntry.java:253)
 at com.dropbox.core.v2.files.UploadSessionFinishBatchResultEntry$Serializer.deserialize(UploadSessionFinishBatchResultEntry.java:205)
 at com.dropbox.core.stone.StoneSerializers$ListSerializer.deserialize(StoneSerializers.java:265)
 at com.dropbox.core.stone.StoneSerializers$ListSerializer.deserialize(StoneSerializers.java:244)
 at com.dropbox.core.v2.files.UploadSessionFinishBatchResult$Serializer.deserialize(UploadSessionFinishBatchResult.java:124)
 at com.dropbox.core.v2.files.UploadSessionFinishBatchJobStatus$Serializer.deserialize(UploadSessionFinishBatchJobStatus.java:236)
 at com.dropbox.core.v2.files.UploadSessionFinishBatchJobStatus$Serializer.deserialize(UploadSessionFinishBatchJobStatus.java:190)
 at com.dropbox.core.stone.StoneSerializer.deserialize(StoneSerializer.java:66)
 at com.dropbox.core.v2.DbxRawClientV2$1.execute(DbxRawClientV2.java:104)
 ... 36 common frames omitted

I think there are acually two problems. I seems it seems, that there are two colliding versions of the com.fasterxml.jackson library (we use the Spring Boot framework).

Nevertheless the main problem is, that the StoneSerializer.java is not able to parse the message and throws exception where the intended error message should be this:

expected field 'success', but was: 'name'

Greg-DB · ‎12-06-2016

That looks like a bug that was fixed in version 2.1.2. Please make sure you're using the latest version of the SDK, and let me know if you're still seeing that.

michal_k · ‎12-07-2016

My bad, I just followed the installation guide and didn't checke the MVN Repo.

Thank you for your help.