Out of memory with java sdk during uploading (unlimited upload buffer)

[SimoneBuzzi]

Hi,

I'm tring to use the java sdk to move file from pc to dropbox space.

I've written a method that upload file without problem while they are sufficently small (50MB), but generate a java.lang.OutOfMemoryError: Java heap space when the file is bigger

This is the trace thati I obtain 

Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOf(Arrays.java:2271)
at java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:118)
at java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:93)
at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:153)
at sun.net.www.http.PosterOutputStream.write(PosterOutputStream.java:78)
at com.dropbox.core.util.IOUtil.copyStreamToStream(IOUtil.java:59)
at com.dropbox.core.util.IOUtil.copyStreamToStream(IOUtil.java:68)
at com.dropbox.core.util.IOUtil.copyStreamToStream(IOUtil.java:43)
at com.dropbox.core.http.HttpRequestor$Uploader.upload(HttpRequestor.java:98)
at com.dropbox.core.DbxUploader.uploadAndFinish(DbxUploader.java:95)
at com.dropbox.core.v2.DbxUploadStyleBuilder.uploadAndFinish(DbxUploadStyleBuilder.java:92)
at it.xxx.cloudLib.CopyToDropBox.upload(CopyToDropBox.java:45)

Is there the possibiliy to configure the library buffer?

Thank for your help

[Rich]

Moved to the API forum.

[Greg-DB]

Thanks for the report! I don't believe the buffer is configurable, but we'll be happy to look into this issue.

First, what version of the SDK are you using? If you're not already using the latest version (currently v3.0.3) please update to that and try again.

If you're still seeing this issue on the latest version, please let me know, and share the code to reproduce the issue. Thanks in advance!

[SimoneBuzzi]

Hi Greg,

you can find a sample project that show the same problem here

https://www.dropbox.com/s/54b86x1hgxpup99/dropboxProblem.zip?dl=0

The error generated is 

Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOf(Arrays.java:2271)
at java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:118)
at java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:93)
at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:153)
at sun.net.www.http.PosterOutputStream.write(PosterOutputStream.java:78)
at com.dropbox.core.util.IOUtil.copyStreamToStream(IOUtil.java:59)
at com.dropbox.core.util.IOUtil.copyStreamToStream(IOUtil.java:68)
at com.dropbox.core.util.IOUtil.copyStreamToStream(IOUtil.java:43)
at com.dropbox.core.http.HttpRequestor$Uploader.upload(HttpRequestor.java:98)
at com.dropbox.core.DbxUploader.uploadAndFinish(DbxUploader.java:95)
at com.dropbox.core.v2.DbxUploadStyleBuilder.uploadAndFinish(DbxUploadStyleBuilder.java:92)
at it.xxx.cloudLib.CopyToDropBox.upload(CopyToDropBox.java:45)
at it.xxx.cloudLib.CopyFile.main(CopyFile.java:17)

I'm using the last SDK version as you can find in the pom.

During my test I have experienced the same issue using one of 2.x.x version

I'm experiencing the problem using

Linux XXX 3.2.0-126-generic #169-Ubuntu SMP Fri Mar 31 14:15:21 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

Java

java version "1.7.0_80"
Java(TM) SE Runtime Environment (build 1.7.0_80-b15)
Java HotSpot(TM) 64-Bit Server VM (build 24.80-b11, mixed mode)

I'm compiling for 1.7 version

I obtain the error trying to uploading a file of 1.1 GB, I know that the maximum size for a file is 5GB, do I hope I can use the SDK till this line

Thanks for your help

[Greg-DB]

Thanks! That's helpful. We'll look into it.

[Greg-DB]

I just took a look over your last reply and the supplied code, and I notice you mentioned you're trying to upload a 1.1 GB file, but you're using UploadBuilder. For large files, that is, anything bigger than 150 MB, you need to use "upload sessions", per the documentation for uploadBuilder. (You may even want to use upload sessions for files smaller than 150 MB, depending on the scenario.)

There's an example of using upload sessions here. For reference, using this method, there isn't an overall file size limit. You can upload files of any size, including larger than 5 GB. (Note that this only applies to the Dropbox side. Your local filesystem may have its own limit.)

[SimoneBuzzi]

Thank you for your help.

I've created a method that analyzes the file and decides which method to use.
I think that create a single method that implement this logic could be usefull also for other.
Do you see any limitation to this approach?

[Greg-DB]

Programmatically checking the file size to determine which style of uploading to use if a good practice. The example does that here:

https://github.com/dropbox/dropbox-sdk-java/blob/master/examples/upload-file/src/main/java/com/dropb...