<?xml version="1.0" encoding="UTF-8"?>
<rss xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#" xmlns:taxo="http://purl.org/rss/1.0/modules/taxonomy/" version="2.0">
  <channel>
    <title>topic unreliable data transfer w/ API in Dropbox API Support &amp; Feedback</title>
    <link>https://www.dropboxforum.com/t5/Dropbox-API-Support-Feedback/unreliable-data-transfer-w-API/m-p/55551#M1693</link>
    <description>&lt;P&gt;Hi,&lt;BR /&gt;
I'm using API to transfer data from my computer to Dropbox. For 50 500MB files 5 have failed with the error like this:&lt;/P&gt;

&lt;PRE&gt;&lt;CODE&gt;Uploading "/var/lib/condor/execute/dir_2220/oneA.e15.oswrk136_500GB.data" to "/bigWrkA/oneA.e15.oswrk136_500GB.data"..*** FAILED
An error occurred requesting /chunked_upload
&lt;/CODE&gt;&lt;/PRE&gt;

&lt;P&gt;Can you advice me how to overcome this issue?&lt;BR /&gt;
Thanks&lt;BR /&gt;
Jan&lt;/P&gt;</description>
    <pubDate>Wed, 29 May 2019 09:42:04 GMT</pubDate>
    <dc:creator>Jan B.23</dc:creator>
    <dc:date>2019-05-29T09:42:04Z</dc:date>
    <item>
      <title>unreliable data transfer w/ API</title>
      <link>https://www.dropboxforum.com/t5/Dropbox-API-Support-Feedback/unreliable-data-transfer-w-API/m-p/55551#M1693</link>
      <description>&lt;P&gt;Hi,&lt;BR /&gt;
I'm using API to transfer data from my computer to Dropbox. For 50 500MB files 5 have failed with the error like this:&lt;/P&gt;

&lt;PRE&gt;&lt;CODE&gt;Uploading "/var/lib/condor/execute/dir_2220/oneA.e15.oswrk136_500GB.data" to "/bigWrkA/oneA.e15.oswrk136_500GB.data"..*** FAILED
An error occurred requesting /chunked_upload
&lt;/CODE&gt;&lt;/PRE&gt;

&lt;P&gt;Can you advice me how to overcome this issue?&lt;BR /&gt;
Thanks&lt;BR /&gt;
Jan&lt;/P&gt;</description>
      <pubDate>Wed, 29 May 2019 09:42:04 GMT</pubDate>
      <guid>https://www.dropboxforum.com/t5/Dropbox-API-Support-Feedback/unreliable-data-transfer-w-API/m-p/55551#M1693</guid>
      <dc:creator>Jan B.23</dc:creator>
      <dc:date>2019-05-29T09:42:04Z</dc:date>
    </item>
    <item>
      <title>Re: unreliable data transfer w/ API</title>
      <link>https://www.dropboxforum.com/t5/Dropbox-API-Support-Feedback/unreliable-data-transfer-w-API/m-p/55552#M1694</link>
      <description>&lt;P&gt;What's the actual error you're getting when making the failed calls? What chunk size are you using?&lt;/P&gt;</description>
      <pubDate>Tue, 16 Jun 2015 02:49:18 GMT</pubDate>
      <guid>https://www.dropboxforum.com/t5/Dropbox-API-Support-Feedback/unreliable-data-transfer-w-API/m-p/55552#M1694</guid>
      <dc:creator>Greg-DB</dc:creator>
      <dc:date>2015-06-16T02:49:18Z</dc:date>
    </item>
    <item>
      <title>Re: unreliable data transfer w/ API</title>
      <link>https://www.dropboxforum.com/t5/Dropbox-API-Support-Feedback/unreliable-data-transfer-w-API/m-p/55553#M1695</link>
      <description>&lt;P&gt;What’s  the actual error you're getting when making the failed calls?&lt;/P&gt;

&lt;P&gt;The stderr reports for last 3 chunks of this file:&lt;/P&gt;

&lt;PRE&gt;&lt;CODE&gt;+ CHUNK_PARAMS='upload_id=0B3fybsi14V1MmpZ18As_Q&amp;amp;offset=209715200'
++ utime
+++ date +%s
++ echo 1434394232
+ curl -s --show-error --globoff -i -o /tmp/du_resp_debug --upload-file /tmp/du_chunk_1035 'https://api-content.dropbox.com/1/chunked_upload?upload_id=0B3fybsi14V1MmpZ18As_Q&amp;amp;offset=209715200&amp;amp;oauth_consumer_key=xeg3vxakd86svsh&amp;amp;oauth_token=jm624faonumhhxl9&amp;amp;oauth_signature_method=PLAINTEXT&amp;amp;oauth_signature=&amp;lt;redacted&amp;gt;%&amp;lt;redacted&amp;gt;&amp;amp;oauth_timestamp=1434394232&amp;amp;oauth_nonce=14453'
+ check_http_response
+ CODE=0
+ case $CODE in
+ grep -q 'HTTP/1.1 400' /tmp/du_resp_debug
+ grep -q '^HTTP/1.1 200 OK' /tmp/du_resp_debug
+ print '*'
+ [[ 0 == 0 ]]
+ echo -ne '*'
+ let UPLOAD_ERROR=0+1
+ [[ 1 -gt 2 ]]
+ [[ 209715200 != 524288000 ]]
+ let OFFSET_MB=209715200/1024/1024
+ dd if=/var/lib/condor/execute/dir_2220/oneA.e15.oswrk136_500GB.data of=/tmp/du_chunk_1035 bs=1048576 skip=200 count=100
+ [[ 209715200 != 0 ]]
+ CHUNK_PARAMS='upload_id=0B3fybsi14V1MmpZ18As_Q&amp;amp;offset=209715200'
++ utime
+++ date +%s
++ echo 1434394296
+ curl -s --show-error --globoff -i -o /tmp/du_resp_debug --upload-file /tmp/du_chunk_1035 'https://api-content.dropbox.com/1/chunked_upload?upload_id=0B3fybsi14V1MmpZ18As_Q&amp;amp;offset=209715200&amp;amp;oauth_consumer_key=xeg3vxakd86svsh&amp;amp;oauth_token=jm624faonumhhxl9&amp;amp;oauth_signature_method=PLAINTEXT&amp;amp;oauth_signature=&amp;lt;redacted&amp;gt;%&amp;lt;redacted&amp;gt;&amp;amp;oauth_timestamp=1434394296&amp;amp;oauth_nonce=5737'
+ check_http_response
+ CODE=0
+ case $CODE in
+ grep -q 'HTTP/1.1 400' /tmp/du_resp_debug
++ sed -n -e 's/{"error": "\([^"]*\)"}/\1/p' /tmp/du_resp_debug
+ ERROR_MSG=
+ case $ERROR_MSG in
+ grep -q '^HTTP/1.1 200 OK' /tmp/du_resp_debug
+ print '*'
+ [[ 0 == 0 ]]
+ echo -ne '*'
+ let UPLOAD_ERROR=1+1
+ [[ 2 -gt 2 ]]
+ [[ 209715200 != 524288000 ]]
+ let OFFSET_MB=209715200/1024/1024
+ dd if=/var/lib/condor/execute/dir_2220/oneA.e15.oswrk136_500GB.data of=/tmp/du_chunk_1035 bs=1048576 skip=200 count=100
+ [[ 209715200 != 0 ]]
+ CHUNK_PARAMS='upload_id=0B3fybsi14V1MmpZ18As_Q&amp;amp;offset=209715200'
++ utime
+++ date +%s
++ echo 1434394336
+ curl -s --show-error --globoff -i -o /tmp/du_resp_debug --upload-file /tmp/du_chunk_1035 'https://api-content.dropbox.com/1/chunked_upload?upload_id=0B3fybsi14V1MmpZ18As_Q&amp;amp;offset=209715200&amp;amp;oauth_consumer_key=xeg3vxakd86svsh&amp;amp;oauth_token=jm624faonumhhxl9&amp;amp;oauth_signature_method=PLAINTEXT&amp;amp;oauth_signature=&amp;lt;redacted&amp;gt;%&amp;lt;redacted&amp;gt;&amp;amp;oauth_timestamp=1434394336&amp;amp;oauth_nonce=21291'
+ check_http_response
+ CODE=0
+ case $CODE in
+ grep -q 'HTTP/1.1 400' /tmp/du_resp_debug
++ sed -n -e 's/{"error": "\([^"]*\)"}/\1/p' /tmp/du_resp_debug
+ ERROR_MSG=
+ case $ERROR_MSG in
+ grep -q '^HTTP/1.1 200 OK' /tmp/du_resp_debug
+ print '*'
+ [[ 0 == 0 ]]
+ echo -ne '*'
+ let UPLOAD_ERROR=2+1
+ [[ 3 -gt 2 ]]
+ print ' FAILED\n'
+ [[ 0 == 0 ]]
+ echo -ne ' FAILED\n'
+ print 'An error occurred requesting /chunked_upload\n'
+ [[ 0 == 0 ]]
+ echo -ne 'An error occurred requesting /chunked_upload\n'
+ ERROR_STATUS=1
+ return
+ (( i++  ))
+ (( i&amp;lt;6 ))
+ remove_temp_files
+ [[ 1 == 0 ]]
+ exit 1
&lt;/CODE&gt;&lt;/PRE&gt;

&lt;H1&gt;====================================================================================&lt;/H1&gt;

&lt;H1&gt;====================================================================================&lt;/H1&gt;

&lt;P&gt;What chunk size are you using?&lt;/P&gt;

&lt;P&gt;CHUNK_SIZE=100&lt;/P&gt;

&lt;P&gt;It is in MB.  The code sending data looks like this:&lt;/P&gt;

&lt;PRE&gt;&lt;CODE&gt;#Chunked file upload
#$1 = Local source file
#$2 = Remote destination file
function db_chunked_upload_file
{
    local FILE_SRC=$(normalize_path "$1")
    local FILE_DST=$(normalize_path "$2")

    print " &amp;gt; Uploading \"$FILE_SRC\" to \"$FILE_DST\"

    local FILE_SIZE=$(file_size "$FILE_SRC")
    local OFFSET=0
    local UPLOAD_ID="
    local UPLOAD_ERROR=0

    local CHUNK_PARAMS="

    #Uploading chunks...
    while ([[ $OFFSET != $FILE_SIZE ]]); do

        let OFFSET_MB=$OFFSET/1024/1024

        #Create the chunk
        dd if="$FILE_SRC" of="$CHUNK_FILE" bs=1048576 skip=$OFFSET_MB count=$CHUNK_SIZE 2&amp;gt; /dev/null

        #Only for the first request these parameters are not included
        if [[ $OFFSET != 0 ]]; then
            CHUNK_PARAMS="upload_id=$UPLOAD_ID&amp;amp;offset=$OFFSET"
        fi

        #Uploading the chunk...
        $CURL_BIN $CURL_ACCEPT_CERTIFICATES -s --show-error --globoff -i -o "$RESPONSE_FILE" --upload-file "$CHUNK_FILE" "$API_CHUNKED_UPLOAD_URL?$CHUNK_PARAMS&amp;amp;oauth_consumer_key=$APPKEY&amp;amp;oauth_token=$OAUTH_ACCESS_TOKEN&amp;amp;oauth_signature_method=PLAINTEXT&amp;amp;oauth_signature=$APPSECRET%26$OAUTH_ACCESS_TOKEN_SECRET&amp;amp;oauth_timestamp=$(utime)&amp;amp;oauth_nonce=$RANDOM" 2&amp;gt; /dev/null
        check_http_response

        #Check
        if grep -q "^HTTP/1.1 200 OK" "$RESPONSE_FILE"; then
            print "."
            UPLOAD_ERROR=0
            UPLOAD_ID=$(sed -n 's/.*"upload_id": *"*\([^"]*\)"*.*/\1/p' "$RESPONSE_FILE")
            OFFSET=$(sed -n 's/.*"offset": *\([^}]*\).*/\1/p' "$RESPONSE_FILE")
        else
            print "*"
            let UPLOAD_ERROR=$UPLOAD_ERROR+1

            #On error, the upload is retried for max 3 times
            if [[ $UPLOAD_ERROR -gt 2 ]]; then
                print " FAILED\n"
                print "An error occurred requesting /chunked_upload\n"
                ERROR_STATUS=1
                return
            fi
        fi

    done

    UPLOAD_ERROR=0

    #Commit the upload
    while (true); do

        $CURL_BIN $CURL_ACCEPT_CERTIFICATES -s --show-error --globoff -i -o "$RESPONSE_FILE" --data "upload_id=$UPLOAD_ID&amp;amp;oauth_consumer_key=$APPKEY&amp;amp;oauth_token=$OAUTH_ACCESS_TOKEN&amp;amp;oauth_signature_method=PLAINTEXT&amp;amp;oauth_signature=$APPSECRET%26$OAUTH_ACCESS_TOKEN_SECRET&amp;amp;oauth_timestamp=$(utime)&amp;amp;oauth_nonce=$RANDOM" "$API_CHUNKED_UPLOAD_COMMIT_URL/$ACCESS_LEVEL/$(urlencode "$FILE_DST")" 2&amp;gt; /dev/null
        check_http_response

        #Check
        if grep -q "^HTTP/1.1 200 OK" "$RESPONSE_FILE"; then
            print "."
            UPLOAD_ERROR=0
            break
        else
            print "*"
            let UPLOAD_ERROR=$UPLOAD_ERROR+1

            #On error, the commit is retried for max 3 times
            if [[ $UPLOAD_ERROR -gt 2 ]]; then
                print " FAILED\n"
                print "An error occurred requesting /commit_chunked_upload\n"
                ERROR_STATUS=1
                return
            fi
        fi

    done

    print " DONE\n"
}
&lt;/CODE&gt;&lt;/PRE&gt;</description>
      <pubDate>Tue, 16 Jun 2015 06:44:51 GMT</pubDate>
      <guid>https://www.dropboxforum.com/t5/Dropbox-API-Support-Feedback/unreliable-data-transfer-w-API/m-p/55553#M1695</guid>
      <dc:creator>Jan B.23</dc:creator>
      <dc:date>2015-06-16T06:44:51Z</dc:date>
    </item>
    <item>
      <title>Re: unreliable data transfer w/ API</title>
      <link>https://www.dropboxforum.com/t5/Dropbox-API-Support-Feedback/unreliable-data-transfer-w-API/m-p/55554#M1696</link>
      <description>&lt;P&gt;Thanks for the additional information. I don't think I see the actual response or error in there though. If you get a response from the API, it should contain an error message in the body. If you don't (e.g., if the connection fails) I would expect curl to show a network error of some sort. Can you print those out?&lt;/P&gt;

&lt;P&gt;In any case, 100 MB is a pretty large chunk size, so I recommend trying something smaller, perhaps 50 MB or smaller. (API calls for large chunks can time out.) Let me know if that doesn't help.&lt;/P&gt;</description>
      <pubDate>Wed, 17 Jun 2015 01:37:22 GMT</pubDate>
      <guid>https://www.dropboxforum.com/t5/Dropbox-API-Support-Feedback/unreliable-data-transfer-w-API/m-p/55554#M1696</guid>
      <dc:creator>Greg-DB</dc:creator>
      <dc:date>2015-06-17T01:37:22Z</dc:date>
    </item>
  </channel>
</rss>

