Need to see if your shared folder is taking up space on your dropbox 👨‍💻? Find out how to check here.

Forum Discussion

Jan B.23's avatar
Jan B.23
New member | Level 1
11 years ago

unreliable data transfer w/ API

Hi,
I'm using API to transfer data from my computer to Dropbox. For 50 500MB files 5 have failed with the error like this:

Uploading "/var/lib/condor/execute/dir_2220/oneA.e15.oswrk136_500GB.data" to "/bigWrkA/oneA.e15.oswrk136_500GB.data"..*** FAILED
An error occurred requesting /chunked_upload

Can you advice me how to overcome this issue?
Thanks
Jan

3 Replies

Replies have been turned off for this discussion
  • Greg-DB's avatar
    Greg-DB
    Icon for Dropbox Community Moderator rankDropbox Community Moderator
    11 years ago

    What's the actual error you're getting when making the failed calls? What chunk size are you using?

  • Jan B.23's avatar
    Jan B.23
    New member | Level 1
    11 years ago

    What’s the actual error you're getting when making the failed calls?

    The stderr reports for last 3 chunks of this file:

    + CHUNK_PARAMS='upload_id=0B3fybsi14V1MmpZ18As_Q&offset=209715200'
    ++ utime
    +++ date +%s
    ++ echo 1434394232
    + curl -s --show-error --globoff -i -o /tmp/du_resp_debug --upload-file /tmp/du_chunk_1035 'https://api-content.dropbox.com/1/chunked_upload?upload_id=0B3fybsi14V1MmpZ18As_Q&offset=209715200&oauth_consumer_key=xeg3vxakd86svsh&oauth_token=jm624faonumhhxl9&oauth_signature_method=PLAINTEXT&oauth_signature=<redacted>%<redacted>&oauth_timestamp=1434394232&oauth_nonce=14453'
    + check_http_response
    + CODE=0
    + case $CODE in
    + grep -q 'HTTP/1.1 400' /tmp/du_resp_debug
    + grep -q '^HTTP/1.1 200 OK' /tmp/du_resp_debug
    + print '*'
    + [[ 0 == 0 ]]
    + echo -ne '*'
    + let UPLOAD_ERROR=0+1
    + [[ 1 -gt 2 ]]
    + [[ 209715200 != 524288000 ]]
    + let OFFSET_MB=209715200/1024/1024
    + dd if=/var/lib/condor/execute/dir_2220/oneA.e15.oswrk136_500GB.data of=/tmp/du_chunk_1035 bs=1048576 skip=200 count=100
    + [[ 209715200 != 0 ]]
    + CHUNK_PARAMS='upload_id=0B3fybsi14V1MmpZ18As_Q&offset=209715200'
    ++ utime
    +++ date +%s
    ++ echo 1434394296
    + curl -s --show-error --globoff -i -o /tmp/du_resp_debug --upload-file /tmp/du_chunk_1035 'https://api-content.dropbox.com/1/chunked_upload?upload_id=0B3fybsi14V1MmpZ18As_Q&offset=209715200&oauth_consumer_key=xeg3vxakd86svsh&oauth_token=jm624faonumhhxl9&oauth_signature_method=PLAINTEXT&oauth_signature=<redacted>%<redacted>&oauth_timestamp=1434394296&oauth_nonce=5737'
    + check_http_response
    + CODE=0
    + case $CODE in
    + grep -q 'HTTP/1.1 400' /tmp/du_resp_debug
    ++ sed -n -e 's/{"error": "\([^"]*\)"}/\1/p' /tmp/du_resp_debug
    + ERROR_MSG=
    + case $ERROR_MSG in
    + grep -q '^HTTP/1.1 200 OK' /tmp/du_resp_debug
    + print '*'
    + [[ 0 == 0 ]]
    + echo -ne '*'
    + let UPLOAD_ERROR=1+1
    + [[ 2 -gt 2 ]]
    + [[ 209715200 != 524288000 ]]
    + let OFFSET_MB=209715200/1024/1024
    + dd if=/var/lib/condor/execute/dir_2220/oneA.e15.oswrk136_500GB.data of=/tmp/du_chunk_1035 bs=1048576 skip=200 count=100
    + [[ 209715200 != 0 ]]
    + CHUNK_PARAMS='upload_id=0B3fybsi14V1MmpZ18As_Q&offset=209715200'
    ++ utime
    +++ date +%s
    ++ echo 1434394336
    + curl -s --show-error --globoff -i -o /tmp/du_resp_debug --upload-file /tmp/du_chunk_1035 'https://api-content.dropbox.com/1/chunked_upload?upload_id=0B3fybsi14V1MmpZ18As_Q&offset=209715200&oauth_consumer_key=xeg3vxakd86svsh&oauth_token=jm624faonumhhxl9&oauth_signature_method=PLAINTEXT&oauth_signature=<redacted>%<redacted>&oauth_timestamp=1434394336&oauth_nonce=21291'
    + check_http_response
    + CODE=0
    + case $CODE in
    + grep -q 'HTTP/1.1 400' /tmp/du_resp_debug
    ++ sed -n -e 's/{"error": "\([^"]*\)"}/\1/p' /tmp/du_resp_debug
    + ERROR_MSG=
    + case $ERROR_MSG in
    + grep -q '^HTTP/1.1 200 OK' /tmp/du_resp_debug
    + print '*'
    + [[ 0 == 0 ]]
    + echo -ne '*'
    + let UPLOAD_ERROR=2+1
    + [[ 3 -gt 2 ]]
    + print ' FAILED\n'
    + [[ 0 == 0 ]]
    + echo -ne ' FAILED\n'
    + print 'An error occurred requesting /chunked_upload\n'
    + [[ 0 == 0 ]]
    + echo -ne 'An error occurred requesting /chunked_upload\n'
    + ERROR_STATUS=1
    + return
    + (( i++  ))
    + (( i<6 ))
    + remove_temp_files
    + [[ 1 == 0 ]]
    + exit 1
    

    ====================================================================================

    ====================================================================================

    What chunk size are you using?

    CHUNK_SIZE=100

    It is in MB. The code sending data looks like this:

    #Chunked file upload
    #$1 = Local source file
    #$2 = Remote destination file
    function db_chunked_upload_file
    {
        local FILE_SRC=$(normalize_path "$1")
        local FILE_DST=$(normalize_path "$2")
    
        print " > Uploading \"$FILE_SRC\" to \"$FILE_DST\"
    
        local FILE_SIZE=$(file_size "$FILE_SRC")
        local OFFSET=0
        local UPLOAD_ID="
        local UPLOAD_ERROR=0
    
        local CHUNK_PARAMS="
    
        #Uploading chunks...
        while ([[ $OFFSET != $FILE_SIZE ]]); do
    
            let OFFSET_MB=$OFFSET/1024/1024
    
            #Create the chunk
            dd if="$FILE_SRC" of="$CHUNK_FILE" bs=1048576 skip=$OFFSET_MB count=$CHUNK_SIZE 2> /dev/null
    
            #Only for the first request these parameters are not included
            if [[ $OFFSET != 0 ]]; then
                CHUNK_PARAMS="upload_id=$UPLOAD_ID&offset=$OFFSET"
            fi
    
            #Uploading the chunk...
            $CURL_BIN $CURL_ACCEPT_CERTIFICATES -s --show-error --globoff -i -o "$RESPONSE_FILE" --upload-file "$CHUNK_FILE" "$API_CHUNKED_UPLOAD_URL?$CHUNK_PARAMS&oauth_consumer_key=$APPKEY&oauth_token=$OAUTH_ACCESS_TOKEN&oauth_signature_method=PLAINTEXT&oauth_signature=$APPSECRET%26$OAUTH_ACCESS_TOKEN_SECRET&oauth_timestamp=$(utime)&oauth_nonce=$RANDOM" 2> /dev/null
            check_http_response
    
            #Check
            if grep -q "^HTTP/1.1 200 OK" "$RESPONSE_FILE"; then
                print "."
                UPLOAD_ERROR=0
                UPLOAD_ID=$(sed -n 's/.*"upload_id": *"*\([^"]*\)"*.*/\1/p' "$RESPONSE_FILE")
                OFFSET=$(sed -n 's/.*"offset": *\([^}]*\).*/\1/p' "$RESPONSE_FILE")
            else
                print "*"
                let UPLOAD_ERROR=$UPLOAD_ERROR+1
    
                #On error, the upload is retried for max 3 times
                if [[ $UPLOAD_ERROR -gt 2 ]]; then
                    print " FAILED\n"
                    print "An error occurred requesting /chunked_upload\n"
                    ERROR_STATUS=1
                    return
                fi
            fi
    
        done
    
        UPLOAD_ERROR=0
    
        #Commit the upload
        while (true); do
    
            $CURL_BIN $CURL_ACCEPT_CERTIFICATES -s --show-error --globoff -i -o "$RESPONSE_FILE" --data "upload_id=$UPLOAD_ID&oauth_consumer_key=$APPKEY&oauth_token=$OAUTH_ACCESS_TOKEN&oauth_signature_method=PLAINTEXT&oauth_signature=$APPSECRET%26$OAUTH_ACCESS_TOKEN_SECRET&oauth_timestamp=$(utime)&oauth_nonce=$RANDOM" "$API_CHUNKED_UPLOAD_COMMIT_URL/$ACCESS_LEVEL/$(urlencode "$FILE_DST")" 2> /dev/null
            check_http_response
    
            #Check
            if grep -q "^HTTP/1.1 200 OK" "$RESPONSE_FILE"; then
                print "."
                UPLOAD_ERROR=0
                break
            else
                print "*"
                let UPLOAD_ERROR=$UPLOAD_ERROR+1
    
                #On error, the commit is retried for max 3 times
                if [[ $UPLOAD_ERROR -gt 2 ]]; then
                    print " FAILED\n"
                    print "An error occurred requesting /commit_chunked_upload\n"
                    ERROR_STATUS=1
                    return
                fi
            fi
    
        done
    
        print " DONE\n"
    }
    
  • Greg-DB's avatar
    Greg-DB
    Icon for Dropbox Community Moderator rankDropbox Community Moderator
    11 years ago

    Thanks for the additional information. I don't think I see the actual response or error in there though. If you get a response from the API, it should contain an error message in the body. If you don't (e.g., if the connection fails) I would expect curl to show a network error of some sort. Can you print those out?

    In any case, 100 MB is a pretty large chunk size, so I recommend trying something smaller, perhaps 50 MB or smaller. (API calls for large chunks can time out.) Let me know if that doesn't help.

About Dropbox API Support & Feedback

Node avatar for Dropbox API Support & Feedback
Find help with the Dropbox API from other developers.

The Dropbox Community team is active from Monday to Friday. We try to respond to you as soon as we can, usually within 2 hours.

If you need more help you can view your support options (expected response time for an email or ticket is 24 hours), or contact us on X, Facebook or Instagram.

For more info on available support options for your Dropbox plan, see this article.

If you found the answer to your question in this Community thread, please 'like' the post to say thanks and to let us know it was useful!