cancel
Showing results for 
Show  only  | Search instead for 
Did you mean: 
Announcements
Share your feedback on the Document Scanning Experience in the Dropbox App right here.

Apps and Installations

Have a question about a Dropbox app or installation? Reach out to the Dropbox Community and get solutions, help, and advice from members.

cancel
Showing results for 
Show  only  | Search instead for 
Did you mean: 

Re: Some files not syncing from Mac computer to Ubuntu but others do

Some files not syncing from Mac computer to Ubuntu but others do

BrunoJ
Explorer | Level 3

Dear all,

I have been banging my head against the wall with this one...

 

Some of my files do not sync to my Ubuntu computer if I create them in my Mac computer  -other do without any problem- but all files I create in my Ubuntu  show up in my Mac computer. I have the global settings on synchronize all files and folders in both computers.

 

I have cleaned both dropbox caches, I have stopped and restarted dropbox in both computers. They all show up in the dropbox site so they are all being synchronized to the dropbox servers. It's just the direction from Mac to Ubuntu that sometimes works and sometimes not!

 

Any ideas? Thoughts?

 

Thanks in advance for any help you might provide,

Bruno

20 Replies 20

Greg-DB
Dropbox Staff

@Здравко Unfortunately I don't think the API has anything that would be useful for this. Anyone seeing an issue like this may want to contact support directly.

Здравко
Legendary | Level 20

@Greg-DB wrote:

... Anyone seeing an issue like this may want to contact support directly.


😁😁😁 Greg, that has to be the funny part. Do you really think somebody there can help? Rhetoric, of course. It's clear there is an application bug, at least.

Anyway...

My question was - as there are some criteria/flags forcing application to sync or not, associated to each file (2 copy of the same file - one sync the other no and without visible reason way) - is there a way those criteria/flags be read (might appear somewhere as part of some call result). I will interpret your answer as "mentioned feature is not accessible throughout Dropbox API and only available for internal use".

Thanks.

Здравко
Legendary | Level 20

Hi @FesT,

As a temp workaround (till possible solution, if any) you can use following script:

#!/usr/bin/bash
##############################################################################
#                  Workaround for Dropbox broken sync
#                  ----------------------------------
# Verified on Dropbox v159.4.5870
# Just make it executable (if need, using
#   $ chmod a+x dropbox_syncfix
# ) and run it.
# Author: Здравко
#   www.dropboxforum.com/t5/user/viewprofilepage/user-id/422790
##############################################################################

# General command line check.
if [ $# -eq 0 ]
then
  echo "Command line should be 'dropbox_syncfix <access token>'."
  echo "The access token seems missing!"
  echo "You can get it here: https://dropbox.github.io/dropbox-api-v2-explorer/#check_user"
  exit 1
fi
if [ $# -gt 1 ]
then
  echo "Command line should be 'dropbox_syncfix <access token>'."
  echo "Unexpected aguments passed!"
  exit 1
fi

##############################################################################
# Access token validity confirmation.
randstr=$(
tmpbase='0123456789abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ'
for ((i=0; i<16; ++i)); do echo -n "${tmpbase:RANDOM%${#tmpbase}:1}"; done
)
res=`curl -X POST https://api.dropboxapi.com/2/check/user -H "Authorization: Bearer $1" -H 'Content-Type: application/json' -d "{\"query\":\"$randstr\"}" 2> /dev/null | jq -r .result`
if [[ $res != $randstr ]]
then
  echo "The first argument ($1) is NOT valid access token!"
  exit 1
fi
access_token="$1"

##############################################################################
# Figuring out local Dropbox folder location.
dbx_dir=`jq -r .personal.path ~/.dropbox/info.json`
if [ $? -ne 0 ]
then
  echo "Cannot detect Dropbox application installed and worked local."
  exit 1
fi
if [ ! -d "$dbx_dir" ]
then
  echo "Incorrect Dropbox folder path ($dbx_dir)! Only individual Dropbox account is supported."
  exit 1
fi

##############################################################################
# Single file sunc. $1 - relative file path in your account; $2 - local Dropbox absolute path
sync_to_local() {
  local path="$2$1"
  echo "Processing file: '$path'..."
  local error=`curl -o /tmp/dropbox_tmp.file -D /tmp/dropbox_tmp.dump -X POST https://content.dropboxapi.com/2/files/download -H "Authorization: Bearer $access_token" -H "Dropbox-API-Arg: {\"path\":\"$1\"}" 2> /dev/null`
  if [ $? -ne 0 ]
  then
    echo "👺 Trouble fetching '$1' from Dropbox"
    return
  fi
  local file_date=`grep 'Dropbox-Api-Result' /tmp/dropbox_tmp.dump | sed 's/^Dropbox-Api-Result *\: *//' | jq -r .client_modified`
  if [[ "$file_date" == "null" ]]
  then
    echo "👹 Download '$1' error: $error"
    return
  fi
  touch -d "$file_date" /tmp/dropbox_tmp.file
  local res=`curl -X POST https://api.dropboxapi.com/2/files/delete_v2 -H "Authorization: Bearer $access_token" -H 'Content-Type: application/json' -d "{\"path\":\"$1\"}" 2> /dev/null`
  if [[ `echo "$res" | jq -r '.metadata[".tag"]'` != "file" ]]
  then
    echo "🤷 Cannot remove '$1', error: $res"
    return
  fi
  sleep 1
  while [[ `curl -X POST https://api.dropboxapi.com/2/files/get_metadata -H "Authorization: Bearer $access_token" -H 'Content-Type: application/json' -d "{\"path\":\"$1\"}" 2> /dev/null | jq .error_summary` == "null" ]]
  do
    sleep 1
  done
  mv -u /tmp/dropbox_tmp.file "$path"
  echo "'$path' processed."
}

##############################################################################
# Processing single enumeration step. $1 - current listing result; $2 - current folder relative path
enumerate_files() {
  local oldIFS="$IFS"
  IFS=$'\n'
  for path in $(echo "$1" | jq -r ".entries[] | select(.[\".tag\"] == \"file\").name")
  do
    if [ ! -f "$dbx_dir$2/$path" ]
    then
      sync_to_local "$2/$path" "$dbx_dir"
    fi
  done
  for path in $(echo "$1" | jq -r ".entries[] | select(.[\".tag\"] == \"folder\").name")
  do
    parse_folder "$2/$path"
  done
  IFS="$oldIFS"
}

##############################################################################
# Processing single folder (indirect recursion). $1 - relative folder path
parse_folder() {
  local res=`curl -X POST https://api.dropboxapi.com/2/files/list_folder -H "Authorization: Bearer $access_token" -H 'Content-Type: application/json' -d "{\"path\":\"$1\",\"include_non_downloadable_files\":false}" 2> /dev/null`
  echo "$1"
  enumerate_files "$res" "$1"
  while [[ "`echo "$res" | jq .has_more`" == "true" ]]
  do
    res=`curl -X POST https://api.dropboxapi.com/2/files/list_folder/continue -H "Authorization: Bearer $access_token" -H 'Content-Type: application/json' -d "{\"cursor\":$(echo "$res" | jq .cursor)}" 2> /dev/null`
    echo "$1"
    enumerate_files "$res" "$1"
  done
}

# Starting enumeration  with the account root.
parse_folder ''

##############################################################################
# Wiping the 'dust'.
rm -f /tmp/dropbox_tmp.file /tmp/dropbox_tmp.dump
echo
echo "Complete! 😉"
echo

 Save the script in a file with name like dropbox_syncfix on you Linux. After that, open a terminal window and change the current folder to the one where you have script saved. Execute in the same shell a command like:

 

chmod a+x dropbox_syncfix

 

Now the script is ready to start. As the only argument you need access token. You can get such from here. Make sure at the same moment you're logged in the same account (in the web browser) as you used on local machine (I don't check it)! Click 'Get Token' button and a sequence of symbols will appear in field labeled 'Access Token'. Execute the script as follows:

 

./dropbox_syncfix PUT_YOUR_TOKEN_HERE

 

The script will enumerate your entire account tree folder by folder and when meet some troublesome file will try to fix it. Such a process can (and usually will) take a lot of time, so be patient. You can let your Linux machine work alone or do some other job there. The script is neither processor nor network load intensive. 😉

Hope this helps to some extent.

BrunoJ
Explorer | Level 3

Hi @Здравко

 

Many thanks for your solution! It looks awesome,  but it seems to be me (a basic user) it feels too much of a hassle to do all that for I service that I pay a considerable amount of money per year. Frankly, if the problem was much bigger than just a few photoshop files I would just consider changing to another cloud/backup system. I work at an university so I could easily change to an internal system we have... I'm just lazy and as long as dropbox works I'll keep using it instead 😉

 

Hopeful this trend and your work will help the dropbox developers with fixing this issue?

 

Best regards and many thanks for your work,

Bruno

Здравко
Legendary | Level 20

@BrunoJ wrote:

...

Hopeful this trend and your work will help the dropbox developers with fixing this issue?

...


🙂🤔 Hm.. Good wish, but better don't rely on. My experience shows - if the application doesn't stop completely and some part of the UI doesn't become too 'ugly', Dropbox development doesn't take care. 🤷 Linux is NOT first class citizen here, so everybody should save himself, unfortunately.

My script is definitely not a final solution; it's rather workaround using 'brute force' file fixing (cosa there's not know what exactly force wrong behavior). Nowhere can be seen any documentation describing the possible file attributes leading to such thing. That's why, in my script, I'm clearing all files attributes (accessible or not - in fact files regeneration).

 


@BrunoJ wrote:

...,  but it seems to be me (a basic user) it feels too much of a hassle to do all that for I service that I pay a considerable amount of money per year. ...


Fully agree that once a user's paying (and not only), the service should be the same everywhere (on every supported platform). 😡 Unfortunately, it's not and, as I said, everybody is looking for solution outside Dropbox (or directly another service - there are competitors).

 


@BrunoJ wrote:

... Frankly, if the problem was much bigger than just a few photoshop files ...


Especially for you I made some changes in above script and I'm posting it here:

#!/usr/bin/bash
##############################################################################
#                  Workaround for Dropbox broken sync
#                  ----------------------------------
# Verified on Dropbox v159.4.5870
# Just make it executable (if need, using
#   $ chmod a+x dropbox_syncfix
# ) and run it.
# Author: Здравко
#   www.dropboxforum.com/t5/user/viewprofilepage/user-id/422790
##############################################################################

# General command line check.
if [ $# -eq 0 ]
then
  echo "Command line should be 'dropbox_syncfix <access token> [<folder>]'."
  echo "The access token seems missing!"
  echo "You can get it here:"
  echo "https://dropbox.github.io/dropbox-api-v2-explorer/#check_user"
  exit 1
fi
if [ $# -gt 2 ]
then
  echo "Command line should be 'dropbox_syncfix <access token> [<folder>]'."
  echo "Unexpected number of arguments passed!"
  exit 1
fi

##############################################################################
# Access token validity confirmation.
randstr=$(
tmpbase='0123456789abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ'
for ((i=0; i<16; ++i)); do echo -n "${tmpbase:RANDOM%${#tmpbase}:1}"; done
)
res=`curl -X POST https://api.dropboxapi.com/2/check/user \
  -H "Authorization: Bearer $1" -H 'Content-Type: application/json' \
  -d "{\"query\":\"$randstr\"}" 2> /dev/null | jq -r .result`
if [[ $res != $randstr ]]
then
  echo "👆 The first argument ($1) is NOT valid access token!"
  exit 1
fi
access_token="$1"

##############################################################################
# Figuring out local Dropbox folder location.
dbx_dir=`jq -r .personal.path ~/.dropbox/info.json`
if [ $? -ne 0 ]
then
  echo "Cannot detect Dropbox application installed and worked local."
  exit 1
fi
if [ ! -d "$dbx_dir" ]
then
  echo "Incorrect Dropbox folder path ($dbx_dir)!"
  echo "Only individual Dropbox account is supported."
  exit 1
fi

##############################################################################
# Dealing with start folder path.

# Setting default path - the account root (empty string).
start_folder=''

# Checking for explicit start folder path.
if [ $# -eq 2 ]
then
  if [ ! -d "$2" ]
  then
    echo "😯 The second argument ($2) does NOT point a folder."
    exit 1
  fi
  start_folder=`realpath "$2"`
  if [[ "${start_folder:0:${#dbx_dir}}" != "$dbx_dir" ]]
  then
    echo -n "🤦 The pinted folder ($2) does NOT reside within your Dropbox "
    echo "folder ($dbx_dir)."
    exit 1
  fi
  start_folder="${start_folder:${#dbx_dir}}"
fi

##############################################################################
# Single file sunc.
# $1 - relative file path in your account
sync_to_local() {
  local path="$dbx_dir$1"
  echo "Processing file: '$path'..."
  local error=`curl -o /tmp/dropbox_tmp.file -D /tmp/dropbox_tmp.dump \
    -X POST https://content.dropboxapi.com/2/files/download \
    -H "Authorization: Bearer $access_token" \
    -H "Dropbox-API-Arg: {\"path\":\"$1\"}" 2> /dev/null`
  if [ $? -ne 0 ]
  then
    echo "👺 Trouble fetching '$1' from Dropbox"
    return
  fi
  local file_date=`grep 'Dropbox-Api-Result' /tmp/dropbox_tmp.dump | \
              sed 's/^Dropbox-Api-Result *\: *//' | jq -r .client_modified`
  if [[ "$file_date" == "null" ]]
  then
    echo "👹 Download '$1' error: $error"
    return
  fi
  touch -d "$file_date" /tmp/dropbox_tmp.file
  local res=`curl -X POST https://api.dropboxapi.com/2/files/delete_v2 \
    -H "Authorization: Bearer $access_token" \
    -H 'Content-Type: application/json' -d "{\"path\":\"$1\"}" 2> /dev/null`
  if [[ `echo "$res" | jq -r '.metadata[".tag"]'` != "file" ]]
  then
    echo "🤷 Cannot remove '$1', error: $res"
    return
  fi
  sleep 1
  while [[ `curl -X POST https://api.dropboxapi.com/2/files/get_metadata \
    -H "Authorization: Bearer $access_token" \
    -H 'Content-Type: application/json' -d "{\"path\":\"$1\"}" 2> /dev/null \
    | jq .error_summary` == "null" ]]
  do
    sleep 1
  done
  mv -u /tmp/dropbox_tmp.file "$path"
  echo "'$path' processed."
}

##############################################################################
# Processing single enumeration step.
# $1 - current listing result
# $2 - current folder relative path
enumerate_files() {
  local oldIFS="$IFS"
  IFS=$'\n'
  for path in `jq -r '.entries[] | select(.[".tag"] == "file").name' <<< "$1"`
  do
    if [ ! -f "$dbx_dir$2/$path" ]
    then
      sync_to_local "$2/$path"
    fi
  done
  for path in `jq -r '.entries[] | select(.[".tag"] == "folder").name' <<< "$1"`
  do
    parse_folder "$2/$path"
  done
  IFS="$oldIFS"
}

##############################################################################
# Processing single folder (indirect recursion).
# $1 - relative folder path
parse_folder() {
  local res=`curl -X POST https://api.dropboxapi.com/2/files/list_folder \
    -H "Authorization: Bearer $access_token" \
    -H 'Content-Type: application/json' \
    -d "{\"path\":\"$1\",\"include_non_downloadable_files\":false}" 2> /dev/null`
  echo "$1"
  enumerate_files "$res" "$1"
  while [[ `jq .has_more <<< "$res"` == "true" ]]
  do
    res=`curl -X POST https://api.dropboxapi.com/2/files/list_folder/continue \
      -H "Authorization: Bearer $access_token" \
      -H 'Content-Type: application/json' \
      -d "{\"cursor\":$(jq .cursor <<< "$res")}" 2> /dev/null`
    echo "$1"
    enumerate_files "$res" "$1"
  done
}

##############################################################################
# Starting enumeration  with the start folder.
parse_folder "$start_folder"

##############################################################################
# Wiping the 'dust'.
rm -f /tmp/dropbox_tmp.file /tmp/dropbox_tmp.dump

echo
echo "Complete! 😉"
echo

 The primary change (addition in fact) is the ability to provide folder path in your Dropbox folder. So if your troublesome files are located on limited part of the account folders tree and you know exactly where, providing path (second argument) can speed up script significant. The time needed can be just few minutes or, on very few files, less than a minute. 😉 Now you can use a call like:

./dropbox_syncfix PUT_YOUR_TOKEN_HERE "Dropbox/path/to/the/troubles"

Everything else is like on my previous post above.

 

PS: I may have forgotten to mention - in the script is used 'jq' tool (able handle JSON format used by Dropbox). The tool is relatively specific and that's why usually it's not preinstalled. You may need to install it explicitly, using your preferable package manager or using following command on Ubuntu (and compatible; other distributions have equivalent commands):

sudo apt-get install jq

 

FesT
Helpful | Level 7

@Здравко 

thank you for this script! Could you briefly explain what it does exactly?

Здравко
Legendary | Level 20

@FesT wrote:

... Could you briefly explain what it does exactly?


As I said already, main trick is recreation of every problematic file in the Dropbox namespace, so any undesirable attributes (assigned to the former file) get gone. This is the most brief variant.

Within the script itself are many comments, describing what exactly each part of the script does. You need only basic shell scripting knowledge to understand them. About the algorithm - enumeration starts from a target folder (by default the account root) and when non matching file comes up - passed for 'refresh and undressing' (if I can say so). After that the nested folders are processed in recursion, following the same rule of course. The enumeration (and other things, like download, etc) gets performed using corresponding Dropbox API access point. You can follow the links for more information. This is less brief variant.

The same is valid for both script variants. 👍 If something isn't clear enough, clarify your question.

Hope this helps.

FesT
Helpful | Level 7

Hi @Здравко,

thank you for the short explanation. So far I had already understood it, what I do not understand exactly is what for "undesirable attributes" are removed, and in what way they change the files.

 

Здравко
Legendary | Level 20

@FesT wrote:

..., what I do not understand exactly is what for "undesirable attributes" are removed, ...


Hm... 🤔🤷 To be honest I'm not sure what actually you are asking for. The attributes are undesirable because they break the syncing and that why they are removed! If these attributes are Ok for you, don't take a look on any of the scripts any more (you don't need them). As far as I understand, sync breaking is an issue. Am I right, or I might have misunderstanding something? 🧐

 


@FesT wrote:

..., and in what way they change the files.


The files themself are not changed in any way. File attributes are things "around" the file (things associated to a file, not file's integral part). Seems some Dropbox application instances (the one on Mac) attach to some files such attributes on upload (don't ask me why; seems nobody knows - including Dropbox staff). Removing these undesired attributes (hope you know already why 'undesired') seems resume syncing.

FesT
Helpful | Level 7

HAHA

 


@Здравко wrote:
...(don't ask me why; seems nobody knows - including Dropbox staff).

this is exactly what I some how tried to figure out... where these attributes are coming from, why there are there, and what happens if there are not there anymore (except that syncing is working again – which obviously is good 😁)

Anyhow thank you a lot for this great script!  

Need more support?