Synchronizing encrypted files

[xtremebytes]

Hi, here is a synchronization problem.

I have a list of files. Each is encrypted using a AES block cipher in CTR (counter) mode with its own password, a salt and an initialisation vector, where the latter two are randomly chosen at the time of encryption. What that means is that E(A, k1)_t0 is going to look very different from E(A, k2)_t1 even though A has not changed (where E(M, k)_t is the AES CTR mode encryption of plaintext M at time t with some encryption key k).

If the file has actually changed from A to A + delta_A, then it is not easy to detect the delta change in the encrypted domain. This is because the initialisation vector and salt are different every time, resulting in a ciphertext that is entirely different even for a single bit change. Thus, one cannot just patch separate blocks of the ciphertext and expect the decryption to work.

Assuming the file names on both the local and the Dropbox server side are the same, what I could do:

1. Compute the Dropbox content hash on the local file.
2. Request the metadata of the remote file (without downloading it), and obtain the remote content hash.
3. If the two content hashes mismatch, then compare the local write time with the ServerModifed time on the remote metadata.
   1. If the local write time is newer than the remote version then upload the file to Dropbox.
   2. If the local write time is older then overwrite the local version with the remote version.

The strategies in 3.1 and 3.2 create a problem when the Dropbox version is say, A + delta1_A and the local version is A + delta2_A, i.e., both sides have been modified but we cannot tell how to merge the deltas in the encrypted domain. One constraint here: decrypting both files locally and comparing and applying deltas is not an option I am willing to take, for various application specific reasons.

Question 1: Is my only synchronisation strategy to ask the user in both cases, which copy the user wants to keep?

Question 2: Is there a better way to compare files to detect the newer one instead of comparing with their write times when the content hashes mismatch? Can I compare Dropbox file revisions numerically? How can I compute the local revision of a file? Or, is comparing local write time and ServerModified time a reasonable idea?

Thank you!

[Greg-DB · Accepted solution]

1: This sounds more like a product decision, so we can't offer a "right" answer. For comparison, Dropbox has to deal with essentially the same scenario. That is, when both the local and remote copies of a file have changed before the file could be synced. Dropbox creates "conflicted copies" in this case, in order to make sure neither version of the data is lost, and to let the user themselves decide which (or both or neither) to keep, since Dropbox can't know which one(s) they actually want.

2: Using the modified time isn't necessarily recommended for this, as clocks can differ. The Dropbox API offers two things you can use: a) the content hash, to confirm the exact data, and b) the file "rev". You can store the last known rev and compare it to the current remote rev. If the rev changed, you know that the content may have changed. If the rev hasn't changed, you know that the content did not change. The rev can't be locally computed though.

[xtremebytes]

Thanks very much for the clarification.