Dropbox uploads TOO fast

[gsheltonj]

Hi All,

This may sound like a strange glitch. But, I use dropbox to send important files to paying clients, so I want to try to figure out what is going on to make sure that I don't send people broken links or corrupted files.

Let me explain. I added a 25GB video file to my dropbox folder as I normally would. Files this size typically take 12-24 hours to upload to dropbox. I don't have gig speed internet. However, today, dropbox reported that the file was uploaded within 5 minutes!

Windows explorer and the sync queue all indicate that the file is uploaded. When I open dropbox on my browser and my mobile phone I'm able to watch the video in it's entirety. As one additional check I opened it in a different browser where I'm not logged into dropbox and it even appears to play fine there. Although it only allows me to watch the first hour of content since I'm not logged in there.

I have not tested watching the video on another computer outside my home. 

While I would be thrilled to upload massive files in minutes, I want to be sure that the files aren't getting corrupted or clipped. I don't want to send a link to a paying client only to find out the file never uploaded. I'm particularly concerned because I uploaded the same file yesterday (under a different name) and the client said it came through corrupted, Yesterday it took 14 hours to upload. 

I can think of a couple of possibilities right now: First is that I uploaded an identical file yesterday (with a different name). Is it possible that dropbox somehow recognizes that these are identical files and just uses the first one as a stand in? 
Second, is that dropbox is glitching out but since I'm checking the file from devices on the same wifi network, it's reading them over the local network.

Has anyone else experienced anything like this? If so, any help would be appreciated!

Thanks!

[Jay]

Hi @gsheltonj, thanks for posting today!

Did you move the file into the Dropbox folder, in a new location, or did you copy the file from one part of the Dropbox folder to another?

Regarding the corrupted 'upload', was it also added to a Dropbox folder, or uploaded via the site itself?

This will help me to assist further!

[gsheltonj]

Moved it into the Dropbox folder from a different location on my hard drive, outside of Dropbox.

The corrupted file was added through the Dropbox folder on my desktop

[Jay · Accepted solution]

If you add a file to your Dropbox file, which is identical to another file on your account, then the Dropbox desktop application recognizes they are the same, so it creates a duplicate of the original file on the site and instantly 'syncs' the file.

This works even if the new file has a different name. I've tested on on my device with a 1 GB video.

[gsheltonj]

I didn't know that. Super useful. Thanks!

[KMT333]

I have noticed something similar with audio files. I will upload 20GB of audio files which takes a while. I then make a tiny change to the audio in those audio files.

This audio is a bunch of 30 minute wave files of a tv program split into its various components, dialogue, music, effects, full mix etc. I change one tiny 5 second segment. That is the only thing that has changed.

I erase the old files in DB, I recreate the files, give them different names and they have different timestamps. They are almost identical to the old files as far as content goes except for the changed 5 secs of audio across all the wave files, the names and the time stamps.

DB seems to upload these very very quickly. Is DB so clever that it compares the content of the new and old files and updates the original uploaded files only altering the changed 5 secs in the whole file, plus the name and the time stamp?

[Rich]

---

@KMT333 wrote:

Is DB so clever that it compares the content of the new and old files and updates the original uploaded files only altering the changed 5 secs in the whole file, plus the name and the time stamp?

---

Yes. Before every file is uploaded, it's split into 4MB chunks and then hashed. Before uploading the chunks, Dropbox compares the hash of each chunk to the hashes of all the chunks in your account. If there's a match, Dropbox doesn't upload that chunk. It already exists online, so no need to upload it.

Now the next part is dependent on the applications writing your files. If, when a file is modified, an application rewrites the entire file or just writes data to the beginning of the file, it's likely that every chunk of that file (and every hash) will be different, resulting in Dropbox having to re-upload the entire file. But if the application only rewrites a portion of the file, or just adds new data to the end of the file, the majority of your file remains the same, so most of the hashes remain the same, and Dropbox only needs to upload the chunks that have changed. In such a case, it will appear as though Dropbox has re-uploaded the entire 20GB file when in reality it's only uploading a few 4MB chunks.

[KMT333]

Thank you for your reply!

That's impressive. I did actually totally recreate of all the files using exactly he same method that I used to create the original files so they are the same except for the tiny changed section, name and time info.

The software I use must be incredibly / perfectly accurate in its creation / recreation to the files, which is comforting :)! 

[projectdeliverypartners]

This is true, I’ve had to do a lot of digging around for a somewhat similar problem with this method of uploading/downloading in ‘chunks’ adopted by Dropbox is often referred to as block mode.

This works great for most users which I expect are working with everyday small and uncomplicated programs, word, excel, one note etc. and seems to make good sense to me however be careful and seek advice from someone appropriate if you use apps that rely on a database of some kind for whatever reason. Lots of 3d modelling / GIS programs work this way and I’ve tested uploading the same data to the major players in this space. I found any service that adopts block storage as Dropbox does had issues uploading files or rather, the software that opens the files doesn’t see what it expects and then problems occur. Usually like @Rich said, the only thing that’s changed is a small line of text/code in a file at the end, this will often not be uploaded but sometimes I found it was. Downloads didn’t appear to have the same issues from the experiments I completed anyway.
I have created some power apps, shell code etc to try and work around and the convulsion I came to is this: for programs like I’ve mentioned, use databases which reference other files and point to various locations within the projects it’s ‘root’ folder the safest was is to use a service that doesn’t adopt block storage which will take longer of course, but I would suspect the difference in minutes is negligible compared to missing or corrupt data. Programs like this really need a check in and out system where the pc or the user can specify dependants which would normally be, the project tile or root folder it self. Ie anytime I check out Project.xyz, every file found as of that moment in time (a new list needs to be created) needs to also I’ve checked and then uploaded once finished making the changes. The only way I’ve come close with Dropbox and almost all the other provides I’ve used is to pause the syncing of the cloud app while working on the project so to speak. When you’re finished, saved and closed down the what you need, then resume syncing. It shouldn’t make too much of a difference but it seems to. I suspect some kind of indexing process happens when a services resumes after being in a period of paused/inactive. That had been my experience anyway.