Local folder structure and online structure is out of sync

Thanks for the reply. Seeing the DB app drop out and restart multiple times I went ahead and rebooted the machine (just for completeness: Dell Precision 2x8 Core Xeon / 48GB / 2TB SSD / local folders located at c:/dropbox on primary NTFS drive). 

Right clicking on the Dropbox tray icon now indicates recent activity. There is little/no write activity is noted in the last 24 hours, so I'm going to assume it's scanning. I've got ~960k files (no symlinks, no active database files like .pst/.ost), so it will take some time.  Syncback takes a good 40-60 minutes to scan prior to backup, so I'll just let it run for a while and check back later today. 

If it chokes or hangs I'll try a re-install of DB; hopefully it was just a hiccup. 

Hey jordan T.9,

It's good to hear that the app seems to be updating now with recent activity, hopefully you'll see it progressing soon!

I'll be here ready and waiting should you run into any issues - Cheers!

For those playing along at home (or finding this in the future); Dropbox appears to be exclusively single threaded for Indexing and has, in 10 hours, indexed roughly 175,000 of the 500,000 files I uploaded. During this time, network activity (even with the 300GB pending upload) has been trivial (in the 5kB/s range. most of which is receive rather than send) so it seems that the processes are serialized.  At this rate, 18 more hours before I can upload anything or see any of the the file structure mirrored throughout my devices. 

Also, files which have been uploaded from other clients are not yet downloading on my machine, so it's convenient that I took the week off and don't need to collaborate with anyone over DropBox!

Thanks for updating us here jordan T.9!

I'm glad to hear that the indexing is currently progressing and you'll see the upload start after this process has finished.

If you're seeing a slight degradation in the performance of the desktop app, it's most likely just due to the number of files you have syncing in your Dropbox folder. While the soft limit is around 300,000 files, this highly depends on your OS specifications too so your device might be able to handle a larger number before you see this.

However, if you'd like to try and speed up the uploading process when that gets started, I can suggest checking that your bandwidth settings for the desktop app aren't limited.

I hope this helps - Cheers!

Even with the PC running 24/7, the online directory structure still has not synchronized 90 hours later.  The network connections are not limited (not that it matters, DB is using a trivial amount of bandwidth). It continues to index and fail (tray icon turns gray, notifications area becomes empty), then restart. 

I'm in the process of re-installing dropbox, but am not hopeful. 

Edit: On re-install I got a message (email) that I had just deleted 215,000 files. This is not promising. 

--

I've noticed that there is no tecnihcal information on the Dropbox site - or in any help files - about the 300k file limit. This is fairly critical information, especially given how easy it is to exceed this limit in a 2TB box - I have 950k files+folders in just 1.4TB of combined personal and consulting files. It must make Dropbox for Business awful to manage. I'm going to attempt some pruning in hopes of getting this process moving.  and hope that I don't end up with data loss. 

Hey jordan T.9,

Thanks for getting back to me here with your current process.

If you were to click the Dropbox icon, are you seeing any error messages within the sync status at all?

With issues in the desktop app performance where there are a large number of files syncing, we generally suggest using selective sync to alleviate some of the load on the application. As I mentioned though, the soft limit does depend highly on your current OS and specifications as some computer systems can handle more than others.

Finally, about the email that you received, if you just want to check on that event, I can suggest checking your Events page to see this event. However, as Dropbox sees moves as an addition to the new location and deletion from the original location, I believe that the email may have stemmed from syncing the moves you intially made.

Keep me posted on how it goes - Thanks!

Okay, the syncing kerfuffle is nearly at an end. After endless failures and restarts, the system is somewhat back in sync. Again, for those finding this in the future:

1) Do NOT unlink your account and re-link it if you have files folders out of sync. The re-link process will treat any two folders which were out of sync as unique data sets. The result is that you will have files in both places (the old one, referenced in the Cloud version and the new one, referenced on your local drive).  The conflict will result in a s**t-ton of file-duplicates whic you will have to manually de-duplicate (likely my next two days of work). It will also increase your DB volume - both total bytes and total files. 

2) If you should be foolish enough to put over 330,000 files on dropbox, never let a Windows PC see the entire database. If you have to move large qualities of files around, do it on the web interface. As much as it sucks, the Dropbox app is unable to efficiently process massive moves and will crash. Repeatedly. 

AFAICT - and you can correct me if I'm wrong, Daphne - the Windows app is 32 bit and cannot access datasets in excess of 4GB in size. Dropbox reserves somewhere on the order of 10-12kB of space for file information and tracking. Once you hit 330k files, you've hit the 4GB limit and the app fails.  The reason Dropbox is cagey about this is that there the OSX app is 64 bit and Linux has both 32 and 64 bit options, meaning that Windows is the only OS, despite the large numbers of users, which only has a 32 bit app.  As a result, even a professional workstation running windows (I'm running a 16 core Xeon with 48GB RAM) cannot process any more files than a single core celeron with 4GB of RAM. And, since the sync process appears to be single-threaded under windows, it can't process appreciably faster either. 

I'll end this just by saying that Dropbox is good at warning/preventing you from exceeding your disk quota (2GB/1TB/2TB), but has no ability in the UI to prevent high file-count choking. I would request that Dropbox consider re-working the ability to address large filesets but, in the meantime, add a warning about moving or adding total file counts over 330k. At 2TB space, that's 6MB per file, and it's easy to have a large number of files smaller than that in a typical user account. 

Hey jordan T.9,

Thanks for updating me here and giving such a great amount of detail into your findings! I'm sure that this will be helpful to others who might have a similar issue you experienced.

What you mentioned about syncing a directory larger than 300,000 files is indeed helpful too as long as we keep in mind that this is a soft limit. Also, as you know, the post you created seperately for requesting a way for the desktop app to notify you when you reach this many files has been passed along to our dev team too.

Again, thanks for taking the time to provide all this info and I hope you have a great start to the week - Thanks!

Hi Daphne,

While passing information to dev team about this issue, may be it's good to be added the fact that Dropbox don't recognize file move from file copy and delete! This is very important and noted multiple times in the forum, but seems without enough attention from Dropbox side. File moving is generic operation and is very fast while in border on single (Dropbox) folder. Exception from above could be on very specific cases which most of the users will/had never see(n). The Dropbox application count every file move as copy-delete operation, which slow down the work significantly! Probably this is in the base of the current thread issue. If dev team introduce file move recognition and serving it independently, then issues like the current one will never happens again. Even more: network traffic (client and server sides) will decrease (no reupload of already uploaded files).

Hope this can improve the application quality. :wink: