You might see that the Dropbox Community team have been busy working on some major updates to the Community itself! So, here is some info on what’s changed, what’s staying the same and what you can expect from the Dropbox Community overall.

Forum Discussion

XionicFire's avatar
XionicFire
Collaborator | Level 9
3 years ago

How to fix being stuck on Indexing forever loop of death on multi million file DBs after db crash

Ok so I've used Dropbox for almost as long as its existed and recently due to frustration with the never finishing indexing bug I was forced to find out why this kept happening so I could prevent it.

 

Bear with me on this long post but trust me its worth it, what I found was mind blowing and game changing.

 

So our business Dropbox is more than 9 million files strong, I've noticed REALISTICALLY any machine handling over 2 million will just enter an indexing loop at some point from which it will never recover, after its happened 5 times in the last week I was pissed enough to that I decided I was going to find out why this is happening, I know I'm pushing the limits but we've had machines with 2.5m files running fine for years, why some work fine and some don't was a mystery, one I was determined to find out.

 

When you add stuff to your Dropbox, Dropbox has to index it so  it can know what to do with it.

If you add "too much stuff"  (copying 200,000 files of small size in one go, coders know what I'm talking about)

or

do it "too fast" (changing access permissions on 1.5 million files located within Dropbox in one go in less than 5 minutes)

on a computer with too many files (1-2+m)

this causes Dropbox to start indexing them all at once causing the system to slow to a crawl

however if you don't let Dropbox finish before doing something else (like adding more files or using the computer for other tasks)

or

someone else adds a bunch of files on another machine

its almost for sure going to cause Dropbox to crash and restart during this process (it happens quite frequently),

this crashing and restarting triggers a full reindexing of the ENTIRE database, ALL files, and since the machine is already trying to download or upload some of the new files while trying to reindex the current millions of files, doing both at once overtaxes it, which causes it to crash, and were back to square one with the infinite indexing crash loop.

 

This kept happening to us all the time, the only solution was to unlink and relink the Dropbox account so all pending changes were lost, we got a bunch of conflicted copies and spent days sorting out the mess.

 

So I figured I needed to see what was going on with Dropbox, what it was doing when it was "indexing" to find out what was causing the crashes.

 

So after looking for a while I found using Microsoft's sysinternals process monitor (https://docs.microsoft.com/en-us/sysinternals/downloads/procmon) configured in a certain way allowed me to look at EXACTLY what dropbox was doing and I found out all I just told you and another very important piece of info.

(if you want to see how awesome it looks in action check out THIS video:)

 

When you click PAUSE FILE SYNCING you would think Dropbox pauses and ceases all operations, but you are wrong!, it pauses all incoming and outgoing transfers but any INDEXING tasks keep going!

 

This is an absolute game changer! now if I see a machine that says "Indexing" for a long time, I turn on the process monitor, hit pause on the file syncing and watch as the machine does the indexing at super high speed (5-10 times faster than doing it while downloading), it usually finishes doing the full file reindexing in a couple of minutes once its done I can hit back resume and keep going, I've never had the app crash while doing this "offline" or "paused" indexing, thus avoiding the inevitable crash, and reindex loop.

 

I have been successful in recovering 4 machines from the indexing loop of death using this method, where before I was screwed and had to eat the duplicate files and cleanup for a week and a TON of annoyed users in the office.

 

Basically if your machine is taking too long indexing or is stuck indexing after a crash just hit "pause for 1 hour" and forget about it, it will still be working on the indexing in the background, and when it restarts it should have finished the re-index avoiding a crash when trying to download/upload the new files.

 

Id wish Dropbox would have told us this, I never expected it to keep indexing while paused, I assumed pause was PAUSE, as in, cease all operations, it would have saved me so many headaches.

 

All they need to do now is let us have a "log viewer" or something so we can tell when its done doing its thing and we can hit resume, also show us, even when in pause, when its indexing and when its not, so when its done we know we can restart it safely, or the better yet, set it to where if Dropbox has to index a large volume of files (say over 100), it will automatically pause all other disk operations until the indexing is complete, then restart the downloads, trying to do both does not work, i know you want it to but it just doesn't, and just causes the whole thing to explode non stop in a loop of death, maybe enable this on a setting somewhere? or auto enable it on machines with over 500k files? something has to be able to be done.

 

TLDR:

If your Dropbox is stuck indexing, hit pause 30 mins, and let it do its thing until its done, it will keep on doing it even when paused, you wont know if its doing anything or working unless you use procmon, but its working, and try avoid using the hard drive or the machine until its done, (usually less than 30 mins), and your indexing/crashing problem will be fixed. 

 

Message to Dropbox:

Dear Dropbox, Please give us a way to view this info without having to resort to third party programs, this way we can help auto troubleshoot our Dropbox issues and take a lot of load off your customer service guys.

Something like: Enabling a setting  somewhere saying "activate/enable troubleshoot/server mode" or something that allows us to turn on an always shown (ALWAYS, NOT ONLY WHEN MOUSE OVER, BUT ALWAYS!!!) 3 tab little window,  containing:

 

Indexing files. (with a current list of the exact files being indexed and their speed (x files per sec)/paths)

Downloading Files.(with a current list of the exact files being downloaded and their speed/paths)

Uploading Files (with a current list of the exact files being uploaded and their speed/paths)

 

There's another issue with slow uploads due to Dropbox connections stuck in a "stagnant state" (force closing the TCP socket connection using netmon restarts the download/upload and speed goes back up again) but that's another problem for another time.

 

I hope this was helpful to some other sysadmin and sorry for the long message but it needed explaining.

  • XionicFire's avatar
    XionicFire
    Collaborator | Level 9

    Someone just suggested to me another great way to find out if its done indexing, when your dropbox is indexing on crappy computers it can use up to 50% of the CPU power, hit pause, you will notice the CPU will not come down, it will stay high, for a while then after a few minutes it will go down to 1-2% this is when you know that its finished indexing and you can hit resume.

     

    I know ill be using that tip from now on, its much easier than procmon

     

    Just open your task manager or type "resmon" into your windows console, when you see the Dropbox CPU usage go down like this picture, you know its done indexing and its safe to hit resume.

     

  • XionicFire's avatar
    XionicFire
    Collaborator | Level 9

    Also worth nothing, this is not a silver bullet, theres other cases where the computer is just stuck indexing forever, those are usually caused by locked files, corrupt files, naming errors, permissions issues or other random issues having to to with the exact file in question, if this is your case this fix will not help you, but procmon will, you will know exactly what file is causing the problem instead of the "super usefull" /rolleyes "indexing 4 files" message, and you will be able to find out whats causing it and fix it.

  • DaveHolmes's avatar
    DaveHolmes
    Explorer | Level 4

    As another long term user I also have this issue and it is fricking annoying. I use two PCs and Android, so things not syncing causes major issues. 

    A full sync takes over a week on high speed internet using ethernet, so I don't want to unlink, plus there is always the fear that my latest changes would somehow roll back to the earlier changes.  Confilicting file issues would also take me ages to sort out. I am trying the pause technique here to see if it fixes it for me as I have been stuck on 93 files indexing for 2 weeks.  


    Most of my changes are folders moving. I noticed they did changed things a little and fixed one problem, back in the day when you moved a folder it would count as extra in your allwance until it synced and removed it from where it was. but now I guess each file has a unique id and it no longer gives me over limit warnings which used to cause it stop syncing too.

    • Walter's avatar
      Walter
      Icon for Dropbox Staff rankDropbox Staff

      Hey DaveHolmes, sorry to hear about this.

       

      If you're still stuck at syncing, could you let us know the device's exact OS and the version of the desktop app installed there? 

       

      A screenshot of the app's status as shown in your system tray would also help, so feel free to include one. 

       

      On a side note, have you tried adjusting your bandwidth settings through the app's preferences to see if it helps at all?

  • DaveHolmes's avatar
    DaveHolmes
    Explorer | Level 4
    I couldn't get my dropbox to sync and helpdssk were utterly useless, just copy posting scripts really. no real effort to unravel my own issue. after more than 10 years with £20 a month and two accounts I have had enough. The Dropbox era is over. I paid £20 for Google One and I get 5TB instead of 3 and everything syncs fine and I can see what files are syncing at any time.

    Good luck everyone
  • hawaiikeith's avatar
    hawaiikeith
    New member | Level 2

    This sounds helpful and appreciate the response. Might try the pause sync to see if it catches up.

     

    Today, I had a similar issue on the Mac.

    It asked to upgrade last night, so I did.

    Well, that was a bad idea.

     

    Got all hung up and froze me out of Chrome, no downloading email attachments, Preview didn't work, some files I couldn't open, icons next to files on the desktop with the DB cloud symbol, etc. and gobbled up resources.

    Lots of restarts and confusion as it scrambled my HD and folders/files.

     

    Support said 1-3 hours and still waiting.

     

    Tonight I deleted the app and reinstalled it.

    Some functionality returned and it started re-indexing 120k files.

     

    There is a glitch with the latest update.

    I'm on Monterey OS 12.6

     

    Seems like this was released prior to testing.

     

    This also happened last year.

     

    Believe before they let users update, they need to verify and validate the release works on all OS ______ versions.

    Because today, my machine was relatively useless.

     

    Any solutions, I welcome them.

    • ArcticAnna's avatar
      ArcticAnna
      Explorer | Level 3

      Thank you! 

      I KEEP on having this problem. 

      Apple wiped my mac in a fix so I started added folders one by one (4.3TB of mostly small files)

      As you can imagine, this took me weeks

      And then it just stalled.

      I kept thinking since it was just 3.8GB trying to change that it would sort itself out and as I waited and watched, of course I kept fixin  things as I went

      But no go.

      DB support---zero support...other than to unlink..which I clearly didn't want to do to go back to ground zero.

      But that is now where I find myself. I didn't read this in time, unlinked, relinked, tried to link just a few folders to start off with...and we are just indexing. Oh boy. I am also thinking about looking elsewhere but I hate google and a few years ago, at least, Microsoft had issues syncing between mac and pc...

      • XionicFire's avatar
        XionicFire
        Collaborator | Level 9

        Well it happened again... out of nowhere with no major changes, syncing forever with no reason why, just never finishing I tried my usual tricks and nothing

         

        Using resmon to see if any files where open by some app that were holding up the sync?

        Nope

         

        Using resmon to check for disk activity to see what files were being written or read that could be causing the problem?

        Nope

         

        Checking the folders to see what folder had the "sync" blue icon and figured one huge file was syncing?

        Nope

         

        Suspend and wait for re-indexing to do a re-index and find out if that fixes it?

        Nope

         

        I was running out of ideas and I came across this post:

        https://www.dropboxforum.com/t5/Apps-and-Installations/Desktop-client-stuck-a-quot-Syncing-quot/td-p/434215

         

        And I figured could it be that easy?

        so I went to the Dropbox\.dropbox.cache\old_files\ folder and sure enough a 15 gig file was there, I deleted it and like magic the problem plaguing me for more than a week fixed itself.

         

        So if you tried my previous solutions and the sync issue still persists try this one

         

        For those that care:

         

        The reason this happens is every time you right click and select a file or folder to convert to "online-only" dropbox doesn't just automatically delete it, it moves it to a temporary location for further processing.

         

        Basically what dropbox does when you do this is it moves the file immediately to Dropbox\.dropbox.cache\old_files\ under a random generic number and puts an online only link where the file was before as a placeholder, then once it does this it starts hashing out the file, meaning it reads it all completely, then creates a hash of it, then compares it to the hash on the server, if the hashes match, then it deletes it for real, if it doesn't match, it uploads the file and then deletes it, this ensures the online only'ed file is always the most up to date version.

         

        This has two benefits:

         

        The first, as described above if the file were to be different it would correct the situation by uploading it.

         

        Second, if you accidentally hit online only, on these files, you have until dropbox does this process to "undo" your mistake without having to redownload the files again, so its kinda useful.

         

        I'm guessing what happened is something borked up on the dropbox hashing routine, making it never be able to hash the file, since it never hashed it couldn't be deleted, since it was not deleted the sync icon just stayed there forever.

         

        Deleting the file causes dropbox to "error" out saying "file not found" and skipping the hashing procedure, just assuming the last file he has on the server is the newest one, and leaving it at that.

         

        This causes the sync to finally finish

         

        So now you know, if it happens to you, try this, it might fix it. 

  • Nicholas M.4's avatar
    Nicholas M.4
    Helpful | Level 5

    Thanks, good to know

    I want to add a folder that has millions of small files in it. I tried  a few years back and Dropbox just couldn't cope. I might give it another go armed with this information.

  • INDIcisive's avatar
    INDIcisive
    New member | Level 2

    XionicFire,

    Thank you for the well written easily understood post. I've experienced the same problem many years ago only to end up in LOOP hell. Fighting against the machine only caused increased aggravation and lockups.

    Dealing with a multitude of files and/or excessively large files can be a nightmare to the point of overtaxing the system. Glad to know that purgatory exists in the form of pausing sync and walking away for a bit.

     

    Absolutely agree that Dropbox needs to supply an under the hood monitor. Plenty of IT folks would thank them, but until that happens keep 3rd party tools attached to your Batman utility belt.

    Thanks again.

  • Aylagrace43's avatar
    Aylagrace43
    Explorer | Level 3
    Hi XionicFire,
    Thanks for writing your post. This is literally the 1st time I've visited the Dropbox Community Forum. I've had a Dropbox Plus (personal, 2TB) account for years now, but haven't explored this section of Dropbox before tonight.
    Your post doesn't answer the question I've been looking for help with, but in a round about way it's related. I've been in contact with Dropbox's Customer Support regarding my issue and they offered literally *zero* assistance.
    A hopefully short summary of my issue: I currently have personal accounts with 3 cloud providers- Dropbox, Microsoft OneDrive, and GoogleOne. I'm learning the intricacies of syncing between my phone/laptop/cloud and other general tech topics to be a well-functioning adult :). Unfortunately, it was quite a while before I realized that I was syncing all my data/back-ups to all 3 cloud providers. So, now I have almost 6TB data between the 3 providers, many duplicate files, my filing/folder systems are a disorganized mess, and I can't seem to find answers as to how to straighten this out.
    What I want, to begin with, is an Index of my files/folders from each cloud provider. Do you know how I can obtain these?
    Thanks again for your post!
    Michele

About Dropbox tips & tricks

Node avatar for Dropbox tips & tricks

Learn how to get the most out of Dropbox with other users like you.

393 PostsLatest Activity: 17 hours ago
325 Following

If you need more help you can view your support options (expected response time for an email or ticket is 24 hours), or contact us on X or Facebook.

For more info on available support options for your Dropbox plan, see this article.

If you found the answer to your question in this Community thread, please 'like' the post to say thanks and to let us know it was useful!