cancel
Showing results for 
Show  only  | Search instead for 
Did you mean: 
Announcements
Want to learn some quick and useful tips to make your day easier? Check out how Calvin uses Replay to get feedback from other teams at Dropbox here.

Dropbox API Support & Feedback

Find help with the Dropbox API from other developers.

cancel
Showing results for 
Show  only  | Search instead for 
Did you mean: 

Re: Speed block when scanning multiple folders

Speed block when scanning multiple folders

Keith B.7
Helpful | Level 7

Hello,

 

For the latest version of our app, we changed the file format, and ever since doing so, Dropbox sync has slowed significantly. I'm wondering if there is anything I can do about this. Some background:

 

- Our app uses a "file package" format - that is, each project created in the app creates a folder full of files and folders on disk.

 

- I use the Objective-C Dropbox API.

 

- The user can choose which folder inside their Dropbox folder our app syncs with.

 

- I check the files inside the folder using -listFolder:.. and -listFolderContinue:, checking the results against what is on disk to see what needs uploading and downloading.

 

Our old file format looked like this:

 

- Project

- -  Docs

- - - File 1 Text

- - - File 1 Notes

- - - File 2 Text

- - - File 2 Notes

- - - File 3 Text

- - - File 3 Notes

- - - Etc

 

Our new file format looks like this (for various reasons):

 

- Project

- - Docs

- - - File 1 Folder

- - - - Text

- - - - Notes

- - - File 2 Folder

- - - - Text

- - - - Notes

- - - File 3 Folder

- - - - Text

- - - - Notes

 

So, in our older format, a file containing, say, 100 documents, would have them all together inside a single folder. In our new format, they are split among other subfolders.

 

The result of this seems to be that Dropbox syncs much more slowly - a project containing 15,000 files using our old format syncs as fast as a project containing only 3,000 files using our new project. And most of this extra time seems to be spent on -listFiles:... and -listFilesContinue...

 

Of course, the new format will have more files in it since it breaks each file into folders, so that a project using the older format with 100 files may have those 100 files split into 50 format in our new format, so that would add 50 extra folders to be checked. But even taking this into account, the sync is slower. Even a file with, say, 200 files in the older format and 200 files including all the new folders added in the newer format will sync much more slowly for the newer format.

 

So, my question is, what is the reason for this, and is there anything I can do about it? I'm guessing that it is putting more load on -listFiles:... because it now has to search through more subfolders, but even taking that into account, the slowdown is significant. Is there any way of speeding this process up?

 

Many thanks,

Keith

3 Replies 3

Greg-DB
Dropbox Staff

Hi Keith, increasing how much nesting there is could certainly have an impact on performance.

 

Are you using the 'recursive' option on listFolder though, or are you calling listFolder/listFolderContinue for each folder and subfolder? If you're not using 'recursive=true' please try that out to see if it works faster in your case.

Keith B.7
Helpful | Level 7

Hi Greg,

 

Thanks for the reply. Unfortunately I am already calling 'recursive=true'. A long shot, but are there any techniques or tricks that can speed up the recursion?

 

Thanks and all the best,
Keith

Greg-DB
Dropbox Staff
Thanks for confirming that.

No, unfortunately there aren't any techniques for speeding this up, but I'll pass this along as feedback.
Need more support?