API v2 - Long polling

[Frank R.15]

I'm in the process of migrating from the V1 Sync API to the V2 "DIY" API and I would really appreciate some feedback on whether I'm really approaching this correctly, especially with regards to reacting to remote changes and the seemingly insanely heavy network traffic requried by my "solution".

I'm using Dropbox to keep my iOS and Mac apps synched, so that both show the same "event logs". On the Mac those logs are stored in "My Folder" folder.

On startup, the iOS version (Swift) does a files.listFolder for "My Folder" and subsequent files.listFolderContinue until it has a complete file list, then for each file it does a files.getMetaData and compares the rev (revision) field in the meta with the rev value stored in my local DIY rev-to-file-path database. If the revs are different, it does a files.download for the file and writes to local storage.

After all local files are up-to-date,  it does a files.listFolderLongPoll and waits for any updates, if an updates occurs it restarts the entire sync process.

All this results, for n log files, in at least n+1 network requests when there's nothing to by synched and 2*n+1 network requests in the worst case. There's a log file for each day of use, so after a year, we have 366 to 731 network requests.. 

Question 1: Is this really the most efficient way that this can be done?

Question 2: Is there are way of polling only for changes to a specific directory?

BTW I really appreciate the great job the support team is doing; even as I'm perplexed by the new API 🙂

[Greg-DB · Accepted solution]

Hi Frank, I'll be happy to help here. 

"Question 1: Is this really the most efficient way that this can be done?"

Reading through your description (thank you for the detailed write up, it really helps), I believe there are a few optimizations to make:

a) "does a files.listFolder" ... "files.listFolderContinue" ... "then for each file it does a files.getMetaData and compares the rev":

If you've already done files.listFolder/files.listFolderContinue, you don't also need to call files.getMetadata for each file. That just returns the same Metadata objects as were already returned in ListFolderResult.entries from files.listFolder/files.listFolderContinue.

Part of the confusion may be that you need to cast the Metadata to e.g., FileMetadata to access FileMetadata.rev though, as shown here.

b) "if an updates occurs it restarts the entire sync process":

You don't need to start totally over. When files.listFolderLongPoll indicates something changed, you should just call back to files.listFolderContinue with the latest cursor you recieved, and it will tell you only about what has changed since then. (You should just continue applying the same steps from the files.listFolder documentation to apply those additional changes to your local state to get up to date.)

"Question 2: Is there are way of polling only for changes to a specific directory?"

No, unfortunately files.listFolderLongPoll will notify of changes in the account even if the changes only occurred outside of the path originally specified in the path set in the original files.listFolder call. We'll consider that a feature request, but I don't know if/when that would be implemented. (It would be non-trivial due to the way these pieces of infrastructure work on the backend.)

Hope this helps, and thanks for the kind words about the support!

[Frank R.15]

Cheers, that's awesome and makes a lot of sense. I did miss the metadata casting and did wonder "why on earth bother sending back metadata at all if the revision isn't in there?"

I'm getting rid of code left, right and center and getting better performance into the bargain.

I'm so glad I posted here!

Cheers,

Frank