longpoll workflow / flowchart / advice how to do it right

[latimba]

Hello Dropbox API team,

I built an app that loads and saves information from a single text file in Dropbox.
That app could be used from several devices, mobile and desktop, and therefore has to detect changes in Dropbox and update the app state as well as pushing changes to the dropbox file.
My research indicates that list_folder/longpoll would be the right way to approach this. 
I implemented a simple longpoll where I would get the latest cursor, create a longpoll connection and listen to changes. I update the changes in the app if the revision in files_getmetadata is different to my last stored one, and restart the longpoll. I also restart longpoll if the request returns with "nochange". All that works fine.

But I'm having trouble designing a complete longpoll implementation which covers all edge cases. Here are some questions/examples:

-what is the best way to handle longpoll if an error is returned, either when getting the latest cursor or during the longpoll? An example would be a lack of internet connection. Currently, I postpone the next longpoll for 30 seconds (and in the case of a backoff response, for the seconds which are requested). Is that a good idea, or should I handle errors differently, perhaps depending on the error I get?

-supposed that 30 seconds delay is a good idea: In those 30 seconds (or it could be much longer, like on a plane), the file in Dropbox could have been changed by another client. If the longpoll resumes, the cursor has changed without the app knowing about it, right? Should I also store and watch the last cursor locally, so that I know something changed? Or use a different method?

-it gets more complicated when I save / push updates to that file in rapid succession (which I do): While the app is busy with handling a change from the longpoll, the next change could be on the way and I'm not sure it gets detected between the one longpoll exiting with a change response and the next longpoll not yet being established.

So my question is:
Is there a workflow or flowchart or an example for a complete implementation how to handle longpolling?

I found many examples for Dropbox longpoll and other longpolls in Java, Ruby or javascript (this is what I use), but they all just show how to listen for changes, they don't show the workflow (or whatever it's called) how to do longpolling right or complete.

So if you could point me to an implementation example or flowchart that details when to get the cursor, when to restart the longpoll, when to get metadata and perhaps when to restart the whole process: That would really help me a lot.

Thanks in advance for any reply, 
l

Note: None of the tags really match, it's an API matter. The app is build with  Cordova which is supposed to run on all the platforms I selected. I'm using the javascript SDK of Dropbox, but it doesn't matter to me which language is used in an answer.

[Greg-DB]

Using the /2/files/list_folder/longpoll endpoint, filesListFolderLongpoll in the JavaScript SDK, is the right way to monitor for changes in a connected Dropbox account with low latency from a client-side application.

I don't believe we have a particular example or flowchart that would cover this use case very well, but we'll be happy to help with your questions.

Overall, it sounds like one part you're missing though is that you are using filesGetMetadata to get the file metadata each time, and are perhaps also getting the "latest" cursor using filesListFolderGetLatestCursor each time, instead of making use of listFolder and listFolderContinue.

If you use listFolder and listFolderContinue, you can keep track of changes over time, without the risk of missing anything, e.g., due to race conditions. For example, a typical flow would look like this:

• Call listFolder and listFolderContinue until FilesListFolderResult.has_more is false to get all of the current file/folder state and save the last cursor (or use filesListFolderGetLatestCursor for this initial step if that makes sense for your use case).
• Call filesListFolderLongpoll with that cursor to monitor for changes.
• When there are changes, call listFolderContinue to retrieve just the changes since you last called it, based on the cursor.
• (and repeat the previous two steps)

To answer your specific questions though:

-what is the best way to handle longpoll if an error is returned, either when getting the latest cursor or during the longpoll? An example would be a lack of internet connection. Currently, I postpone the next longpoll for 30 seconds (and in the case of a backoff response, for the seconds which are requested). Is that a good idea, or should I handle errors differently, perhaps depending on the error I get?

Yes, you'll likely want to handle each kind of error differently. For a network-level error like you describe, you'll probably just want to wait and try again, and perhaps notify the user of the issue in the UI, if applicable.

For an API-level issue, it depends on the error returned. For example, FilesListFolderLongpollError.reset means your cursor is invalid, so you'll need to start over to get a new one. 

-supposed that 30 seconds delay is a good idea: In those 30 seconds (or it could be much longer, like on a plane), the file in Dropbox could have been changed by another client. If the longpoll resumes, the cursor has changed without the app knowing about it, right? Should I also store and watch the last cursor locally, so that I know something changed? Or use a different method?

Your app should store the last cursor it received. As long as it passes that cursor back to listFolderContinue, the Dropbox API will return all of the changes that the app needs to know about since that cursor was returned.

-it gets more complicated when I save / push updates to that file in rapid succession (which I do): While the app is busy with handling a change from the longpoll, the next change could be on the way and I'm not sure it gets detected between the one longpoll exiting with a change response and the next longpoll not yet being established.

Using the flow described above will help avoid any such race conditions. You can also use the 'rev' or 'content_hash' values to determine if you have the latest file data or not.

Hope this helps! 

[marimba]

Dear Greg,

thank you very much for that extensive answer, has been very helpful!

[elsigh]

I wonder if one thing not covered in this flow is, say, if the folder name is changed in Dropbox.. Maybe that's why they (and I) want to call `filesGetMetadata` as well on the folder in addition each time. Is there another/more efficient way to catch and handle that from the client side? I will double check that the longpoll doesn't return changes to the folder itself.. (it looks very `entities` focused)

[Greg-DB]

@elsigh Renaming a file or folder will be reflected by listFolder/listFolderContinue as a deletion of the item at the original path, and then an addition of the item at the new path.

The file ID doesn't change when renaming an item in Dropbox though, so tracking that value is the best what to correlate these.

[elsigh]

Hey @Greg-DB  - ok that's great to know. I haven't gotten to testing renaming/deletion so I'll look out for that. But does that cover the folder being polled? e.g. if I'm polling a folder A and a user goes and renames folder A to folder B I would catch that somewhere?

I've been implementing as you suggested here but I'm wondering about two scenarios I'm seeing.

In one case I'm receiving a response with `changes: false` - I'm guessing in this case I should start the poll again with the same cursor?

In another case it seems that I'm getting `changes: true` so I subsequently call listFolderContinue with the cursor but then get a response that has `entries: []` but does come back with a new cursor.. in other words it seems like I'm not actually getting any changes so I wonder what's going on there? This pattern happens over and over without me making any changes within that folder. Is that expected?

[Greg-DB]

---

@elsigh wrote:

does that cover the folder being polled? e.g. if I'm polling a folder A and a user goes and renames folder A to folder B I would catch that somewhere?

Yes, if you listFolder for a path a particular folder and are polling using the returned cursor, you'll get `changes: true` if/when that folder itself is renamed. If you then listFolderContinue with that cursor, you'll accordingly get a `path/not_found` error, since that path no longer exists (or a `reset`, if you originally listed the folder via file ID). Alternatively, if you were listing a parent folder of the renamed folder, you'd see `changes: true` and the deletion/addition in the listFolderContinue result.

In one case I'm receiving a response with `changes: false` - I'm guessing in this case I should start the poll again with the same cursor?

That's correct, if/when you get `changes: false`, that means nothing has changed yet but that poll operation timed out, so you should start another poll call if you wish to keep monitoring for changes.

In another case it seems that I'm getting `changes: true` so I subsequently call listFolderContinue with the cursor but then get a response that has `entries: []` but does come back with a new cursor.. in other words it seems like I'm not actually getting any changes so I wonder what's going on there? This pattern happens over and over without me making any changes within that folder. Is that expected?

That sounds unexpected. Can you open a thread with the relevant code/steps to reproduce this, and the output, so we can investigate? Thanks in advance!