Dropbox Datastore API Alternatives

[Clifton L.]

I'm sure you've all heard Dropbox's news that they will be deprecating the Datastore API. I'm curious to know what everyone is going to move over to.

I originally chose the Datastore API for a few reasons:

• It can gracefully handle large datasets (my users have thousands of records each)
• Importing lots of data is very fast (thousands of records in less than 10 seconds; doing the same thing on Parse takes ~25 minutes).
• Dropbox is well known and trusted by many

I am currently considering a few other options, but I'd love to hear what you all are looking to do.

1. Firebase - firebase.com - This seems to be the most similar to the Datastore API so far. Developers claim it performs really well with large datasets. The downside is that it costs money for more concurrent connections (but the prices are reasonable).

2. Couchbase - couchbase.com - This is open source and seems pretty good. The downside is having to get your own server environment up and running.

3. Helios - helios.io - This seems to provide a lot of flexibility, but you have to do some of your own back-end setup, and it doesn't seem to provide an object-base sync interface. Instead, you have to use Core Data and its REST API.

4. Parse - parse.com - There is a lot to like about this option, and it can handle large datasets, but doing basic database operations like importing data and summing the numbers in a column is surprisingly difficult (for me, at least).

Anyone else found an alternative they like?

Good luck with your BaaS migration. 😕

[Adam E.1]

"Parse is unique in that it has a huge following and tremendous financial support from Facebook, so I see it being around for a long time and only getting better."

Parse is shutting down. Clifton it seems you don't have much luck!

[Clifton L.]

Haha... yep. I was shocked when Parse made that announcement. 🙂

I have since switched to a new data storage solution where everything gets stored in SQLite and I sync the SQLite databases between devices using Dropbox.

I sure hope Dropbox doesn't shut down their main API. 🙂

[Raheel S.]

Is syncing latest versions of SQLite db really the best option? 

I thought I'll just try normal file sync and figure out how the linking would work.

[Clifton L.]

For my situation and what I'm trying to accomplish, it seems to be a good option. 

[Raheel S.]

Any special logic your applying (if you don't mind sharing)?

[Clifton L.]

All the syncing is done via SQL queries, so it's very custom. Here's how it works:

Step 1: Download the remote Dropbox SQLite file.

Step 2: Open the local SQLite file and the remote SQLite file and attach them so they can be queried simultaneously.

Step 3: Perform queries that add new records from the remote database to the local SQLite database, delete records marked as deleted, and update records which have a newer "updated" time (I have a timestamp column)

Step 4: Once the local database reflects all the changes, upload the local database and replace the remote Dropbox SQLite file

[Denis B.]

I switched to Firebase as soon as the announcement was made and its a perfect substitute, I would definitely suggest you take a look!

(and no, I'm not in any way affiliated with them, just really love their product :D)

[Raheel S.]

@Clifton: Thanks for sharing

I'm really struggling to find a good way to switch away from Datastores. I have a simple Notes with Tags system. 

At first I thought, I'll jump to the usual "creating" raw files in an app folder on Dropbox wherein each DBRecord is now a text file. But I have thousands of those records (short notes). I don't know how efficient it would actually be. I'm gonna test this out though. 

To get back to your syncing I'm assuming:

Every "Remote db" is actually the fully updated "local db" of  some device that ran a session prior to the latest. 

OR

Is the Remote DB a set of queries of "transactions" that occurred in "a" particular session. and perhaps your storing a differently named "truth" db file separately on dropbox?

One more question Is:

Are you uploading the dbFile at every "save" call?

[Clifton L.]

Yes, the local and remote DBs are the same structure, they just get compared for new/changed/deleted records with each sync. I don't keep a log of transactions, I just compare the databases.

And yes, each sync requires a download and an upload of the database file.

So the downsides:

1. It's "expensive" to sync (download time, database comparison processor power, upload time)

But the upsides are many:

1. I can use SQLite queries to dissect and display the data how I need to.
2. Troubleshooting for customers is easy because they can send me their whole database and I can run queries in it and inspect the data.
3. SQLite is available to both mobile and web platforms
4. All the data is available offline

If you don't need to do any querying against your note records, then I would recommend you take a look at Apple's CloudKit (unless you need to do an Android app). You might also look at Firebase.

[Raheel S.]

Your right. I'm gonna try your approach and stick with Dropbox (for now atleast). 

Thankfully, I also used SQLite as my local storage and kept DBDatastore for syncing. Only need to figure out backup and sync architecture. (which is ... so not easy).

Oh well, thanks for help 🙂