TouchDB replication against a CouchDB view

I have been hacking away with CouchDB and TouchDB for quite some time now. I have been building a publishing platform to help onboard print publications into Newstand in the iOS App Store. During this time i have learned (the hard way) all the interesting bits about how the replication engine in CouchDB works.

After lots of research into the replication engine and our current architecture it became clear that the replication engine just wasnt for us. Let me be clear. The replication engine in CouchDB is FANTASTIC. However our architecture is a little different and it lead us to efficiency issues on the iPad that needed to be addressed.

Let me explain

Currently we have a single master CouchDB database that holds a number of documents. The size of the database is not large, we are talking under 10MB of data with around 200-300 documents at any given time. There are various jobs that run during the day to update the documents in the database. We then have multiple iPads that all sync to the database. These ipads are not doing any writes, they are simply a viewer of all of the documents in the DB. Not only that we dont care about any revisions or deleted items. We simply want the iPad to sync its TouchDB instance with exactly whats in CouchDB.

The problem with this scenario is that the replication engine was built for master to master syncing. In our situation users that download and launch the app further from when the CouchDB instance was created causes them to have a very long initial sync.

The replication engine uses the _changes feed to notify any people listening about what has happened with the database. It is the main endpoint for the replication engine. The issue here is that in our architecture the users must go through every change that ever happened to the database since its inception. This is terribly inefficient when we just want the current 200-300 docs sitting in the database.

There had to be a better way

After having some long discussions with Cloudant it was pretty clear we needed to do things differently.

So i set out to write a new replication path in TouchDB-ios that would allow us to do the following.

• Replicate against the _all_docs view of the database
• Replicate against any remote CouchDB view that mimics the format of the _all_docs view
• Replicate any number of remote CouchDB views against any number of internal TouchDB views

Over the past 48 hours i have experienced some great success adding some of these features to the TouchDB library. You can check outthe progress in Github here. Please note i am making changes in the public-api branch.

More to follow in some upcoming blog posts but currently I can successfully replicate against the _all_docs feed as well as any remote CouchDB view that i define that mimics the output format of the _all_docs view. Pretty exciting!