IRC discussion about filtered replication on views/security plugin

irc.txt

2:48 PM <nolanlawson> wendall911: looks like someone on stackoverflow is interested in something similar to your "encrypted pouch" idea: http://stackoverflow.com/q/23670733/680742
2:49 PM — wendall911 looks
2:50 PM <wendall911> nolanlawson: cool, that's actually a perfect use-case right there
2:52 PM <wendall911> nolanlawson: I'm going to start a doc on this and share soonish. Several use-case scenarios to start a discussion around
2:54 PM <nolanlawson> wendall911: oh, you have something coded already?
2:54 PM <wendall911> I wonder if there would be an efficient way to mark fields to be encrypted... maybe a prefix on the key that gets removed when the data is decrypted and added back again on a put
2:54 PM <wendall911> nolanlawson: no nothing coded, just been thinking about it
2:54 PM — nolanlawson is pumped for more pouch plugins
2:54 PM <nolanlawson> wendall911: good, you get to benefit from the new plugin seed project I threw together today :)
2:54 PM <nolanlawson> https://github.com/pouchdb/plugin-seed
2:54 PM <wendall911> nolanlawson: should be farily simple to code, I just want to make sure I have a good idea what the use-case scenarios are before I do anything
2:55 PM <wendall911> nolanlawson: sweet!
2:55 PM <nolanlawson> I would make the default that every field gets encrypted, except for _id, _rev, _attachments, and _deleted obviously
2:55 PM <nolanlawson> that's probably what most users will want
2:56 PM <wendall911> maybe, not sure ;)
2:56 PM <wendall911> but, yeah, could easily be configurable for whatever
2:56 PM <nolanlawson> hm, yeah
2:56 PM <wendall911> I like the idea of having keys that get encrypted
2:56 PM <wendall911> not sure I like the overhead of encrypt/decrypt on every doc
2:56 PM <nolanlawson> pouch.setSecurityConfig({optIn: false, encryptedFields: ['secret', 'anotherSecret']})
2:56 PM <wendall911> would work for hello world, but might not scale too well
2:57 PM <wendall911> nolanlawson: ah, yeah, that's nice
2:57 PM <nolanlawson> encrypting ids is tricky, because it would mess with ordering
2:57 PM <wendall911> yeah
2:57 PM <nolanlawson> if only certain docs need to be encrypted, you could look for something like {encrypted: true} on the doc
2:57 PM <wendall911> would be some gotchas in the incremental views
2:57 PM <nolanlawson> yeah, incremental views would basically not work
2:58 PM <wendall911> basically would need to disallow any fields configured to be encrypted
2:58 PM <nolanlawson> you cannot sort on fields that are encrypted
2:58 PM <wendall911> yeah
2:58 PM <nolanlawson> or you can, but it'll just be garbage
2:58 PM <nolanlawson> unless you create an encryption scheme that preserves ordering, but that would make it really insecure
2:58 PM <wendall911> yeah
2:58 PM <wendall911> would defeat the purpose
2:59 PM <wendall911> I'm mostly thinking about just protecting sensitive data
2:59 PM <wendall911> and you typically don't do view indexes for passwords or other sensitive data (at least I'd hope not)
3:00 PM <nolanlawson> yeah, absolutely
3:00 PM <nolanlawson> I think just listing the fields that you want encrypted (like passwords) would be good enough
3:00 PM <nolanlawson> in general, it's a really exciting idea for a plugin. I was looking through the irc logs and someone else was asking about it like 2 days ago
3:00 PM <wendall911> yeah, and maybe a config value for everything
3:00 PM <wendall911> just flat disable views with that
3:01 PM <nolanlawson> you can't really disable views on encrypted data. I think users just have to understand that it will be garbage
3:01 PM <wendall911> maybe even just throw if the entire doc is encrypted (outside of _ fields) if doing views
3:01 PM <wendall911> hmm, yeah, I guess that works
3:01 PM <nolanlawson> same for encrypted _ids and allDocs
3:02 PM <wendall911> yeah
3:02 PM <nolanlawson> also it would be neat to allow different encryption algorithms. or asymmetric hash functions for things like passwords, where encryption is usually a bad idea
3:02 PM <nolanlawson> so cool!
3:02 PM <wendall911> agreed
3:02 PM <wendall911> :)
3:02 PM <Kxepal> iirc webcrypto should solve that problem
3:03 PM <wendall911> Kxepal: yeah, an interim method for this would be nice
3:03 PM <wendall911> even could implement a shim as well
3:03 PM <nolanlawson> Kxepal: neat, didn't know about this. http://html5.creation.net/webcrypto-api/
3:03 PM <wendall911> http://polycrypt.net/
3:04 PM <Kxepal> btw https://github.com/GlobalSign/PKI.js
3:04 PM <nolanlawson> holy crap, though, I just had a crazy thought
3:04 PM <Kxepal> wendall911: I heard polycrypt isn't good one
3:04 PM <nolanlawson> if this plugin worked, it would kinda sorta solve the "couchperuser" problem in couchdb
3:04 PM <wendall911> Kxepal: yeah, just saying a polyfill would be the best approach
3:04 PM <nolanlawson> you could have a single public database shared by all users, but everything's encrypted
3:05 PM <wendall911> nolanlawson: yep
3:05 PM <wendall911> nolanlawson: that's exactly what we're trying to solve
3:05 PM <nolanlawson> you could even do views across user's public data, so no need for a daemon to pull together different DBs
3:05 PM <nolanlawson> that's amazing; I feel like I just achieved enlightenment
3:06 PM <nolanlawson> do you realize how long I've agonized over the couchperuser problem, without ever thinking of that solution?
3:06 PM <Kxepal> also webcrypto is available in nightly safari/webkit
3:06 PM <Kxepal> just been noted about
3:07 PM <wendall911> Kxepal: any idea about FF?
3:08 PM <Kxepal> wendall911: no, but as I see WIP: https://bugzilla.mozilla.org/show_bug.cgi?id=865789
3:09 PM <Kxepal> probably it'll happened before few more releases
3:09 PM  ↔ klaemo nipped out  
3:10 PM <nolanlawson> wendall911: another solution to the problem of sorting by encrypted fields is to replicate to an unencrypted database that's only sorted locally
3:10 PM <nolanlawson> if your db is small, you could even just keep it entirely in-memory, once we release the memdown plugin
3:10 PM <nolanlawson> hm, although then your view would also only be in memory, never mind
3:11 PM <wendall911> nolanlawson: yeah, not useful for anything I work with
3:11 PM <nolanlawson> *stored locally
3:11 PM <nolanlawson> fair nuff
3:11 PM <wendall911> unless you want to store several gigs in memory
3:11 PM <nolanlawson> but basically yeah, two databases: one encrypted and shared, the other unencrypted and unshared
3:11 PM <wendall911> which is possible, but probably won't work in browser context
3:12 PM <nolanlawson> yeah, the storage quotas are way less than that: http://www.html5rocks.com/en/tutorials/offline/quota-research/#toc-safari
3:13 PM <nolanlawson> looks like cross-browser, the most you can hope for is about 50MB
3:14 PM <wendall911> nolanlawson: oh, you know users should only have a small subset of data anyhow
3:14 PM ⇐ klaemo quit (~klaemo@g229036025.adsl.alicedsl.de) Ping timeout: 258 seconds
3:14 PM <nolanlawson> ok yeah, I was about to say, are your users storing movies as attachments or something ;)
3:14 PM <wendall911> nolanlawson: not an entire db of users, so that would work. You'd need to be able to share a segment of the view from the server, which is totally possible
3:15 PM <wendall911> would be neat to fetch a startkey, endkey from a remote view and patch it in locally to the pouchdb
3:16 PM <wendall911> trust that the collation is correct
3:16 PM <nolanlawson> not sure what kind of data you're storing, but assuming the startkey and endkey includes the userId somewhere, that sounds totally possible
3:16 PM <nolanlawson> although I assume filtered replication would be easier than creating a view on the remote CouchDB
3:16 PM <wendall911> yeah, would have to be just the keys, and omit any data
3:16 PM <wendall911> hmm, probably
3:17 PM — wendall911 likes views
3:17 PM <wendall911> maybe a bit too much
3:17 PM <wendall911> indexes are crazy fast to share around, especially liberal use of arrays as keys
3:17 PM <wendall911> fetch docs if you need them and don't replicate
3:18 PM <wendall911> but I'll rethink that for this :)
3:18 PM <nolanlawson> but then you're basically bypassing replication?
3:18 PM <wendall911> yep
3:18 PM <wendall911> replication is too slow
3:18 PM <nolanlawson> I was thinking you could do filtered replication with the "everybody can read, but you can only write to your own documents" scheme: https://github.com/nolanlawson/pouchdb-authentication#everybody-can-read-only-some-can-write-some-things
3:19 PM <nolanlawson> if your documents never change, I guess using a view would be fine, but consider this:
3:19 PM <nolanlawson> 1) replication from HTTP to local will do an allDocs anyway for 1st gen docs
3:19 PM <nolanlawson> 2) views are always slower than allDocs in CouchDB, guaranteed
3:19 PM <nolanlawson> 3) you have to write more code :)
3:20 PM <wendall911> well, it's fine until you try to scale that, then views are orders of magnitude faster
3:20 PM <wendall911> say I want to see if a username is unique
3:20 PM <nolanlawson> that has not been my experience with CouchDB, although maybe you're using BigCouch?
3:21 PM <nolanlawson> anyway yeah, if you want to operate on massive scale, then we're back to one database per user...
3:21 PM <wendall911> well, it isn't even that massive
3:21 PM <wendall911> once you get to several hundred thousand users, the simpler models just fail
3:21 PM <wendall911> views using include_docs
3:22 PM <wendall911> fast and reliable
3:22 PM <nolanlawson> hm, how does filtered replication work? I guess it doesn't create a view, right - it just iterates through all the documents and checks some field
3:22 PM ⇐ Yakulu quit (~Yakulu@78.194.3.29) Quit: WeeChat 0.4.3
3:23 PM <nolanlawson> yeah okay, that seems correct: https://wiki.apache.org/couchdb/Replication#Filtered_Replication
3:23 PM <nolanlawson> unfortunate though
3:23 PM <wendall911> nolanlawson: yeah, if filtered replication could work like views with complex keys, that would be ideal
3:23 PM <nolanlawson> I'll hop over to #couchdb and ask
3:24 PM <wendall911> it's a good question, but I've always avoided filtered replication just for this reason
3:24 PM  → shimaore joined  •  smathy_ → smathy  
3:25 PM <wendall911> nolanlawson: I think most folks are late in the day or sleeping by now
3:26 PM <wendall911> would be nice to continue the discussion on this tomorrow
3:26 PM <nolanlawson> yeah, I'm on Pacific time, always forgetting that
3:26 PM <nolanlawson> anyway, your plugin idea sounds amazing
3:27 PM <wendall911> nolanlawson: ok, my co-worker is writing up his thoughts on this soon, I'll share once that's in my hands
3:27 PM <wendall911> nolanlawson: oh yeah! rcouch added that
3:28 PM <wendall911> well that will solve the slowness issue
3:28 PM <nolanlawson> potentially we could also implement this on the pouchdb side
3:28 PM <wendall911> I think that would be sweet
3:28 PM <nolanlawson> i.e. replicate from a couchdb view, exactly as you said
3:28 PM <nolanlawson> there's no technical reason we couldn't, I think. it'd just be difficult
3:29 PM <wendall911> mostly it's for complex relationships in groups for permissions
3:29 PM <wendall911> say you have several levels of access
3:29 PM <nolanlawson> yeah, or even just for docs with a "user" field, this would be useful
3:29 PM <wendall911> [level1, level2, level3, userId, somethingElse]
3:29 PM <nolanlawson> right