IRC discussion on PouchDB defaults

pouchdb_defaults.txt

12:52 PM <daleharvey> nolanlawson: we cant have express-pouchdb depend on pouchdb, thats broken
12:52 PM <daleharvey> pouchdb-server should depend on pouchdb
12:55 PM ⇐ ncthom91 quit (~ncthom91@mpk-nat-7.thefacebook.com) Ping timeout: 264 seconds
12:56 PM <nolanlawson> daleharvey: hm, yeah, that wouldn't be so bad
12:56 PM <nolanlawson> Less duplicated config in express and server
12:57 PM <nolanlawson> However gets weird with all the places Pouch is instantiated, e.g. all-dbs
12:57 PM <nolanlawson> Would require your default constructor configuration
12:58 PM <nolanlawson> I see where you're coming from
12:58 PM <daleharvey> nolanlawson: yup, but its not just like a nice to have, it currently makes pouchdb-express pretty much useless outside of pouchdb-server
12:59 PM <nolanlawson> Why
1:00 PM → jrhicks joined (0cb375fd@gateway/web/freenode/ip.12.179.117.253)
1:02 PM <daleharvey> because express-pouchdb is to help people build apps with an ambedded http that their application also operated on, we cant have 2 (differing versions) of pouchdb operating on the same data
1:05 PM <nolanlawson> hm, I wouldn't be too concerned about that. if we say we're compliant with couch, couchbase, cloudant, etc., then we better be compliant across versions, or at least minor versions
1:06 PM <daleharvey> I am not talking about the http api, thats just want express pouchdb provides, I am talking about 2 differing versions of pouchdb (my applications and express pouchdbs) operating on the same data
1:06 PM <daleharvey> thats broken (and also crazy annoying in terms of build times)
1:06 PM <nolanlawson> ah okay, you mean the server version of pouchdb
1:07 PM <nolanlawson> okay yeah, good point
1:07 PM <daleharvey> I noticed it when I was building pppaste.com
1:07 PM <nolanlawson> still, this can be solved without modifying pouch itself
1:08 PM <nolanlawson> default config options for express-pouchdb (like I already have) plus passing in the pouchdb dep
1:08 PM <nolanlawson> done
1:09 PM <nolanlawson> and even better, because it will work with pouch 2.2+
1:12 PM ⇐ dwhly2 quit (~dwhly@91.82.208.7) Remote host closed the connection
1:13 PM <daleharvey> can you give an example in the bug? I dont think it should be the responsibility of express-pouchdb to configure anything about how pouchdb behaves
1:14 PM <nolanlawson> https://github.com/pouchdb/express-pouchdb/blob/38/README.md
1:14 PM <nolanlawson> search for setPath and setBackend
1:15 PM <nolanlawson> could also wrap them up in a single cinfig object
1:15 PM <nolanlawson> *config
1:17 PM <nolanlawson> I still like keeping that logic out of pouch for reasons I already stated. express-pouchdb and pouchdb-server will have a million special needs, don't want that leaking into PouchDB. worried it'll be allDbs all over again
1:18 PM <daleharvey> nolanlawson: we discussed the {dir: } option long before express-pouchdb existed, just been lazy
1:20 PM → dwhly2 joined (~dwhly@91.82.208.7)
1:21 PM <daleharvey> and I think its the wrong seperation of concerns to have express know about pouch configuration, pouch should know pouch config, express should just use it
1:22 PM <nolanlawson> daleharvey: the problem arises because adapters also have their own config. with riak et al it will explode. with localstorage-down I have plans for more opts (e.g. in-key memories vs. btree)
1:23 PM <daleharvey> nolanlawson: thats specifically why express should deal with them
1:23 PM <nolanlawson> I'd rather pouch remain as dumb as possible and just pass a string into leveldown. the leveldown constructor takes a string as input; that's all pouch needs to know about
1:24 PM <daleharvey> it just means patching express every time do something with pouchdb's config
1:24 PM <nolanlawson> if we do the config within express, you won't need to upgrade your version of pouch and express at the same time. just upgrade express
1:25 PM <nolanlawson> also we already have to do stuff like mkdirp and file path checking; look at the path.resolve() stuff I already have to do
1:25 PM <nolanlawson> that would just be duplicated in pouch, plus a dependency on path which only makes sense in Node...
1:25 PM <daleharvey> if you had {memkeys: true} for memdown pouch, it means you need to patch express to add support for that
1:26 PM <nolanlawson> well, patch pouchdb-server, yeah. setBackend(require('memdown').withMemKeys())
1:26 PM <nolanlawson> basically I want the exact same config object as you, but I want it in express, not pouch
1:28 PM <daleharvey> As said, that means we need to update express every time we change configuration options in pouch, and it applications need to figure out how to configure both pouchdb and express to do the same thing in different ways
1:30 PM <daleharvey> one sec, will code an example
1:30 PM <nolanlawson> I don't think that's true, because pouch config options won't change. we'll continue using the 2.2 API forever
1:30 PM <nolanlawson> ok
1:32 PM  → ncthom91 joined  ↔ klaemo nipped out  
1:35 PM <daleharvey> https://gist.github.com/daleharvey/437a6dc2887df5da7389
closedaleharvey — 28 May 2014
1 2 3 4 5 6 7 8 9 	

var pouchdb = require('PouchDB');
var expressPouchDB = require('express-pouchdb');
 
var backend = require('memdown');
 
var db = new PouchDB('adb', {db: backend});
expressPouchDB.setBackend(backend);
 
app.use(expressPouchDB);

view raw gistfile1.txt hosted with ❤ by GitHub
1:35 PM <daleharvey> requires the user understanding 2 different ways to configure the same thing
1:35 PM  → klaemo_ joined  ⇐ klaemo quit  
1:36 PM <nolanlawson> hm, well, three if you count leveldown + pouchdb + express-pouchdb
1:36 PM <daleharvey> also I dont understand using the 2.2 API forever, we will add features to pouchdb, pouchdb-express users will want to use those configurations
1:36 PM ⇐ ncthom91_ quit (ncthom91@64.125.189.90) Ping timeout: 276 seconds
1:37 PM <nolanlawson> I said that because currently the 2.2. API gives us {db: leveldown} plus a string that's passed to the leveldown constructor. this is a backdoor that gives us everything
1:37 PM <daleharvey> putting the configuration in express puts a coupling between express-pouchdb and pouchdb, as configurations are added, express needs updated for users to make use of them, if the options are opaque, then there is no coupling
1:38 PM <daleharvey> ie if we change the name of 'db' in pouchdb, then express needs updated, if we use opaque opts, it doesnt
1:39 PM <nolanlawson> how about this: https://gist.github.com/nolanlawson/60c863ffd8334bf1c5d5
closenolanlawson — 28 May 2014
1 2 3 4 5 6 7 	

var expressPouchDB = require('express-pouchdb');
 
expressPouchDB.setBackend(require('memdown'));
 
app.use(expressPouchDB);
 
var pouch = expressPouchDB.getPouchDB('dbname');

view raw index.js hosted with ❤ by GitHub
1:39 PM wilhol → wilhol_away
1:40 PM <nolanlawson> also the 'db' name better not change in pouchdb, because we put it in the public API docs
1:40 PM <nolanlawson> at least not without a major version change
1:41 PM <nolanlawson> I just want two things:
1:41 PM <nolanlawson> 1) a stable pouch version for express-pouch to depend on, so it doesn't change out from under our feet and break users who probably installed pouchdb-server with npm install -g
1:42 PM <nolanlawson> 2) for pouch to stick with leveldown + a single string, so that no matter how crazy the leveldown adapters get, pouch doesn't need to know about it; only express-pouchdb needs to know about it
1:42 PM <daleharvey> as I said, express-pouchdb cannot depend on pouchdb at all
1:42 PM  → klaemo joined  ⇐ klaemo_ quit  
1:43 PM <wendall911> Sorry to interrupt, have a quick question.
1:43 PM <wendall911> I'm looking to add docs locally and have continuous replication on docs added locally with a remote db. I see that I can pass a list of doc_ids, but I won't know those in advance. Any suggestions here? I'm trying to avoid starting a new replication every time I add docs locally.
1:43 PM <nolanlawson> daleharvey: okay, then: https://gist.github.com/nolanlawson/7cebec7bc8f99115fe3e
closenolanlawson — 28 May 2014
1 2 3 4 5 6 7 	

var expressPouchDB = require('express-pouchdb')(require('pouchdb'));
expressPouchDB.setBackend(require('memdown'));
app.use(expressPouchDB);
var pouch = expressPouchDB.getPouchDB('dbname');

view raw index.js hosted with ❤ by GitHub
1:44 PM <nolanlawson> wendall911: just use non-continuous replication? then you can trigger it whenever you'd like
1:44 PM <wendall911> nolanlawson: I'd like it to be continuous
1:45 PM <nolanlawson> the definition of continuous replication is that it starts replicating as soon as new docs are added...
1:45 PM <wendall911> When I add a doc locally from the remote, sync that, but I don't want to pull in the entire remote db locally. Only if it exists..
1:45 PM <daleharvey> nolanlawson: I dont really understand what is motivating that, it now means users need to learn new ways to both configure and create pouch's
1:45 PM <nolanlawson> wendall911: only if the remote doc exists or the remote db exists?
1:46 PM <wendall911> nolanlawson: I'm setting up a local cache db. I check if it exists locally, if not, fetch from remote and push local. I'd then like to monitor the remote for changes
1:46 PM <wendall911> nolanlawson: trying to use PouchDB to do this, I have something already, but want to use sync if possible
1:46 PM <wendall911> or replication from pouchdb rather
1:47 PM <nolanlawson> daleharvey: I don't understand what's motivating the desire to stop express-pouchdb from depending on pouchdb. I think getPouchDB() is a neat solution; express-pouchdb can rely on e.g. pouchdb 3.1.1 because somebody broke something in 3.1.2, and that's fine, the user of express-pouchdb can use 3.1.1 which is the same that express-pouchdb is using
1:47 PM <daleharvey> "I am talking about 2 differing versions of pouchdb (my applications and express pouchdbs) operating on the same data"
1:48 PM <wendall911> nolanlawson: I'll just maintain a local filter, so no worries
1:48 PM <daleharvey> we do a migration, the user is using a new pouchdb, express is using the old version, stuff is broken
1:49 PM <nolanlawson> daleharvey: yes, in this case it would be the same version, because it was require()d by express-pouchdb. expressPouchDB.getPouchDB('dbname') would return a new PouchDB using the same version that express was using
1:49 PM <nolanlawson> wendall911: interesting, let us know how it goes. sounds like you may be inadvertently implementing some speedups for replication. :)
1:50 PM <wendall911> nolanlawson: that's exactly why I'm doing this
1:50 PM <nolanlawson> wendall911: sweet!
1:50 PM <wendall911> nolanlawson: keeping a local metadata db that can survive refresh
1:51 PM <wendall911> so I can iterate all configured dbs, start replication. If I keep a list of docs to filter, I start replicator with filter list
1:51 PM — nolanlawson points to PouchDB blog, which has been empty for one month
1:51 PM <nolanlawson> ;)
1:52 PM <wendall911> nolanlawson: would probably be nice to have a _replicator db or something, but this is intended to be dynamic configuration for the filter
1:53 PM <daleharvey> nolanlawson: that means we need to update express with every pouchdb release, users cant just add or remove pouchdb from they application or use the version of pouchdb they want to
1:53 PM <daleharvey> *their
1:53 PM <nolanlawson> daleharvey: I would prefer that, to avoid problems like what express-pouchdb had before
1:53 PM <nolanlawson> concern #1: express-pouchdb and pouchdb-server should work out-of-the-box when you npm install them
1:53 PM <nolanlawson> concern #2: users can use their own server-side pouch in Node alongside it, and it works the same way
1:54 PM <nolanlawson> imo
1:54 PM <nolanlawson> average use-case of both pouchdb-server and express-pouchdb is to spin up a quick and dirty couch replacement, and then talk to it from the client
1:55 PM <nolanlawson> if all else fails, you can talk to it with nano or http
1:55 PM <daleharvey> thats the use case of pouchdb-server
1:56 PM <daleharvey> express-pouchdb is to add a compatible http api to your node application that is using pouchdb
1:57 PM <nolanlawson> okay, I understand better what your use-case is
1:57 PM <nolanlawson> however allDbs will simply not work if you are calling new PouchDB() in your own code before you call require('express-pouchdb')
1:58 PM <nolanlawson> other stuff like ongoing replications and uuids should be okay, I think
1:58 PM <nolanlawson> but yeah, hm, it is a tough problem
1:58 PM <daleharvey> it will, the allDbs behaviour came up as part of the api decision
1:59 PM <nolanlawson> welp, the way I implemented it, it will not ;)
2:00 PM <daleharvey> pouch has a default data directory, allDbs does a filescan on that to implement allDbs, if you specify {dir: ...} differently from that directory, it wont be listed in alldbs
2:00 PM <nolanlawson> I implemented allDbs using PouchDB.on('created') and PouchDB.on('destroyed') so that it would work with memdown/riak/etc.
2:00 PM <nolanlawson> it creates a special db called pouch__all_dbs that is placed in the same dir
2:01 PM <nolanlawson> or in memory in the case of memdown
2:02 PM <nolanlawson> ok, I'm starting to get convinced, but I think the default config should have "prefix" rather than "path"
2:02 PM <nolanlawson> "path" implies pouchdb does path resolution, also doesn't make sense for riak/mysql URLs
2:02 PM <nolanlawson> only other default option is the default "db", right?
2:03 PM <daleharvey> part of the reason to do it is its confusing to overload name and the filepath
2:04 PM ⇐ Yakulu quit (~Yakulu@78.194.3.29) Quit: WeeChat 0.4.3
2:04 PM <nolanlawson> fair point
2:04 PM <nolanlawson> okay, so this is the only thing that needs to be merged Pouch-side, right?
2:04 PM <nolanlawson> https://github.com/pouchdb/pouchdb/pull/2074/files
2:04 PM <nolanlawson> then in PouchDB Server I would do e.g.
2:05 PM <nolanlawson> PouchDB.defaults({db: require('riakdown'), prefix: 'riak://localhost:8000/my/crazy/prefix'})
2:05 PM <nolanlawson> then give that to express-pouchdb
2:06 PM <nickcolley> nolanlawson, I think the uses cases of pouchdb-server and express-pouchdb might be better explained. I'm still kinda confused on it.
2:06 PM <daleharvey> yup something along those lines
2:08 PM <nolanlawson> okay, I can get behind that. thanks for talking me through your thought process
2:08 PM <nolanlawson> users will still need to be advised to call new PouchDB() AFTER they've called require('express-pouchdb'), but other than that I don't see any potential pitfalls
2:09 PM <nolanlawson> also would like to know how you feel about PouchDB.defaults() clearing the defaults every time, vs. e.g. PouchDB.setDefaults(), PouchDB.getDefaults()
2:10 PM <daleharvey> I commented in the PR, I really want to avoid that api (although its kinda complicated)
2:10 PM <nolanlawson> daleharvey: "Its tied to the other comment, if we create a wrapper then we dont need to clear the options since you only ever create specific constructors, if you want to clear it you use the default one"
2:11 PM <nolanlawson> some code samples would help explain what you mean
2:11 PM <daleharvey> heh I was just typing one out
2:11 PM <nolanlawson> if I know what changes we want for pouchdb, I could probably implement all this stuff in a few hours
2:11 PM <nolanlawson> sweet
2:11 PM drsm79|out → drsm79
2:12 PM <nolanlawson> nickcolley: biggest thing about pouchdb-server IMO is that we should not make grandiose claims about it. it's not production-ready until it passes our test suite and couchdb's, and even then I would be skeptical for a long time. however it is amazing for quick testing and prototyping
2:12 PM <daleharvey> I think the {dir: '', prefix: ''} stuff still needs a little more discussion, not sure how my proposal interacts with the level plugins very well
2:13 PM <nolanlawson> level plugins take strings as input, which is why I recommended "prefix" since it only implies stringiness
2:13 PM <nickcolley> nolanlawson, I just mean its not immediately obvious why there's two different projects
2:13 PM <nolanlawson> even prefix is shaky; it presumes every leveldown constructor string has the most significant parts on the left, which is only true because currently they all use URLs or filepaths
2:14 PM <nolanlawson> nickcolley: yeah, that can be solved with one sentence: "pouchdb-server is backed by express-pouchdb."
2:15 PM <nolanlawson> daleharvey: we can discuss, but I think prefix is the way to go. pouchdb-server can do all the fancy require('path').resolve(), mkdirp, etc. as it currently does; pouchdb itself is none the wiser
2:22 PM <daleharvey> nolanlawson: kinda hard to make a code example of my point - https://gist.github.com/daleharvey/ded340cf7637907db858
closedaleharvey — 28 May 2014
1 2 3 4 5 6 7 	

var PouchDB = require('pouchdb');
var express = require('express-pouchdb')(PouchDB);
 
PouchDB.defaults({some: thing});
 
// We very possibly introduced options which broke express / made it behave in ways we dont want
app.use(express);

view raw gistfile1.txt hosted with ❤ by GitHub
2:22 PM <daleharvey> point is if we have a global defaults option, its very easy to break things that use pouch because you can change their behaviour with no indication
2:23 PM <daleharvey> whereas https://gist.github.com/daleharvey/ded340cf7637907db858 is much more robust
closedaleharvey — 28 May 2014
1 2 3 4 5 6 	

var PouchDB = require('pouchdb');
var express = require('express-pouchdb')(PouchDB);
 
var myCustomPouch = PouchDB.defaults({some: thing});
 
app.use(express);

view raw gistfile1.txt hosted with ❤ by GitHub
2:24 PM <daleharvey> its inspired from the node requests api - https://github.com/mikeal/request
2:25 PM <nickcolley> nolanlawson, express-pouchdb exists so you can create a custom express app with pouchdb right?
2:25 PM <daleharvey> nickcolley: yup, its to add a http api to your node (express) app thats using pouchdb
2:26 PM <nickcolley> Whereas pouchdb-server is a restricted couchdb dropin that uses express-pouchdb
2:27 PM <nolanlawson> daleharvey: I do not understand the difference. myCustomPouch is not itself a PouchDB object, right? what's the difference between the two code snippets?
2:28 PM <daleharvey> myCustomPouch is a PouchDB object, that when used as a constructor uses those default options
2:28 PM  → ncthom9__ joined  ⇐ ncthom91 and dwhly2 quit  ↔ ncthom91_ popped in  
2:36 PM <daleharvey> and nickcolley yup exactly
2:37 PM <nolanlawson> ah, currying
2:37 PM <nolanlawson> so new PouchDB('foo', {db: require('memdown')}) is the same as new MyCustomPouch('foo')
2:38 PM <nolanlawson> so it's MyCustomPouch, not myCustomPouch
2:45 PM → dwhly2 joined (~dwhly@91.82.208.7)
2:45 PM <daleharvey> sorry yup
2:47 PM <nolanlawson> okay, I understand now
2:47 PM <nolanlawson> I'll paste this into that PR and then rewrite my other PRs. I'll probably do one massive PR for all three projects that combines in-memory with being able to set the path