private npm repo namespace idea so that private registries don't have to replicate the entire public NPM registry or you don't have to do some sort of conditional proxying.

package.json_private_module

// A module published to a private registry would optionally have a "registry" 
// property that is a reference to the registry where this module is published. 
//
// A `npm publish` would publish to the registry in the registry property.
//
// Otherwise, `npm --registry http://private.me:5984/registry/_design/app/_rewrite publish`
// 

{
  "name": "foo",
  "description": "a module published to a private registry",
  "version": "1.0.1",
  "dependencies": {},
  "registry": "http://private.me:5984/registry/_design/app/_rewrite"
}

package.json_ref_private_reg

// If this module, `with-some-privates` depends on modules from a private registry (could be public too),
// it can register module namespaces that will be used when resolving dependencies by specifying the 
// namespace and the URL of the registry in the `registries` property.
//
// The namespaces are meaningless outside of this package.json, they are simply and alias to the registry
// for dependencies. 
//
// Dependencies would reference modules in the private registry by the namespace prefix and some delimiter,
// a hash (`#`) perhaps. 
//
// After installing the module you could require it without the namespace like
// `require('foo')  --- possible conflicts, but simplifies other things (?)
//

{
  "name": "with-some-privates",
  "description": "a module with private registry references",
  "version": "0.2.0",s
  "dependencies": {
    "request": "2.9.x",
    "myprivates#foo": "1.0.1",
    "another#bar": "0.2.0"
  },
  registries: {
    "myprivates": "http://private.me:5984/registry/_design/app/_rewrite",
    "another": "http://another.com:5984/registry/_design/app/_rewrite"
  }
}

The colon in dependency names is a bit visually noisy, but I really like where you're going with this.

We definitely need to evolve private registries and make it a real thing without making npm into Maven.

I don't hate this.

Perhaps with '#' instead of ':'. See https://gist.github.com/3180050

I find this more visually distinct and easier to separate from the surrounding json markup.

@dshaw good point on the delimiter, it wasn't until I looked at it with fresh eyes that I appreciated it. A : delimiter within a string in a JSON block is not a good idea. The # seems like a good option. I updated the gist.

What do you think about the namespace not being exposed outside of the package.json?

For example, let's say you decide to name a module in a private registry request which obviously overlaps with the name of the public npm request module.

  "dependencies": {
    "myprivates#request": "1.10.1",
  },
  registries: {
    "myprivates": "http://private.me:5984/registry/_design/app/_rewrite",
  }

To use this module you would require('request'). I think that's OK, but I would highly discourage naming a private registry module a name that clearly conflicts with a public registry module. Even if you fork a public registry module and publish to a private registry I think you should rename it.

Therefore this would cause problems because require('request') would be ambiguous. I think the NPM client should treat this the same way as it would treat listing the request module twice in the dependencies.

  "dependencies": {
    "myprivates#request": "1.10.1",
    "requiest": "0.2.0"
  },
  registries: {
    "myprivates": "http://private.me:5984/registry/_design/app/_rewrite",
  }

I prefer this because it would require minimal changes to how the NPM client works today. It would also hopefully require no registry changes. It's also completely opt-in.

Also, a public module should NEVER depend on a private registry module. A private module may depend on other private modules, even across private registries but that's up to the authors of those modules.

Other NPM client functions like info or search would require you to specify the registry like you would today:

npm --registry http://private.me:5984/registry/_design/app/_rewrite

or

npm config set registry http://private.me:5984/registry/_design/app/_rewrite

Resisting the urge to jump into the npm client code to make this work before getting a big more feedback.

Thanks!

I really like this idea, obviously. Here are some additional thoughts:

• In package.json of the private module, you can already point at a specific registry. But it doesn't seem to work for me on NPM v1.1.21. Maybe it works on the latest version, but I haven't tried.
• Also, dealing with name conflicts might get a little hairy during an NPM install. What if you depend on a private module called "request" and a public module that depends on the public "request" module. I think (according to the algorithm) that NPM would recognize that "request" was already installed as a top-level dependency, and then ignore any submodules that reference "request" (because it thinks it already got it). Now, I think that the versions would have to be the same as well... but it could happen. So you'd almost want to either disallow an install if there are conflicts, or use the private registry name when checking for already installed dependencies.

Does all that make sense? :)

I like this, definitely a problem worth solving and this seems like a fine solution. Perhaps the npm client could even (optionally?) warn if a private registry module has a name conflict with public?

I'd be tempted to flip it so it reads left to right and uses @... as in "modulename@registrypointer"

 "dependencies": {
    "request@myprivates": "1.10.1"
  },
  "registries": {
    "myprivates": "http://private.me:5984/registry/_design/app/_rewrite",
  }

but that is nit-picky... and overall I like it.

Also, an alternative might be to support defining multiple registries in the .npmrc and/or via "npm config add myregistry http://registry.myregistry.com:5984"... and keeping the registry definitions out of the package.json all together

01:16 -!- Irssi: Starting query in freenode with JasonSmith
01:16 <JasonSmith> Do you have any thoughts about this private registry proposal? https://gist.github.com/3180014
05:30 <isaacs> i dont hate the syntax.
05:30 -!- JasonSmith is away: Auto-away
05:30 <isaacs> but i haven't thought much about it
05:53 <isaacs> JasonSmith i don't like the aliasing aspect of it
05:53 -!- JasonSmith is away: Auto-away
05:54 <isaacs> like, it's very very nice that Object.keys(pkg.dependencies) --> fs.readdir('node_modules')
05:54 <isaacs> changing that will have subtle bugs all over the damn place, i'm sure.
05:54 <isaacs> i would almost rather put the registry name/url/something in the value of the dependency somehow
05:55 <isaacs> "dependencies":{"foo":{"registry":"http://myprivates:5984/","version":"1.2.3"}}
05:55 <isaacs> with maybe some shorthand for that somehow
05:56 <isaacs> "dependencies":{"foo":{"registry":"myprivates","version":"1.2.3"}},"registries":{"myprivates":"url"}
05:56 <isaacs> "dependencies":{"foo":{"npm://myprivates#1.2.3"}},"registries":{"myprivates":"url"}
05:56 <isaacs> i dunny
05:56 <isaacs> *dunno

@isaacs Good point on the package dependencies name and contents of node_modules.

What if we moved those details into the value of the dependency as you suggested. The private repo could be aliased by properties in the registries property or npm:// or npms:// protocols can used literally like npm://{http_reg_host}/{name}/{version}.

For example, I pretty much have this shoe stringed together and working:

"dependencies": {
  "foo":"myprivates#1.2.3",
  "bar":"npm://my.privates.com/bar/1.0.6",
  "baz":"npms://my.privates.com/bar/1.0.6"
},
"registries":{
  "myprivates":"http://my.privates.com"
}

you could implement this on the registry or the client side.
another factor to take into account is auth,
as this will be private code in the non private repos,
and you'll need auth for that anyway.

hmm, that is gonna need a change to the npm client anyway.
as you'll need support for different credentials per registry.

and if you are gonna configure the auth in the config, maybe you should configure the url there too.

[registry.joyent]
url= ...
auth= ...

and then when "trade-secrets@joyent": "1.0" appears in the package.json npm will know that it's at the url you've specified. otherwise, if you want to move the registry or something, you'll have to go and update ALL your modules.

@dominictarr The auth config is a good point. Currently when we've used a private registry at Pearson it was behind a firewall without auth to resolve modules. So how about if it finds _auth config for the alt registry it will always use it otherwise it will try without it.

So far in my hacked up npm client code I can have config that looks like this:

registry.myprivates.registry = http://registry.foo.org
registry.myprivates._auth = ....
registry.myprivates.username = ....
registry.myprivates._password = ....

If a package.json has a version dependency that's not a protocol URL but contains a hash, then it will assume the left side of the hash is the alias name of the alternate registry.

"dependencies": {
  "foo":"myprivates#1.2.3",
}

Can anyone think of any ramifications of specifying the dependency like "foo":"myprivates#1.2.3", with the alternate registry alias in the value of the dependency rather than the property name?