Balance of Modularity and Packaging

the-problem.md

So I have this question I asked on twitter and irc. I got some useful feedback, but thought putting the question clearly in a gist would lessen the confusion.

The Problem

As many of you know, I'm working on js-git right now. I've come across a question about how to package the various modules that make up the library and am unsure which is the best path.

The modules in question are things like:

• pkt-line - A min-stream codec for deframing and framing git protocol data over TCP
• list-pack - A min-stream codec for consuming a stream of git PACKfile data and emitting raw objects
• fetch-pack - A high-level min-stream codec for speaking the fetch-pack protocol. Depends on list-pack, pkt-line, and others.
• inflate - A min-stream codec for inflating the deflated bits inside PACKfile, dependes on list-pack
• apply-delta - Apply git deltas used only within the git PACKfile format.
• etc...

Modules that I've already broken out of js-git and made into standalone packages/projects are:

• min-stream - A collection of min-stream helpers to make using min-stream codecs easier.
• min-stream-node - Adapter for node.js exposing streams as min-stream compatible.
• min-stream-chrome - Same as min-stream-node, but for chrome packaged apps using chrome.* APIs
• http-codec - A min-stream codec for converting between TCP and HTTP protocols (lets you implement HTTP clients and servers)

I've also broken out the sample applications that use js-git into their own repos.

• js-git-node - A CLI tool for working with git repos (replaces the git command)
• js-git-app - A chrome packaged app that allows working with git on chromebooks.

As you can see I'm already taking anything generic out into it's own module.

My question is which is better for the js-git community?

Solution A

Keep all the git-specific code in a single js-git repository as a single meta-package. This is how bops, min-stream, and continuable are all packaged and it works well for them. The individual modules within the package can still be used independently of the rest. If you're unable to use just part of a package that's an issue with your tooling, not my package.

This does mean that some of Chris Dickinson's code will need to be assimilated into the main js-git repo. It's currently scattered across his various repos (mostly starting with git-*)

Solution B

Use many small individual repos. Each with it's own independent owner, repo, CI integration, package, npm name, version, issues, etc. Each module will have it's own code style and act like an independent project.

This also has many benefits, but makes combining everything quite the hassle.

One interesting constraint of the targets that js-git supports is I really want to be able to develop js-git on a chromebook without going into dev mode (which has nasty security implications and a really ugly warning at every boot). As stated in my kickstarter, I want to help kids get into programming easier. The less barriers the better. Chromebooks are the #1 selling laptop on Amazon and have been so pretty much since they came out. Most of these are bought as gifts to kids. They often can't put their device in dev-mode and install linux, but they can run chrome apps. In this environment, there is no node.js, no npm, no browserify, and no git. I'm fixing the problem with no-git, but I can't depend on node, bash, or any normal command-line tools for packaging.

Tell me what you think.

I'm especially interested in hearing from my backers to hear what they expect.

Keep in mind that with both proposals, the modules themselves will be standalone code with independent tests. With either route, there will be a pre-packaged version for people who just want to use js-git. The question is about what contributors want. Do we want independent issue trackers, CI integration hooks, packages, or do you see js-git as a single project with many modules that make it up?

The stuff that's not git specific will already be pulled out into independent projects, so that's not in question.

I'll +1 Solution B for the following advantages which outweigh the disadvantage of scattered issues and outlined challenges IMHO:

• isolated bug reports/fixes are a good thing
• contributors can be added to a repo on a per feature basis instead of on the entire git project
• sub packages can be reused as stand alone (you never know who may need certain functionality)
• tests can be isolated
• smaller code base/package makes it easier for contributors to understand problems and apply isolated patches with appropriate tests
• changes applied in a sub package can be properly versioned and thus selectively applied to the main repo
• issues and PRs are more focused since they will be made in single concern packages

As an added caveat, the packages in question already exist in isolation(search for 'git').
So pulling all these into one package would essentially duplicate code and either cause duplicate maintenance effort or result in these packages to become obsolete due to maintenance only being performed in the main package.

IMHO this would be a step backwards, i.e. de-modularizing pieces of code that are currently nicely separated into single concern packages.

B could be merged with A using subtree merges and automating that. You end up with one master repo that has complete history, and individual repos that contain the individual bits.

Solution A for me. Too much hassle to have multiple repos in both setup and future updates.

I really like having it broken down into seperate modules, usable for specific tasks, but don't feel they need to be seperate repos. Just make a list a list in the readme just like you've done above?

Hassle for what? As an end-user you would use the pre-built packages, as a developer the build system would solve it for you.

A.

It's more important to do what's right than what makes the node community happy.

@sztanpet, What about people wanting to develop js-git and other libraries apps on chromebooks? You know, the main use case for this library in the first place.

Unsurprisingly, I vote B.

• A. seems like it's solving for the symptom of a larger problem -- one that may prove transitory with @dominictarr and @maxogden's efforts to build an in-browser version of npm and browserify.
• A. implies that all of my existing code will have to be rewritten to match style. Not a huge cost, but, I feel, not a cost which is justified by gains.
• I'm not switching development of my packages to be against the primary js-git repo -- I have no chromebook/chromeos to test against, so for me it would be exceedingly painful for, again, little gain.
• When you say some packages are "too small" to be useful -- git-read-pkt-line and git-write-pkt-line for example -- they're actually very useful when you frame it against the problem of having to write fetch-pack, upload-pack, and receive-pack, which can depend on those packages OR swap them out if the abstraction for one proves unfeasible for the rest. Likewise if they don't match for smart-http-protocol.
• Again, look at list-objects / objectify-pack -- they can be used for fetch-pack or generate-packidx.
• Others, like git-apply-delta and git-create-delta will probably be renamed out from under the git namespace -- they're generally useful! they implement a coder and decoder for xdelta binary diffs. separating the two into separate packages with separate concerns has actually been useful -- apply-delta being separate means that i can make forward progress on things like fetch-pack while being somewhat blocked on making create-delta better. Plus I can include apply-delta as a devDep of create-delta for testing, at a pinned version.
• As a maintainer of these packages, I would dearly love for patches to make it upstream to js-git. But even more so, I'd love for upstream bug-fixes and improvements from js-git to make it back into these packages. Approach A means I need to wade through js-git's commit history constantly looking for bugfixes. (IOW, @st-luke, it's not just about making the "node community" happy, it's about writing the best packages for all communities with as little duplication of effort as possible)
• It seems to be glossed over, but it was always my plan, at least, to have the js-git repo have a dist/ directory with precompiled versions of js-git for chromeos, browsers not using browserify, etc -- which would address the concerns of multiple users in multiple ecosystems at once!
• For licensing concerns -- I'm tempted to say "let that become a problem", but we could be proactive and just use npm ls plus a script to fetch the license info from the various package.jsons to generate a VENDOR.txt/LICENSES.txt.
• There's a lot left to do in js-git -- more than we can reasonably guess at at the moment. Preemptively making a single large "meta" package could prove unmaintainable very quickly as we start to implement the merge algorithms, rebasing, diffing, submodules, etc, etc, etc -- and each of those smaller sub-packages solve interesting problems that can be reusable (see walk-refs and walk-tree -- they're both DAG walking algorithms -- so they can both rely on dag-walk which implements a simple min-stream that's usable elsewhere in the browser/node communities). By going with strategy B -- we can not only precisely identify the problems with chromebook development, we can solve them in a reusable way; similarly, we can not only build the various tooling that js-git needs to exist, we can actively improve and expand the solution space available to other javascript programmers for problems that may only be tangentially related to git itself -- at no extra cost to ourselves, and maybe more importantly, this can be completely transparent to end users of the library (who will be just directed to the dist/ downloads, a la jquery).

whew. sorry for the wall of text.

anyway, I'm for approach B. :)

@st-luke

It's more important to do what's right than what makes the node community happy.

This is all about doing what's right and facilitate ease of reuse and community contributions. If that makes people happy as well that is just an added benefit ;).

@thlorenz I'm referring about doing what makes it easier for people without a command line to use this, not what makes it easier for already capable programmers.

I think I'll go for option B for now. It seems the negative parts will be temporary as better tooling is written for the other environments. I really don't want to duplicate efforts by maintaining a fork of Chris's code.

It should be noted that

independent issue trackers,

Can be solved with a single meta github repo to centralize issues. I would imagine you want fine grained issues on individual bugs and higher level issues on js-git itself

B

Hello from 2017! Would anyone from the original project care to write a retrospective? I myself have found the result to be incredibly frustrating. As an outsider trying to comprehend the project & how to use it, what I've experienced is dozens upon dozens of modules (Tim has 68 results for source repositories matching "git" and Chris has 32 results for source repositories matching "git"), with varying degrees of documentation, and no website explaining how each module relates to each other. I've literally been using a hyper-media driven approach, looking at each module on npmjs.org and following the "Dependencies" and "Dependents" links. I think the root node for @creationix is js-git and the root node for @chrisdickinson is git-fs-repo but those are just best guesses.