Create a gist now

Instantly share code, notes, and snippets.

What would you like to do?
The way of the substack

The way of the substack

With over 200 modules in npm, and many such as browserify, dnode, optimist, etc. relied upon day-to-day by developers all around the world, substack is a pretty damn productive guy.

Plus, he's got an awesome philosophy on programming to boot (yes, there is a programming philosophy! ... no comprende? Let me explain later).

BTW, how do I know this? I got to talk to him at campjs as he wrote & published a module (lexical-scope to be exact). He was super friendly and shared alot of his thoughts on many topics.

All in all, it was really cool to meet someone that's completely congruent with what he says & lives out as a programmer. It almost felt Bret Victor-like.

Tools of the trade

Substack uses:

  • a small-ish Lenovo laptop
  • unix: not Mac
  • node/npm: duh!
  • vim
  • xmonad: keyboard friendly window manager that tiles windows automatically with virtual workspaces
    • Why? Floating windows that overlap aren't generally useful, so you might as well tile them, use multiple workspaces and make it all accessible with the keyboard.

That's pretty much it. As you'd expect, he knows alot of node modules, far too many to mention here.

How substack creates modules

  1. substack codes as usual:
    • Uses console.dir alot to inspect & debug
    • Didn't TDD - creates an example.js that requires the module and tests its functionality
    • Codes in vim & switches to a terminal every so often to run the example.js script
  2. If he notices an opportunity to modularise, he immediately moves the function into a new file & changes the function declaration to module.exports = ... (When I say immediately, I really do mean immediately).
  3. When he's happy with the module, npm install tape tap
  4. Refactors the example.js file as a set of tests (see below).
  5. Writes README.markdown from scratch with introduction, API documentation & license (API documentation is quick & easy to write when you have small modules).
  6. Runs pkginit to create package.json.
  7. Create GitHub repo and npm publish

Test example:

t.plan(numberOfTests)
// do stuff
t.same ... // asserts

Now... if you've already done a few modules, the above set of steps might seem like nothing special to you.

The secret sauce is that substack is fast - really fast! He thinks fast, he codes fast... he's fast!

As the adage goes, "practise makes perfect".

Philosophy

I have to admit something... in the past, I used to evaluate packages by looking at the Github page to see when it was last updated. If I see something that is more recent, it gets my tick of approval.

Bad Chris!

Now substack doesn't do that. He just wants the most dependable, easy to understand and smallest module he can find (or make) to fulfil his objective.

In other words, he subscribes to the UNIX philosophy. I thought I did as well, until I talked to him.

So as an example, everyone sees nodejs as quite progressive. Features like domains, streams, etc. are being improved upon at the moment.

However, substack wants to see nodejs as 100% complete and dependable one day.

That means no more work on it and zero issues, just like the UNIX tools we use - cat, grep, date, etc.

Hopefully what I said makes total sense to you. But for me, what he said sounded like crazy talk at first. I mean, can't we have nodejs v42.6.7 one day?

Over time, it eventually sunk in and made total sense.

Lean Mean Fighting Machine

This goes on to my next point - substack practises extreme modularity.

He tries to keep each module he writes under 200 lines of code, although he knows it's not easy in all cases.

But he doesn't say it's impossible. To him, it's about taking the time to understand the abstractions. And once you do, refactor to modules.

If this is done right, you have modules that are:

  • Easier to add to projects without feeling guilty of bloat (e.g. jquery, ember, etc.)
  • Easier to contribute to
  • Easier to read & understand
  • Has independent versioning

Wow, doesn't that sound awesome! Why doesn't everyone code like that?

The more astute among you might say, "That sounds great, but if you go crazy modular, you're going to have massive dependency graphs with replicated functionality everywhere."

Sure you might, but substack says he's never really encountered that. I haven't encountered that. Therefore, we're right!

No, jokes aside, a smart package manager (i.e. npm) with a whole bunch of micro-libraries is usually still a lot smaller... and more importantly, easier to understand than an all-purpose framework.

Modularity over convention

So what about all-purpose frameworks?

My heading was a play on the words "convention over configuration". When I queried substack about his thoughts on CoC, he was slightly perturbed. He then gently explained that how you glue modules together essentially is the "configuration".

To summarise, Dominic Tarr put it well when he came over at one stage and briefly quipped, "configuration is a smell".

All in all

There were other topics discussed, but this post is getting a bit past 100 lines :-). All in all, it was super interesting to talk to substack.

If you learnt something useful (or think I'm full of it), I'll happily receive your comments below!

Related sources

History

  • 09-Mar-13: Updated to be more correct in terms of how substack writes modules (e.g. test.js to example.js)
  • 22-Feb-13: Wrote the article
@alexyoung

This is great, thanks for sharing! I like hearing about what makes prolific developers like substack tick. He should have one of those usesthis.com posts.

The only negative thing small modules and high modularity has caused for me in the past is maintenance headaches. Particularly when maintaining projects that last for years. This is because I find myself dealing with a lot of freelance projects at any one time just to make a living, so it might be just down to my poor management skills. Approaches like semantic versioning can help mitigate most of the API change related issues that this usually boils down to, but what usually helps a lot more is just saying 'no' to unnecessary features.

Whenever I'm at a meetup/conference I'm always surprised by the Apple laptop dominance. Node's great on Unix and Windows, so there's no reason you can't use a PC laptop.

@wagenet
wagenet commented Feb 23, 2013

How does this fit with the fact that we have a myriad of different Unix releases? Some of the underlying modules don't see much change, but we certainly see changes to the overall package.

@jlogsdon

@wagenet, that still fits in with what substack wants: there may be a myriad of different Unix releases, but they are all just different "configurations" of the stable programs. Obviously there are higher level things going on, but the basis of all linux distributions is the same base stack of simple, single-purpose programs and libraries.

@khoomeister
Owner

@wagenet @jlogsdon, interesting... @substack raised this when I was talking to him. I actually wrote a section about it, but then left it out because the article was going too many places.

So let me repeat it here. The example is not 'nix related, but hopefully I'm drilling down to the core of the issue.

There's a common complaint in the npm community that there are too many packages that duplicate the same functionality (e.g. try searching "promises" in npm).

But @substack says that is a good thing.

Why?

You have choice, which is better than very little or no choice.

So you do a npm search and it comes back with a squillion packages. You have a local optimization problem.

There are many ways to resolve this, but let's just say you find one that works for you.

As people use the packages and more information is gathered on what works and what doesn't, a consensus will eventually be reached. That becomes the defacto standard for a time, and people will naturally move on to other abstractions.

To tie it back, modularity goes hand in hand with choice. Modularity at all levels of the stack encourages participation, which increases the breadth of ideas and sets a better foundation for the future.

Aww man, I could keep on going on and on, but I hope you get the point!

@khoomeister
Owner

@alexyoung - I'm not sure about the specifics, but the great thing about modular code is that it's easy to fork and maintain yourself. For example, try reading @substack's code and compare it to jQuery... worlds of difference!

@sstur
sstur commented Feb 23, 2013

That was a good read! I will have to remember to take better mental notes the next time I get a chance to chat with an industry guru.

@Floby
Floby commented Feb 28, 2013

Did you ask @substack about the reason behind choosing non-self-explanatory names for some modules?

This is the kind of things that annoys me.

@mgan59
mgan59 commented Feb 28, 2013

Seen Substack present twice now and it is always a treat. I adopted his

module.exports = function(){
    // write module code here
};

as my default way to write node code.

@khoomeister
Owner

@mgan59 - yes I wish more module systems would allow for that!

@khoomeister
Owner

@Floby - naming is one of those really really difficult things to get right - what makes total sense to one group might be totally confusing to another group. Plus, terminology changes over time.

A great example of this is "virtual machine", which could either mean programming language VM (e.g. Java, Dart, etc.) or a virtual image (e.g. VMware, VirtualBox, etc.).

I'm sure you can make suggestions to @substack on github issues and see what he says.

If you really don't like it, you can always fork! That's the beauty of open source.

Chris

@denis-sokolov

Didn't TDD - creates an example.js that requires the module and tests its functionality
Codes in vim & switches to a terminal every so often to run the example.js script

How is creating a runnable script that requires the module and uses it, then running that script every so often not TDD?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment