splitting built and imported files out of a project tree

Splitting.md

Splitting files out of a Mix project tree

I've been playing with splitting built and imported files out of a Mix (eg, Elixir, Phoenix) project tree.

Motivation

Although Mix works very nicely, its approach involves importing (or creating) a very large number of files in the project's file tree. This means that:

• copying the file tree is tedious and wastes a lot of space
• code searching may be tedious and bring up false positives

In addition, although I understand the rationale for keeping some assets commingled (eg, css, js), keeping 166 MB of node_modules in my editable work space seems rather a bit much. To say nothing of several MB of _build files...

Early Results

I tried moving the biggest sub-trees to a shelf tree, getting to them by means of symlinks. This reduced the size of my bench tree by more than 97% (99% if you ignore the unusually large collection of TOML files):

$ du -sk . *
184160	.
  4128  bench
180024  shelf

$ du -sk bench/{*,PA_elixir/*}
    36  bench/PA_about
   984  bench/PA_elixir
   856  bench/PA_elixir/phx_http
   116  bench/PA_elixir/ref_data
  3100  bench/PA_toml

iex(1)> 4128/184160
0.022415291051259775
iex(2)> 984/184160
0.005343179843614249

Here are the stats on the shelf tree. Almost all of the content comes from my Phoenix app (phx_http) in general and node_modules in particular:

$ du -sk shelf/*
179432  shelf/phx_http
   584  shelf/ref_data

$ du -sk shelf/phx_http/*
  7256  shelf/phx_http/_build
  6120  shelf/phx_http/deps
166056  shelf/phx_http/node_modules

$ du -sk shelf/ref_data/*
   432  shelf/ref_data/_build
   152  shelf/ref_data/deps

BBEdit's "Multi-File Search" still took a while and brought up a lot of false positives. However, a simple filter resolved that:

no_goo
Matching: Any term may match
Path  Does not contain  /deps/

Snafu

When I played with the RefData app in IEx, it worked fine. However, my Phoenix app (PhxHttp) choked:

ERROR in ./js/app.js
Module not found: Error: Can't resolve 'phoenix_html'
in '.../PA_sym/bench/PA_elixir/phx_http/assets/js'
 @ ./js/app.js 11:0-22
 @ multi ./js/app.js

I suspect that some part of the Phoenix tooling doesn't like symlinks, but this is getting beyond my pay grade...

Implementation

My working tree is in PA_all; the experimental one is in PA_sym. Here is some (iffy, reverse-engineered) shell code for the rework:

$ cd .../PA_hax
$ cp -Rp PA_all PA_sym

$ cd PA_sym
$ mkdir -p bench shelf/{phx_http,ref_data}
$ mv PA* bench
$ s1=../../../shelf

$ pushd bench/PA_elixir/phx_http
$ s2=$s1/phx_http
$ mv _build $s2
$ ln -s $s2/_build .
$ mv deps $s2
$ ln -s $s2/deps .

$ cd assets
$ mv node_modules ../$s2
$ ln -s ../$s2/node_modules .
$ popd

$ pushd bench/PA_elixir/ref_data
$ s2=$s1/ref_data
$ mv _build $s2
$ ln -s $s2/_build .
$ mv deps $s
$ ln -s $s2/deps .
$ popd