This will hopefully one day be a blog post on how to identify and remove
transitive compile-time dependencies using mix xref graph and mix xref trace.
Elixir 1.13 ships with a simple tool to find problematic dependencies.
Running it on the hexpm repository gives:
$ mix xref graph --label compile-connected
lib/hexpm/billing/hexpm.ex
└── lib/hexpm/billing/billing.ex (compile)
lib/hexpm/billing/local.ex
└── lib/hexpm/billing/billing.ex (compile)
lib/hexpm/emails/bamboo.ex
├── lib/hexpm/accounts/email.ex (compile)
└── lib/hexpm/accounts/user.ex (compile)
lib/hexpm/store/gcs.ex
└── lib/hexpm/store/store.ex (compile)
lib/hexpm/store/local.ex
└── lib/hexpm/store/store.ex (compile)
lib/hexpm/store/s3.ex
└── lib/hexpm/store/store.ex (compile)
lib/hexpm_web/controllers/api/docs_controller.ex
└── lib/hexpm_web/auth_helpers.ex (compile)
lib/hexpm_web/controllers/api/key_controller.ex
└── lib/hexpm_web/auth_helpers.ex (compile)
lib/hexpm_web/controllers/api/organization_controller.ex
└── lib/hexpm_web/auth_helpers.ex (compile)
lib/hexpm_web/controllers/api/organization_user_controller.ex
└── lib/hexpm_web/auth_helpers.ex (compile)
lib/hexpm_web/controllers/api/owner_controller.ex
└── lib/hexpm_web/auth_helpers.ex (compile)
lib/hexpm_web/controllers/api/package_controller.ex
└── lib/hexpm_web/auth_helpers.ex (compile)
lib/hexpm_web/controllers/api/release_controller.ex
└── lib/hexpm_web/auth_helpers.ex (compile)
lib/hexpm_web/controllers/api/repository_controller.ex
└── lib/hexpm_web/auth_helpers.ex (compile)
lib/hexpm_web/controllers/api/retirement_controller.ex
└── lib/hexpm_web/auth_helpers.ex (compile)
lib/hexpm_web/controllers/blog_controller.ex
└── lib/hexpm_web/views/blog_view.ex (compile)
lib/hexpm_web/endpoint.ex
├── lib/hexpm_web/plug_parser.ex (compile)
└── lib/hexpm_web/session.ex (compile)
Each of the above is an issue.
The first up: lib/hexpm/billing/billing.ex is a compile-time dependency for a few files, and itself has dependencies and hence introduces transitive dependencies.
If we ask for a list of the dependencies, we get a
$ mix xref graph --source lib/hexpm/billing/billing.ex
lib/hexpm/billing/billing.ex
├── lib/hexpm/accounts/audit_log.ex (export)
│ ├── lib/hexpm/accounts/email.ex (export)
│ │ ├── lib/hexpm/accounts/auth.ex
│ │ │ ├── lib/hexpm/accounts/key.ex
│ │ │ ├── lib/hexpm/accounts/keys.ex
│ │ │ │ ├── lib/hexpm/accounts/audit_log.ex (export)
│ │ │ │ ├── lib/hexpm/accounts/key.ex (export)
│ │ │ │ ├── lib/hexpm/context.ex (compile)
│ │ │ │ ├── lib/hexpm/repo.ex
│ │ │ │ └── lib/hexpm/shared.ex (compile)
│ │ │ ├── lib/hexpm/accounts/user.ex
│ │ │ ├── lib/hexpm/accounts/users.ex
│ │ │ │ ├── lib/hexpm/accounts/audit_log.ex (export)
│ │ │ │ ├── lib/hexpm/accounts/email.ex (export)
│ │ │ │ ├── lib/hexpm/accounts/key.ex
│ │ │ │ ├── lib/hexpm/accounts/organization.ex
│ │ │ │ ├── lib/hexpm/accounts/password_reset.ex
│ │ │ │ │ ├── lib/hexpm/accounts/auth.ex
│ │ │ │ │ ├── lib/hexpm/accounts/user.ex
[700+ more lines...]
We can get the number of files that are dependencies (direct or indirect) of lib/hexpm/billing/billing.ex with the stats formatter:
$ mix xref graph --source lib/hexpm/billing/billing.ex --format stats
Tracked files: 132 (nodes)
[...]
This means that any kind of change to any of these 132 files appearing in the huge graph above will force a recompilation for all files having a compile-time dependency on lib/hexpm/billing/billing.ex. This is almost 70% of the 192 elixir files in the project!
We can get a list of these files with the --sink option and restricting to compile-time dependencies only with --label compile:
$ mix xref graph --sink lib/hexpm/billing/billing.ex --label compile
lib/hexpm/billing/hexpm.ex
└── lib/hexpm/billing/billing.ex (compile)
lib/hexpm/billing/local.ex
└── lib/hexpm/billing/billing.ex (compile)
lib/hexpm/emails/bamboo.ex
├── lib/hexpm/accounts/email.ex (compile)
└── lib/hexpm/accounts/user.ex (compile)
lib/hexpm_web/controllers/api/docs_controller.ex
└── lib/hexpm_web/auth_helpers.ex (compile)
lib/hexpm_web/controllers/api/key_controller.ex
└── lib/hexpm_web/auth_helpers.ex (compile)
lib/hexpm_web/controllers/api/organization_controller.ex
└── lib/hexpm_web/auth_helpers.ex (compile)
lib/hexpm_web/controllers/api/organization_user_controller.ex
└── lib/hexpm_web/auth_helpers.ex (compile)
lib/hexpm_web/controllers/api/owner_controller.ex
└── lib/hexpm_web/auth_helpers.ex (compile)
lib/hexpm_web/controllers/api/package_controller.ex
└── lib/hexpm_web/auth_helpers.ex (compile)
lib/hexpm_web/controllers/api/release_controller.ex
└── lib/hexpm_web/auth_helpers.ex (compile)
lib/hexpm_web/controllers/api/repository_controller.ex
└── lib/hexpm_web/auth_helpers.ex (compile)
lib/hexpm_web/controllers/api/retirement_controller.ex
└── lib/hexpm_web/auth_helpers.ex (compile)
lib/hexpm_web/controllers/blog_controller.ex
└── lib/hexpm_web/views/blog_view.ex (compile)
lib/hexpm_web/endpoint.ex
├── lib/hexpm_web/plug_parser.ex (compile)
└── lib/hexpm_web/session.ex (compile)
There are 14 such files.
Making a change to any of the 132 files (e.g. lib/hexpm/accounts/audit_log.ex) will force a recompilation of these 14 files.
If you think about it, there is no good reason why modifying a schema like the audit log should affect the code of a bunch of web controllers and require them to be recompiled.
This forced recompilaton is easy to verify:
> mix compile
[...]
> echo "# recompile me" >> lib/hexpm/accounts/audit_log.ex
> mix compile --verbose
Compiling 15 files (.ex)
Compiled lib/hexpm/billing/local.ex
Compiled lib/hexpm_web/controllers/blog_controller.ex
Compiled lib/hexpm_web/controllers/api/repository_controller.ex
Compiled lib/hexpm_web/controllers/api/retirement_controller.ex
Compiled lib/hexpm_web/controllers/api/owner_controller.ex
Compiled lib/hexpm_web/controllers/api/package_controller.ex
Compiled lib/hexpm_web/controllers/api/docs_controller.ex
Compiled lib/hexpm_web/controllers/api/organization_user_controller.ex
Compiled lib/hexpm_web/controllers/api/organization_controller.ex
Compiled lib/hexpm_web/controllers/api/key_controller.ex
Compiled lib/hexpm_web/controllers/api/release_controller.ex
Compiled lib/hexpm/billing/hexpm.ex
Compiled lib/hexpm_web/endpoint.ex
Compiled lib/hexpm/emails/bamboo.ex
Compiled lib/hexpm/accounts/audit_log.ex
What we would like to see is "Compiling 1 file (.ex)", the audit_log.
Out first issue stems from lib/hexpm/billing/billing.ex. In this file, you will find a behaviour (the first 10 lines) and code that supplements implementations of that behaviour (everything else).
The behaviour spec is always a compile-time dependency, and the implementations refers to the rest of the app. These must be separated to avoid the issue.
Simply moving the behaviour code to a new lib/hexpm/billing/behaviour.ex and changing the references @behaviour Hexpm.Billing to @behaviour Hexpm.Billing.Behaviour fixes the issue.
Similarly, the behaviour code in lib/hexpm/store/store.ex must be isolated to its own file.
Once this is done, mix xref graph --label compile-connected will no longer list either lib/hexpm/billing/billing.ex nor lib/hexpm/store/store.ex:
$ mix xref graph --label compile-connected
lib/hexpm/emails/bamboo.ex
├── lib/hexpm/accounts/email.ex (compile)
└── lib/hexpm/accounts/user.ex (compile)
lib/hexpm_web/controllers/api/docs_controller.ex
└── lib/hexpm_web/auth_helpers.ex (compile)
lib/hexpm_web/controllers/api/key_controller.ex
└── lib/hexpm_web/auth_helpers.ex (compile)
...
Note that there remains other issues, and so modifying many files (including the adit_log) will still force recompilation of a bunch of files.
Let's skip the issue with lib/hexpm/emails/bamboo.ex, we'll come back to it at the end.
Let's examine the following issue, why the auth_helpers are a compile-time dependencies of some controllers.
Can you spot what is the cause of the compile-time dependency?
It is not obvious, so we can use the new trace command to see which lines introduce compile time dependencies to what:
$ mix xref trace lib/hexpm_web/controllers/api/docs_controller.ex --label compile
lib/hexpm_web/controllers/api/docs_controller.ex:2: call Hexpm.Shared.__using__/1 (compile)
lib/hexpm_web/controllers/api/docs_controller.ex:2: call HexpmWeb.__using__/1 (compile)
lib/hexpm_web/controllers/api/docs_controller.ex:10: call HexpmWeb.AuthHelpers.organization_access/2 (compile)
lib/hexpm_web/controllers/api/docs_controller.ex:10: import HexpmWeb.AuthHelpers.organization_access/2 (compile)
lib/hexpm_web/controllers/api/docs_controller.ex:10: call HexpmWeb.AuthHelpers.organization_billing_active/2 (compile)
lib/hexpm_web/controllers/api/docs_controller.ex:10: import HexpmWeb.AuthHelpers.organization_billing_active/2 (compile)
lib/hexpm_web/controllers/api/docs_controller.ex:18: call HexpmWeb.AuthHelpers.organization_billing_active/2 (compile)
lib/hexpm_web/controllers/api/docs_controller.ex:18: import HexpmWeb.AuthHelpers.organization_billing_active/2 (compile)
lib/hexpm_web/controllers/api/docs_controller.ex:18: call HexpmWeb.AuthHelpers.package_owner/2 (compile)
lib/hexpm_web/controllers/api/docs_controller.ex:18: import HexpmWeb.AuthHelpers.package_owner/2 (compile)
Lines 10 & 18 refer to some functions from AuthHelpers when calling plug, at compile-time.
We can resolve this by adopting the common {module, function_name, argument} style (a.k.a MFA, see Phoenix's live_session/3 for example) instead.
This makes the interface a bit cleaner too, as the functions previously needed to share the given opts.
Did you see the magic trick? This change should not have made a difference, as the module is still used, at compile-time, as part of the options.
That should still introduce a compile-time dependency, the same way as adding IO.inspect(AuthHelpers) would.
But plug is a macro that actually goes through the given options and uses voodoo incantations to avoid adding a compile-time dependencies, just so we can pass modules this way.
Once this is done, we have just a few things remaining:
$ mix xref graph --label compile-connected
[...bamboo, fixed later...]
lib/hexpm_web/controllers/blog_controller.ex
└── lib/hexpm_web/views/blog_view.ex (compile)
lib/hexpm_web/endpoint.ex
├── lib/hexpm_web/plug_parser.ex (compile)
└── lib/hexpm_web/session.ex (compile)
Next issue concerns lib/hexpm_web/views/blog_view.ex. It finds, at compile-time, the blog templates that have been created, and this is also needed by the controller.
This wouldn't be an issue if the view wasn't also referring to the rest of the app. We can simply do the same thing as we did for the behaviour's code: isolate it in it's own file that doesn't refer to the app itself.
After this, we get dependency:
$ mix xref graph --label compile-connected
[...bamboo, fixed later...]
lib/hexpm_web/endpoint.ex
├── lib/hexpm_web/plug_parser.ex (compile)
└── lib/hexpm_web/session.ex (compile)
The issue with lib/hexpm_web/endpoint.ex is that it passes the two modules as arguments to plug here and here.
Wait a minute, didn't we see this awesome voodoo trick that made it possible to pass a module to plug? Well, yes, but it wasn't the same plug, and this one can't run the same voodoo for module plugs in general; it would need a specialized version for Phoenix (PR welcome). Moreover, using a module attribute would make the trick not work.
We see the same problem with the lib/hexpm/emails/bamboo.ex issue we fixed. That file calls defimpl and provides modules as arguments and this introduces compile-time dependencies. It is a known issue.
What makes this complicated is that, because plug / defimpl are macros, they could, if they so desired, use the modules passed to them at compile time. They don't though in these cases and ideally should be careful not to introduce these dependencies.
Until this is fixed, there is an ugly workaround: specifying a module without the compiler knowing. Modules are atoms, so this entails in building the correct atom without using the form that the compiler recognizes.
I know of two (ugly) such ways:
:"Elixir.HexpmWeb.Session"Module.concat(HexpmWeb, Session)
Replacing the references to the modules this way will remove the last transitive dependencies:
$ mix xref graph --label compile-connected
(no output)
You could add to your CI a command that will fail unless this remains the case:
mix xref graph --label compile-connected --fail-above 0
We have seen how to identify problematic dependencies with mix xref graph and mix xref trace
Modifying most of hexpm's files (and probably the ones most subject to changes) forced the useless recompilation of 14 extra files.
Solutions we used to fix these are:
- isolate the code that refers to the app from the code that is needed at compile-time
- use MFA (Module-Function-Argument) instead of function references where allowed
- avoid refering to modules by cheating
These changes fixes all useless recompilations; this can be enforced in the future with --fail-above 0 option
If this turns into a blog post, please let me know, I'd like to promote it in Elixir Radar. :)