Magento has recently merged an architecture proposal with the goal of removing non-composer modules.
At first this seems like a good idea, but I think there are some problems.
Magento listens to the community. That is a good thing.
However, the loudest voices in the developer ecosystem are extension vendors.
Both Fooman and Yireo
have blogged about why module development in app/code is "wrong" or "bad".
I love Christof and Jisse - this is not a personal attack, they are doing outstanding work. But they fall pray to our usual human bias.
They are heavily involved in their extension businesses, while I've mainly worked as a freelancer in merchant developer teams over the last three years.
They fail to see the perspecive of non-extension vendor developers.
There are no evangelizing blog posts by merchant developers that I know of. In general, merchant developers seem a lot
quieter and not as involved in the community.
The set of constraints a merchant developer is working under is very different from the set of constraints of an extension vendor or an SI developer.
I feel that Magento is missing that perspective.
Composer is a natural fit for an extension vendor, and much has been written on the benefits of using composer to manage modules. I'll focus on the merchant developer point of view in this post.
For a merchant developer, using composer to manage modules dependencies has zero benefits and is costly.
Let's have a look why I say such a preposterous thing.
The big difference is that for a merchant developer, custom modules will only ever be installed in one Magento instance.
The constraints for a developer working for a merchant are:
- Modules are developed together, not stand-alone
- Modules will only ever be installed on one Magento instance
- Modules will be upgraded together with Magento
These differences negate all the benefits of working with custom modules as individual packages installed by composer.
So, what are the benefits of listing composer dependencies on individual modules, and why don't they apply to development in a merchant team context?
- Installing dependencies, i.e. finding compatible Magento module versions.
The dependency will always be the current Magento instance. The modules will never be installed on different Magento versions. Dependencies on custom modules are already resolved because they are right there, inapp/code, too. - Module conflicts don't happen, as all merchant modules are developed together.
For agency developers it depends on their business model and client structure.
It can make sense to follow an approach
where modules are developed individually, or it can make sense from a cost and development perspective to keep all custom modules together in app/code.
Using composer to manage module dependencies is costly, firstly in time, which as we all know translates directly to money:
-
Tracking dependencies on Magento.
Every time I write code that uses a part of the Magento platform I have to stop and think if I already have the appropriate dependency in the modulecomposer.json.
And even if that module or framework dependency is already listed, I have to ask myself if the code I introduced causes a change in the dependency version constraint.
This breaks my flow and forces me to spend time checking the module service contracts and Magento rules for module version dependencies. -
Keeping dependencies between custom merchant modules in sync.
When I change a module, I have to make a release so composer will see the update. That means I have to keep track of the type of changes I make and how other custom module use my module, and upgrade those dependencies, accordingly. -
More moving parts, more can go wrong.
Having to track all those additional dependencies increases the risk of getting into dependency hell (see below). -
Build time
The more packages in the dependency graph, the longer composer takes to create a build. This might not be a lot for individual builds, but it sure adds up over time. -
Creating new modules takes longer and is more complex.
Instead of just running a scaffolding tool or creating the files manually inapp/codeand be done, I have to create a stub modulecomposer.json, add a path repository configuration to the rootcomposer.json, install the module with composer, exclude the new folder under vendor from being tracked in PHPStorm, and then I can finally start writing some code. Again, this isn't too bad once, but it does add up over time a lot.
I have yet to meet a developer who knows all of Magento's module dependency rules. In addition to those rules, there also are the rules how versions must change according to code base changes. And finally there is also the modularity section in the technical guidelines.
If you haven't read these pages, go, read them now. Then continue to read this post.
The rules are too complicated to keep in mind while focused on solving a business problem.
During trainings, I've found it is hard enough for developers to wrap their head around when to use require and when to use suggest.
The sad thing is, if I follow those rules and specify all individual module dependencies accordingly, I can't even run unit tests in CI after a composer install, let alone integration tests...
The rules are counter-intuitive and hard to apply. Most of the time my modules end up with mostly patch level dependencies pretty quickly.
Most of the outspoken people in the PHP developer ecosystem argue that a deep understanding of composer and semver are a basic requirement nowadays. But that isn't my experience. Most PHP developers I meet aren't that deeply familiar with composer. Many have to google basic comands every time they use them.
Forcing Magento developers to memorize and apply all those rules, or maybe even only a subset of those rules, makes it even harder to find PHP developers that already "know" Magento.
Arguing that composer is a standard and as such it's easy to find talent is simply not true, if pages and pages of rules have to be specified in addition to "we use composer".
Let's think about the steps to upgrade Magento.
Regarding composer, we run a variation of this, where the exact version and meta package might differ:
composer require magento/product-community-edition=2.3.1 --no-update
composer update
bin/magento setup:upgrade
This will make composer find all the versions of all installed packages that are compatible with the meta-package and install them.
And if there is no compatible module, the upgrade will be aborted.
This is great from an extension developers point of view.
But from merchant developers point of view, this is the start of a world of pain if module dependencies are tracked with composer.
In order to do a test upgrade, I have to run the above command and analyze the output of composer to figure out which of my modules
have incompatible version constraints. Then I have to go into each one, change the constraints so Magento can be upgraded.
I have to do that for all custom modules at once.
And then I can check if the version conflict actually broke a custom module or if the change doesn't really affect the custom module.
Just going through the composer output and changing module dependencies so Magento can be upgraded takes hours on a large instance.
If a version constraint prohibits Magento from being updated, that should indicate the package in question is not compatible.
However, most of the time, that module actually still is compatible.
Code changes in a package that cause an incompatible version bump don't mean the package actually is incompatible!
This is the major downfall of semver.
What upgrading Magento boils down to, is that we always have to test if custom modules are compatible with an upgrade.
It simply is impossible to express runtime code dependencies in semver.
As a merchant developer, there is no net benefit in composer aborting an upgrade if a version constraint in a custom module is incompatible.
The same amount of testing is required to see if the new version of Magento really is compatible or not, with or without that dependency declaration.
The base problem is that from a merchant developer perspective, all custom modules implicitly depend on the meta package version, and not individual packages.
This is an implicit dependency, since we are talking about Magento modules. They will only ever be used in the context of Magento.
So there is no benefit in specifying even the meta package version dependency in custom modules.
Without composer dependencies listed in custom modules, doing a test upgrade is a lot simpler (and much much faster).
Simply running the composer command to install the new Magento version enables a merchant developer to check for broken modules, without having to spend hours updating version constraints to get to that point.
Another argument that is often used against keeping modules in app/code is that of non-Magento dependencies.
When a custom module depends on a non-Magento package - for example something from packagist - composer won't see that dependency in the modules composer.json file and it won't be installed.
Luckily this isn't a problem thanks to the great composer merge plugin.
Nuff said, let's move on.
Since I've used this term, I would like to briefly specify what I mean.
In terms of composer, dependency hell is when I want to install or update a package, but composer refuses to do that because of conflicting version constraints.
Usually it is triggered by two or more packages requiring incompatible versions of another package.
If these are direct dependencies, it's bad enough, but if they are version conflicts in transitive dependencies, it gets really ugly.
This can lead to hours of yak shaving, where fixing one version constraint only leads to the next conflict.
Anton Krill argues:
My understanding is that if we solve the performance problem, it will reduce complexity:
- less framework code to maintain
- less things for a new Magento developer to learn
- less boilerplate code
I disagree. It is easier to use module.xml and registration.php than to apply Magento's module dependency rules and semver.
For an extension vendor and some SIs it may be worth the cost, but for merchant developer teams it most certainly isn't.
Taking away non-composer modules will increase development cost for merchants and make it harder to hire Magento developers.
@Vinai @tmotyl It is nice to discuss some practices on the local composer repos bit. Indeed a
composer.lockgit conflict could still occur and referencing more modules in the lock file increases this risk. However, I would only recommend using the local composer repos with version@dev(and not specific versions) because a local path doesn't have git tags or multiple version paths or something. Because of this, you never need to runcomposer update foobar/my-local-path-module. And with this approach, I've not heard of any git conflicts.@Vinai The point on many project modules needing to over-maintained when using composer, landed with me. I'm convinced in that case, the
app/codeapproach is much easier.