There is quite the thread on clojureverse where a lot of this is hashed out (both advocates and critics).
I think the historical motivation is well captured in Alex Miller's talk A Trick of the Tool. It primarily focuses on the emergence of tools.build.
From the QA section "The Beast of Obstacle":
"We really spent a long time chasing a declarative way to define builds, and the attributes that you use in your builds, and get that into declarative data, and deps.edn...we came at it from several different directions, I build things that worked several different ways, and we eventually threw it all in the bin...and decided that...it's very tempting to take [the] common build scenario like compile stuff, make a jar, that's it, where there's only like 3 variable things in there, and I can pull them out and put them in attributes like declaratively drive some set of tasks...It just falls down immediately as soon as you need to do anything special...in my experience, any project that lives for more than 6 months does something special...it was just brittle...there are so many problems that can be solved with writing just one line of clojure to format a string instead of building some declarative task infrastructure that formats strings."
I think there was a fundamental impedance mismatch with what Cognitect was doing (seemed correlated with Datomic development and use) and what extant tooling like lein (and later boot) provided.
-
Extending lein with custom plugins and behavior e.g. to script some functionality into a new task was perhaps more than trivial.
-
The primacy of maven artifacts (e.g. jar files) as opposed to source-based dependencies (far more typical....just a bunch of clojure source files that are evaluated at runtime...). Elevating source-based dependencies (e.g. a git repo using a commit SHA as its coordinate) to first class dependencies.
-
The conflation of tasks available with lein's declarative veneer: lein can (still) do very many things beyond dependency resolution and setting up the classpath. The universe of plugins enable it to do more. Some would argue (via creating alternate tooling like boot [defunct] and now tools.deps/clojure cli/tools.build etc.) that it already did "too much" to be simple and composable.
In my experience, I think the closer you are to doing custom operations (think managing devops) the more control and flexibility you probably want to manage these things. If you can do all this in the language you wrote in the first place to tackle complex problems [clojure], why not? The prospect is that you strip down the tasks lein was performing in a bundled/coupled fashion, and provide a clojure library that anyone can leverage to write clojure programs to accomplish those tasks. That means you get to leverage clojure to write as simple or as complex of a build process as you'd like. This sounds great. Build complex stuff from orthogonal simple pieces (sounds familiar...).
The tradeoff I have seen (still primarily using lein, have used deps [and far less tools.build] quite a bit) is that all the stuff lein was doing still has to be done somehow.
The dependency management piece is handled via tools.deps and deps.edn files (arguably elegantly and simply), with aliases providing a lot of configuration flexibility.
You get a way to script via the CLI if you want to hook this stuff into a larger ecosystem as well. The workflow can be a bit simpler, in that you don't need to create a project to get a repl...you can just dump source files or even eval expressions pretty trivially from the CLI. (ex. I often just spin up a quick repl these days with clj that pulls in a default set of dependencies, as opposed to jacking in to a project).
Beyond that, you are exposed to tools.build which provides the primitives necessary to compile, bundle, and deploy artifacts (e.g. jar files). So if you are not going the source-based dependency route, and you need to deploy artifacts (e.g. jars, uberjars) then you handle this yourself via tools.build, typically exposed in a colocated build.clj file. More control, but also more required on your part, but it's all written in Clojure.
Since these are primitives, higher order tasks naturally emerge (like setting up a project structure based off a template - e.g. lein templates) which are denoted as "tools". The clj-new tool is an example of a tool that creates new project structure for you. These are akin to lein plugins, with the exception that project dependencies are not on their path, and in theory (I have not written any) they are more independent and accessible via an exposed API.
So the "old" reality via lein: you have project.clj, through which you can configure a great many options; there are community plugins that provide additional commands or tasks or features; the command line structure is akin to git with lein some-command maybe some args etc. It comes with built in commands like new to create and populate projects, including test stubs; jar to build jar files, uberjar to bundle all dependencies as a single jar file with an optional main entrypoint, repl to prepare the classpath and launch a repl, with-profile to load alternate configurations (e.g. dependencies and the like) and perform tasks in that context etc.
The "new": you have tools.deps (leveraged via the clojure CLI). You have a deps.edn file which defines your dependencies and aliases. You invoke either clj or clojure from the command line to get a repl with all the dependencies specified from the deps.edn (and including the local /src folder by default, or any source file locations declared in deps.edn). The CLI has a substantial surface area of options to allow configuration via scripting, to include launching a clojure source file as a "main" without added ceremony.
You control your build process via the tools.build tooling, and a colocated build.clj. Where once there was implicit tooling like depstar, now you have build-clj where you delegate work to a tool, and invoke it with the clojure CLI. Tasks are exposed as functions; tools are exposed as aliases. You invoke tools with the -T command line switch (there are several other switches, some of which the semantics have changed over time as the CLI continues to develop....).
From my perspective, the barrier to entry is - on the whole - a bit higher on the tools.deps route. There is more surface area intentionally exposed in the build process, with some effort to mitigate that with higher abstractions and wrappers emerging. The tradeoff is, as Alex said, the ability to be "special" and trivially deviate from the happy path that a declarative task approach provides.