Polylith migration strategy

notes.clj

;; (1): root/components/logger
(ns com.example.logger.protocol)

(defprotocol Logger
  (log [this level msg data]))


;; (2): root/components/logger
(ns com.example.logger.api)

(defn info [logger msg data]
  ;; Some extra validations here that are always true for the API,
  ;; irrespective of the stateful component implementation
  ;; Also hides the use of protocols from the end-user.
  (protocol/log logger :info msg data))


;; (3): root/components/logger-unilog
(ns com.example.logger.comp.unilog
  (:require [com.example.logger.protocol :as protocol]))

(defrecord UnilogLogger
    ;; 1. stateful component; implements Lifecycle methods
    ;; 2. implements Logger protocol
    )


;; (4): root/components/logger-aws
(ns com.example.logger.comp.aws
  (:require [com.example.logger.protocol :as protocol]))

(defrecord AWSLogger
    ;; 1. stateful component; implements Lifecycle methods
    ;; 2. implements Logger protocol
    )


;; (5): project includes deps for (1), (3), and (4)

(ns com.example.project.handler
  (:require
   [com.example.logger.api :as log]
   [com.example.system.api :as system]))

(deftest example-handler
  (system/with-system [{:keys [:aws-logger :unilog-logger]} (system/start-system! config)]
    (log/info aws-logger "log something to disk" {:foo :bar})
    (log/info unilog-logger "log something else to other system" {:more :data})))


;; Migration strategy questions:
;;
;; (Q1) I expect (2) will become a poly interface `com.example.logger.interface`.
;;
;; (Q2) Can I still use internal protocols for implementing stateful components, like in (1)?
;;
;; (Q2A) If so, where would this file be located to be shared?
;; Perhaps `com.example.logger.interface.protocol` and the prior is `com.example.logger.interface.api` -
;; but no idea if protocol sharing is supported like this?
;;
;; (Q2B) If not, I assume we just replace the protocol methods (ie. `log`) with new functions that exist
;; as interfaces in each of (3) and (4)?
;; When you do want to dispatch on type, protocols are more natural than poly interfaces.
;;
;; (Q3) If I convert (3) and (4) to poly components, then I need to either:
;;
;; (Q3A) use a common interface and `:profiles`,
;; but cannot have both on the classpath and in the project at the same time
;;
;; (Q3B) move them to a "lib" folder and treat them as regular `:local/root` deps,
;; but then I assume they will not participate in the poly dependency logic for checking
;; which code is stable, tests needed to run, etc?
;;
;; (Q3C) some other solution?

Q1: Yes, let it become com.example.logger.interface.
Q2: Yes, you can keep the protocol. You can go for Q2A. Then you either create a sub interface com.example.logger.interface.protocol as you suggest, or you can put it directly in com.example.logger.interface (if it doesn't look too messy). What is best is up to you to decide!
Q3: I would let both implementations live inside the logger component under two separate implementing top namespaces, e.g. com.example.logger.unilog and com.example.logger.aws because it sounds like you want to be able to use both of them within the same project, and the code is not shared elsewhere.

@tengstrand - Thanks for following up!

My only concern with the Q3 approach, is that we've got multiple deployable projects, where the dependencies are mix-and-match. If we put everything under one component, we've got a shared global classpath. This is OKish for development, but not great for production.

Example: :dev project uses AWS + Unilog, one deploy artifact requires only AWS, and a different deploy artifact requires both AWS + Unilog.

I think it helps to explain we deploy production artifacts for different clients, based on the same monorepo and mix-and-match system config that chooses which components get pulled in (and abstracted via the described protocol dispatch). We potentially even have cases where these classpaths could interfere (e.g. one project pulls in the MSSQL component with some JVM deps, while another pulls in MySQL component with other JVM deps).

I think what we could do is a hybrid approach, where:

1. If stateful components don't have any conflicting deps, we put them all under the same component. Just kind of a shame that classpath will pull in unneccessary dependencies in production.
2. If stateful components do have conflicting deps, then each must be separate component (with shared poly interface) and we nudge this way that these component bricks are exclusive-or in a project.

PS. The way we've been resolving this potential issue up to now, is:

1. Each project after it pulls in the deps/components it wants, can run the full suite of tests.
2. Heavily relying on :override-deps to ensure we have same deps, wherever this was feasible (but project artifacts could still override these defaults if that was a necessity).

We've adopted components/<name>/src/<top-ns>.<name>.interface.protocols for any protocols we've migrated into components and then we've pushed any record types down into the implementation -- so the implementation ns can require the protocol ns (it can't require the main interface ns!). Then we have a public "constructor" function in the main interface that constructs the record using impl/->SomeRecord or whatever. Hiding those concrete types is nice because then it "forces" your code to operate on just the interface functions and the protocol functions.

For the multiple logging components, I think I'd be tempted to have each as a separate component (with a unique name) and then have a general logging interface, implemented in various "glue" components that map that generic interface onto each specific logger (or combination of). That lets you mix'n'match logger components as needed without dragging in extra dependencies and still provides a clean separation of the "logging" component from its specific services.

Interesting, I was just going to suggest something very similar! It's good that Sean can validate that this works well in practice also! In your example, you will end up with two implementing components, e.g. unilog-logger and aws-logger, and then we need somewhere to put the generic interface, the defprotocol statement, maybe in a logger component? I haven't tried this myself yet, so I'm not sure.

I was suggesting more components:

• a unilog-logger with its natural interface
• an aws-logger with its natural interface
• a generic logging component -- this is the interface the combo components (below) will implement
• a combo component for each logger or set of loggers you want, which implement the generic logging component interface using the specific logger or loggers.

Each combo component will depend on the natural interface of the one or more loggers that it acts as a bridge for.

Projects can then depend on the combo components. This allows for a mix'n'match of the loggers by introducing a layer between the generic logging interface and the specific logger or loggers, plural.

Then in :dev you can depend on each independent logger (since they all have unique names) and you can have a combo component for logging to some or all of them as needed for the development.