Various, unsorted notes from using and struggling with Pathom 3 with Fulcro.
When something doesn’t work, I try to simplify it as much as possible. Often it starts working at some point. Then I try to find the exact point where it starts/stop working by bringing the broken and working alternatives closer together. And this is exactly what I did when my auto-generated RAD id-resolver did not return the expected data. It might be useful to know the steps along this path for any future troubleshooting.
This problem has two dimensions - the parser and the resolver. Here I move mostly along the parser dimension, while in the end I discovered the problem was with the resolver. Still, it is a valuable knowledge.
This returned as expected (notice the [::person/addresses] w/o a join):
(parser {} [{[::person/id "ann"] [::person/addresses]} ])
; =>
{[::person/id "ann"] {::person/addresses [{::address/street "First St."}
{::address/street "Second St."}]}}but if I changed it to specify I only wanted …/street for each address:
(parser {} [{[::person/id "ann"] [{::person/addresses [::address/street]}]} ])
; =>
{[::person/id "ann"] #::person{:addresses [{} {}]}}no data was returned in the addresses.
I knew what my resolver was returning from a log, so was able to use that.
(ns p3-to-f-rad
(:require
edn-query-language.core
[com.fulcrologic.rad.pathom3 :as rad.p3]
com.wsscode.pathom3.plugin
[com.wsscode.pathom3.connect.indexes :as pci]
[com.wsscode.pathom3.connect.operation :as pco]
[com.wsscode.pathom3.entity-tree :as entity-tree]
[com.wsscode.pathom3.interface.eql :as p.eql]))
;; 1. WORKS Simples: raw `process`, with the entity in the input
(p.eql/process
(-> {} ; (pci/register some-resolver)
(entity-tree/with-entity
#:person{:id "ann" :addresses [#:address{:id "a-one", :street "First St."} ; <- data from the log
#:address{:id "a-two", :street "Second St."}]}))
[:person/id {:person/addresses [:address/street]}])
;; 2. WORKS RAD's P3 parser uses not process but boundary-interface => try that
;; (it takes a map with :entity and :ast inputs)
((p.eql/boundary-interface {})
{}
#:pathom{:entity #:person{:id "bob" :addresses [#:address{:id "a-one", :street "First St."}
#:address{:id "a-two", :street "Second St."}]}
:ast (edn-query-language.core/query->ast
[:person/id {:person/addresses [:address/street]}])})
;; 3. WORKS - Use the full RAD parser as-is, but without any resolvers or plugins with the exception of
;; a custom plugin that inserts the entity data
(com.wsscode.pathom3.plugin/defplugin
insert-entity
{::p.eql/wrap-process-ast
(fn [process] (fn [env ast] (process
(entity-tree/with-entity env
#:person{:id "ann" :addresses [#:address{:id "a-one", :street "First St."}
#:address{:id "a-two", :street "Second St."}]})
ast)))})
((rad.p3/new-processor nil identity [insert-entity] [#_resolvers])
{} ; env
[:person/id {:person/addresses [:address/street]}])
;; WORKS - 4. Use the full RAD parser as-is w/ all the plugins (though no resolvers) as in the failing test case plus the
;; custom entity insertion plugin
(let [conn (d/connect "asami:mem://test123")]
((rad.p3/new-processor
{}
(-> (attr/wrap-env all-attributes)
(form/wrap-env (asami/wrap-save) (asami/wrap-delete))
(asami/wrap-env (fn [env] {:production conn})))
[insert-entity]
[#_resolvers])
(entity-tree/with-entity {} ; this call is likely useless and should have been just `{}`
#:person{:id "ann" :addresses [#:address{:id "a-one", :street "First St."}
#:address{:id "a-two", :street "Second St."}]})
[:person/id {:person/addresses [:address/street]}]))
;; WORKS - 5. Use the full RAD parser as-is w/ all the plugins AND resolvers + entity insertion plugin
(let [conn (d/connect "asami:mem://test123")]
((com.fulcrologic.rad.pathom3/new-processor
{}
(-> (attr/wrap-env all-attributes)
(form/wrap-env (asami/wrap-save) (asami/wrap-delete))
(asami/wrap-env (fn [env] {:production conn})))
[insert-entity]
[(asami/generate-resolvers all-attributes :production) form/resolvers cities-resolver])
(entity-tree/with-entity {} ; this call is likely useless and should have been just `{}`
#:person{:id "ann" :addresses [#:address{:id "a-one", :street "First St."} #:address{:id "a-two", :street "Second St."}]})
[:person/id {:person/addresses [:address/street]}]))
;; WORKS - 6. Use the full RAD parser as-is w/ all the plugins AND resolvers, this time w/o entity insertion, adding
;; instead an extra resolver returning hadcoded data
;; NOTE: I should have tried this case waaaay earlier, likely right after experiment 1.
(pco/defresolver
fake-person-resolver [in]
{::pco/input [::person/id]
::pco/output [::person/name ::person/addresses]
::pco/batch? true}
(println "JHDBG: fake-person-resolver" in)
[#::person{:id "ann" :full-name "Ann" :addresses [#::address{:id "a-one", :street "First St."} #::address{:id "a-two", :street "Second St."}]}])
(let [conn (d/connect "asami:mem://test123")]
((rad.p3/new-processor
{}
(-> (attr/wrap-env all-attributes)
(form/wrap-env (asami/wrap-save) (asami/wrap-delete))
(asami/wrap-env (fn [env] {:production conn})))
[]
[fake-person-resolver (asami/generate-resolvers all-attributes :production) form/resolvers cities-resolver])
{}
;; NOTE: We must change the query (to be same as in the failing test) to include the entity iden
;; b/c now, for the first time, there is no pre-inserted entity and a resolver must be used
[{[::person/id "ann"] [::person/full-name {::person/addresses [::address/street]}]}]))
;; BROKEN - 7. The extracted failing test code, using the generated id-resolver reading from the DB
(let [; cleanup so we can safely run this repeatedly w/o leftover state:
_ (do (d/delete-database "asami:mem://test123") (swap! d/connections dissoc "asami:mem://test123"))
conn (d/connect "asami:mem://test123")]
@(d/transact conn {:tx-data [{:id [::person/id "ann"]
::person/id "ann"
::person/addresses [{::address/id "a-one"
::address/street "First St."}
{::address/id "a-two"
::address/street "Second St."}]}]})
((rad.p3/new-processor
{}
(-> (attr/wrap-env all-attributes)
(form/wrap-env (asami/wrap-save) (asami/wrap-delete))
(asami/wrap-env (fn [env] {:production conn})))
[]
[#_fake-person-resolver (asami/generate-resolvers all-attributes :production) form/resolvers cities-resolver])
{}
[{[::person/id "ann"] [::person/full-name {::person/addresses [::address/street]}]}]))At this point I am quite sure the problem is in the auto-generated resolver, b/c the same code works if I replace it with a custom one, returning hardcoded data. Strangely if I change the auto-generated resolver to return the very same data, it still does not behave. So there is some difference that is not in the return value…
;; As discovered above, I know that using following resolver leads to the desired parser output:
(pco/defresolver
fake-person-resolver [in]
{::pco/input [::person/id]
::pco/output [::person/name ::person/addresses]
::pco/batch? true}
(println "JHDBG: fake-person-resolver" in)
[#::person{:id "ann" :full-name "Ann" :addresses [#::address{:id "a-one", :street "First St."} #::address{:id "a-two", :street "Second St."}]}])
;; while the auto-generated resolver does not. Let's bring them closer to each other!
;; ✅ Experiment 1 (not shown): print the `::pco/output` of the generated auto-resolver and use that in the fake one; still works
;; Note: ✅ means that the test is passing, i.e. we haven't found where the code breaks yet
;; ✅ Exp.2: Create resolver closer to the autogenerated one in fulcro-rad-asami:
(pco/defresolver fake-person-resolver [in]
{::pco/input [:person/id]
;:com.wsscode.pathom3.connect.operation/output [:person/name :person/addresses]
; output copied from the auto-gen. one:
::pco/output [:person/full-name :person/email :person/nicks #:person{:primary-address [:address/id]} #:person{:addresses [:address/id]} :person/role :person/permissions #:person{:things [:thing/id]} :person/account-balance]
::pco/batch? true}
(println "JHDBG: fake-person-resolver" in)
[#:person{:id "ann" :name "Ann" :addresses [#:address{:id "a-one", :street "First St."} #:address{:id "a-two", :street "Second St."}]}])
;; ✅ Exp.3: Use the actual rad-asami code to generate the resolver, returning hard-coded data
(def asami-p3-resolver
(cz.holyjak.rad.database-adapters.asami.pathom3/make-pathom3-resolver
'asami-p3-resolver :person/id
[:person/full-name :person/email :person/nicks #:person{:primary-address [:address/id]} #:person{:addresses [:address/id]} :person/role :person/permissions #:person{:things [:thing/id]} :person/account-balance]
(constantly [#:person{:id "ann", :addresses [#:address{:id "a-one", :street "First St."} #:address{:id "a-two", :street "Second St."}]}])
nil))
(let [...]
((rad.p3/new-processor
{}
(-> (attr/wrap-env all-attributes)
(form/wrap-env (asami/wrap-save) (asami/wrap-delete))
(asami/wrap-env (fn [env] {:production conn})))
[]
[;(asami/generate-resolvers all-attributes :production)
asami-p3-resolver])
{}
[{[::person/id "ann"] [::person/full-name {::person/addresses [::address/street]}]}]))
;; Exp.3.b Inline what the resolver gen function does:
(def asami-p3-resolver-diy
(pco/resolver
{::pco/op-name 'asami-p3-resolver-diy
::pco/batch? true
::pco/input [:person/id]
::pco/output [:person/full-name :person/email :person/nicks #:person{:primary-address [:address/id]}
#:person{:addresses [:address/id]} :person/role :person/permissions
#:person{:things [:thing/id]} :person/account-balance]
::pco/resolve (constantly [#:person{:id "ann", :addresses [#:address{:id "a-one", :street "First St."}
#:address{:id "a-two", :street "Second St."}]}])}))
;; ❌ Exp.4 (not shown) modify the auto-generated resolver to use the same `(constantly ...)` instead of modifying
;; its resolver-fn to return hardcoded data
;; => the diff really is not in the resolve fn / returned data but somewhere else in the resolver...
;; ✅ Exp.5: Try to use _only_ the relevant auto-generated resolver (which I see from logs is the 1st one):
((rad.p3/new-processor ... [(-> (asami/generate-resolvers all-attributes :production) first)] ...)
;; => the person RESOLVER IS JUST FINE! But there is some other resolver messing this up
;; A short binary search later and I see that removing the attribute for :address/street
;; makes the test pass.
;; BONUS: How to build an `env` that works for the generated resolvers w/ a raw `process`:
(let [key->attribute (medley/index-by ::attr/qualified-key all-attributes)
conn (d/connect "asami:mem://test123")]
@(d/transact conn {:tx-data [...]})
(p.eql/process
(-> (pci/register [(asami/generate-resolvers person/attributes :production)])
((cz.holyjak.rad.database-adapters.asami.pathom-common/wrap-env nil (constantly {:production conn})))
(assoc ::attr/key->attribute key->attribute))
[{[::person/id "ann"] [{::person/addresses [::address/street]}]}]))Note: The Pathom version I’ve used is 2022.10.19-alpha
-
Any of the two following changes will prevent the problem and make the parser return the desired data:
-
Change the person/id resolver’s outputs to declare
[ .. :person/addresses ..]instead of[.. #:person{:addresses [:address/id]} ..] -
Change the person/id resolver’s outputs to also include address/street, i.e.
[.. #:person{:addresses [:address/id :address/street]} ..] -
Remove the auto-generated resolver for address/id → address/street
-
Change the person/id resolver to only return address/id and not the street - the address resolver will then be called to provide the street
-
Here is the demonstration, with the 3 possible changes marked:
(def asami-p3-pers-resolver-diy
(pco/resolver
{::pco/op-name 'asami-p3-pers-resolver-diy
::pco/batch? true
::pco/input [:person/id]
::pco/output [#_:1->:person/addresses
#:person{:addresses [:address/id #_:2->:address/street]}]
::pco/resolve (fn [_env in]
(println "JHDBG: asami-p3-pers-resolver-diy" in)
[#:person{:id "ann", :addresses [#:address{:id "a-one", :street "First St."}
#:address{:id "a-two", :street "Second St."}]}])}))
(def asami-p3-addr-resolver-diy
(pco/resolver
{::pco/op-name 'asami-p3-addr-resolver-diy
::pco/batch? true
::pco/input [:address/id]
::pco/output [:address/street]
::pco/resolve (fn [_env in]
(println "JHDBG: asami-p3-addr-resolver-diy" in) ; never called!
[#:address{:id "a-one", :street "First St."}
#:address{:id "a-two", :street "Second St."}])}))
(p.eql/process
(pci/register [asami-p3-pers-resolver-diy #_:3:comment-out-> asami-p3-addr-resolver-diy])
[{[:person/id "ann"] [{:person/addresses [:address/street]}]}])Yes, indeed there is an overhead to Pathom that might get significant depending on the use case. Pathom is more optimal when dealing with complex data source, but with limited number of entities. also there is a significant difference if your main overhead is dealing with IO or if its computational. if its very IO driven, Pathom can do a good job on paralellizing, and given in those scenarios the IO time tend to dominate the cost, it ends up being a good tradeoff. But when you have large collections that need to handle CPU intensive operations, then Pathom overhead might play a significant percentage of the total cost in the operation.
IME in most cases we can avoid this scenario, but when we cant, there is a scape path in Pathom, that is to use ::pco/final true in the sequence meta. when you do that, Pathom will not process that value at all, and then you have to do it instead, if the process you need is simple enough in each item, that might just work fine