This is on-prem. Both transactor and peer is running datomic pro v1.0.6344.
I am encountering a strange (?) OutOfMemoryError with the following simple query:
(d/query {:query {:find '[(count ?tx) .]
:in '[$ ?start ?end]
:where '[[?tx :tx/user-id _]
[?tx :db/txInstant ?inst]
[(.after ^java.util.Date ?inst ?start)]
[(.before ^java.util.Date ?inst ?end)]]}
:args [(d/db conn)
(-> (LocalDate/now)
(.atStartOfDay (ZoneId/of "Europe/Oslo"))
(.toInstant)
(Date/from))
(-> (LocalDate/now)
(.plusDays 1)
(.atStartOfDay (ZoneId/of "Europe/Oslo"))
(.toInstant)
(Date/from))]})The JVM is launched with -Ddatomic.objectCacheMax=1g.
datomic.process-monitor logs the following:
2021-10-04 20:41:41.869 [Datomic Metrics Reporter] INFO datomic.process-monitor - {:tid 22, :ObjectCacheCount 449, :PeerAcceptNewMsec {:lo 0, :hi 0, :sum 0, :count 23}, :AvailableMB 3550.0, :StorageGetMsec {:lo 159, :hi 2120, :sum 229028, :count 251}, :pid 208439, :event :metrics, :ObjectCache {:lo 0, :hi 1, :sum 157, :count 617}, :MetricsReport {:lo 1, :hi 1, :sum 1, :count 1}, :PeerFulltextBatch {:lo 1, :hi 3, :sum 23, :count 21}, :StorageGetBytes {:lo 25480, :hi 45572, :sum 7851346, :count 251}}
2021-10-04 20:42:41.869 [Datomic Metrics Reporter] INFO datomic.process-monitor - {:tid 22, :ObjectCacheCount 739, :PeerAcceptNewMsec {:lo 0, :hi 0, :sum 0, :count 7}, :AvailableMB 3200.0, :StorageGetMsec {:lo 194, :hi 2300, :sum 336667, :count 355}, :pid 208439, :event :metrics, :ObjectCache {:lo 0, :hi 1, :sum 224, :count 869}, :MetricsReport {:lo 1, :hi 1, :sum 1, :count 1}, :PeerFulltextBatch {:lo 1, :hi 1, :sum 7, :count 7}, :StorageGetBytes {:lo 29590, :hi 36498, :sum 11607418, :count 355}}
2021-10-04 20:43:41.868 [Datomic Metrics Reporter] INFO datomic.process-monitor - {:tid 22, :ObjectCacheCount 1029, :PeerAcceptNewMsec {:lo 0, :hi 0, :sum 0, :count 3}, :AvailableMB 2780.0, :StorageGetMsec {:lo 133, :hi 2310, :sum 358370, :count 357}, :pid 208439, :event :metrics, :ObjectCache {:lo 0, :hi 1, :sum 220, :count 859}, :MetricsReport {:lo 1, :hi 1, :sum 1, :count 1}, :PeerFulltextBatch {:lo 1, :hi 1, :sum 3, :count 3}, :StorageGetBytes {:lo 27508, :hi 34842, :sum 11100905, :count 357}}
2021-10-04 20:44:41.870 [Datomic Metrics Reporter] INFO datomic.process-monitor - {:tid 22, :ObjectCacheCount 1301, :PeerAcceptNewMsec {:lo 0, :hi 0, :sum 0, :count 4}, :AvailableMB 2460.0, :StorageGetMsec {:lo 148, :hi 2790, :sum 331181, :count 339}, :pid 208439, :event :metrics, :ObjectCache {:lo 0, :hi 1, :sum 211, :count 825}, :MetricsReport {:lo 1, :hi 1, :sum 1, :count 1}, :PeerFulltextBatch {:lo 1, :hi 1, :sum 4, :count 4}, :StorageGetBytes {:lo 27494, :hi 35016, :sum 10632358, :count 339}}
2021-10-04 20:45:41.871 [Datomic Metrics Reporter] INFO datomic.process-monitor - {:tid 22, :ObjectCacheCount 1184, :PeerAcceptNewMsec {:lo 0, :hi 1, :sum 1, :count 12}, :AvailableMB 2050.0, :StorageGetMsec {:lo 188, :hi 2170, :sum 338793, :count 359}, :pid 208439, :event :metrics, :ObjectCache {:lo 0, :hi 1, :sum 236, :count 902}, :MetricsReport {:lo 1, :hi 1, :sum 1, :count 1}, :PeerFulltextBatch {:lo 1, :hi 1, :sum 12, :count 12}, :StorageGetBytes {:lo 27364, :hi 35287, :sum 11468046, :count 359}}
2021-10-04 20:46:41.870 [Datomic Metrics Reporter] INFO datomic.process-monitor - {:tid 22, :ObjectCacheCount 1184, :PeerAcceptNewMsec {:lo 0, :hi 0, :sum 0, :count 11}, :AvailableMB 1900.0, :StorageGetMsec {:lo 169, :hi 3670, :sum 350245, :count 315}, :pid 208439, :event :metrics, :ObjectCache {:lo 0, :hi 1, :sum 196, :count 757}, :MetricsReport {:lo 1, :hi 1, :sum 1, :count 1}, :PeerFulltextBatch {:lo 1, :hi 1, :sum 11, :count 11}, :StorageGetBytes {:lo 27741, :hi 37233, :sum 9905707, :count 315}}
2021-10-04 20:47:41.871 [Datomic Metrics Reporter] INFO datomic.process-monitor - {:tid 22, :ObjectCacheCount 1184, :PeerAcceptNewMsec {:lo 0, :hi 0, :sum 0, :count 6}, :AvailableMB 1720.0, :StorageGetMsec {:lo 225, :hi 2650, :sum 342457, :count 333}, :pid 208439, :event :metrics, :ObjectCache {:lo 0, :hi 1, :sum 212, :count 827}, :MetricsReport {:lo 1, :hi 1, :sum 1, :count 1}, :PeerFulltextBatch {:lo 1, :hi 1, :sum 6, :count 6}, :StorageGetBytes {:lo 27806, :hi 36923, :sum 10685119, :count 333}}
2021-10-04 20:48:41.871 [Datomic Metrics Reporter] INFO datomic.process-monitor - {:tid 22, :ObjectCacheCount 1184, :PeerAcceptNewMsec {:lo 0, :hi 0, :sum 0, :count 8}, :AvailableMB 1540.0, :StorageGetMsec {:lo 233, :hi 3160, :sum 331388, :count 304}, :pid 208439, :event :metrics, :ObjectCache {:lo 0, :hi 1, :sum 183, :count 721}, :MetricsReport {:lo 1, :hi 1, :sum 1, :count 1}, :PeerFulltextBatch {:lo 1, :hi 1, :sum 8, :count 8}, :StorageGetBytes {:lo 28340, :hi 45048, :sum 9931362, :count 304}}
2021-10-04 20:49:41.871 [Datomic Metrics Reporter] INFO datomic.process-monitor - {:tid 22, :ObjectCacheCount 1185, :PeerAcceptNewMsec {:lo 0, :hi 0, :sum 0, :count 12}, :AvailableMB 1130.0, :StorageGetMsec {:lo 197, :hi 3410, :sum 329495, :count 313}, :pid 208439, :event :metrics, :ObjectCache {:lo 0, :hi 1, :sum 199, :count 774}, :MetricsReport {:lo 1, :hi 1, :sum 1, :count 1}, :PeerFulltextBatch {:lo 1, :hi 1, :sum 12, :count 12}, :StorageGetBytes {:lo 23657, :hi 39356, :sum 10406591, :count 313}}
2021-10-04 20:50:44.839 [Datomic Metrics Reporter] INFO datomic.process-monitor - {:tid 22, :ObjectCacheCount 1186, :PeerAcceptNewMsec {:lo 0, :hi 0, :sum 0, :count 8}, :AvailableMB 5.12, :StorageGetMsec {:lo 676, :hi 836, :sum 5566, :count 7}, :pid 208439, :event :metrics, :MetricsReport {:lo 1, :hi 1, :sum 1, :count 1}, :PeerFulltextBatch {:lo 1, :hi 1, :sum 8, :count 8}, :StorageGetBytes {:lo 24680, :hi 25023, :sum 173911, :count 7}}
java.lang.OutOfMemoryError: Java heap space
at java.base/java.lang.Integer.valueOf(Integer.java:1050)
at clojure.core$hash.invokeStatic(core.clj:5161)
at clojure.core$hash.invoke(core.clj:5154)
at datomic.db.Datum.hashCode(db.clj:349)
at java.base/java.util.HashMap.hash(HashMap.java:339)
at java.base/java.util.HashMap.put(HashMap.java:607)
at java.base/java.util.HashSet.add(HashSet.java:220)
at datomic.datalog$fn__5457.invokeStatic(datalog.clj:570)
at datomic.datalog$fn__5457.invoke(datalog.clj:342)
at datomic.datalog$fn__5313$G__5287__5328.invoke(datalog.clj:64)
at datomic.datalog$join_project_coll.invokeStatic(datalog.clj:129)
at datomic.datalog$join_project_coll.invoke(datalog.clj:127)
at datomic.datalog$fn__5382.invokeStatic(datalog.clj:232)
at datomic.datalog$fn__5382.invoke(datalog.clj:228)
at datomic.datalog$fn__5292$G__5285__5307.invoke(datalog.clj:64)
at datomic.datalog$eval_clause$fn__5919.invoke(datalog.clj:1385)
at datomic.datalog$eval_clause.invokeStatic(datalog.clj:1380)
at datomic.datalog$eval_clause.invoke(datalog.clj:1346)
at datomic.datalog$eval_rule$fn__5945.invoke(datalog.clj:1466)
at datomic.datalog$eval_rule.invokeStatic(datalog.clj:1451)
at datomic.datalog$eval_rule.invoke(datalog.clj:1430)
at datomic.datalog$eval_query.invokeStatic(datalog.clj:1494)
at datomic.datalog$eval_query.invoke(datalog.clj:1477)
at datomic.datalog$qsqr.invokeStatic(datalog.clj:1583)
at datomic.datalog$qsqr.invoke(datalog.clj:1522)
at datomic.datalog$qsqr.invokeStatic(datalog.clj:1540)
at datomic.datalog$qsqr.invoke(datalog.clj:1522)
at datomic.query$q_STAR_.invokeStatic(query.clj:755)
at datomic.query$q_STAR_.invoke(query.clj:742)
at datomic.query$query_STAR_.invokeStatic(query.clj:783)
at datomic.query$query_STAR_.invoke(query.clj:776)
at datomic.query$query.invokeStatic(query.clj:803)
As can be seen from the logs, the ObjectCacheCount stays at ~1200, but available memory keeps going down. Why is that?