Kotlin grouped summaries: mutable seeds and weighted means (AI-assisted, JR / Telemetry)

README.md

Kotlin grouped-summary example

Original article and executable demonstration by JR, creator of Telemetry. Prepared with AI assistance. Fictional data; no account, network request, credential, SDK or hosted integration.

Files: ARTICLE.md, Summaries.kt, test-output.txt.

Run with Kotlin/JVM 2.4.20 and a compatible JDK:

kotlinc Summaries.kt -include-runtime -d summaries.jar
java -jar summaries.jar

Tested locally on Java 15.0.1. The compiler was run with a 384 MiB heap and two active processors; the executable used a 96 MiB heap. Sixteen checks passed. No claim this is production telemetry code or a performance benchmark.

The reducer assumes non-overlapping batches with consistent units and population definitions. It does not provide deduplication, windowing, percentiles or thread safety. Article submission and publication are separate actions recorded outside this package.

Permission: You may copy and adapt the original article and code with attribution to JR and retention of the Telemetry relationship and AI-assistance disclosure. Linked external documentation remains governed by its own terms.

ARTICLE.md

Kotlin grouped summaries: two ways to get a plausible wrong average

By JR, creator of Telemetry, a tool for querying event data with SQL. Prepared with AI assistance; the accompanying Kotlin example was compiled and tested locally. All measurements below are fictional. No service or account is needed.

An average can look reasonable while answering the wrong question. Consider three summaries produced by workers:

Service	Measurements	Total elapsed milliseconds
api	1	1,000
ingest	2	60
api	99	990

For api, there are 100 measurements totaling 1,990 milliseconds. Its mean is 19.9 milliseconds. The ingest mean is 30 milliseconds. A small Kotlin grouping expression can lose both answers before a dashboard ever sees the data.

Mistake one: one mutable seed, several keys

This looks like a compact way to aggregate the rows:

class MutableSummary(var count: Long = 0, var totalMillis: Long = 0)

val result = batches.groupingBy { it.service }
    .fold(MutableSummary()) { accumulator, batch ->
        accumulator.apply {
            count = Math.addExact(count, batch.count)
            totalMillis = Math.addExact(totalMillis, batch.totalMillis)
        }
    }

There is only one MutableSummary() expression. Both keys end up referring to that same object. With the fixture above, each entry reports 102 measurements and 2,050 milliseconds. Checking only that result contains two keys would miss the problem.

The accompanying test makes the aliasing visible:

check(result.getValue("api") === result.getValue("ingest"))

That assertion is supposed to pass for this deliberately broken implementation. It is evidence of the problem, not a correctness check for a production report.

Kotlin documents that the fixed-initial-value overload supplies the same initial value to each group. It also offers an overload that calculates an initial value for each key. Official Grouping API reference.

Fix one: return a new immutable summary

data class Summary(val count: Long = 0, val totalMillis: Long = 0) {
    fun add(batch: Batch) = Summary(
        Math.addExact(count, batch.count),
        Math.addExact(totalMillis, batch.totalMillis)
    )

    fun meanMillis(): Double? =
        if (count == 0L) null else totalMillis.toDouble() / count
}

val result = batches.groupingBy { it.service }
    .fold(Summary()) { accumulator, batch -> accumulator.add(batch) }

Sharing the initial immutable zero value is harmless here. Every update returns a fresh summary; it does not change the seed or another group's current result. This version is easy to inspect, although it allocates an object for each update. The example makes no performance claim about that tradeoff.

Math.addExact is deliberate. A wrapped negative total would still be a valid Long, but it would be an invalid measurement summary. Here, overflow fails explicitly. A real reporting pipeline must decide how to surface that failure rather than silently continuing with corrupted numbers.

Fix two: create one mutable seed per group

If mutation is preferable, use a factory:

val result = batches.groupingBy { it.service }.fold(
    initialValueSelector = { _, _ -> MutableSummary() },
    operation = { _, accumulator, batch ->
        accumulator.apply {
            count = Math.addExact(count, batch.count)
            totalMillis = Math.addExact(totalMillis, batch.totalMillis)
        }
    }
)

The factory starts each group at zero. The operation then adds its first batch, just as it adds subsequent batches. Initializing the seed with the first batch's numbers and then adding that batch again would double-count it. The single-batch test protects that boundary.

These examples process a list sequentially. A separate mutable accumulator per key does not make a map safe for concurrent mutation, and the code is not a streaming-window implementation.

Mistake two: averaging the batch averages

Even after fixing the grouping, this calculation is wrong for a per-measurement mean:

(1000 / 1 + 990 / 99) / 2 = 505 ms

It gives the one-measurement batch as much influence as the 99-measurement batch. Retaining both counts and totals lets us calculate the requested statistic:

(1000 + 990) / (1 + 99) = 19.9 ms

A mean of means can be intentional when the question gives every worker or batch equal weight. Label it that way. It is not interchangeable with the mean duration of all measurements.

Check the data contract too

The runnable example rejects blank service names, negative counts and negative totals. A batch with zero measurements must have a zero total. Its mean is absent, represented by null. Two measured durations totaling zero have a mean of 0.0; they are different data.

The fixture assumes that batches contain disjoint observations, use the same unit and describe the same reporting population. Repeated delivery of a batch violates that contract. This reducer does not deduplicate, align time windows or distinguish failed requests from successful ones. Those decisions belong upstream and should be explicit before totals are combined.

Totals stay in integer milliseconds until the final division. The displayed Double mean is approximate, especially at very large magnitudes. The code does not calculate percentiles: a count and total alone cannot recover a latency distribution.

Run the checks

The companion Summaries.kt uses only Kotlin's standard library and JVM APIs:

kotlinc Summaries.kt -include-runtime -d summaries.jar
java -jar summaries.jar

The local run used Kotlin/JVM 2.4.20 and Java 15.0.1. Its 16 checks cover the shared-reference failure, both fixes, a one-batch group, unequal batch sizes, empty input, zero observations, zero duration, invalid data and arithmetic overflow. They also verify that reordering this fixture and splitting its 99-measurement batch preserve the correct result.

Expected final lines:

16 checks passed
api: count=100, totalMs=1990, meanMs=19.9
ingest: count=2, totalMs=60, meanMs=30.0

Those are useful properties to test before sending summaries to any reporting tool. A dashboard can display the fields it receives perfectly and still display the wrong answer.

Summaries.kt

import kotlin.math.abs

// Fictional worker summaries. Each batch describes disjoint measurements.
data class Batch(val service: String, val count: Long, val totalMillis: Long) {
    init {
        require(service.isNotBlank())
        require(count >= 0 && totalMillis >= 0)
        require(count != 0L || totalMillis == 0L)
    }
}

data class Summary(val count: Long = 0, val totalMillis: Long = 0) {
    fun add(batch: Batch) = Summary(
        Math.addExact(count, batch.count),
        Math.addExact(totalMillis, batch.totalMillis)
    )
    fun meanMillis(): Double? = if (count == 0L) null else totalMillis.toDouble() / count
}

fun summarize(batches: List<Batch>): Map<String, Summary> =
    batches.groupingBy { it.service }.fold(Summary()) { accumulator, batch ->
        accumulator.add(batch)
    }

// Deliberately wrong: the same mutable seed is reused by every group.
class MutableSummary(var count: Long = 0, var totalMillis: Long = 0)
fun wrongSharedSeed(batches: List<Batch>): Map<String, MutableSummary> =
    batches.groupingBy { it.service }.fold(MutableSummary()) { accumulator, batch ->
        accumulator.apply {
            count = Math.addExact(count, batch.count)
            totalMillis = Math.addExact(totalMillis, batch.totalMillis)
        }
    }

// Also valid: create a fresh seed for each key, then mutate only that key's value.
fun summarizeWithFactory(batches: List<Batch>): Map<String, MutableSummary> =
    batches.groupingBy { it.service }.fold(
        initialValueSelector = { _, _ -> MutableSummary() },
        operation = { _, accumulator, batch ->
            accumulator.apply {
                count = Math.addExact(count, batch.count)
                totalMillis = Math.addExact(totalMillis, batch.totalMillis)
            }
        }
    )

fun main() {
    var passed = 0
    fun test(name: String, block: () -> Unit) {
        block(); passed++; println("PASS $name")
    }
    fun expectIllegal(block: () -> Unit) {
        try { block() } catch (_: IllegalArgumentException) { return }
        error("Expected IllegalArgumentException")
    }
    fun expectOverflow(block: () -> Unit) {
        try { block() } catch (_: ArithmeticException) { return }
        error("Expected ArithmeticException")
    }
    val input = listOf(Batch("api", 1, 1000), Batch("ingest", 2, 60), Batch("api", 99, 990))
    test("shared mutable seed demonstrably corrupts groups") {
        val wrong = wrongSharedSeed(input)
        check(wrong.getValue("api") === wrong.getValue("ingest"))
        check(wrong.getValue("api").count == 102L)
        check(wrong.getValue("ingest").totalMillis == 2050L)
    }
    test("immutable groups stay separate") {
        check(summarize(input) == mapOf("api" to Summary(100, 1990), "ingest" to Summary(2, 60)))
    }
    test("weighted mean instead of mean of batch means") {
        val mean = summarize(input).getValue("api").meanMillis()!!
        check(abs(mean - 19.9) < 1e-10)
        val meanOfMeans = (1000.0 + 10.0) / 2
        check(meanOfMeans == 505.0 && mean != meanOfMeans)
    }
    test("factory gives each group a separate mutable value") {
        val result = summarizeWithFactory(input)
        check(result.getValue("api") !== result.getValue("ingest"))
        check(result.mapValues { Summary(it.value.count, it.value.totalMillis) } == summarize(input))
    }
    test("first element is processed once with zero factory seed") {
        val result = summarizeWithFactory(listOf(Batch("api", 3, 12))).getValue("api")
        check(result.count == 3L && result.totalMillis == 12L)
    }
    test("empty input has no invented groups") { check(summarize(emptyList()).isEmpty()) }
    test("empty batch produces null mean") { check(summarize(listOf(Batch("api", 0, 0))).getValue("api").meanMillis() == null) }
    test("zero duration is an observed value") { check(summarize(listOf(Batch("api", 2, 0))).getValue("api").meanMillis() == 0.0) }
    test("negative count rejected") { expectIllegal { Batch("api", -1, 0) } }
    test("negative total rejected") { expectIllegal { Batch("api", 1, -1) } }
    test("positive total with no observations rejected") { expectIllegal { Batch("api", 0, 1) } }
    test("blank key rejected") { expectIllegal { Batch(" ", 1, 1) } }
    test("count overflow is explicit") { expectOverflow { summarize(listOf(Batch("api", Long.MAX_VALUE, 0), Batch("api", 1, 0))) } }
    test("total overflow is explicit") { expectOverflow { summarize(listOf(Batch("api", 1, Long.MAX_VALUE), Batch("api", 1, 1))) } }
    test("all permutations preserve totals") {
        for (a in input.indices) for (b in input.indices) for (c in input.indices) {
            if (setOf(a,b,c).size == input.size) check(summarize(listOf(input[a],input[b],input[c])) == summarize(input))
        }
    }
    test("splitting a disjoint batch preserves the result") {
        val split = listOf(Batch("api", 1, 1000), Batch("ingest", 2, 60), Batch("api", 40, 400), Batch("api", 59, 590))
        check(summarize(split) == summarize(input))
    }
    println("$passed checks passed")
    summarize(input).toSortedMap().forEach { (key, summary) -> println("$key: count=${summary.count}, totalMs=${summary.totalMillis}, meanMs=${summary.meanMillis()}") }
}

test-output.txt

PASS shared mutable seed demonstrably corrupts groups
PASS immutable groups stay separate
PASS weighted mean instead of mean of batch means
PASS factory gives each group a separate mutable value
PASS first element is processed once with zero factory seed
PASS empty input has no invented groups
PASS empty batch produces null mean
PASS zero duration is an observed value
PASS negative count rejected
PASS negative total rejected
PASS positive total with no observations rejected
PASS blank key rejected
PASS count overflow is explicit
PASS total overflow is explicit
PASS all permutations preserve totals
PASS splitting a disjoint batch preserves the result
16 checks passed
api: count=100, totalMs=1990, meanMs=19.9
ingest: count=2, totalMs=60, meanMs=30.0