Skip to content

Instantly share code, notes, and snippets.

@ZacSweers
Last active January 31, 2023 00:24
Show Gist options
  • Star 2 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save ZacSweers/5884834475f90a2e5037f458bd97373d to your computer and use it in GitHub Desktop.
Save ZacSweers/5884834475f90a2e5037f458bd97373d to your computer and use it in GitHub Desktop.
MergeFileTask Patch
buildscript {
dependencies {
classpath("com.slack.gradle:sgp-monkeypatch-agp:0.5.3") // <-- Before AGP!
classpath("<android gradle plugin coordinates>")
}
}
// Then add this to gradle.properties (or however you want to set your system props)
// systemProp.com.slack.sgp.sort-merge-files=true

The Problem

While doing some remote build cache fidelity testing in #66841, I found that our lint analysis tasks were consistently getting cache misses due to an esoteric intermediate proguard file contents change.

After a little digging, I found these files were generated by AGP as a "merged" file of all that project's generated proguard files. This was most notable in projects using Moshi, which generates proguard rules on the fly.

My hunch was that these files were being merged with non-deterministic order, as I couldn't find anything that ensured ordering and this input was purely a file contents check (so the same rules in different order would still constitute a miss).

I was able to verify this was the case via snagging merged proguard.txt files via github actions artifact uploads.

I filed this issue for it here: issuetracker.google.com/issues/266403349 (note it's been made private for some reason).

The Fix

To "fix" this now without waiting for a new AGP release, and to verify that sorting them would indeed fix the cache misses, I monkeypatched the fix via some Gradle classpath trickery. We've done this before to fix memory leaks in KGP, but TL;DR is that if we create the exact same class signature and put it on the buildscript classpath before AGP itself, Gradle will use our patched version instead and AGP will be none-the-wiser.

The patch fixes were made in these commits while testing (and gated by a system property).

Results

Before the patch, running both the first and second job in the original PR would both take ~16min to run (that's compilations + lint + some other analysis checks). After the patch, the second job only takes ~5min to run (!!).

The number of cacheable tasks with differing inputs went from hundreds to just 14 (!!).

This should fix lint caching issues and give us a nice win on CI, though not local dev (no one runs lint locally) or TTM (lint is no longer the long tail).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment