how-to-leak.md

If you take a look at the dump here: https://dl.dropboxusercontent.com/u/20288797/temp/memory-dump.tar.bz2

And that you look at the classes, you will see several copies of the class org.codenarc.rule.Violationororg.codehaus.groovy.antlr.parser.GroovyLexer`. The multiple copies come from different classloaders, which is fine. What is not is that those classes are not retained anymore, nor is their classloader (they are all softly or weakly reachable), but they are not garbage collected, leading to the leak.

To reproduce the PermGen leak, first make sure you use JDK 7 or JDK 6, but the two should be tested independently because Groovy doesn't use the same way to store global class information (ClassValue since JDK 7).

1. Checkout this branch: https://github.com/melix/gradle/tree/cc-oom-codenarc
2. Compile a local version of Gradle that includes the changes in that branch: ./gradlew clean install -Pgradle_installPath=/path/to/local/gradle/build
3. Execute the failing integration test using that local version of Gradle JAVA_HOME=/opt/jdk1.7.0_75 /path/to/local/gradle/build/bin/gradle reporting:intTest --tests org.gradle.api.reporting.plugins.BuildDashboardPluginIntegrationTest

It will fail with an OutOfMemoryError (PermGen space).

The test has been manually updated to fail early, but it happens without modifications of the code base too. It seems to be related to CodeNarc (problem shows up starting from CodeNarc 0.23) and Gradle 2.3+. It is going worse starting from Groovy 2.4.4 compared to Groovy 2.3.10, but we already rules Groovy out of the game, because the OOM occurs even with 2.3.10. I have made some changes to org.gradle.api.internal.project.DefaultIsolatedAntBuilder which is where we create multiple classloaders which have different goals, but the idea is to have a separate classloader for each Ant execution task with a given classpath. Those classloaders are supposed to be cached, but I disabled caching temporarily to put this out of the game.

I am afraid your tool lied to you. In your dump I have searched for org.codenarc.rule.Violation class. It is loaded by 3 class loaders. And every one of them is strongly reachable from GC roots via multiple paths. Here is the example:

ClassValue is handled by the JVM as a SoftReference, its meant to store data on a Class, a bit like ThreadLocal. So even if it is referenced, it's not a problem for the GC. I was wondering the same as you, and I tried to clear all class value things too, without success.

The leak can actually be reproduced with a much simpler example (update the path to Groovy accordingly):

package doNotCommit;

import java.io.File;
import java.net.URL;
import java.net.URLClassLoader;

public class PermGenLeak {

    public static final String GROOVY_JAR = "/home/cchampeau/.gvm/groovy/2.4.4/lib/groovy-2.4.4.jar";

    public static void main(String[] args) throws Exception {
        int i = 0;
        try {
            while (true) {
                i++;
                URLClassLoader loader = new URLClassLoader(
                    new URL[] { new File(GROOVY_JAR).toURI().toURL() },
                    ClassLoader.getSystemClassLoader().getParent());
                Class system = loader.loadClass("groovy.lang.GroovySystem");
                system.getDeclaredMethod("getMetaClassRegistry").invoke(null);
                loader.close();
                System.gc();
                System.out.println("That's your chance to take a heap dump!");
                Thread.sleep(4000);
            }
        } catch (OutOfMemoryError e) {
            System.err.println("Failed after " + i + " loadings");
        }
    }
}

I also used a lot of trickery to try to clear class values/thread locals/caches, without success:

package doNotCommit;

import java.beans.Introspector;
import java.io.File;
import java.lang.reflect.Field;
import java.lang.reflect.Method;
import java.net.URL;
import java.net.URLClassLoader;
import java.util.Iterator;

public class PermGenLeak {

    public static final String GROOVY_JAR = "/home/cchampeau/.gvm/groovy/2.4.4/lib/groovy-2.4.4.jar";

    static void removeClassFromGlobalClassSet(Class<?> classInfoClass) throws Exception {
        Field globalClassValueField = classInfoClass.getDeclaredField("globalClassValue");
        globalClassValueField.setAccessible(true);
        Object globalClassValue = globalClassValueField.get(null);
        Method removeFromGlobalClassValue = globalClassValueField.getType().getDeclaredMethod("remove", Class.class);
        removeFromGlobalClassValue.setAccessible(true);

        Field globalClassSetField = classInfoClass.getDeclaredField("globalClassSet");
        globalClassSetField.setAccessible(true);
        Object globalClassSet = globalClassSetField.get(null);
        globalClassSetField = globalClassSet.getClass().getDeclaredField("items");
        globalClassSetField.setAccessible(true);
        Object globalClassSetItems = globalClassSetField.get(globalClassSet);
        Class<?> metaClassClass = classInfoClass.getClassLoader().loadClass("groovy.lang.MetaClass");
        Method setStrongMetaClass = classInfoClass.getDeclaredMethod("setStrongMetaClass", metaClassClass);

        Field cachedClassRefField = classInfoClass.getDeclaredField("cachedClassRef");
        cachedClassRefField.setAccessible(true);
        Method clearcachedClassRefMethod = cachedClassRefField.getType().getMethod("clear");
        clearcachedClassRefMethod.setAccessible(true);

        Field artifactClassLoaderField = classInfoClass.getDeclaredField("artifactClassLoader");
        artifactClassLoaderField.setAccessible(true);
        Method artifactClassLoaderMethod = cachedClassRefField.getType().getMethod("clear");
        artifactClassLoaderMethod.setAccessible(true);

        Field clazzField = classInfoClass.getDeclaredField("klazz");
        clazzField.setAccessible(true);


        Iterator it = (Iterator) globalClassSetItems.getClass().getDeclaredMethod("iterator").invoke(globalClassSetItems);

        while (it.hasNext()) {
            Object classInfo = it.next();
            Object clazz = clazzField.get(classInfo);
            System.out.println("clazz = " + clazz);
            setStrongMetaClass.invoke(classInfo, new Object[] { null });
            clearcachedClassRefMethod.invoke(cachedClassRefField.get(classInfo));
            artifactClassLoaderMethod.invoke(artifactClassLoaderField.get(classInfo));
            removeFromGlobalClassValue.invoke(globalClassValue, clazz);
            it.remove();
        }
    }

    public static void main(String[] args) throws Exception {
        int i = 0;
        try {
            while (true) {
                i++;
                loadGroovy();
                System.gc();
                //System.out.println("That's your chance to take a heap dump!");
                //Thread.sleep(5000);
            }
        } catch (OutOfMemoryError e) {
            System.err.println("Failed after " + i + " loadings");
        }
    }

    private static void loadGroovy() throws Exception {
        URLClassLoader loader = new URLClassLoader(
            new URL[]{new File(GROOVY_JAR).toURI().toURL()},
            ClassLoader.getSystemClassLoader().getParent());
        Class system = loader.loadClass("groovy.lang.GroovySystem");
        system.getDeclaredMethod("getMetaClassRegistry").invoke(null);
        system.getDeclaredMethod("stopThreadedReferenceManager").invoke(null);
        removeClassFromGlobalClassSet(loader.loadClass("org.codehaus.groovy.reflection.ClassInfo"));

        // clear org.codehaus.groovy.runtime.GroovyCategorySupport
        Class support = loader.loadClass("org.codehaus.groovy.runtime.GroovyCategorySupport");
        Field threadLocal = support.getDeclaredField("THREAD_INFO");
        threadLocal.setAccessible(true);
        Method remove = ThreadLocal.class.getDeclaredMethod("remove");
        remove.invoke(threadLocal.get(null));

        Field declaredMethodCache = Introspector.class.getDeclaredField("declaredMethodCache");
        declaredMethodCache.setAccessible(true);
        Object cache = declaredMethodCache.get(null);
        cache.getClass().getDeclaredMethod("clear").invoke(cache);

        Class<?> threadGroupContextClass = Class.forName("java.beans.ThreadGroupContext");
        Method getContextMethod = threadGroupContextClass.getDeclaredMethod("getContext");
        getContextMethod.setAccessible(true);

        Method clearBeanInfoCacheMethod = threadGroupContextClass.getDeclaredMethod("clearBeanInfoCache");
        clearBeanInfoCacheMethod.setAccessible(true);
        clearBeanInfoCacheMethod.invoke(getContextMethod.invoke(null));

        loader.close();
    }
}

Actually I may have found a workaround. This version works properly, the Groovy classes get unloaded!

package doNotCommit;

import java.io.File;
import java.lang.reflect.Field;
import java.lang.reflect.Method;
import java.net.URL;
import java.net.URLClassLoader;
import java.util.Iterator;

public class PermGenLeak {

    public static final String GROOVY_JAR = "/home/cchampeau/.gvm/groovy/2.4.4/lib/groovy-2.4.4.jar";

    static void removeClassFromGlobalClassSet(Class<?> classInfoClass) throws Exception {
        Field globalClassValueField = classInfoClass.getDeclaredField("globalClassValue");
        globalClassValueField.setAccessible(true);
        Object globalClassValue = globalClassValueField.get(null);
        Method removeFromGlobalClassValue = globalClassValueField.getType().getDeclaredMethod("remove", Class.class);
        removeFromGlobalClassValue.setAccessible(true);

        Field globalClassSetField = classInfoClass.getDeclaredField("globalClassSet");
        globalClassSetField.setAccessible(true);
        Object globalClassSet = globalClassSetField.get(null);
        globalClassSetField = globalClassSet.getClass().getDeclaredField("items");
        globalClassSetField.setAccessible(true);
        Object globalClassSetItems = globalClassSetField.get(globalClassSet);

        Field clazzField = classInfoClass.getDeclaredField("klazz");
        clazzField.setAccessible(true);


        Iterator it = (Iterator) globalClassSetItems.getClass().getDeclaredMethod("iterator").invoke(globalClassSetItems);

        while (it.hasNext()) {
            Object classInfo = it.next();
            Object clazz = clazzField.get(classInfo);
            removeFromGlobalClassValue.invoke(globalClassValue, clazz);
        }

    }

    public static void main(String[] args) throws Exception {
        int i = 0;
        try {
            while (true) {
                i++;
                loadGroovy();
            }
        } catch (OutOfMemoryError e) {
            System.err.println("Failed after " + i + " loadings");
        }
    }

    private static void loadGroovy() throws Exception {
        URLClassLoader loader = new URLClassLoader(
            new URL[]{new File(GROOVY_JAR).toURI().toURL()},
            ClassLoader.getSystemClassLoader().getParent());
        Class system = loader.loadClass("groovy.lang.GroovySystem");
        system.getDeclaredMethod("getMetaClassRegistry").invoke(null);
        system.getDeclaredMethod("stopThreadedReferenceManager").invoke(null);
        removeClassFromGlobalClassSet(loader.loadClass("org.codehaus.groovy.reflection.ClassInfo"));
        loader.close();
    }
}