Micro-benchmarks about indy

Benchmark.java

/*
 * Copyright 2003-2012 the original author or authors.
 *
 * Licensed under the Apache License, Version 2.0 (the "License");
 * you may not use this file except in compliance with the License.
 * You may obtain a copy of the License at
 *
 *     http://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing, software
 * distributed under the License is distributed on an "AS IS" BASIS,
 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 * See the License for the specific language governing permissions and
 * limitations under the License.
 */

import java.lang.invoke.MethodHandle;
import java.lang.invoke.MethodHandles;
import java.lang.invoke.MethodType;
import java.lang.reflect.Method;
import java.util.concurrent.TimeUnit;

public class Benchmark {
    int x = 0;
    private static final MethodHandle CACHED_HANDLE;

    static {
        MethodHandle handle = null;
        try {
            handle = MethodHandles.lookup().findVirtual(Benchmark.class, "foo", MethodType.methodType(Void.TYPE));
        } catch (NoSuchMethodException | IllegalAccessException e) {
            handle = null;
        }
        CACHED_HANDLE = handle;
    }
    private static final int REPEAT = 500_000_000;

    public void foo() {
        x = 2*x+1;
    }

    public static interface Code {
        int call() throws Throwable;
    }

    public static void time(String label, Code code) throws Throwable {
        long start = System.nanoTime();
        int res = code.call();
        long dur = System.nanoTime() - start;
        System.out.println(label +"[" + res + "] " + TimeUnit.MILLISECONDS.convert(dur, TimeUnit.NANOSECONDS) + "ms");
    }

    public static void main(String[] args) throws Throwable {
        time("Classic for loop", () -> {
            Benchmark b = new Benchmark();
            for (int i = 0; i < REPEAT; i++) {
                b.foo();
            }
            return b.x;
        });

        time("InvokeExact", () -> {
            final Benchmark b = new Benchmark();
            for (int i = 0; i < REPEAT; i++) {
                CACHED_HANDLE.invokeExact(b);
            }
            return b.x;
        });

        time("InvokeExact (local method handle)", () -> {
            final Benchmark b = new Benchmark();
            MethodHandle handle = MethodHandles.lookup().findVirtual(Benchmark.class, "foo", MethodType.methodType(Void.TYPE));
            for (int i = 0; i < REPEAT; i++) {
                handle.invokeExact(b);
            }
            return b.x;
        });

        time("bindTo+invokeExact", () -> {
            final Benchmark b = new Benchmark();
            final MethodHandle handle = CACHED_HANDLE.bindTo(b);
            for (int i = 0; i < REPEAT; i++) {
                handle.invokeExact();
            }
            return b.x;
        });

        time("insertArgument+invokeExact", () -> {
            final Benchmark b = new Benchmark();
            final MethodHandle handle = MethodHandles.insertArguments(CACHED_HANDLE, 0, b);
            for (int i = 0; i < REPEAT; i++) {
                handle.invokeExact();
            }
            return b.x;
        });

        time("invoke", () -> {
            final Benchmark b = new Benchmark();
            final MethodHandle handle = CACHED_HANDLE;
            for (int i = 0; i < REPEAT; i++) {
                handle.invoke(b);
            }
            return b.x;
        });

        time("bindTo+invoke", () -> {
            final Benchmark b = new Benchmark();
            final MethodHandle handle = CACHED_HANDLE.bindTo(b);
            for (int i = 0; i < REPEAT; i++) {
                handle.invoke();
            }
            return b.x;
        });

        time("insertArgument+invoke", () -> {
            final Benchmark b = new Benchmark();
            final MethodHandle handle = MethodHandles.insertArguments(CACHED_HANDLE, 0, b);
            for (int i = 0; i < REPEAT; i++) {
                handle.invoke();
            }
            return b.x;
        });

        time("Reflect", () -> {
            final Benchmark b = new Benchmark();
            final Method method = Benchmark.class.getDeclaredMethod("foo");
            for (int i=0; i<REPEAT; i++) {
                method.invoke(b);
            }
            return b.x;
        });

    }
}

I'm a bit surprised by the results. I would have expected invokeExact to be faster and bindTo+invokeExact even faster since the receiver is fixed. But no, results are not as good as I would have expected:

Classif for loop[-1] 333ms InvokeExact[-1] 7430ms bindTo+invokeExact[-1] 12392ms invoke[-1] 142804ms bindTo+invoke[-1] 168037ms

Note that I'm testing with OpenJDK Runtime Environment (build 1.8.0-ea-lambda-nightly-h3419-20130219-b78-b00)

invokeExact just fails with a Java 7 runtime.

You should extract each time call into a method of its own, even though on-stack-replacement happens to work.

Also, invoke() calls have type conversion checks, while invokeExact() does not :-)

The use of lambdas makes the code elegant BTW.

Sure, I expected invoke to be slower. But not invokeExact. I thought it could be inlined. And in theory, bindTo+invokeExact could be faster than regular java if you cache the method address :)

Maybe you could try generating invokedynamic call sites in bytecode, too. The JIT may get it differently.

Even stranger, java.lang.Reflect is faster. There's definitely something wrong here :( BTW, how does writing invokedynamic callsites would help? In the end, I would have to manipulate a method handle, right? Are you suggesting that since the MH would be generated in a bootstrap method, it would be handled differently by the JVM?

Writing things like in revision 4 makes it faster. As suggested by Jochen Theodorou, I made the MethodHandle static, which leads to the same performance as Java for the "invokeExact" scenario. All other scenarios are much slower, even slower than reflection:

Classic for loop[-1] 173ms
InvokeExact[-1] 169ms
InvokeExact (local method handle)[-1] 3711ms
bindTo+invokeExact[-1] 6304ms
insertArgument+invokeExact[-1] 6124ms
invoke[-1] 62966ms
bindTo+invoke[-1] 83338ms
insertArgument+invoke[-1] 83423ms
Reflect[-1] 1226ms

Tested with lambda b83 without more success.

I think the JIT can inline currently only if it finds the method handle reference to be constant (from a static final field), otherwise the invokeExact call site can only dispatch dynamically. But you can build your own constant dispatch trees by using MethodHandles.guardWithTest which are quite fast in my experience.