Instantly share code, notes, and snippets.

@melix /
Last active Oct 18, 2018

What would you like to do?
Reproducing the benchmarks of the Maven vs Gradle performance page

Gradle vs Maven performance reproduction instructions

Install the Gradle Profiler

This page explains how to reproduce the Gradle vs Maven performance numbers yourself. For that, you need to install the Gradle profiler, a tool which will automate benchmarking by running several builds in a row, with the appropriate options (it also runs Maven builds).

Generate the test projects

Our performance comparison uses 4 test projects:

  • a large, monolithic app
    • single project
    • 50000 source files and 50000 test source files
  • a medium sized, multi-project build
    • 100 subprojects
    • each subproject has 100 source files and 100 test source files
  • a large, multi-project build
    • 500 subprojects
    • each subproject has 100 source files and 100 test source files
  • Apache Commons Lang

The first 3 can be found in the Gradle repository. To generate a test project, checkout the Gradle sources, then execute:

  • ./gradlew largeMonolithicJavaProject for the large monolithic build
  • ./gradlew mediumJavaMultiProject for the medium multiproject build
  • ./gradlew largeJavaMultiProject for the large multiproject build

The test projects will then be found in subprojects/performance/build/<test project name>.

The last project is a real-world project (Apache Commons Lang that needs to be setup separately:

git clone commons-lang --branch commons-lang --depth 1
cd commons-lang

This will clone the repository which is a copy of the original Commons Lang repository, but containing our converted to Gradle build and the scenario file.

Running a performance test

Make sure you have installed the Gradle Profiler and that it's available on your PATH.

Measuring the performance of Gradle

Change directory to a test project and run:

gradle-profiler --gradle-version 4.0 --benchmark --scenario-file performance.scenarios --project-dir .

This will execute all scenarios by default, which can take a significant amount of time. If you want to run one or more specific scenarios, just add them explicitly to the command line:

gradle-profiler --gradle-version 4.0 --benchmark --scenario-file performance.scenarios --project-dir . <scenario1> <scenario2> ...

Measuring the performance of Apache Maven

First make sure you have properly set the MAVEN_HOME and MAVEN_OPTS environment variables:

export MAVEN_HOME=/path/to/apache-maven-3.5.0
export MAVEN_OPTS="-Xms2g -Xmx2g"

Then execute:

gradle-profiler --benchmark --scenario-file performance.scenarios --project-dir . --maven --warmups 2 --iterations 2

Note: For the large monolithic case, increase Maven memory to 3g instead of 2g, otherwise it won't run. Gradle will use 2g for this scenario.

You will notice that we only use 2 warmups and runs for Maven. There are multiple reasons for this:

  • Maven doesn't have a daemon, so doesn't benefit from subsequent runs of the same build
  • Maven is significantly slower. A test that would take minutes to complete with Gradle would take several hours with Maven

Note about the scenarios for Maven

You will notice that for Apache Maven, we run clean compile instead of compile. It may, at first glance, look like an unfair comparison, but it's for an important reason: Maven's incremental compiler is completely broken and will miss changes. An example of serious bug can be found here.


This comment has been minimized.

Copy link

ph-redknee commented Oct 18, 2018

@melix what about the result?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment