Skip to content

Instantly share code, notes, and snippets.

@samrocketman
Last active November 24, 2019 06:47
Show Gist options
  • Save samrocketman/332ca683835e59499f8d5082dd277510 to your computer and use it in GitHub Desktop.
Save samrocketman/332ca683835e59499f8d5082dd277510 to your computer and use it in GitHub Desktop.

Matrix building in scripted pipeline

In my project, Jervis, it supports Matrix building via YAML using Jenkins scripted pipeline. However, it is quite easy to calculate matrix building using the cartesian product on a list of lists containing maps (the matrix axes).

For user pipelines the following script approvals must be allowed:

staticMethod org.codehaus.groovy.runtime.DefaultGroovyMethods combinations java.util.Collection

The following Jenkins scripted pipeline will build combinations across two matrix axes. However, adding more axes to the matrix is just as easy as adding another entry to the Map matrix_axes.

// you can add more axes and this will still work
Map matrix_axes = [
    PLATFORM: ['linux', 'windows', 'mac'],
    BROWSER: ['firefox', 'chrome', 'safari', 'edge']
]

@NonCPS
List getMatrixAxes(Map matrix_axes) {
    List axes = []
    matrix_axes.each { axis, values ->
        List axisList = []
        values.each { value ->
            axisList << [(axis): value]
        }
        axes << axisList
    }
    // calculate cartesian product
    axes.combinations()*.sum()
}

// filter the matrix axes since
// Safari is not available on Linux and
// Edge is only available on Windows
List axes = getMatrixAxes(matrix_axes).findAll { axis ->
    !(axis['BROWSER'] == 'safari' && axis['PLATFORM'] == 'linux') &&
    !(axis['BROWSER'] == 'edge' && axis['PLATFORM'] != 'windows')
}

// parallel task map
Map tasks = [failFast: false]

for(int i = 0; i < axes.size(); i++) {
    // convert the Axis into valid values for withEnv step
    Map axis = axes[i]
    List axisEnv = axis.collect { k, v ->
        "${k}=${v}"
    }
    // let's say you have diverse agents among Windows, Mac and Linux all of which have proper labels for their platform and what browsers are available on those agents.
    String nodeLabel = "os:${axis['PLATFORM']} && browser:${axis['BROWSER']}"
    tasks[axisEnv.join(', ')] = { ->
        node(nodeLabel) {
            withEnv(axisEnv) {
                stage("Build") {
                    echo nodeLabel
                    sh 'echo Do Build for ${PLATFORM} - ${BROWSER}'
                }
                stage("Test") {
                    echo nodeLabel
                    sh 'echo Do Build for ${PLATFORM} - ${BROWSER}'
                }
            }
        }
    }
}

stage("Matrix builds") {
    parallel(tasks)
}

Matrix axes contain the following combinations:

[PLATFORM=linux, BROWSER=firefox]
[PLATFORM=windows, BROWSER=firefox]
[PLATFORM=mac, BROWSER=firefox]
[PLATFORM=linux, BROWSER=chrome]
[PLATFORM=windows, BROWSER=chrome]
[PLATFORM=mac, BROWSER=chrome]
[PLATFORM=windows, BROWSER=safari]
[PLATFORM=mac, BROWSER=safari]
[PLATFORM=windows, BROWSER=edge]

Screenshot

Screenshot from 2019-11-24 00-32-18

Adding static choices

Adding choices requires only a few changes to the above script:

Map response = [:]
stage("Choose combinations") {
    response = input(
        id: 'Platform',
        message: 'Customize your matrix build.',
        parameters: [
            choice(
                choices: ['all', 'linux', 'mac', 'windows'],
                description: 'Choose a single platform or all platforms to run tests.',
                name: 'PLATFORM'),
            choice(
                choices: ['all', 'chrome', 'edge', 'firefox', 'safari'],
                description: 'Choose a single browser or all browsers to run tests.',
                name: 'BROWSER')
        ])
}

// filter the matrix axes since
// Safari is not available on Linux and
// Edge is only available on Windows
List axes = getMatrixAxes(matrix_axes).findAll { axis ->
    (response['PLATFORM'] == 'all' || response['PLATFORM'] == axis['PLATFORM']) &&
    (response['BROWSER'] == 'all' || response['BROWSER'] == axis['BROWSER']) &&
    !(axis['BROWSER'] == 'safari' && axis['PLATFORM'] == 'linux') &&
    !(axis['BROWSER'] == 'edge' && axis['PLATFORM'] != 'windows')
}

Which renders the following choice dialog

Screenshot from 2019-11-24 01-14-29

And can customize the matix axes filtered when selected.

Screenshot from 2019-11-24 01-29-22

Adding dynamic choices

For user experience, you'll want your choices to automatically reflex your the matrix axis options you have available. For example, let's say you want to add a new dimension for Java to the matrix.

// you can add more axes and this will still work
Map matrix_axes = [
    PLATFORM: ['linux', 'windows', 'mac'],
    JAVA: ['openjdk8', 'openjdk10', 'openjdk11'],
    BROWSER: ['firefox', 'chrome', 'safari', 'edge']
]

To support dynamic choices your choice and matrix axis filter needs to be updated to the following.

Map response = [:]
stage("Choose combinations") {
    response = input(
        id: 'Platform',
        message: 'Customize your matrix build.',
        parameters: matrix_axes.collect { key, options ->
            choice(
                choices: ['all'] + options.sort(),
                description: "Choose a single ${key.toLowerCase()} or all to run tests.",
                name: key)
        })
}

// filter the matrix axes since
// Safari is not available on Linux and
// Edge is only available on Windows
List axes = getMatrixAxes(matrix_axes).findAll { axis ->
    response.every { key, choice ->
        choice == 'all' || choice == axis[key]
    } &&
    !(axis['BROWSER'] == 'safari' && axis['PLATFORM'] == 'linux') &&
    !(axis['BROWSER'] == 'edge' && axis['PLATFORM'] != 'windows')
}

It will dynamically generate choices based on available matrix axes and will automatically filter if users customize it. Here's an example dialog and rendered choice when the pipeline executes.

Screenshot from 2019-11-24 01-40-10

Screenshot from 2019-11-24 01-40-37

See the full script with dynamic choices (click to expand)
// you can add more axes and this will still work
Map matrix_axes = [
    PLATFORM: ['linux', 'windows', 'mac'],
    JAVA: ['openjdk8', 'openjdk10', 'openjdk11'],
    BROWSER: ['firefox', 'chrome', 'safari', 'edge']
]

@NonCPS
List getMatrixAxes(Map matrix_axes) {
    List axes = []
    matrix_axes.each { axis, values ->
        List axisList = []
        values.each { value ->
            axisList << [(axis): value]
        }
        axes << axisList
    }
    // calculate cartesian product
    axes.combinations()*.sum()
}

Map response = [:]
stage("Choose combinations") {
    response = input(
        id: 'Platform',
        message: 'Customize your matrix build.',
        parameters: matrix_axes.collect { key, options ->
            choice(
                choices: ['all'] + options.sort(),
                description: "Choose a single ${key.toLowerCase()} or all to run tests.",
                name: key)
        })
}

// filter the matrix axes since
// Safari is not available on Linux and
// Edge is only available on Windows
List axes = getMatrixAxes(matrix_axes).findAll { axis ->
    response.every { key, choice ->
        choice == 'all' || choice == axis[key]
    } &&
    !(axis['BROWSER'] == 'safari' && axis['PLATFORM'] == 'linux') &&
    !(axis['BROWSER'] == 'edge' && axis['PLATFORM'] != 'windows')
}

// parallel task map
Map tasks = [failFast: false]

for(int i = 0; i < axes.size(); i++) {
    // convert the Axis into valid values for withEnv step
    Map axis = axes[i]
    List axisEnv = axis.collect { k, v ->
        "${k}=${v}"
    }
    // let's say you have diverse agents among Windows, Mac and Linux
    // all of which have proper labels for their platform and what
    // browsers are available on those agents.
    String nodeLabel = "os:${axis['PLATFORM']} && browser:${axis['BROWSER']}"
    tasks[axisEnv.join(', ')] = { ->
        node(nodeLabel) {
            withEnv(axisEnv) {
                stage("Build") {
                    echo nodeLabel
                    sh 'echo Do Build for ${PLATFORM} - ${BROWSER}'
                }
                stage("Test") {
                    echo nodeLabel
                    sh 'echo Do Build for ${PLATFORM} - ${BROWSER}'
                }
            }
        }
    }
}

stage("Matrix builds") {
    parallel(tasks)
}

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment