How would you run a large test suite in parallel across Jenkins agents?

Parallelism in Jenkins has two separate layers: the pipeline (running stages concurrently on different executors) and the test runner (threads or workers inside one process). For a big suite you usually want both, but the Jenkins layer is what lets you scale beyond one machine.

Pipeline structure

The approach I use: build once, stash the result, fan out to N shards on separate agents, publish results from each shard, and keep a single place where the overall status is decided.

pipeline {
    agent none
    options { timeout(time: 40, unit: 'MINUTES') }
    stages {
        stage('Build') {
            agent { label 'linux' }
            steps {
                sh 'mvn -B -DskipTests package'
                stash name: 'build', includes: 'target/**, pom.xml, src/test/**'
            }
        }
        stage('Tests') {
            matrix {
                axes {
                    axis { name 'SHARD'; values '1', '2', '3', '4' }
                }
                agent { label 'linux && docker' }
                stages {
                    stage('Run shard') {
                        steps {
                            unstash 'build'
                            sh "./scripts/run-shard.sh ${SHARD} 4"
                        }
                        post {
                            always { junit testResults: 'target/surefire-reports/*.xml' }
                        }
                    }
                }
            }
        }
    }
}

The same can be written with an explicit parallel { stage('Shard 1') {...} ... } block. matrix is handy when shards are uniform or when you also want an axis for browsers. In declarative syntax, failFast true (on a parallel block) or parallelsAlwaysFailFast() in options aborts sibling branches when one fails. I usually leave it off for test stages, because you want the full failure picture, and turn it on for cheap gating checks.

Agents

Each parallel branch needs its own executor. Static agents with labels work, but they drift: different browser versions, leftover files, full disks. I prefer ephemeral agents via the Kubernetes plugin (a pod per build) or Docker agents (agent { docker { image '...' } }), so every shard starts from the same image. Capacity planning matters too: four shards on a pool with two free executors just queue.

Sharding strategy

• Naive split by file or class count is easy but uneven; one slow shard sets the wall-clock time.
• Duration-based split uses timings from previous JUnit reports to balance shards. Many runners support sharding natively (Playwright --shard=1/4); for Maven/TestNG you typically generate the shard’s class list in a script.
• Keep tests independent: no shared accounts, no ordering assumptions, unique test data per shard. Parallelism exposes hidden coupling quickly.

Results and diagnosis

Publish junit in every shard; Jenkins aggregates results across the build. Archive per-shard artifacts with the shard number in the path. Track shard durations over time; if one shard drifts, rebalance.

Common mistakes

• Rebuilding the application in every shard.
• Running parallel shards against one shared environment that cannot handle the load, then blaming the tests.
• Using failFast on the test stage and losing results.

What interviewers look for

• Correct use of parallel/matrix and agents in declarative syntax.
• Build-once plus stash/unstash or artifacts.
• Duration-balanced sharding and an emphasis on test independence and environment capacity.