CI/CD Integration

Tests that only run on someone’s laptop protect nothing. The point of automation is that every change is checked the same way, automatically, before it reaches users. Getting tests into a CI/CD pipeline is usually straightforward; getting a pipeline whose results people trust and act on takes more design.

This lesson focuses on the test automation side: what to run when, and how to wire a real suite into GitHub Actions. For pipeline fundamentals, see GitHub Actions for Test Automation.

What runs when

Not every test belongs at every stage. The goal is fast feedback early and broad coverage before release:

Stage	Trigger	Typical tests	Time budget
Pull request	Every push to a PR	Lint, type check, unit, API tests, a UI smoke set	Minutes
Main branch	Merge to main, then deploy to staging	Full API and UI regression against staging	Tens of minutes
Nightly	Schedule	Cross-browser, long journeys, visual and accessibility checks	Can run longer
After deploy	Each production deploy	A small smoke suite against production, read-only	Minutes

Two principles drive this table:

• Fast tests gate early. A developer waiting on a PR will tolerate minutes, not an hour. Put the fastest, most reliable tests on the PR path.
• Each stage tests what changed at that stage. PR checks test the code; post-deploy checks test the deployment (configuration, infrastructure, secrets).

Where the application comes from

UI and API tests need a running system. Common options:

• Start it inside the job. Playwright’s webServer config starts your app before tests and waits for a URL to respond. Good for front-end apps with mocked or containerised back ends.
• Service containers. GitHub Actions services: (or Docker Compose) can start databases and dependencies next to the job.
• Preview environments. Many platforms deploy every PR to its own URL; pass it to tests as BASE_URL.
• A shared staging environment. Simplest to set up, but tests then compete with other runs and people for data, which makes test data management essential.

// playwright.config.ts: start the app locally and in CI unless BASE_URL points elsewhere
import { defineConfig } from '@playwright/test';

export default defineConfig({
  use: { baseURL: process.env.BASE_URL ?? 'http://localhost:3000' },
  webServer: process.env.BASE_URL
    ? undefined
    : {
        command: 'npm run start',
        url: 'http://localhost:3000',
        reuseExistingServer: !process.env.CI,
        timeout: 120_000,
      },
  reporter: process.env.CI ? [['blob'], ['github']] : [['html', { open: 'on-failure' }]],
  retries: process.env.CI ? 2 : 0,
  fullyParallel: true,
  // projects: an 'api' project and a 'chromium' project, as in Framework Architecture
});

A complete GitHub Actions workflow

The workflow below runs API tests first, then UI tests in four parallel shards, then merges the shard results into one HTML report. It uses Playwright’s blob reporter in CI, as in the config above.

# .github/workflows/tests.yml
name: Tests

on:
  pull_request:
  push:
    branches: [main]
  schedule:
    - cron: '0 2 * * 1-5'   # 02:00 UTC, Monday to Friday
  workflow_dispatch:

concurrency:
  group: tests-${{ github.ref }}
  cancel-in-progress: true

jobs:
  api:
    runs-on: ubuntu-latest
    timeout-minutes: 15
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: 22
          cache: npm
      - run: npm ci
      - name: API tests
        run: npx playwright test --project=api
        env:
          API_BASE_URL: https://sdetrole.com/api/mock/
          API_PASSWORD: ${{ secrets.API_PASSWORD }}

  ui:
    needs: api
    runs-on: ubuntu-latest
    timeout-minutes: 30
    strategy:
      fail-fast: false
      matrix:
        shardIndex: [1, 2, 3, 4]
        shardTotal: [4]
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: 22
          cache: npm
      - run: npm ci
      - run: npx playwright install --with-deps chromium
      - name: UI tests (shard ${{ matrix.shardIndex }} of ${{ matrix.shardTotal }})
        run: npx playwright test --project=chromium --shard=${{ matrix.shardIndex }}/${{ matrix.shardTotal }}
        env:
          BASE_URL: ${{ vars.STAGING_URL }}
          TEST_PASSWORD: ${{ secrets.TEST_PASSWORD }}
      - uses: actions/upload-artifact@v4
        if: ${{ !cancelled() }}
        with:
          name: blob-report-${{ matrix.shardIndex }}
          path: blob-report
          retention-days: 1

  report:
    if: ${{ !cancelled() }}
    needs: ui
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: 22
          cache: npm
      - run: npm ci
      - uses: actions/download-artifact@v4
        with:
          path: all-blob-reports
          pattern: blob-report-*
          merge-multiple: true
      - run: npx playwright merge-reports --reporter html ./all-blob-reports
      - uses: actions/upload-artifact@v4
        with:
          name: playwright-report-attempt-${{ github.run_attempt }}
          path: playwright-report
          retention-days: 14

What each part is doing:

• Triggers: PRs and pushes to main for fast feedback; a weekday schedule for a nightly run; workflow_dispatch for manual runs. Check cron expressions with the Cron Expression Explainer; GitHub schedules use UTC.
• concurrency cancels an in-progress run when a newer commit arrives on the same branch, saving runner time.
• needs: api makes the slower UI jobs wait for the fast API tests, so an obviously broken build fails in minutes.
• fail-fast: false keeps other shards running when one fails, so you see every failure in one run.
• npx playwright install --with-deps chromium installs only the browser this project needs, plus its OS libraries.
• if: ${{ !cancelled() }} uploads reports even when tests fail, which is exactly when you need them.
• timeout-minutes stops a hung job from consuming runner time for hours.

Java suites in the same pipeline

A REST Assured or Selenium suite built with Maven fits the same pattern:

  java-api:
    runs-on: ubuntu-latest
    timeout-minutes: 20
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-java@v4
        with:
          distribution: temurin
          java-version: '21'
          cache: maven
      - run: mvn -B test
        env:
          API_BASE_URI: https://sdetrole.com
      - uses: actions/upload-artifact@v4
        if: ${{ !cancelled() }}
        with:
          name: surefire-reports
          path: target/surefire-reports

Maven exits non-zero when tests fail, which fails the job. Surefire writes JUnit XML into target/surefire-reports, which a test-report action or your CI’s test tab can display.

Secrets and configuration

• Store credentials as encrypted secrets (secrets.*) and non-sensitive settings such as URLs as variables (vars.*), at repository or environment level.
• Pass them to steps through env, never on the command line, where they may be echoed. GitHub masks known secret values in logs, but not values derived from them, such as a token your test prints.
• Workflows triggered by pull requests from forks do not receive repository secrets. Design PR checks for external contributors to run without them, or gate those runs behind manual approval.
• Use GitHub environments (for example staging, production) with protection rules when tests or deployments target sensitive systems.

Quality gates and trust

A pipeline only protects anything if a failure blocks something. Make the PR test jobs required status checks in branch protection so a PR cannot merge while they fail.

That only works while people believe failures are real. The fastest way to lose that trust is flaky tests:

• Retries in CI only (retries: process.env.CI ? 2 : 0) absorb rare environmental blips; Playwright still reports such tests as flaky, so they stay visible.
• Quarantine consistently flaky tests: move them out of the gating suite (for example with a @quarantine tag excluded by --grep-invert), create a ticket with an owner, and fix or delete them within an agreed time.
• Never normalise red. A main branch that is “always a bit red” teaches everyone to ignore it, and the next real regression goes unnoticed.

Practice exercise

Your team has a Playwright suite with an api project (60 tests, 1 minute) and a chromium project (400 tests, 24 minutes on one machine). PR checks currently run everything serially and take 26 minutes; developers have started merging before checks finish. Design a pipeline that gives PR feedback in about 10 minutes, keeps full coverage before release, and produces one report per run. Sketch it with the GitHub Actions Generator if useful.

Show a model answer

PR pipeline (target about 10 minutes):

1. api job: npx playwright test --project=api, about 1 minute plus setup.
2. ui job, needs: api, matrix of 4 shards with fullyParallel: true and 2 workers each: about 24 / 8 = 3 minutes of test time per shard, plus browser install and startup. Even with imperfect balancing this fits comfortably inside the target.
3. report job: merges blob reports into one HTML report and uploads it.
4. Branch protection marks api and all ui shards as required checks.

Main branch and nightly: the same jobs run on merge to main against staging after deployment, and nightly with extra projects (firefox, webkit) and slow tests tagged @nightly, which PR runs exclude with --grep-invert @nightly.

Trust measures: retries: 2 in CI only; flaky tests reviewed weekly from the merged report; consistently flaky tests tagged @quarantine, excluded from required checks, with an owner and a deadline.

The arithmetic matters: sharding scales well here because individual tests are short. If a few tests took several minutes each, they would set the floor for each shard, and splitting or speeding up those tests would matter more than adding shards.

Key takeaways

• Match suites to stages: fast, reliable tests on PRs, full regression after merge, broad and slow suites nightly, small read-only smoke tests after production deploys.
• In GitHub Actions, use actions/checkout@v4, actions/setup-node@v4 or setup-java@v4 with caching, shard with a matrix, upload blob reports with if: ${{ !cancelled() }}, and merge them in a final job.
• Keep secrets in secrets.*, pass them through env, and remember fork PRs do not receive them.
• Make test jobs required checks, retry only in CI, and quarantine flaky tests with an owner so red always means something.