Designing CI/CD Quality Gates

A pipeline that runs tests is not the same as a pipeline with quality gates. A gate is a decision point: if this check is not green, the change does not move forward. Gates are where testing turns from information into control, and designing them well is one of the highest-leverage things an SDET does.

A good gate is fast enough that people do not route around it, reliable enough that a red result is believed, and relevant enough that green actually means something. This lesson covers how to arrange tests into gates that meet all three.

Continuous integration, delivery and deployment

• Continuous integration (CI): every change is merged frequently and verified automatically by building and testing it.
• Continuous delivery: every change that passes the pipeline is releasable; a person decides when to release.
• Continuous deployment: every change that passes the pipeline goes to production automatically.

The further right you go, the more the pipeline’s gates are the only thing between a commit and your users. Continuous deployment is only safe when the gates are trustworthy and production has its own safety nets such as monitoring, canaries and fast rollback.

Placing tests in the pipeline

The test pyramid is usually discussed as “how many of each type of test”. In pipeline design it also answers “when does each type run”. Faster, narrower tests run earlier and more often; slower, broader ones run later and on fewer occasions.

Stage	Typical checks	Target duration	Runs on
Pre-commit / local	Formatting, lint, type check, affected unit tests	Seconds	Developer machine
PR: fast gate	Build, lint, unit tests, static analysis, dependency scan	Under ~5 min	Every push to a PR
PR: integration gate	API and contract tests, component tests, focused E2E smoke	Under ~15 min	Every push to a PR
Post-merge	Full E2E across browsers, visual regression, broader integration	Tens of minutes	Merge to main
Pre-release / deploy	Deploy to staging, smoke tests, performance baseline, security scans	Varies	Release candidate
Post-deploy	Smoke and synthetic checks in production, canary analysis	Minutes	Every production deploy
Scheduled	Long soak or load tests, full cross-device runs, dependency audits	Hours	Nightly or weekly

The durations are targets to discuss with your team, not standards. The principle is what matters: a developer waiting for PR feedback should get it while they still have the change in their head.

A useful question for any test: what is the cheapest stage that could catch this bug? A validation rule tested through the UI on every PR belongs as a unit or API test instead; the E2E suite should cover the journeys that only an end-to-end run can prove.

Fail fast

Fail-fast design means the checks most likely to fail, and cheapest to run, go first, and later stages do not start when earlier ones fail.

name: PR checks

on:
  pull_request:

concurrency:
  group: pr-${{ github.event.pull_request.number }}
  cancel-in-progress: true

jobs:
  static:
    runs-on: ubuntu-latest
    timeout-minutes: 10
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with: { node-version: 22, cache: npm }
      - run: npm ci
      - run: npm run lint
      - run: npm run typecheck

  unit:
    runs-on: ubuntu-latest
    timeout-minutes: 10
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with: { node-version: 22, cache: npm }
      - run: npm ci
      - run: npm run test:unit

  api:
    needs: [static, unit]
    runs-on: ubuntu-latest
    timeout-minutes: 15
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with: { node-version: 22, cache: npm }
      - run: npm ci
      - run: npm run test:api

  e2e-smoke:
    needs: [static, unit]
    runs-on: ubuntu-latest
    timeout-minutes: 20
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with: { node-version: 22, cache: npm }
      - run: npm ci
      - run: npx playwright install --with-deps chromium
      - run: npx playwright test --grep @smoke --grep-invert @quarantine

The structure: static and unit run in parallel first; api and e2e-smoke only start when both pass (needs). A type error stops the pipeline in a couple of minutes instead of after a 15-minute browser run.

Fail-fast has a trade-off. Within a single test stage, stopping at the first failure hides other failures and forces a fix-push-wait loop for each one. A reasonable compromise: fail fast between stages, but let each stage run fully (for example fail-fast: false in a matrix) so the developer sees all failures from that stage at once. Playwright’s --max-failures option is useful for very large suites where a broken environment would otherwise produce hundreds of identical failures.

Required checks

A check only gates if the platform enforces it. On GitHub this is done with branch protection rules or rulesets on main: mark specific checks as required, require the branch to be up to date (or use a merge queue) and restrict who can bypass. GitLab, Azure DevOps and Bitbucket have equivalent settings.

Design rules for required checks:

• Make few checks required, and make them reliable. A required check that fails randomly teaches people to re-run until green or to ask for admin bypass. Both destroy the gate.
• Use stable check names. Required checks are matched by name; renaming a job silently removes the gate or leaves PRs waiting forever for a check that no longer runs.
• Beware skipped workflows. If a required workflow is skipped by a path filter, some platforms treat the check as pending, not passed. Prefer a lightweight job that always runs and decides internally what to test.
• Gate on outcomes, not percentages alone. A coverage threshold can be a useful guard against untested new code, but a high coverage number does not prove good tests. Use coverage on changed lines as a prompt for review, not a substitute for it.

Other gates beyond tests

Quality gates can also include static analysis findings above a severity, known-vulnerable dependencies, licence checks, bundle size budgets, accessibility scan results on key pages, and performance budgets on critical endpoints. Each should have an owner and a documented way to handle false positives, or it will be bypassed.

Flaky tests and quarantine

A flaky test passes and fails on the same code. In a gate, flakiness is poison: one flaky test in a required suite makes the whole gate unreliable. You need a process, not just retries.

A workable quarantine process:

1. Detect. Flag tests that fail then pass on retry, or that both pass and fail on the same commit across runs.
2. Quarantine quickly. Move the test out of the gating run with a tag, so it stops blocking unrelated work. In Playwright:

import { test, expect } from '@playwright/test';

test('applies loyalty discount at checkout', { tag: ['@quarantine'] }, async ({ page }) => {
  // Quarantined 2026-11-03, ticket QA-812: intermittent timeout on price recalculation.
  await page.goto('/checkout');
  await expect(page.getByTestId('discount')).toHaveText('-£5.00');
});

3. Keep running it. Run quarantined tests in a separate, non-blocking job (--grep @quarantine) so you collect evidence of whether the fix worked.
4. Assign an owner and a deadline. Every quarantined test gets a ticket, an owner and a time limit. When the deadline passes, fix it or delete it and accept the coverage gap explicitly.
5. Cap the quarantine. If the quarantine list grows past an agreed size, stop adding features until it shrinks. Otherwise it becomes a graveyard of tests that nobody runs.

Common root causes to look for first: fixed waits instead of condition-based waits, shared test data between parallel workers, dependence on test order, animations and timing, and unstable test environments. The last one is not a test bug, but it still needs an owner.

Post-deploy gates

Gates do not stop at merge. After a deploy:

• Run a small smoke suite against the deployed environment; fail the deploy (and roll back automatically if you can) when it fails.
• With canary releases, route a small share of traffic to the new version and compare error rates and latency against the stable version before promoting it.
• Keep synthetic monitoring running so you learn about breakage before users report it.

These ideas are covered in Reliability Engineering for QA.

Practice exercise

A team has one required check: a 45-minute job that runs lint, unit tests, 600 UI tests across three browsers, and a load test. It fails on roughly one run in four for reasons unrelated to the change, and developers routinely click “re-run”. Propose a new gate design.

Show a model answer

1. Split the job into stages: static checks and unit tests first (parallel, minutes), then API/contract tests and a tagged E2E smoke set in Chromium only, gated with needs.
2. Required checks on PRs: static, unit, API and E2E smoke. Keep the check names stable.
3. Move the full three-browser UI suite to post-merge on main (sharded) and the load test to a nightly scheduled run against a production-like environment. Failures there create tickets and notify the owning team.
4. Push tests down: review the 600 UI tests for checks that belong at API or unit level and migrate them.
5. Flakiness: turn on retry reporting, identify the flaky tests, quarantine them with a tag into a non-blocking job, assign owners and deadlines, and cap the quarantine list.
6. Speed: cancel superseded runs with a concurrency group, cache dependencies, shard the post-merge suite.
7. Measure PR feedback time, re-run rate and non-code failure rate before and after, so the team can see the gate becoming trustworthy.

Key takeaways

• A quality gate is a decision point; it must be fast, reliable and relevant to be respected.
• Place each test at the cheapest stage that can catch its bugs: unit and static checks first, broad E2E and load later.
• Fail fast between stages, but let each stage report all of its failures.
• Enforce gates with required checks on stable names, and keep the required set small and reliable.
• Quarantine flaky tests quickly, keep running them, give each an owner and a deadline, and cap the list.
• Gates continue after deploy with smoke tests, canaries and synthetic monitoring.