Design a framework for 2,000 UI tests that must run in under 15 minutes

This is a design question, so I would start by clarifying constraints and doing the arithmetic before naming tools.

Clarify first

• Is the 15 minutes for every pull request, or for the merge or nightly run?
• Which tool and language? Is the app a SPA, and how long does a typical test take today?
• Can we create data through APIs? Is there one shared environment or can we get ephemeral ones?
• What infrastructure budget exists: CI runners, a Selenium Grid, a cloud browser provider, Kubernetes?

The arithmetic

Suppose the average UI test takes 30 seconds including setup. 2,000 tests is about 1,000 minutes of serial work. To finish in 15 minutes with a little overhead for startup and reporting, you need about 12 minutes of execution, so roughly 85 to 100 concurrent sessions, assuming perfect balancing and no flakiness. That number tells you this is mostly an infrastructure, isolation and test-design problem, not a “make the code faster” problem.

Lever 1: run fewer UI tests

• Audit the suite. Many UI tests check validation messages, calculations or permissions that are cheaper and more reliable as API or component tests. Moving even a quarter of them down the pyramid changes the infrastructure bill significantly.
• Tier the suite. A critical-path smoke set on every PR, the full suite on merge to main or in a merge queue, plus impact-based selection (tests tagged by feature, mapped to changed paths) where you can trust the mapping.
• Delete duplicates that exercise the same flow with trivial variations.

Lever 2: make each test fast and independent

• Setup through APIs or seeding, never the UI. Create users, carts and orders with API calls.
• Authenticate once. Log in via API, capture the session cookie or token, and inject it (Playwright’s storageState, or adding cookies in Selenium) instead of filling the login form 2,000 times.
• Unique data per test so tests can run in any order on any worker.
• No fixed sleeps; rely on proper waits and web-first assertions.
• Block irrelevant third parties (analytics, chat widgets) in test environments to reduce load time and noise.

Lever 3: parallelise and shard

• Within a machine, run several workers (Playwright workers, TestNG or JUnit 5 parallel execution with a thread-safe driver).
• Across machines, shard the suite. Playwright supports --shard=3/10 natively; for other runners, split by file list in CI.
• Balance by historical duration, not by file count. One shard full of slow checkout tests will define your wall-clock time. Keep each spec file reasonably small so balancing has granularity.

strategy:
  fail-fast: false
  matrix:
    shard: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12]
steps:
  - uses: actions/checkout@v4
  - run: npx playwright test --shard=${{ matrix.shard }}/12

With 12 machines at 8 workers each, you have about 96 concurrent browsers.

Lever 4: infrastructure that can take the load

• Browsers: containerised browsers on Kubernetes (Selenium Grid with autoscaling nodes, or Playwright workers in pods), or a cloud browser provider. Pin browser versions for reproducibility.
• The application under test must handle around 100 concurrent sessions. A single small staging instance often becomes the bottleneck, which shows up as timeouts that look like flaky tests. Ephemeral per-run environments or a scaled-up test environment help.
• Fast startup: prebuilt Docker images with dependencies and browsers baked in, cached package installs.

Lever 5: keep it reliable at scale

At 2,000 tests, a 0.5% flake rate means about ten random failures per run, which destroys trust.

• Track per-test pass rate and duration over time in a dashboard.
• Quarantine flaky tests to a non-blocking job with an owner and a deadline, rather than blanket retries.
• Allow at most one retry, and report “passed on retry” as a flake signal rather than hiding it.
• Time budgets: flag any test that exceeds, say, 60 seconds for review.
• Collect traces, screenshots and logs on failure only, to keep artefacts manageable.

Reporting

Merge shard results into one report (for example Playwright’s blob reports merged with merge-reports, or Allure results aggregated), and post a summary on the pull request with failures grouped by error.

What interviewers look for

• Clarifying questions and back-of-envelope sizing before tooling.
• Reducing the UI suite, not just throwing hardware at it.
• Independence as the prerequisite for parallelism.
• Awareness that the system under test and test data can be the real bottleneck.
• A concrete plan for flakiness and observability.

Common mistakes

• Answering only “use Selenium Grid with more nodes”.
• Sharding by file count and ending up with one shard taking 40 minutes.
• Enabling three automatic retries to make the build green.