What does testing in production mean, and how do you do it safely?

Testing in production is the set of practices that verify software in the environment where it actually runs, with real infrastructure, configuration, data volumes and traffic. Staging environments always differ in some way, so some classes of defects (configuration drift, scale effects, real third-party behaviour) only appear in production. The point is to find them deliberately and with a small blast radius, rather than through customer complaints.

It is not an excuse to skip pre-production testing. It is the last layer of the strategy.

Separate deploy from release

Feature flags let you deploy code dark and enable it for a controlled audience: employees first, then a percentage of users, then everyone. Testers can validate the real feature in production with their own accounts while customers see nothing. A flag also acts as a kill switch.

Testing considerations:

• Test both flag states before release, and the transition between them.
• Keep the number of live flag combinations under control; remove flags after rollout.
• Treat flag configuration changes like deployments, with audit and review.

Progressive delivery

• Canary releases send a small fraction of traffic to the new version and compare its error rate, latency and business metrics (conversion, payment success) with the stable version. Automated analysis can promote or roll back.
• Blue-green deployments keep the old environment ready for instant switch-back.

As an SDET, I help define the metrics and thresholds that gate promotion, for example “p99 latency within 10% of baseline and 5xx rate below the SLO for 15 minutes”, and make sure the rollback path is itself tested.

Post-deploy verification and synthetic monitoring

• Smoke tests after each deploy: a few read-only or safely reversible checks against production endpoints, run by the pipeline. A failure triggers rollback.
• Synthetic monitoring: scripted journeys (log in, search, add to basket, sometimes checkout with a test payment method) run on a schedule from several regions, alerting when they fail or slow down.

import { test, expect } from '@playwright/test';

test('search returns results in production', async ({ page }) => {
  await page.goto(process.env.PROD_URL!);
  await page.getByRole('searchbox', { name: 'Search' }).fill('headphones');
  await page.keyboard.press('Enter');
  await expect(page.getByTestId('result-card').first()).toBeVisible();
});

Keep synthetic tests extremely stable and fast; a flaky production monitor trains people to ignore pages.

Other techniques

• Shadow traffic (mirroring requests to a new version and discarding its responses) to compare behaviour without user impact, with care around side effects such as sending emails.
• Chaos experiments to verify resilience, run with agreed scope, monitoring and an abort plan.
• A/B tests are product experiments, but their instrumentation also needs verifying.

Safety rules

• Use dedicated, clearly marked test accounts and tag any data they create so it is excluded from analytics, billing and reports, and can be cleaned up.
• Never create side effects on real customers: no real charges, emails or inventory changes outside agreed test paths.
• Respect privacy: production testing does not grant broader access to customer data.
• Have observability in place first: structured logs, metrics, traces and alerts tied to SLOs. Without it you are deploying, not testing.

What interviewers look for

• A clear statement that it complements, not replaces, earlier testing.
• Concrete mechanisms: flags, canaries with metric gates, synthetic monitoring, rollback.
• Safety: blast radius, test data isolation, customer impact.
• The SDET’s role in defining signals and automating verification.

Common mistakes

• Equating testing in production with “we test manually on prod after release”.
• Running write-heavy tests on production without data tagging and cleanup.