How do you decide which tests to automate, and which to leave manual?

I decide based on value over time: how often the check will run and give useful feedback, how much risk it covers, and what it costs to build and keep green. A good automation candidate is repeated, deterministic and important. A poor candidate is a one-off, subjective, or attached to something that changes every sprint. The second decision, which is often more important, is at which layer to automate.

Strong candidates

• Regression of stable, critical flows: login, checkout, payments, permissions. They run on every change, and failures are expensive.
• Data-heavy and combinatorial checks: validation rules, pricing matrices, role-permission combinations. Humans get bored and make mistakes; parameterised tests do not.
• Smoke and post-deployment checks: fast gate for every build and environment.
• Bugs that escaped: each fixed defect gets a test at the lowest layer that would have caught it.
• Things humans cannot practically do: load, concurrency, many browsers in parallel, long soak runs, API contract checks across services.
• Environment and data setup: even for manual testing, scripting the seeding of accounts and orders saves hours.

Poor candidates (keep humans here)

• Exploratory testing, where the value is human judgement and discovering unknowns.
• Usability and look-and-feel: “is this confusing?” has no reliable assertion. Visual regression tools can catch changes, not decide whether a design is good.
• Features in heavy flux: a prototype or an experiment likely to be removed. Automate once the behaviour settles.
• One-off checks such as a data migration verification (unless it will be repeated).
• Tests needing hard-to-control externals (real SMS, third-party CAPTCHA), unless you can stub them.

A rough cost-benefit model

For a candidate, I estimate:

• Expected executions per month (each PR? nightly? per release?).
• Cost of a manual run versus build cost plus ongoing maintenance.
• Risk covered: likelihood and impact of the defect it would catch.
• Flakiness risk: async UI, third-party dependencies, shared data.

I do not pretend these are precise. The value is in the conversation: a test that runs on every PR and guards payments wins easily; a UI test for a rarely used settings page that changes monthly probably does not.

Choose the layer

Most of the automation decision is “where”, not “whether”. Validation rules belong in unit or API tests; a few UI tests show the error is displayed. Moving a check from UI to API often cuts its runtime from tens of seconds to well under a second and removes most flakiness.

Keeping the suite healthy

• Treat test code as production code: reviews, linting, ownership.
• Track flaky tests and quarantine or fix them quickly; an untrusted suite has no value.
• Delete tests that no longer earn their keep.

What interviewers look for

• Clear criteria: frequency, stability, risk, determinism, cost.
• Explicit maintenance cost, not just build cost.
• What should stay manual and why.
• Choosing the right layer, consistent with the test pyramid.

Common mistakes

• “Automate everything” or targeting a percentage of test cases automated as the goal.
• Converting manual test cases into UI scripts one to one.