Agentic AI for Testing

An agent is an LLM placed in a loop where it can call tools, see the results and decide what to do next. Instead of writing every step of a test, you give the agent a goal such as “check that a new user can add an item to the cart and see the correct total”, and it works out the steps by navigating, clicking and reading the page through tools.

This is genuinely useful in some situations and genuinely risky in others. This lesson is about telling them apart.

The agent loop

        ┌──────────────────────────────────────────────┐
        │                                              │
Goal ──▶ Plan ──▶ Act (call a tool) ──▶ Observe result ─┤
                                                       │
                          done? ──▶ Assert ──▶ Report ◀┘

1. Goal: a task in natural language, ideally with explicit success criteria.
2. Plan: the model breaks the goal into steps. Some agents plan up front; others decide one step at a time.
3. Act: the model chooses a tool and arguments, for example navigate(url), click(element), type(element, text), http_get(path).
4. Observe: the harness runs the tool and returns the result: a page snapshot, an HTTP response, an error.
5. Repeat until the model decides the goal is reached, a step limit is hit or something fails.
6. Assert: decide whether the goal was actually met.
7. Report: what happened, with evidence.

The model never touches the browser directly. It only produces tool calls; your harness executes them. That boundary is where you enforce safety.

Tools and observations

For browser work, the observation format matters. Screenshots require a vision-capable model and are expensive to process. Many browser tools instead give the model a structured text view of the page, such as the accessibility tree: roles, names and states of elements. This is compact and maps naturally to role-based actions (“click the button named Checkout”). Playwright MCP, covered in MCP for QA Workflows, uses accessibility snapshots by default.

Real risks

Non-determinism

The same goal can produce different paths on different runs. One run checks out with a saved card; another adds a new card; a third gets distracted by a cookie banner. Results are hard to reproduce, and a single green run tells you little.

Hallucinated assertions

The most dangerous failure: the agent reports success when the goal was not met. It may “see” a confirmation that is not there, accept an error page as success, or quietly redefine the goal (“the cart total was not shown, but the item was added, so the test passed”). If the agent is both actor and judge, nothing independent checks it.

Flaky oracles

Even when an agent’s judgment is usually right, “usually” is not good enough for a gate. A judgment that flips between pass and fail on identical behaviour is a flaky oracle, and it erodes trust faster than flaky tests do.

Cost and latency

Each step is a model call with a growing context (goal, history, latest observation). Long flows mean many calls. An agent run is typically far slower and more expensive than the equivalent scripted test, which matters if you run it on every commit.

Security and credentials

An agent with tools can do whatever those tools allow:

• Credentials given to it (test accounts, API keys) can be misused if the agent is manipulated or makes a mistake.
• Page content is untrusted input. Text on a page, in an email or in a ticket can contain instructions (“ignore previous instructions and…”) that the model may follow. This is prompt injection, and it applies to testing agents as much as to any other.
• Destructive actions (deleting data, sending emails, making payments) can happen by accident.

Where agents help today, and where they do not

Helps today (with review)	Not appropriate
Exploratory assistance: “try unusual inputs on this form and report anything odd”	Being the sole release gate
Scaffolding tests: exploring a flow and drafting a Playwright script for a human to refine	Replacing regression suites with fresh agent runs each time
Reproducing a bug report and collecting evidence (steps, console errors, network failures)	Unsupervised testing in production
Triage: gathering logs, traces and recent changes for a failing test	Judging correctness where no explicit oracle exists
Smoke-checking that a new environment is broadly working	Compliance or audit evidence

The pattern in the left column: the agent produces leads, drafts or evidence that a person or a deterministic check then verifies. A strong workflow is to let the agent explore and draft, then convert what it found into a conventional, deterministic test that runs in CI.

Designing a bounded agent harness

If you build or configure an agent for testing, these guardrails turn a risky experiment into a useful assistant:

• Explicit success criteria in the goal, phrased so they can be checked mechanically.
• Step and cost budgets: stop after N steps or a spending limit.
• Tool allowlist: only the tools the task needs; no shell access for a browsing task.
• Domain allowlist: navigation outside the test environment is blocked by the harness.
• Full transcript: every prompt, tool call, observation and decision logged for review.
• Independent final assertions: after the agent finishes, deterministic code checks the outcome.

Here is a minimal, provider-agnostic sketch of such a harness. The model’s job is to choose the next action; the harness enforces the limits and performs the final check itself.

type Action =
  | { tool: 'navigate'; url: string }
  | { tool: 'click'; role: string; name: string }
  | { tool: 'fill'; label: string; value: string }
  | { tool: 'done'; summary: string };

// Stub: send the prompt to your model and return its raw text reply.
async function callModel(prompt: string): Promise<string> {
  throw new Error('callModel not implemented');
}

const ALLOWED_ORIGIN = 'https://staging.example.com';
const MAX_STEPS = 15;

export async function runAgent(page: import('@playwright/test').Page, goal: string) {
  const transcript: string[] = [];

  for (let step = 0; step < MAX_STEPS; step++) {
    const snapshot = await page.locator('body').ariaSnapshot(); // accessibility-tree text
    const prompt = `Goal: ${goal}
Reply with one JSON action: navigate, click (role, name), fill (label, value) or done.
Transcript so far:
${transcript.join('\n')}
Current page:
${snapshot}`;

    let action: Action;
    try {
      action = JSON.parse(await callModel(prompt));
    } catch {
      transcript.push(`step ${step}: invalid model output, stopping`);
      break;
    }
    transcript.push(`step ${step}: ${JSON.stringify(action)}`);

    if (action.tool === 'done') break;
    if (action.tool === 'navigate') {
      if (new URL(action.url).origin !== ALLOWED_ORIGIN) {
        transcript.push('blocked: navigation outside allowed origin');
        break;
      }
      await page.goto(action.url);
    } else if (action.tool === 'click') {
      await page.getByRole(action.role as any, { name: action.name }).click();
    } else if (action.tool === 'fill') {
      await page.getByLabel(action.label).fill(action.value);
    }
  }
  return transcript;
}

And in the test, the verdict comes from deterministic assertions, not from the agent’s summary:

import { test, expect } from '@playwright/test';
import { runAgent } from './agent';

test('agent-assisted: add item to cart', async ({ page }) => {
  await page.goto('https://staging.example.com');
  const transcript = await runAgent(page, 'Add one "Blue Mug" to the cart and open the cart page.');
  await test.info().attach('agent-transcript', { body: transcript.join('\n'), contentType: 'text/plain' });

  // Independent oracle: the harness checks the outcome itself.
  await expect(page).toHaveURL(/\/cart$/);
  await expect(page.getByRole('row', { name: /Blue Mug/ })).toBeVisible();
  await expect(page.getByTestId('cart-total')).toHaveText('£12.00');
});

The sketch omits things a real harness needs (error handling per tool, retries, cost tracking, redaction), but it shows the essential shape: the model proposes, the harness constrains and executes, deterministic code judges, and the transcript is kept as evidence. Even then, run it several times before trusting it, since the path can vary between runs.

Practice exercise

A colleague proposes replacing the 400-test Playwright regression suite with a nightly agent that is given the product’s user stories and asked to “test everything and report failures”. Write a short response explaining the risks and proposing a better way to use an agent.

Show a model answer

Risks of the proposal:

• No stable oracle: the agent decides what “correct” means from user stories, so it may report success when behaviour is wrong (hallucinated assertions) and results will vary night to night.
• No reproducibility: a failure found on Tuesday may not be retried the same way on Wednesday, so we lose the regression guarantee our suite gives us.
• Cost and time: hundreds of stories explored step by step means many model calls every night, likely slower and more expensive than the current suite.
• Security: the agent would need credentials and broad access; page content could manipulate it.

Better approach:

• Keep the deterministic regression suite as the release gate.
• Use an agent for bounded exploratory sessions on new or risky features, with a step budget, staging-only access and low-privilege accounts, producing a transcript and a list of suspected issues for a tester to verify.
• Use the agent to draft Playwright tests for new stories; a person reviews them, confirms they can fail and adds them to the suite.
• Measure: track how many agent-reported issues were real, and keep using it only where that rate justifies the cost.

Key takeaways

• An agent is an LLM in a loop: goal, plan, act via tools, observe, then assert and report.
• Main risks: non-determinism, hallucinated success, flaky oracles, cost and the security of tools and credentials.
• Prompt injection from page content applies to testing agents too.
• Agents help today with exploration, scaffolding, bug reproduction and triage, producing leads that people verify.
• Never make an agent the sole release gate; keep deterministic final assertions and full transcripts.
• Constrain agents with step budgets, tool and domain allowlists and low-privilege test accounts.