AI-Assisted Test Generation

Use LLMs to draft test ideas, test cases, test data and automation code, with prompts that produce reviewable output and checks that catch bad generations.

Using an LLM to draft tests is now an everyday practice in many teams. It works well when you treat the model as a fast, tireless junior collaborator whose every output gets reviewed. It works badly when you paste a vague request and commit what comes back. This lesson is about the difference.

The workflow that works

Context (requirement, contract, constraints)
   → Prompt asking for a specific technique and a structured format
   → Model drafts
   → Automatic checks (parse, schema, lint, compile)
   → Human review (coverage, correctness, value)
   → Run and confirm the test can fail
   → Commit

Every stage after “model drafts” exists because the draft may be wrong. Skip them and you are shipping unreviewed guesses.

Generating test ideas and cases

Give the model what a human tester would need

A model cannot infer your business rules. Provide:

The requirement or acceptance criteria, verbatim.
Relevant constraints: field formats, limits, roles, states.
What is out of scope.
The test design technique you want applied.
The output format.

You are helping a QA engineer design test cases.

Requirement:
"Users can set a password of 12 to 64 characters. It must contain at least one
letter and one digit. The last 3 passwords cannot be reused. After 5 failed
logins within 15 minutes the account is locked for 30 minutes."

Apply boundary value analysis and equivalence partitioning.
Include negative cases. Do not invent rules that are not in the requirement;
if something is ambiguous, list it under "questions" instead of assuming.

Return JSON only, matching:
{ "cases": [ { "id": string, "title": string, "technique": string,
  "steps": string[], "expected": string } ], "questions": string[] }

Two details carry most of the value. Asking for a named technique produces systematic cases rather than a random list. Asking for questions surfaces ambiguities (is the length counted in characters or bytes? does “within 15 minutes” mean rolling or fixed window?) instead of letting the model silently pick an answer. Those questions are often the most useful output, because they are requirement defects.

Review generated cases

Check the output against the requirement line by line:

Coverage: are the boundaries 11, 12, 64 and 65 all present? Is the lockout window tested at 4 and 5 failures?
Invented rules: did the model add “must contain a special character”? Delete invented expectations.
Duplicates and noise: merge or drop cases that test the same partition.
Missing risk: the model will not know that your system once had a bug with Unicode passwords. Add what you know.

Generating test data, safely

LLMs are good at producing varied, realistic-looking data: names from many cultures, addresses, edge-case strings, product descriptions. Two rules make this safe:

Never ask for or supply real personal data. The output must be synthetic.
Validate every generated record mechanically before using it.

For data that must follow a structure, validate against a JSON Schema. The harness below is provider-agnostic: callModel is a stub for whichever model or local tool you use.

import Ajv from 'ajv';
import addFormats from 'ajv-formats';

// Replace with your provider's SDK or a local model. Must return the raw text response.
async function callModel(prompt: string): Promise<string> {
  throw new Error('callModel not implemented');
}

const customerSchema = {
  type: 'object',
  required: ['name', 'email', 'country', 'age'],
  additionalProperties: false,
  properties: {
    name: { type: 'string', minLength: 1, maxLength: 100 },
    email: { type: 'string', format: 'email' },
    country: { type: 'string', pattern: '^[A-Z]{2}$' },
    age: { type: 'integer', minimum: 18, maximum: 120 },
  },
};

const ajv = new Ajv({ allErrors: true });
addFormats(ajv);
const validate = ajv.compile(customerSchema);

export async function generateCustomers(count: number) {
  const prompt = `Generate ${count} fictional customer records as a JSON array.
Use varied names from different cultures, including names with apostrophes,
hyphens and non-ASCII letters. Emails must use the example.com domain.
Country is an ISO 3166-1 alpha-2 code. Age is an integer from 18 to 120.
Return only the JSON array.`;

  const raw = await callModel(prompt);

  let parsed: unknown;
  try {
    parsed = JSON.parse(raw);
  } catch {
    throw new Error('Model did not return valid JSON');
  }
  if (!Array.isArray(parsed)) throw new Error('Expected a JSON array');

  const valid = [];
  const rejected = [];
  for (const record of parsed) {
    if (validate(record)) valid.push(record);
    else rejected.push({ record, errors: validate.errors });
  }

  const emails = new Set(valid.map((r) => r.email));
  if (emails.size !== valid.length) console.warn('Duplicate emails in generated data');

  return { valid, rejected };
}

Notice what the code does not trust: that the response is JSON, that it is an array, that each record matches the schema, or that values are unique. Log the rejection rate; a rising rate tells you the prompt or model changed.

Once a good dataset is generated and reviewed, commit it as a fixture. Regenerating data on every test run makes your tests non-deterministic for no benefit.

Generating automation code

AI coding assistants can scaffold Playwright or Selenium tests, page objects and API clients quickly. Give them your conventions: existing page objects, locator strategy, fixtures and an example test from your codebase. Without that context they produce generic code that does not fit your framework.

Review checklist for generated test code

Check	What to look for
APIs exist	Methods and options that are real in your framework version, not plausible inventions
Assertions	Meaningful web-first assertions such as `toHaveText`, not just “page loaded”
Waiting	No `waitForTimeout` or arbitrary sleeps; rely on auto-waiting assertions
Locators	Role, label or test ID locators that match your conventions
Independence	No dependence on other tests’ state or order
Data	No hard-coded secrets, no reliance on data that may not exist
Scope	Does the test check what the requirement says, or something adjacent?

Prove the test can fail

A generated test that has never failed is unproven. Use one of:

Run it against a build where the feature is broken or reverted.
Temporarily change the expected value and confirm the failure message is clear.
For unit-level tests, use mutation testing tools (for example, Stryker for JavaScript and TypeScript) to see whether the tests detect injected faults.

Where generation goes wrong

Plausible but wrong expectations: the model guesses a discount rounds up when the rule says it rounds down.
Coverage illusion: 40 generated cases, all in the same equivalence partition.
Outdated APIs: code written for an older framework version, because training data predates current releases.
Consistency with the bug: if you give the model the implementation and ask for tests, it may encode the current (buggy) behaviour as expected. Generate from the specification, not the code, when you want independent tests.

Practice exercise

Write a prompt to generate API test cases for POST /api/transfers, which moves money between two accounts owned by the same user. Rules: amount is a positive decimal with at most 2 decimal places, maximum 10,000.00 per transfer; source and destination must differ; source must have sufficient balance. Then list three things you would check when reviewing the output.

Show a model answer

You are assisting a QA engineer. Design API test cases for POST /api/transfers.

Rules (do not add others; list ambiguities under "questions"):
- Body: { "fromAccountId": string, "toAccountId": string, "amount": number }
- Both accounts must belong to the authenticated user.
- amount > 0, at most 2 decimal places, maximum 10000.00.
- fromAccountId must differ from toAccountId.
- Source balance must be >= amount.

Apply boundary value analysis and equivalence partitioning. Include
authorisation cases (another user's account, no token) and malformed bodies.
For each case give: id, title, request body, expected status, expected effect
on both balances. Return JSON only:
{ "cases": [...], "questions": [...] }

Review checks:

Boundaries: 0, 0.01, 10000.00, 10000.01, three decimal places, negative values, and balance exactly equal to the amount are all present.
Expected effects: every case states balances afterwards, and failure cases expect both balances unchanged, not just an error status.
No invented rules or statuses: if the model assumed a specific error code (say 422 vs 400) that the API contract does not define, move it to a question for the team instead of encoding a guess.

Key takeaways

Give the model the requirement, constraints, technique and an output format; ask it to list ambiguities instead of assuming.
Parse and validate every generation mechanically (JSON parse, JSON Schema, compile, lint) before a human reviews it.
Use synthetic data only, and commit reviewed datasets as fixtures rather than regenerating each run.
Review generated code for real APIs, meaningful assertions, no sleeps and fitting conventions.
Confirm each generated test can fail; generate from the specification to avoid encoding existing bugs.