How do I build an agent to test a website?

What you need before you build

An agreed target and boundaries

Name the staging application, permitted journeys and prohibited actions. Use dedicated test identities and synthetic or approved masked records. Decide how to prevent real payments, customer messages or production changes.

A reference for correctness

Have the product owner and QA lead agree expected outcomes, approval rules and known exceptions. Include access to suitable test APIs or state inspection where the visible page cannot prove the result.

A reproducible starting point

Record the application build, browser/tool versions, test data and setup/reset procedure. Assign ownership of fixtures, accounts and dependencies so later runs can be compared meaningfully.

Build and check the testing agent

1. 1. Turn a work outcome into a test contract

   Choose one journey, such as creating a supplier application and sending it for review. State who performs it, the required inputs, allowed state transitions and expected downstream effect. Saving a draft and approving a supplier are different outcomes, even if both display a success message.

   Write acceptance criteria before generating tests. Include an ordinary case, missing evidence, a conflicting record and a role boundary. Agree the evidence each criterion needs and what should happen when the application is unavailable or the case cannot be completed.

2. 2. Separate the agent from the test executor and checks

   Give the agent a specific job: explore the approved journey, propose a plan, generate test code or investigate a failure. Playwright documents separate planning, generation and healing agents. These are useful starting mechanisms, not evidence that their output matches your business requirements.

   Keep reviewed assertions outside the agent's discretion during an acceptance run. A browser executor performs actions; assertions compare observable results with expected behavior; the agent explains or investigates. A narrative saying 'the flow worked' must not replace those comparisons.

3. 3. Build the environment and reset procedure

   Create fixtures for relevant users, supplier records, required documents and catalog state. Decide which external services are real staging dependencies and which are controlled substitutes. Record those substitutions so nobody treats a mocked integration as a verified production connection.

   Isolate each case's storage and data, restore the starting state and verify cleanup. Playwright's best-practice guidance supports independent tests and user-visible behavior. Your enterprise also needs a clear owner for shared records that browser-session isolation alone cannot reset.

4. 4. Implement observable assertions and useful failure evidence

   Use stable user-facing locators and condition-based assertions instead of arbitrary delays. Playwright provides assertions that retry until the expected condition is met or timeout occurs. Choose the condition deliberately: a toast is not proof that a supplier remains ineligible downstream.

   Capture the case identifier, starting data reference, steps, expected and actual behavior, build and relevant trace. Playwright traces support investigation of browser actions and page state. Add approved state checks when business correctness depends on records outside the current screen; protect sensitive artifacts and access.

5. 5. Challenge the testing system with known failures

   In a controlled test build, introduce reviewed defects or replay previously confirmed bugs. Check whether the tests identify the actual failure, distinguish it from an unavailable dependency and provide enough evidence for another engineer to reproduce it. Also run known-correct cases to expose false alarms.

   An agent that heals a locator may improve maintenance, but changing an assertion can conceal a product regression. Require a reviewed diff and the original failure evidence before accepting repairs. A skipped, blocked or inconclusive case must remain visible rather than count as a pass.

6. 6. Review the artifact and operating handoff

   Produce a reviewed test plan, versioned tests, fixture/reset instructions and a defect or coverage report. Compare execution effort, model/tool calls, repeated failures and human triage effort with your existing approach. More generated tests are not automatically more useful coverage.

   The QA owner accepts the evidence and remaining gaps; the engineering owner maintains the runner, credentials and CI integration. Decide who can update prompts, tools or assertions and which cases must be replayed after changes. Release acceptance remains a separate decision.

Illustrative worked example

Fictional artifact: a supplier-portal test contract

The portal accepts supplier applications for review. Creating an application does not authorize supplier approval.

The work product is the testing agent and its reviewed tests, fixtures and evidence. The table supplies a design target; execution against your application is still required.

What to check

Independent expected results

QA and domain owners approve assertions. Test them with known correct and incorrect application behavior; do not let the executor alone decide whether its work passed.

Missed defects and false alarms

Review both. Track untested conditions and unavailable dependencies separately, including whether repeated runs reproduce the finding.

Changes that hide failure

Inspect automatically healed selectors, weakened assertions, retries and skips. Preserve the original failure and require a reviewer to explain the repair.

Make this real in your enterprise

Scope and limitations

• A functional test does not establish security, accessibility, load behavior or correctness across every browser and integration. Add the relevant specialist checks to the agreed scope.
• Results apply to the versions, cases and conditions exercised. Revisit coverage when requirements, application state, models or tools change. This guide supplies design steps and official references, not verified runnable code or a qualified agent.

Questions about this project

Sources and further reading

Check current product support, permissions and licences for your project. References do not imply a Nuvepro partnership or available connector.

1. Playwright Test Agents

   Documents planning, test generation and test repair as distinct agent roles.

2. Playwright assertions

   Official assertions and retry behavior; business criteria still need review.

3. Playwright best practices

   Official guidance on user-visible behavior, test isolation and resilient tests.

4. Playwright trace viewer

   Official execution inspection. Traces support review; they do not establish correctness.