Reported operating use · Software engineering and QA
AI test failure triage: Rescale example
AWS describes Rescale using a Strands agent, triggered by Argo on EKS after Playwright test failures, to combine historical failure patterns with logs accessed through MCP. It returns structured findings and remediation recommendations, and Rescale engineers validate the classifications.
Classify failed tests and prepare engineer-reviewed remediation reports
Rescale
Reported production triage of nightly end-to-end test failures; engineers validate the automated first pass.
A published third-party example, not a Nuvepro customer case or evidence of a partnership.
The workflow at a glance
Test failures to engineer-reviewed findings
Illustrative summary of the published account
- Trigger
Failed nightly Playwright tests
Argo on EKS triggers a Strands agent after test failures.
- NextAgent evidence
History and MCP-accessible logs
The agent combines patterns and logs into structured findings and remediation recommendations.
- NextPeople and evaluation
Engineers validate classifications
Rescale engineers check the first pass; a PyTest harness supports live and stubbed evaluation.
What the source reports
AWS describes a Strands agent triggered by Argo on EKS after Playwright test failures. It combines historical failure patterns with logs accessed through MCP and returns structured findings and remediation recommendations. Rescale engineers validate the classifications. A PyTest harness supports live and stubbed evaluation. Automated pull-request generation was deferred.
Reported technology: AWS Strands Agents and Amazon Bedrock. Scope and outcomes are source-reported.
What the enterprise environment would need
The following is a proposed project design for an enterprise doing similar work. Published details are marked; the remaining systems, data and scenarios would be agreed for your project.
Proposed practice environment
Prepare a defensible test-failure diagnosis
Illustrative project design to scope with your enterprise.
- 1 · Set up the environment
Versioned failure fixtures
Pair failed tests, repository snapshots, logs and historical classifications; keep expected causes held out.
- 2 · Preparation and agent contributions
Agent drafts a triage finding
Link cause hypotheses and recommended ownership to available evidence.
- 3 · Independent checks
Independent QA checks held-out causes
Inspect class-specific errors, unsupported code references and missing logs; test any suggested repair separately.
- 4 · People decide acceptance
Responsible engineer accepts or defers
Verify cause, references and routing; retain untested boundaries.
A reviewable evidence package
A structured failure report with evidence, proposed owner and unresolved diagnoses.
If a check fails → withhold acceptance.
A code defect resembles infrastructure failure and a required log is missing. Reject unsupported confident classification.
Failure or owner rejection returns to review
- Hold
- → 2 · Revise work
- → 3 · Rerun independent checks
- → 4 · Owner review
Operating boundary. Read-only investigation; no test edits, automatic repairs, merge or release authority.
- Systems and tools
- Published: Playwright, Argo, EKS, Strands, Bedrock and MCP-accessible CloudWatch logs. Proposed: use an approved test pipeline and read-only repository/log access.
- Data and state
- Proposed: pair versioned test failures with repository snapshots and historical classifications. Keep the expected cause separate from the evidence available to the agent.
- People and handoffs
- Published: engineers validate the first-pass triage. Proposed: a QA owner accepts, corrects or defers findings; merge and release authority remain separate.
- Rules and authority
- Proposed: recommendations cannot alter tests, apply repairs or approve a release. Missing log access must produce an explicit evidence gap.
- Normal and exception cases
- Proposed: include application defects, infrastructure failures, flaky tests, incomplete logs and misleading historical matches.
- Reset and replay
- Proposed: replay the same failures after prompt, model or knowledge changes; retain rejected recommendations and reviewer corrections.
A proposed simulation
An illustrative practice design derived from this work, not an executed assessment.
Triage a failed nightly test, assemble the available evidence and prepare a finding for the responsible engineer.
Review gate: Independently check the cause, source references and assigned owner; make unavailable evidence and unresolved diagnoses visible.
Challenge the result: Make a genuine code failure resemble an infrastructure failure and remove a required log. Reject confident classification unsupported by the remaining evidence.
What needs checking
- Published: a PyTest evaluation harness compares classifications with human triage.
- Proposed: independent engineers qualify held-out causes and inspect class-specific errors, unsupported code references and correct ownership routing.
- Proposed: verify a suggested repair with fresh tests; measure investigation and review effort separately from initial classification speed.
Review environment fidelity, human capability and agent performance separately. The enterprise’s accountable owners decide acceptance of the shared workflow and any operating use.
How an enterprise could build similar work
A proposed build order for an enterprise doing similar work, written by Nuvepro from the published account and our interpretation of the environment and checks above. It is not how Rescale built or operates this work.
- Set up the test pipelineThis is a proposed design for a similar project, not Rescale's build. Use an approved test pipeline with read-only repository and log access, so the agent can gather evidence but cannot change tests or code.
- Version failures and answersPair versioned test failures with repository snapshots and historical classifications. Keep the expected cause separate from the evidence available to the agent, so a grade cannot be earned by looking up the answer.
- Limit authority and expose gapsRecommendations cannot alter tests, apply repairs or approve a release. Missing log access must produce an explicit evidence gap. A QA owner accepts, corrects or defers each finding, and merge and release authority stay separate.
- Cover normal and hard casesInclude application defects, infrastructure failures, flaky tests, incomplete logs and misleading historical matches. Replay the same failures after prompt, model or knowledge changes, retaining rejected recommendations and reviewer corrections, so any regression after a change becomes visible and can be traced.
- Qualify results independentlyIndependent engineers qualify held-out causes and inspect class-specific errors, unsupported code references and correct ownership routing. Verify a suggested repair with fresh tests, and measure investigation and review effort separately from classification speed.
- Run the disguised failure testMake a genuine code failure resemble an infrastructure failure and remove a required log. Reject any confident classification that the remaining evidence does not support, and keep unavailable evidence and unresolved diagnoses visible.
Your systems, data, rules and accountable owners decide the real project. Agree acceptance criteria, escalation owners and operating responsibility with the people accountable for the work before anything runs on live records. Treat the first run as a rehearsal on test records: compare the draft output with an independently prepared answer, and record what the people reviewing it change.
Questions about this example
- What does Rescale use AI for in test triage?
- A Strands agent runs after nightly Playwright test failures, combining historical failure patterns with logs accessed through MCP. It returns structured findings and remediation recommendations. Automated pull-request generation was deferred, so the agent does not apply repairs in this account.
- Who checks the AI's test failure classifications at Rescale?
- Rescale engineers validate the classifications, treating the automated output as a first pass. A PyTest evaluation harness, supporting live and stubbed runs, compares classifications with human triage. Automated pull-request generation was deferred.
- How strong is the evidence for this Rescale example?
- The evidence class is reported use: AWS describes production triage of nightly end-to-end test failures. The account reports operating improvements without disclosing the evaluation sample or observation window, and it is not independently audited. Agreement with human classifications is not defect-detection accuracy.
- What would an enterprise need to build something similar?
- This is Nuvepro's proposed design, not Rescale's setup. It would use an approved test pipeline with read-only repository and log access, versioned failures with historical classifications, recommendations that cannot change tests or approve releases, and independent engineers checking held-out causes and routing.
Shape this around your enterprise
Bring a similar workflow and the people responsible for it. Together, define the records and systems to use, the result your team will build or configure, and how its owners will check it. Nuvepro scopes the simulation, environment and preparation. Agree any existing-agent evaluation, implementation assistance and production responsibilities separately.
Sources and limitations
Curation review: . This is the date this source example was reviewed, separate from source publication dates.
- The supplier account reports operating improvements without disclosing the evaluation sample, observation window or independent audit. Agreement with human classifications is not defect-detection accuracy or proof that a repair works.
- AWS: How Rescale built an AI triage agent with Strands Agents and AI-DLCSource publication: 2026-10-22
More software engineering examples
- AI engineering agent coordination: Deel exampleDeel
- AI model porting and optimization: Deepgram exampleDeepgram and AWS Forward Deployed Engineering
- AI synthetic evaluation: Microsoft exampleMicrosoft
- AI chip verification workflows: OpenAI exampleUnnamed semiconductor company