Limited published evidence · AI engineering and evaluation
AI agent evaluation task design: BrendanFoody account
In a September 2025 article on X, BrendanFoody describes participants creating rubrics for an agent's tool-use mistakes. Fewer could challenge it six months later; after tools/tasks expanded, all 100 could contribute again. These are reported observations under changed conditions, not a controlled benchmark or demonstrated training loop.
Create evaluation rubrics and expand tool-use tasks
BrendanFoody evaluation-project account (organization unnamed)
Author-reported historical evaluation project in a September 2025 article. Contributor observations under changing conditions, not a controlled benchmark, named customer deployment or demonstrated RL training loop.
A published third-party example, not a Nuvepro customer case or evidence of a partnership.
The workflow at a glance
Reported contributor observations under changing task conditions
Illustrative summary of the published account
Reported work components. Their arrangement does not imply a fixed execution order or simultaneous work.
- Author-reported initial project
100 participants find tool-use failures and create rubrics
The author describes 100 participants finding a frontier agent’s tool-use mistakes and creating evaluation rubrics. Initially everyone could challenge the model; raw artifacts were not inspected.
- Author-reported later observation
Six months later, 20 could still challenge the model
The count concerns contributors able to challenge the model, not an accuracy rate, task count or customer outcome. Model and task versions are undisclosed.
- Author-reported expanded scope
100 contribute again after tool and task expansion
The team reports adding tools and tasks requiring more than ten hours of human work; all 100 could contribute again under those changed conditions. This is not a controlled comparison or agent-runtime measurement.
What the source reports
BrendanFoody describes a project where 100 participants found mistakes made by a frontier agent using a tool and created evaluation rubrics; initially everyone could challenge the model. Six months later, the account says only 20 people could still challenge it. The team then added tools and pursued longer-horizon tasks described as taking humans more than ten hours to complete; the account says all 100 participants could contribute again. These are three reported observations linked by narrative chronology under changed tool/task conditions. The counts concern contributors, not accuracy rates or customers, and the time qualifier concerns human task effort. The account establishes rubric creation and task/tool expansion, without establishing revision of those rubrics, a source approval workflow, independently checked results or a model-weight training loop.
Source accounts: @BrendanFoody
Reported technology: Frontier agent; model and tool identities undisclosed. Scope and outcomes are source-reported.
What the enterprise environment would need
The following is a proposed project design for an enterprise doing similar work. Published details are marked; the remaining systems, data and scenarios would be agreed for your project.
Proposed practice environment
Build a tool-use task whose success can be checked
Illustrative project design to scope with your enterprise.
Checks designed, not executed. Domain owners must agree the reference results, permissions and acceptance tests before implementation.
- 1 · Set up the environment
Fictional case state and independently agreed target
A domain owner defines a tiny case-board task, allowed stub tools, expected final record/note and versioned rubric. Agree the reference before any evaluator is configured.
- 2 · Task author and test agent
Prepare the task and record a bounded attempt
The task author defines fixtures and permissions; the test agent may propose tool actions. Preserve tool events and resulting state instead of trusting its completion message.
- 3 · Independent checks
A separate checker compares actual state with the reference
Check case identity, owner change, preserved fields, required note, allowed actions and held-out variants. A textual done claim or rubric score alone cannot prove success.
- 4 · People decide acceptance
Domain owner accepts only checked task evidence
Review the independent result, unresolved gaps and exact task/tool/rubric versions. Any changed scope needs new checks; acceptance grants no live access, readiness certification or model-training claim.
A reviewable evidence package
A versioned fictional task, rubric, expected state, observed attempt evidence, mismatch report and bounded owner decision.
If a check fails → withhold acceptance.
The agent says done and receives a favorable rubric score, but the required owner field never changes. Independent state checking rejects success and holds acceptance.
Failure or owner rejection returns to review
- Hold
- → 2 · Repair the task or rubric against the reference
- → 3 · Rerun independent checks
- → 4 · Owner review
Operating boundary. Fictional records and isolated stub tools only. No live system writes, production authority, customer outcomes, RL optimizer or model-weight update. Checks are designed, not executed.
- Systems and tools
- Published: participants evaluated a frontier agent using a tool and later expanded tools/tasks; identities and implementation are undisclosed. Proposed: isolate a fictional case-board fixture with read_case, set_owner and append_note stubs, with no network or live enterprise connection.
- Data and state
- Proposed: pin fictional C1 with owner team-a, status open and no notes; the separate expected state changes owner to team-b and appends exactly one fixed R1 review note. C2 and unrelated fields stay unchanged. Version the request, starting state, target, allowed tools, task, model configuration and rubric before any attempt.
- People and handoffs
- Published boundary: contributors found mistakes and created rubrics; the project organization, job roles and acceptance owner are not disclosed. Proposed: a task author prepares fixtures, a bounded test agent attempts permitted work, a separate checker owns the reference comparison, and a domain owner reviews the exact checked evidence.
- Rules and authority
- Proposed: agree reference semantics and permissions with a domain owner before implementation. Allow only scoped fictional reads and C1/R1 owner/note updates through the stubs. Deny deletion, unrelated record changes, network/live writes, reference edits by the task agent and self-issued approval. No model-weight update, production authority or readiness certification is granted.
- Normal and exception cases
- Proposed: design eight scenarios for successful work, missing evidence, conflicting instructions, stale input or approval, duplicate delivery, false success despite unchanged state, unauthorized actions and recovery. Agent text and favorable rubric scores cannot substitute for persisted-state and permission checks. Eight scenarios are designed and zero executed; domain agreement is pending.
- Reset and replay
- Proposed: retain failed evidence separately, restore the agreed fictional starting state and rerun independent state, permission and held-out checks after repair or owner rejection. Changed task, tool, model or rubric versions invalidate earlier checks and decisions. Return to fresh exact-version owner review; never rewrite past failure as success.
A proposed simulation
An illustrative practice design derived from this work, not an executed assessment.
Proposed: prepare a versioned fictional tool-use task and rubric, record a bounded attempt, and expose false success by comparing actual state and tool events with a separate reference.
Review gate: Proposed: after domain agreement, independently check the saved state, required note, allowed actions and exact versions; a human domain owner can accept or reject only that fictional evidence packet. Failed checks or owner rejection require repair, independent rechecks and fresh owner review. This grants no production or model-training authority.
Challenge the result: Proposed: keep the agreed target owner at team-b but plant a false-success packet where the agent says done, a rubric awards success and actual C1 owner remains team-a. The separate state reference must reject it; preserve the mismatch, hold acceptance, repair, reset and independently recheck before owner review.
What needs checking
- Published boundary: three contributor observations and two narrative chronology relationships are supported as reported. No operational arrows, source approval procedure or causal model-training effect is established.
- Proposed: a separate checker compares actual saved C1/C2 state, exactly one R1 note, request identity and allowed tool events with the reference agreed independently of the executing agent. The checker must inspect actual effects rather than trust a done message or self-score.
- Proposed: keep missing/conflicting evidence, duplicate identities, stale versions and denied actions visible. A changed task/tool condition is a new condition, not evidence of a controlled performance improvement.
- Proposed: hold failed or rejected work, preserve its history, repair or obtain domain adjudication, independently recheck from the agreed start and then return to owner review. Eight acceptance scenarios are designed, zero executed, and domain agreement is pending. Checks designed, not executed.
Review environment fidelity, human capability and agent performance separately. The enterprise’s accountable owners decide acceptance of the shared workflow and any operating use.
How an enterprise could build similar work
A proposed build order for an enterprise doing similar work, written by Nuvepro from the published account and our interpretation of the environment and checks above. It is not how BrendanFoody evaluation-project account (organization unnamed) built or operates this work.
- Agree a fictional task and its authorityChoose one observable case-board request with a domain owner. Define the starting records, requested owner change and required note, the people allowed to act, and the boundary between a proposed action and acceptance. Treat these as fictional semantics requiring agreement, not a reconstruction of the author’s undisclosed project.
- Freeze a separate expected-state referenceBefore configuring an evaluator, have someone other than the task-executing agent prepare the reference: C1 changes from team-a to team-b, remains open and contains exactly one fixed R1 review note. C2 and unrelated fields stay unchanged. Pin the reference, request identity, tool rules, model configuration and rubric versions.
- Expose only bounded stub tools and retain evidenceBuild isolated read_case, set_owner and append_note stubs for the fictional scope. Deny unrelated record changes, deletion, network access, live-system writes and reference edits by the task agent. Save actual state and tool events alongside any narrative response; no environment or task attempt has been implemented for this example.
- Challenge false success with independent checksUse a separate checker to compare persisted state and allowed events with the fixed reference. Include a convincing done message and favorable score while C1 still belongs to team-a: that mismatch must hold acceptance. Also design missing, conflicting, stale, duplicate and unauthorized cases. The eight scenarios remain unexecuted.
- Return repaired work through checks and owner reviewPreserve failed evidence or owner rejection, repair the task or rubric against the agreed reference, and restore the fictional starting state. Rerun independent state, permission and held-out checks before a fresh exact-version owner decision. Acceptance covers only the rehearsal evidence, not production operation, human readiness or a training result.
Your systems, data, rules and accountable owners decide the real project. Agree acceptance criteria, escalation owners and operating responsibility with the people accountable for the work before anything runs on live records. Treat the first run as a rehearsal on test records: compare the draft output with an independently prepared answer, and record what the people reviewing it change.
Questions about this example
- What work does BrendanFoody describe in this AI evaluation example?
- In an article on X, BrendanFoody describes participants finding a frontier agent’s tool-use mistakes and creating evaluation rubrics. The team later added tools and longer tasks. These are reported project observations; raw tasks, rubrics, model configurations and execution traces were not inspected.
- Do the 100, 20 and 100 figures measure model accuracy?
- No. The figures concern contributors able to challenge the model or contribute under changed tool/task conditions. The six-month interval is relative, and more than ten hours describes human task effort. The account does not establish a matched benchmark, agent runtime or measured speed gain.
- Who approved the work in the reported project?
- No review step stated. The account names participants finding mistakes and creating rubrics, but does not disclose an acceptance owner or approval procedure. The project organization is unnamed. The separate checker and domain owner below belong to Nuvepro’s proposed fictional design, not the source project.
- What would an enterprise need to build something similar?
- Nuvepro’s proposal is a fictional case-board task with pinned starting state, a separate expected owner/note reference, narrowly permitted stub tools, independent state and event checks, and exact-version owner review. Eight acceptance scenarios are designed and zero executed; domain agreement is pending. It grants no live access, production authority or model-weight training claim.
Shape this around your enterprise
Bring a similar workflow and the people responsible for it. Together, define the records and systems to use, the result your team will build or configure, and how its owners will check it. Nuvepro scopes the simulation, environment and preparation. Agree any existing-agent evaluation, implementation assistance and production responsibilities separately.
Sources and limitations
Curation review: . This is the date this source example was reviewed, separate from source publication dates.
- This is one first-person project account in a September 2025 article, not a named customer implementation, independent operating result or current model-performance claim. The project organization is unknown; the author profile does not identify it.
- The complete article capture and relevant original clauses were read on 1 November 2026 and reused for curation on 2 November 2026. This is prior captured-original review, not a fresh X scan; the source publication date remains 15 September 2025.
- 100, 20 and 100 concern contributor participation under different task/tool conditions, not accuracy percentages, task counts or customer counts. Six months is relative; more than ten hours describes human task effort, not observed agent runtime or a measured speed gain.
- Actual task/rubric artifacts, execution traces, model/tool identities and versions, matched comparison conditions, permission enforcement and source acceptance procedures were not inspected. Rubric creation does not establish revision of existing rubrics.
- The broader article discusses RL and richer work environments, but the project anecdote does not demonstrate an RL optimizer, reward calculation, model-weight update, controlled comparison or training-loop implementation.
- The fictional fixture, roles, reference oracle, held-out variants and acceptance rules are proposed designs requiring domain agreement. No environment or business scenario has been implemented or exercised; no task correctness, human readiness or production authorization is established.
- BrendanFoody: The Economy will become an RL Environment MachineSource publication: 2025-09-15
More software engineering examples
- AI engineering agent coordination: Deel exampleDeel
- AI model porting and optimization: Deepgram exampleDeepgram and AWS Forward Deployed Engineering
- AI synthetic evaluation: Microsoft exampleMicrosoft
- AI test failure triage: Rescale exampleRescale