For engineering teams · AI in practice
Measure an AI task through to acceptance
A practical record of model use, repairs and review helps an engineering team compare coding-agent workflows and identify gaps in its cost picture.
A coding task can consume model calls during implementation, another attempt after a failed check, and further work during review. The first completion message captures only one point in that path. A useful comparison follows the work until its outcome is known.
Start with a task whose result someone can assess. For an illustrative API change, require an input field and test valid and missing values. Agree the expected responses before comparing approaches, keeping the task boundary and acceptance checks fixed. The example below contains no benchmark figures.
Figure 1 as text
Record usage during implementation. Checks and independent review assess the work. If review finds a problem, record its repair and the eventual human decision; an unsuccessful task stays in the comparison scope.
Trace the work, including repair
In Devmojos's internal development practice, an operator prepares a bounded brief with relevant context and acceptance checks. Machine checks examine the brief before a worker starts. The worker implements the assigned change in an isolated project copy and reports what it changed and checked.
Independent review and operator verification inform acceptance. If a check or review finds a problem, repair remains part of the same task record. Record the finding, the next attempt and the eventual outcome. A task that is rejected, stopped or abandoned belongs in the declared comparison scope too.
This structure around an agent is a harness: the task instructions, tools, context, handoffs and review that support its work. A passing test provides evidence about the behaviour it checked. People still decide whether the agreed work is acceptable and whether to release it.
Figure 2 as text
Agree the task, gather relevant context, choose an approach, implement defined work, check and independently review, then make the human acceptance decision. Feedback may return to implementation; a non-accepted result may return to task definition.
Build a record with distinct measures
The following is a proposed worksheet for the illustrative API change. It is not a dashboard of measurements already collected by Devmojos. Leave a field unknown when its source is missing.
Figure 3 as text
Read model usage, tools/infrastructure, elapsed time, review/rework and outcome separately. The article's worksheet proposes a fuller cost record; it is not a report of all measurements currently collected.
| Record for the task | Unit or form | Include |
|---|---|---|
| Outcome and acceptance checks | Recorded status and findings | Accepted, rejected or abandoned; the checks and review behind that status |
| Model use | Provider-reported usage categories | Implementation, repair and review calls; identify cache categories where reported |
| Monetary cost | Currency, with billing or allocation basis | Known provider/tool charges and attributable infrastructure cost; mark exclusions |
| Elapsed time | Time from assignment to the declared outcome | Waiting, attempts and review, using consistent start/end definitions |
| Human work | Recorded time and activity | Preparation, review, verification and repair support |
| Rework | Attempts and reasons | Failed checks, review findings, continuations and reopened work |
Tokens, minutes and statuses describe different things. Convert monetary components using an explicit pricing or allocation basis before adding them, and avoid counting the same charge twice. Plan subscriptions or local hardware need an allocation decision; missing cost data is not evidence of zero cost.
Change one part of a comparable task
Context should give the agent the requirements, relevant code and checks it needs. A brief that omits a constraint can create repair work. Additional material should earn its place by helping the task. Anthropic's context-engineering guidance discusses this selection problem.
Model choice also belongs to the task. Capability, response time, operating constraints and the work's difficulty all matter. Compare the resulting attempts and review findings alongside usage. Caching belongs in the usage record where available; a cache-hit percentage alone does not establish the change in a bill.
For a comparison, declare which completed and unsuccessful tasks are included, what remained comparable, and what changed. Record excluded or unknown costs. If one approach produces a usable result but needs more review, the record should make that trade-off visible rather than collapse it into a token total.
What our internal practice establishes
Our internal routing readout includes verified acceptance timing, failed verifications and rework. It does not establish complete monetary cost per accepted feature or repeatable client savings. The fuller worksheet above is a measurement recommendation. Anthropic's harness research is useful related reading, not evidence of Devmojos results.
Bring one recurring engineering task, its current tools and acceptance checks, and the point where review or repair becomes difficult. Devmojos's AI Engineering offer can begin with a process assessment and scoped adoption experiment, helping your team decide what to retain.