An AI agent can succeed once and still be unreliable. On AppWorld, a ReAct agent using GPT-4.1 succeeded on 77.4% of runs across five repetitions, yet completed all five runs for only 53.0% of tasks. IBM Research describes this 24.4-percentage-point difference as a consistency gap.
Measuring repeatability
Most evaluations emphasize Mean@k: the average success rate across repeated attempts. The new work also measures Pass^k, the share of tasks completed successfully on every run. Unlike Pass@k, which asks whether at least one attempt works, Pass^k captures whether a user can expect the same result repeatedly.
The source defines the consistency gap as Mean@k minus Pass^k. It argues that this is separate from overall capability: an agent may be strong on average while remaining unstable on particular tasks.
Finding flip-prone decisions
The proposed Consistency Analyzer examines a recorded agent trajectory and identifies decision points vulnerable to change. At each step, it requests multiple completions—five by default—using the existing context. This requires one additional model call per decision step, performed offline, without new tool calls, environment interactions, or a second end-to-end task run.
The method is black-box. It does not require logits, model internals, or special instrumentation beyond the trace. The underlying explanation is that near-tied token probabilities can make choices sensitive to small platform-level variations. Across a long chain of decisions, those local changes can alter the final outcome even when the agent runs at temperature 0.0.
From diagnosis to guidelines
The analyzer feeds its findings into ALTK-Evolve, a system that converts an agent’s past trajectories into reusable guidelines and injects them during inference. Newly identified instability becomes a consistency guideline.
In one AppWorld example, generated guidance told the agent to count checkbox-style markers with a line-anchored regular expression and to verify that a note search returned the intended result. These instructions target recurring decision risks rather than merely recording the details of one task.
Reported evaluation results
On AppWorld test_normal, comprising 168 tasks, the system was evaluated with GPT-4.1 using one baseline trajectory per task and five fresh test runs. Aggregate Pass^5 increased from 53.0% to 69.0%, while Mean@5 rose from 77.4% to 81.0%. The consistency gap therefore narrowed from 24.4 points to 12.0 points without reducing average accuracy.
Pass^5 gains were reported across difficulty levels: +12.2 percentage points for easy tasks, +22.9 for medium tasks, and +14.3 for hard tasks. On related tasks, the improvement was +13.0 points. With gpt-oss-120b, same-task Pass^5 increased from 10.1% to 16.1%, while similar-task performance improved by +8.7 points.
The open-source ALTK-Evolve repository now includes the Consistency Analyzer and consistency-guideline generation. The source also points to a technical report on arXiv and a two-minute demonstration.
Original source: huggingface.co
This article was produced by NeonPulse.today using human and AI-assisted editorial processes, based on publicly available information. Content may be edited for clarity and style.
