Evaluation #1: when do robot failures actually begin?

Every robot pilot report leads with a success rate. Say a vendor tells you the robot picked at 94%. You still do not know the two things that decide your economics. What do the failures look like? And when do they begin? This is our first public evaluation. It answers both, on open data, with every number checkable.

The data

We scored the LeRobot PushT dataset [1]: 52 recorded trials of a simple pushing task, where a robot pushes a T-shaped block onto a T-shaped target. We fixed our pass mark before scoring: a peak coverage reward of at least 0.9. By that bar, 25 of 52 trials pass, a 48.1% rate. The median passing trial runs 13.6 s. The median failing trial runs 10.5 s. A real-robot PushT set from Open X-Embodiment [4] scores 8 of 8 successes, but read that with care: those episodes were scored by a different and stricter rule, a reward pulse reaching 1.0, so the two numbers are not comparable. The platform's other 248 of 308 episodes have no automatic pass mark, so they stay unscored. We would rather publish that than invent a number.

Correction: our bar is looser than the benchmark's

An outside fact-check of this article caught a real problem, and we are fixing it in public rather than quietly. Our 0.9 bar is not the benchmark's bar. The PushT environment ends an episode successfully at coverage above 0.95, and its reward is coverage divided by 0.95, so a reward of 0.9 is roughly 0.855 coverage [2]. Ours is the looser test.

It gets starker. The dataset ships its own success flag, and we checked it: it is false for all 206 episodes in the source data. The highest reward any episode reaches is 0.949. Under the benchmark's own criterion, zero of these trials succeed. Our 48.1% exists only because we chose a lower bar. Across the full 206 episodes our bar passes 92, or 44.7%, so the 52-trial subset behaves normally. The arithmetic was never the issue. The label was.

The same applies to the word failure. Peak reward across all 206 episodes runs from 0.813 to 0.949, a narrow band, and no episode reaches the benchmark. So the 27 we call failures are labelled by the same arbitrary bar as the 25 we call passes. Below we keep the words for readability, but read them as stronger half and weaker half of one set of human demonstrations.

What still holds: everything below compares trials that did better against trials that did worse on the same signal, and that comparison does not depend on where the bar sits. We are leaving the numbers visible rather than restating them, because our own buyer's guide warns that a bar set to flatter the result bends against you, and this is what that looks like when it applies to us.

The method: learn success, then flag the first real deviation

The recipe is standard in execution monitoring, in the spirit of Park et al., ICRA 2016 [3]. Build the reference from the passing runs only. At each 0.1 s step, the envelope is the 10th percentile of coverage across those 25 trials. A failed trial gets flagged at the first moment it stays below that envelope for 0.5 s. One rule, two settings, both fixed before we looked at any failure.

Some failed trials never leave the envelope at all. For those, the marker falls back to the coverage peak, the point after which progress never improved. That fallback turned out to be the finding.

Finding: failures come in two shapes

Only 8 of the 27 failures went wrong mid-task. Their median flag lands at 8.0 s, about halfway through the run, or 52.2% of episode time. The range is 5.0 to 11.9 s. The other 19 failures looked normal the whole way. They were flagged at their coverage peak, at a median of 97.8% of episode time. In plain terms: two-thirds of these failures were not doing anything visibly wrong. They ran out of time.

The two shapes need different fixes. Going wrong mid-task is a control problem, and an operator watching a live dashboard could act on it. Running out of time is something else. It points at time budgets, thresholds and task design. A single success rate hides that split completely.

Correction, August 2026: the detector does not work

A second outside check found something worse than the bar problem above, and it is fatal to the method as applied here. We verified it against our own data and it holds.

Put the two numbers below side by side, which we never did. The rule fires on 8 of the 27 trials we called failures, a 29.6% hit rate. It fires on 10 of the 25 we called passes, a 40% false-alarm rate. It fires MORE often on the good trials than the bad ones. A rule that does that has no discriminative power at all. Its precision when it fires is 44.4%, which is worse than the 51.9% you would get by labelling every trial a failure without looking.

There is a reason, and it is structural rather than a tuning problem. Peak reward across these episodes has a mean of 0.89 and a standard deviation of 0.025, and our 0.9 bar sits near the middle of that. We were not separating two populations of good and bad runs. We were splitting one tight population near its median and then asking a detector to recover a boundary that was never there.

Two further claims in the original version do not survive either. We described 19 trials as having run out of time. There is no time limit in this data: episode lengths run from 49 to 246 frames with only one at the maximum, so nothing was truncated by a clock. And these are not robots. The source is a 2D physics simulation driven by a person with a mouse, which our platform records correctly but this article described loosely.

We are leaving the original analysis below so the record is legible, but read it as a worked example of a method failing, not succeeding.

The fix: ask whether anything is predictable at all

Retracting is not enough, so we ran the test we should have run first. Instead of asking when our rule fires, ask a prior question: watching the first part of a trial, how well can you rank which trials end well? The standard measure is AUC, where 0.5 is a coin flip and 1.0 is perfect.

The answer is that nothing is predictable until the end. After watching 40% of a trial the AUC is 0.51, a coin flip. At 95% it is still 0.67. It only reaches 1.0 at 100%, and that is circular, because the outcome is defined by the final value.

We also checked the signals that are not the labelling variable, since a multimodal method is supposed to draw on independent channels. Agent speed and tracking error, measured over the first 40%, score AUC 0.44 to 0.51. The best feature of any kind was simply how long the trial ran, at 0.59. None of that is a signal.

So the conclusion is not that our detector needed tuning. It is that no early-warning method can work on this dataset, and any method that appears to work here is fitting noise. That is a stronger and more useful result than the one we originally published, and it is the test we now run before believing any predictive monitoring claim, including our own.

For a buyer, this becomes a question to ask a vendor selling predictive maintenance or early fault detection: show the curve above for your system. If it only rises at the end, the product is reporting failures, not predicting them.

We measured our own detector too

Run the same rule on the passing trials. It fires on 10 of the 25 at some point. That is a 40% false-alarm rate at this threshold, because passing runs also dip below the line and recover. One caveat we owe you: the envelope is built from those same 25 trials, so this is an in-sample estimate. A held-out set would almost certainly score worse. We publish the number on purpose. Every monitoring rule trades sensitivity against false alarms. An evaluation that hides its own error rate is marketing.

Check our work

A script in the repository regenerates every number and curve from the source data. It cross-checks against the telemetry we ship, and all 27 failure markers reproduce exactly, with zero mismatches. The episodes are browsable in the Episodes view, with video. The dataset is public and the method is stated, so you can disagree with us using the same data.

Limitations, stated plainly

• This is a simulated task with two degrees of freedom. The trials are human demonstrations, not autonomous policy runs. The method carries over. The numbers do not.
• One scored dataset of 52 trials, plus an 8-trial real-robot set. Small n, and we say so.
• The detector watches one signal. Watching several would catch failures this one misses.
• We fixed the threshold in advance rather than tuning it, and the false-alarm rate is measured in-sample, on the same trials that define the envelope.

Sources

1. Dataset: lerobot/pusht on Hugging Face, a conversion of the Diffusion Policy Push-T demonstrations (Chi et al., Robotics: Science and Systems 2023, arxiv.org/abs/2303.04137). 206 human teleoperation episodes.
2. Success criterion and reward definition: huggingface/gym-pusht, gym_pusht/envs/pusht.py. The environment terminates successfully at coverage above 0.95, and reward is coverage divided by 0.95, clipped to 1.0.
3. Execution-monitoring formulation: Park et al., Multimodal execution monitoring for anomaly detection during robot manipulation, ICRA 2016.
4. Contrast set: Columbia PushT real-robot recordings via Open X-Embodiment, scored on the source dataset's own reward pulse.

This is the shape of evaluation we think buyers deserve on their own pilots. Pass mark fixed first. Every trial instrumented. Failure timing analyzed, not averaged away. Error rates disclosed. Everything reproducible from raw data. If the reports on your desk do not look like this, it is worth asking why.