How to run your own robot evaluation: our full methodology, free
This is our whole evaluation method, given away. It is not complicated, it needs no special software, and once you have read it you could run it yourself with a spreadsheet.
We are aware that publishing it looks odd for a company that sells evaluations. Our view is that the readers who try it will discover which parts are hard, and the ones who never try were never going to hire anyone anyway.
Step 1: write the pass mark down before you start
Decide what counts as success, in numbers, and record it before a single trial runs. Pick rate, cycle time, uptime, how often a person has to step in. Whatever matters to you.
This sounds obvious and it is the step people skip. A bar set after everyone has watched the robot run will drift toward the result the room wants. There is no malice needed for this to happen.
Use the accepted bar for your task if one exists. We will show you in a moment what happens when you do not.
Step 2: record every trial, especially the bad ones
Log every run, not the good ones. Failures carry almost all of the diagnostic value, and a log with the embarrassing runs deleted cannot tell you anything you did not already believe.
Set afterwards, it bends toward the result you want.
Failures carry the diagnostic information.
Without a time series you can only count, not diagnose.
Needed to tell a mid-task failure from running out of time.
An unrecorded intervention turns a failure into a success.
So the numbers can be recomputed and disputed later.
The single most valuable item on that list is the timestamped progress signal. Without a time series you can count outcomes, which tells you a robot failed. With one you can see when it started failing, which tells you why.
Step 3: build the reference from the runs that worked
Take your successful trials and, at each moment in time, find the level of progress that most of them beat. We used the 10th percentile across the passing trials, so the reference line is the pace a slow-but-successful run keeps [1].
This is a standard idea in execution monitoring, and it is decades old [4]. Model what good looks like, then watch for departures from it.
Step 4: flag the first sustained deviation
A trial is flagged at the first moment it falls below that reference line and stays below it. We required half a second of persistence, so one noisy reading does not trigger an alarm [1].
Fix both numbers, the percentile and the persistence, before you look at the failures. Otherwise you are tuning the detector to produce the story you expected.
Step 5: measure your own detector, and publish that too
Run the same rule against your successful trials. Every time it fires there, it is a false alarm. Ours fired on 10 of 25, a 40% rate, and we published it [1].
Be honest about one more thing. Our envelope was built from the same 25 trials we then measured against, so that figure is in-sample and a held-out set would score worse. A monitoring system that reports no error rate is not reporting confidence, it is withholding it.
What this produced, on real data
On 52 recorded trials of a 2D pushing task, taken from an open dataset of human demonstrations [2], 25 cleared our bar. Of the 27 that did not, the rule flagged only 8 mid-task. The other 19 were marked at their peak instead [1].
We originally called that split the payoff of the method. Given the correction above, it is not: with the rule firing more often on passing trials than failing ones, the split cannot be read as a finding about failure modes. It is what the rule produced, not what the robots did.
Now the part we got wrong
We chose a pass mark of 0.9 on the task's reward scale. The benchmark's own bar is coverage above 0.95, and because reward is coverage divided by 0.95, our 0.9 works out near 0.855 coverage [3]. Ours was the looser test, and we did not say so at first.
Scored at the benchmark's bar, the same trials produce zero passes. No episode in the source data ever exceeds a peak reward of 0.949, and the dataset ships its own success flag which is false for all 206 episodes [3]. An outside check caught this, and we corrected it in public rather than quietly.
We include it here because it is the most useful thing in this article. Step 1 is not advice we are handing down from a position of having always got it right. It is the step we ourselves got wrong, and the figure above is what that mistake looks like when you draw it.
Correction: our worked example fails its own test
After publishing this, a second check found that the detector described in steps 3 and 4 does not discriminate on our data. It fires on 40% of passing trials and only 29.6% of failing ones. Firing more on the good runs than the bad ones means it carries no signal, and its precision is below what you would get by calling everything a failure.
The cause is upstream of the detector. Peak performance in this dataset is one tight cluster, mean 0.89 with a standard deviation of 0.025, so our pass mark cut a single population near its middle rather than separating two. Step 5 is what caught it, eventually. Step 5 done properly, comparing the false-alarm rate against the hit rate rather than reporting it alone, would have caught it immediately.
So take the five steps, and take this with them: a detector that fires more often on your good runs than your bad ones is not a detector. Compare the two rates. We reported ours side by side only after someone else did it for us.
Step 6, which we learned the hard way: prove the detector beats a coin flip
Before trusting any early-warning rule, including your own, check two things. First, does it fire more often on bad runs than good ones? If not, it carries no signal, whatever its hit rate looks like alone.
Second, and better, measure whether the outcome is predictable at all. Watch the first part of each trial, rank the trials by whatever feature you like, and compute the AUC against what actually happened. On our data that number sits at a coin flip for most of the trial. When we ran it properly, the answer was that nothing could be predicted early, so no rule could have worked.
This is the step that turns an evaluation into evidence. Without it you cannot tell a detector from a coin, and we could not either.
What is actually hard
- Getting a usable signal out of the robot at all. Many systems expose a dashboard and no export.
- Holding the pass mark when the number comes back badly and the room wants it moved.
- Running enough trials for the answer to mean anything. Ours is 52, which is small, and we say so.
- Covering peak conditions, not just a quiet Tuesday.
- Resisting the temptation to tune the detector until the story looks tidy.
Why give it away
Because the alternative is what the sector has now. In the largest automation failure on record, neither the operator nor the vendor ever published a single throughput or utilisation figure [6]. Buyers are asked to spend hundreds of millions against demonstrations, and buyers say they want to see a system running before they commit [5].
A market where more buyers measure their own pilots is a better market for anyone selling honest evaluation, and a worse one for anyone selling a demo reel. Take the method. If you get stuck at the parts listed above, that is the point at which people call us.
Sources
- Robot Eval, Evaluation #1: when do robot failures actually begin? (this site). The worked example, the detection rule and the correction described here all come from that evaluation. Every figure in it regenerates from scripts/build-eval-report.py, which cross-checks against the source data and fails on any mismatch.
- Dataset: lerobot/pusht on Hugging Face, a conversion of the Diffusion Policy Push-T demonstrations (Chi et al., Robotics: Science and Systems 2023, arxiv.org/abs/2303.04137). 206 human teleoperation episodes.
- Success criterion and reward definition: huggingface/gym-pusht, gym_pusht/envs/pusht.py. The environment terminates successfully at coverage above 0.95, and reward is coverage divided by 0.95, clipped at 1.0. We read the source data directly: no episode exceeds a peak reward of 0.949, and the dataset's own success flag is false for all 206 episodes.
- Execution-monitoring formulation: Park et al., Multimodal execution monitoring for anomaly detection during robot manipulation, ICRA 2016. Build the reference model from successful executions, then flag the first sustained deviation.
- Interact Analysis Mobile Robots Buyer Survey, July 2024 (300 buyers): buyers expect to see a system in action, through a demonstration, a reference site visit or a supplier demo center, before purchasing. Via The Robot Report, therobotreport.com/what-do-customers-expect-from-mobile-robots
- Robot Eval, The true cost of a robot that doesn't work (this site), on Kroger's $2.5 billion write-off and the absence of any published operating metric in the sector's largest automation failure