When a robot arm reaches across a table, wraps its gripper around a mug, and sets it down on a tray, it looks like a solved problem. Nowadays, clips like that circulate constantly, and they are genuinely impressive.
Then someone swaps the mug for a slightly taller one, moves a lamp, and runs the same instruction again. This time the gripper closes four centimeters short of the handle.
A robot gripper positioned over a coffee cup, the kind of demonstration that looks effortless until conditions change.
Nothing about the model changed between those two runs. What changed was the situation, and that gap between what a system appears to do and what it reliably does is the central problem in evaluating a vision language action model, or VLA.
These models take in what a robot sees, read an instruction written in plain language, and output the movements needed to carry it out. Testing one properly means finding out how often that works, not whether it can work once.
That turns out to be much harder than testing a language model. A text benchmark can be rerun a thousand times for the cost of some compute, and the questions stay identical every time.
But a robot has to be measured against gravity, friction, lighting, and sensor noise, and each attempt needs a human or a script to physically reset the scene before the next one begins.
A wrong answer from a chatbot is an inconvenience. A failed grasp on a factory floor is a dropped part, a damaged component, or a safety incident.
In this article, we’ll explore what a vision language action model is, why evaluating one is so hard, how robotics simulation helps, and which benchmarks teams are using today. Let’s get started!
A vision language action model is a model that takes a robot from perception all the way to motion. It receives what the robot’s cameras see, an instruction written in everyday language, and the robot’s own physical state, then produces the movement commands needed to carry that instruction out.
Each input does different work. Camera observations show what is in front of the robot and where. The instruction says what to do, in words a person would actually use, like “pick up the red mug and place it on the tray.”
Meanwhile, robot state, also called proprioception, tells the model where its own joints are and whether the gripper is open. Without it, the model would know what the scene looks like but not where its arm sits inside it.
What comes out is a sequence of commands called an action chunk, usually covering eight to fifty future steps. The robot runs through the chunk while the model computes the next one, which keeps a reach, a grasp, and a lift moving as one motion rather than a stutter.
A vision language action model takes camera input, an instruction, and robot state, and outputs actions the robot can execute.
VLA models have gained traction in robotic applications recently because of what sits in the middle. Robot policies used to be built one task at a time. For example, if you wanted an arm to sort bolts, an engineer trained something specific to sorting bolts, and that work transferred to nothing else.
A VLA model replaces that with a pretrained foundation model backbone, the same kind used for describing images, paired with a smaller component that turns its output into robot commands. The backbone arrives already knowing roughly what a mug is, and demonstration data teaches it how to move.
The robot foundation model space has moved quickly, with new releases arriving every few months from research labs and product teams alike.
Here’s a closer look at the vision language action models shaping the field right now:
Every one of these is built around generality, meaning one model operates across many tasks and many robots. That is also what makes them hard to evaluate.
A task-specific policy could be tested on its one task. A general model has no natural boundary, so any test you design covers a small slice of what it claims to do.
When you test a VLA model, you aren’t testing just the model. You are testing the controller that turns its output into motor commands, the camera setup feeding it images, and the scene it works in.
A poor result might mean the policy is weak, or it might mean a camera sits ten centimeters from where the model expects it. Benchmarks measure the whole stack.
Within that stack, teams look at several dimensions, including whether the task gets completed, how the model handles unfamiliar objects and lighting, whether it follows instructions phrased in new ways, and how it recovers when something goes wrong.
Then come the practical limits. Every episode runs in real time and needs a physical reset before the next one, so a hundred trials becomes a day of work rather than a compute job. Coverage stays narrow as a result, with most models tested on one arm in one fixed scene.
A Look at the Challenges of Evaluating a VLA Model
Beyond this, scoring is its own problem. Under pass or fail, a task that breaks down at step nine looks identical to one that fails immediately, which is why teams score long tasks phase by phase instead.
Borrowed metrics mislead too, since strong scores on general vision and language benchmarks don’t reliably predict how a model performs on manipulation. These vision language action model constraints make thorough physical testing slow and expensive, which is why most teams do as much of their evaluation as possible before a robot ever moves.
The constraints we just discussed are why most vision language action model evaluations now start in simulation. A simulated scene resets instantly, runs without supervision, and costs nothing when a grasp fails.
Two environments dominate this work. MuJoCo, maintained by Google DeepMind, is widely used for modeling contact and control, while NVIDIA’s Isaac Sim pairs GPU-accelerated physics with photorealistic rendering.
Both now support GPU backends that run thousands of parallel environments on a single graphics card, which means variations in object position, lighting, and clutter can be tested at a scale no physical lab could match.
Simulation lets teams test a dual-arm robot cell before touching hardware, with the simulated scene matched to the real one. (Source)
The catch is the sim-to-real gap. Results in simulation don’t automatically hold on hardware, and the reason is rarely just physics.
Rendered images look different from camera footage. Real sensors add noise and lag. Timing that is exact in simulation drifts on a real controller. A policy can also learn to exploit quirks of the simulator that have no counterpart in the world.
Setup matters just as much. If your simulated cameras, joint layout, or action format don’t match what the model expects, you end up measuring your own integration work rather than the model. That is why most teams treat simulation as a filter rather than a verdict, using it to narrow a field of candidates and then running physical trials on the ones that survive.
There is no single standard benchmark for VLA models yet, in the way that language models have a shared set of tests everyone reports against.
Part of the reason is hardware. A vision language action model benchmark built around one arm with one gripper says little about a model running on a different robot, and cross-embodiment transfer is its own problem.
An example of how robots are often tested, with one arm working through instructions in a fixed tabletop scene. (Source)
The rest is protocol. Each vision language action model benchmark ships its own observation format, task definitions, and scoring rules, so teams end up maintaining separate evaluation code for each one and results rarely line up.
What exists instead is a set of simulation suites that most published numbers come from. Here’s a closer look at the ones that appear most often:
Real-world evaluation is harder to standardize. FurnitureBench is one of the more deliberate attempts, using 3D printable furniture parts and a documented setup so other labs can reproduce the same assembly tasks, and it ships with over 200 hours of teleoperation data.
LeRobot, meanwhile, has become a common software layer for running policies on affordable hardware rather than a benchmark itself. Alongside this, the simulation numbers keep climbing.
Several leading models now report LIBERO scores at or near 100 percent, which says more about the benchmark being saturated than about manipulation being solved. That is why a credible report states how many trials were run, what varied between them, and how partial progress was scored. Without a trial count, a success rate is a claim rather than a result.
It’s easy to think of a benchmark as software, but every one of them rests on data that people collected and labeled. For instance, LIBERO’s task suites are built on roughly 50 demonstrations per task.
Also, FurnitureBench ships with more than 200 hours of teleoperation recordings. Someone operated those robots, someone decided which attempts counted as successful, and someone defined where one subtask ended and the next began.
That is where evaluation and data annotation turn out to be the same discipline pointed at different ends of the vision language action model lifecycle. Training data needs a rule for what a successful grasp looks like, and so does a benchmark.
Both need consistent boundaries between the phases of a long task. The judgment calls are identical, and so are the failure modes. If two annotators disagree about whether a mug was placed correctly, two evaluations of the same model will disagree too.
Simply put, you can’t do one of these well without understanding the other, which is why we, at Objectways, spend time on both.
To understand what evaluation really involves, our team recently worked with MolmoAct2-BimanualYAM, a current bimanual VLA model, running it against a simulated YAM (Yet Another Manipulator) robot setup.
It was part of a broader internal effort to build vision language action model evaluation infrastructure that holds up across physical AI projects rather than just one model.
Here’s a closer look at what that involved:
Working through this reinforced something we see on the data side every day. Deciding what counts as a usable action, where a task begins and ends, and what a correct setup looks like are the same judgment calls that go into writing good labeling guidelines.
That’s why our robotics work spans perception annotation, demonstration data collection, scenario design, and human-in-the-loop review, rather than treating any one of them as separate.
Building a vision language action model evaluation setup gets easier with a few principles in place. The first is to treat evaluation as infrastructure rather than a milestone. Models change every few months, so a setup you can rerun on the next one is worth far more than a single result produced at the end of a project.
From there, reporting is just as important as testing. Document how many trials were run and what varied between them, because a success rate without those numbers tells a reader very little about reliability.
Good evaluation is built and maintained like infrastructure, not run once at the end of a project.
Along the same lines, check that your simulation matches what the model expects, from camera placement to joint layout to action format. When those don’t line up, you end up measuring your integration work instead of the model.
It’s also worth budgeting for both kinds of testing rather than choosing between them. Simulation narrows a field of candidates quickly and cheaply, while physical trials are what confirm a result actually transfers to hardware. Most teams need both, and planning for only one usually means discovering the gap late.
On top of this, apply your strictest quality control to evaluation data specifically. Noisy training data slows learning down, and more data can partly make up for it. Noisy evaluation data produces confident numbers that are quietly wrong, and no amount of extra testing will catch that.
Right now, vision language action model capability claims are moving faster than the industry’s ability to check them. A model can be announced, demonstrated, and written about long before anyone outside the team that built it knows how often it actually works.
VLA benchmarks are the correction to that, and the recent push toward shared task suites, trial counts, and phase-level scoring is what turns an impressive clip into a result someone else can rely on.
That correction depends on data. Every benchmark, every scoring rule, and every judgment about whether a grasp succeeded traces back to people who collected demonstrations and labeled them carefully. As vision language action models move from research labs into warehouses and factory floors, the teams that can measure their own systems honestly will be the ones that deploy with confidence.
Working on a physical AI or robotics project? Objectways supports the full physical AI data pipeline behind evaluation, from demonstration collection and perception annotation to scenario design and human-in-the-loop review. Get in touch and let’s talk about the data behind your VLA model evaluation setup.
VLA stands for vision language action. It describes a robot control policy that takes in camera images, a plain language instruction, and the robot’s own joint state, then outputs the motor commands needed to carry that instruction out.
Explore AI data governance and how quality, security, privacy, and availability in training data define the reliability and compliance of next-gen AI systems.
Explore what NVIDIA GTC 2026 revealed about physical AI, the data gap in robotics, and how egocentric data improves real-world performance.
