A robot can repeat the same task thousands of times in simulation, but that doesn’t mean it can handle the same task on a real factory floor. That difference is becoming clearer to the robotics companies building systems to collect real-world robot data.
Recreating a factory floor in software is easier than recreating what happens on one.
In June 2026, Apptronik opened a newly expanded robot facility in Austin and called it Robot Park. It spans nearly 90,000 square feet, where Apollo 2 humanoids work through logistics, manufacturing, and retail tasks throughout the day. The building was once a Dell server plant.
Two months earlier, Tutor Intelligence had moved 100 semi-humanoid robots into a former cotton mill in Watertown, Massachusetts, with 45 to 50 remote supervisors in Mexico and the Philippines handling teleoperation.
A Look at a Robot Data Factory (Source)
Every physical demonstration runs in real time, needs a reset, and wears down the hardware. A simulator, by comparison, can run thousands of environments in parallel for the cost of GPU time. Yet these companies are choosing the expensive version anyway.
It comes down to the sim2real gap (simulation to reality). A policy trained in simulation learns the simulator’s version of the world. On a real robot, friction can change, sensors can be noisy, and an action that worked in simulation can produce a different result.
Better physics engines and cheaper compute have narrowed the sim2real gap, though not evenly. Simulation now transfers well for locomotion, navigation, and much of perception.
But it still struggles when a gripper touches an object, and the outcome depends on friction, slip, or deformation. In fact, contact is where most of the new physical AI infrastructure – the data facilities, the teleoperation fleets – is being pointed.
In this article, we’ll look at where the sim2real gap exists, how the robotics industry is compensating, and what that means for robot data collection. Let’s get started!
The sim-to-real gap is usually described as the difference between a simulator and the real world. That is true, but it doesn’t tell us much about why a robot can work in simulation and still fail on hardware. So, it is more useful to treat the sim2real gap as three separate mismatches, because each one needs a different fix.
Here’s a look at what the sim2real gap is composed of:
Three standard fixes can be used to work around these sim2real gap problems. For example, domain randomization assumes you don’t know the real parameters, so it trains across a range wide enough to include them, at the cost of a more cautious policy.
System identification, by contrast, measures the real robot and tunes the simulator to match, though that has to be redone for every new machine. Finally, augmentation and style transfer make rendered images look more like camera footage, while the physics stays untouched.
A research team at ETH Zurich showed how far the second approach can go. They trained a neural network to model the ANYmal quadruped’s actuators using data recorded from the real hardware, so the simulator responded to commands the way the physical machine did. Policies trained entirely in simulation then transferred onto the robot and held up.
The ANYmal Sim to Real Pipeline (Source)
In this case, the sim2real gap fix focused on the actuator, not the contact model. The same limit applies to all three solutions above. None of them makes a rigid-body contact model behave like real friction on a real surface, which is why the sim2real gap has closed so unevenly.
If a simulation can recreate a robot’s environment, you may be wondering whether it actually transfers well enough to be useful. Navigation and locomotion are good examples of where it does, since physics fidelity is sufficient for walking and balancing, while varying terrain, friction, and payload helps policies hold up outdoors.
Perception pretraining works in simulation for a similar reason. Changing lighting, texture, and viewpoint in simulated images gives models more variation to learn from before real images are used to fine-tune them. This is one of the main reasons synthetic data for robotics has become a routine part of training rather than a fallback.
Simulation is also useful for rare and dangerous scenarios that are difficult to collect in the real world. A pedestrian stepping out from behind a parked van or a load shifting during a lift can be created repeatedly without putting people or equipment at risk.
What those three cases have in common is that they sit on the rendering and sensing mismatches, or they involve generating situations rather than reproducing exact physics. Those are the parts a simulation’s randomization and scene-building tools were built to handle.
One interesting example of a program built on exactly these strengths is Waymo’s autonomous driving system.
Simulating a Lane Change Around a Slower Vehicle (Source)
The Waymo Driver has covered nearly 200 million autonomous miles on public roads while running billions of miles in virtual worlds. These simulated miles include wrong-way drivers, sudden cut-ins, and pedestrian crossings that rarely happen during normal driving but still need to be handled reliably.
So why do these fixes still leave a sim2real gap? Because they adjust numbers, and the remaining problems aren’t about numbers.
Randomization widens the range of values a policy trains across. System identification measures a real robot and matches those values in the simulator. Neither one changes how the simulator decides what happens when two objects touch. That part is built into the engine, and it stays simplified either way.
That sim2real limit shows up in four places:
The last point is the failure mode teams notice last, because a saturated benchmark still returns a number. In one recent study, nearly all policies tested on LIBERO, a widely used simulation suite for manipulation, scored between 90% and 95%, even though they spanned the full range from strong to weak on real hardware.
Real Evaluation Scenes and Their Simulated Replicas (Source)
Simply put, the sim2real gap isn’t the kind that closes with more compute or a better random seed. It closes with real demonstrations on real hardware, which is slower, costlier, and the reason companies are building floors full of robots to collect them.
That leaves anyone building a robot with a practical sim2real problem. The demonstrations have to come from real hardware, and there are now three ways teams are collecting robot data.
One approach is what Apptronik’s Robot Park and Tutor’s Data Factory 1 represent. Build a facility, fill it with robots, staff it, and run demonstrations as a dedicated operation.
Data center operators are getting into it too. Hyperscale Data, a data center company, owns a 617,000-square-foot building in Dowagiac, Michigan, put up in 1972 for manufacturing and later used for bitcoin mining. Roughly 100,000 square feet is being turned into robotics operations and teleoperation bays, and the company has bought 143 humanoid robots from Agibot for $13.4 million to work alongside its own staff, generating data for humanoid robot training.
Agibot Humanoid Robots Bound for the Michigan Facility (Source)
The second approach skips the staging entirely. Samsung announced a 19 trillion won investment in Gumi in July that includes a robot data factory, drawing on data accumulated from its own manufacturing lines. The robots do work that needs doing anyway, and the training data becomes a byproduct.
Then there is the option of simply buying the data. Telexistence launched a Motion Data Factory in January that generates motion datasets to order. Its pricing is based on hours multiplied by task complexity, and the company describes the service as infrastructure in the same category as electricity.
All three options cost more than renting GPUs, and companies are choosing them anyway. That is the clearest signal available that simulation isn’t closing the robot data gap on its own.
Another recent answer to the physical AI data problem is to generate it. But researchers have quickly found that a video which looks right doesn’t necessarily behave right, and policies trained on generated footage still land well short of the same policy trained on real demonstrations. Let’s take a closer look at this.
Video world models such as NVIDIA’s Cosmos and Google DeepMind’s Genie can create controllable, photorealistic scenes from a prompt, and the appeal is obvious. You can create more training situations without building an additional robot facility.
Here’s a glimpse of their features:
Photorealism is the easy part. The harder question is how much real-world experience a model has been checked against.
WorldArena is a benchmark built to answer exactly that. It ran 14 world models through tests of whether they’re useful for robot work rather than whether the video looks convincing, including their use as data sources for policy training.
The researchers trained policies on trajectories generated by six different world models, then compared them against the same policy trained on real demonstrations. The real-data version scored 77% and 66% on two tasks, and most generated-data versions landed far below that. Their conclusion is that current world models aren’t yet reliable data sources for training policies.
Overall, world models can extend a dataset rather than creating one. They can multiply what you already collected across new objects and lighting, but the demonstrations underneath still have to come from a real robot.
We’ve seen where simulation holds up and where it stops being trustworthy. So how should teams decide when real robot data collection needs to start?
Rigid pick-and-place can stay mostly in simulation, with real capture coming later for fine-tuning. Transfer is strongest here, and simulated evaluation can still give a useful sense of which policies are likely to work.
Contact-heavy work is different. Real capture needs to start sooner because success can depend on friction, slip, or how a material gives. When the simulator gets those details wrong, it can get both the training and the evaluation wrong.
There is another part of the strategy that is easy to overlook: how the data is collected. More episodes of the same task may seem like the obvious way to build a larger dataset, but variety can be more crucial.
For instance, a study on data scaling in imitation learning ran over 15,000 real robot rollouts to test this directly. Diversity of environments and objects mattered far more than raw demonstration count, and past a threshold, extra demonstrations from the same environment added almost nothing.
Their working recipe was 32 environments, each with a different object and about 50 demonstrations, which four collectors gathered in a single afternoon. The resulting policies scored around 90 percent in places they had never seen.
Twelve Environments Where the Same Task Was Recorded (Source)
That changes how the budget looks. Real robot data collection needs to be planned as infrastructure from the start, with the number of environments covered often more useful than the number of hours logged.
Everything related to the sim2real gap comes back to one thing. Simulation is enough to get a policy working, but not enough to trust it.
A policy can work well in simulation and still struggle when an object weighs differently, light comes in from the wrong side, or an object sits two centimeters further left than it ever did in training. Those small differences are key in tasks that involve contact, and they are the hardest thing for a simulator to reproduce accurately.
Closing the sim2real gap takes more than a simulator. It involves collecting demonstrations across many different environments, keeping teleoperation consistent across operators, and checking the data before it goes into training. Small inconsistencies compound quickly at that scale.
That’s a lot to stand up in-house, so many teams bring in a data partner instead. At Objectways, that is exactly where we can help you.
We run teleoperation and egocentric data collection for the contact-heavy tasks that simulation can’t cover, along with the annotation, validation, and quality assurance needed to turn those recordings into usable training data.
The sim-to-real gap hasn’t closed, but it has changed shape. Simulation now works well for locomotion, navigation, and much of perception, making it a sensible first step for many robot training tasks. Tricky issues still appear when a robot touches something, where friction, slip, and deformation change the outcome.
That is why companies are converting buildings into robot data factories, turning their own production lines into data sources, and selling motion datasets by the hour. The part simulation struggles with is the part that needs real hands on real objects.
For teams planning their next phase of robot training, keep in mind that real data belongs in the plan from the start. How early it needs to begin depends on how much the task depends on contact, and spreading collection across environments can be more essential than recording more episodes.
If you’re building a robotics or physical AI system, Objectways can support your demonstration capture, teleoperation, annotation, and data validation workflows. Contact our team to learn more.
Sim2real is short for simulation to reality. It describes moving a robot policy trained in a simulator onto physical hardware, and the drop in performance that usually shows up when it gets there.
Explore why evaluating a vision language action model is so hard, which benchmarks teams use today, and what a credible benchmark report should contain.
Explore AI data governance and how quality, security, privacy, and availability in training data define the reliability and compliance of next-gen AI systems.
