People often learn everyday tasks, such as making a sandwich, by watching someone else first. We watch a demonstration, follow along, and then try it ourselves. Can a robot do the same?
It can certainly see the same things we do, like the ingredients, the hands, the utensils, and the order the steps happen in. But seeing a task and being able to repeat it are two different problems. A video can’t show how tightly fingers close around the knife, how grip shifts halfway through, or how long a hand pauses before the next move. That is the gap datasets like EgoDex are built to close.
A typical dataset can contain hundreds of hours of video, but still provide limited information for robot training. So, what type of information does a robot model actually need for training?
Mostly, it comes down to how the data was captured. It should contain video with added information, such as what people are asked to do, which sensors are used, what is tracked, and how the different signals are synchronized and described.
EgoDex is Apple’s egocentric dataset, and it was created with those questions in mind. It holds 829 hours of first-person video across 338,000 episodes, all recorded on Apple Vision Pro across a broad range of tabletop tasks. Each recording pairs the video with body and hand tracking and a natural-language description, so a physical AI model gets both the action and the context behind it.
In this article, we’ll look at what EgoDex does differently, what it still doesn’t solve, and how to evaluate an egocentric dataset before using it for robot training. Let’s get started!
EgoDex is a publicly available egocentric dataset released in 2025 to capture more than what a person sees (visual data). Egocentric data is recorded from a first-person perspective, typically capturing what a person sees and does from a camera worn on their head or body. But EgoDex goes a bit further.
The dataset was collected using ARKit on Apple Vision Pro across 194 tabletop tasks, producing 829 hours of 1080p video at 30 Hz, or roughly 90 million frames. More importantly, the recordings include frame-by-frame 3D tracking of the head, upper body, and hands across 68 joints, along with camera parameters.
This information is particularly useful for robotics, especially in hand pose estimation. A video can show someone picking up a bottle, but without hand pose estimation data and finger movement, it’s tricky to understand how the person physically interacted with the object.
Specifically, EgoDex includes activities such as tying shoelaces, folding laundry, unscrewing bottle caps, and plugging in chargers. These everyday tasks involve detailed finger movements and physical contact with objects, making them essential to the fine-grained manipulation problems that robots still struggle with. EgoDex also pairs each episode with natural-language descriptions, adding a semantic layer that helps connect what a person does with what they are trying to accomplish.
Earlier egocentric datasets such as the Ego4D dataset and the EPIC-KITCHENS dataset were built mainly for video understanding rather than robot manipulation. That difference is crucial because ‘what someone is doing’ isn’t the same as ‘how to do it’. A robot needs detailed movement and interaction data to reproduce actions.
EPIC-KITCHENS is a good example of this. Hours of kitchen footage show people cooking, but the recordings don’t carry hand position, finger position, or movement patterns. A robot watching those videos knows a meal was made, but it still has no way to map the movement onto its own hands.
Task distribution adds a second limitation. Passive recordings capture whatever happened, so long stretches turn out to be idle or irrelevant for robotic execution. Natural head movement and camera drift land in the footage too, even though they were never meant to be part of a demonstration.
So, when it comes to robot training data, passive video isn’t deliberate demonstration data, and a useful egocentric dataset has to capture the signals that explain how an action happens.
That is exactly what EgoDex does. EgoDex stands out because several of its egocentric data collection choices move in this direction. The dataset captures detailed hand and body movements alongside the video and provides text descriptions that add context to each action.
A dataset doesn’t become useful for robotics by accident. It comes down to decisions made before data collection even starts, like what the sensors track, which tasks get recorded, and what information gets stored alongside videos.
EgoDex gets three of these right, and each one closes a gap that plain videos leave wide open. Let’s take a closer look at some of the data collection choices that make the EgoDex dataset more useful for physical AI systems.
One of EgoDex’s biggest strengths is that hand pose estimation comes captured rather than inferred. For a robot, the hands are where most of the interaction with the environment happens, so getting that recording right carries more weight than almost anything else in the dataset.
The tracking runs frame by frame across 68 joints, covering the hands, head, and upper body.
That gives a model something a video file can’t. It shows how the fingers move, when they close, and how the grip changes during a task, which is what makes mapping a human demonstration onto a robot end effector possible.
The tasks in a dataset decide what a model can learn from it. Alongside a base of pick-and-place work, EgoDex includes a large share of tasks that demand fine finger control.
For instance, removing a screw from a fixture requires the fingers to grip, turn, release, and take hold again several times over. The grip has to change as the screw backs out. The hand keeps correcting itself the whole way through, and none of that appears in a task where a robot only has to move an object from one place to another.
Unzipping a Ziploc bag works the same way. It asks for steady pressure along a seam while both hands hold the bag taut. Neither task is difficult for a person, and both are hard for a robot.
Contact-rich work like this is where dexterous manipulation still breaks down, which is why building a dataset around it is essential.
The third useful choice is pairing demonstrations with natural-language descriptions. A sequence of hand movements can show what happened, but not necessarily what the person was trying to accomplish. So, a description adds that missing layer of intent.
This becomes increasingly critical as robotics moves toward vision-language-action models. They need to connect visual observations and physical actions with natural-language instructions.
When the text descriptions are paired with the episodes, the semantic information (information about meaning and intent) is tied directly to each demonstration rather than added later. Together, video, hand pose, and language provide a much richer description of an action than a simple video file.
EgoDex improves on several things earlier egocentric datasets left out, but it doesn’t solve every problem related to turning human demonstrations into robot training data.
The most obvious issue is physical feedback. For instance, the dataset captures vision and hand pose, but not force, torque, or tactile signals. A video can show someone gripping and turning a bottle cap, for example, but it can’t tell a robot how much force was applied or exactly when the grip started to slip.
The embodiment gap is another problem. Human hands differ from robot hands in shape, joint structure, range of motion, and control mechanisms. So human hand trajectories can’t simply be copied onto a robot. In addition to this, EgoDex’s focus on tabletop tasks (like tying shoelaces) limits how well it represents robots operating in areas like warehouses, factories, or other less controlled environments.
On top of this, failure and recovery are limitations. Demonstration datasets are generally built around successful execution, and EgoDex is no exception. That leaves little information about what a robot should do when a grasp fails, or an object slips.
Finally, a dataset can be technically sound and still be off limits for your project. EgoDex is free to download, but it comes with a CC BY-NC-ND license. In plain terms, that means non-commercial use with credit, and no sharing of modified versions of the data.
Everything EgoDex gets right doubles as a way to interrogate any other egocentric dataset. Here are five questions that cover most of what you need to know before committing to one:
Those five questions all assume you are judging one dataset. In case you are working with cross-embodiment datasets like the Open X-Embodiment dataset, expect a few of them to change.
Hardware coverage becomes the thing to check rather than annotation depth, since the value comes from how many embodiments are represented and how closely any of them match your own. That coverage adds useful range, though it doesn’t replace data collected around your specific robot, environment, and tasks.
Typically, when you are putting a robot training pipeline together, you don’t need to choose between public datasets and custom collection. The stronger approach uses a large public dataset to give a model its starting point, then fine-tunes on data matching the robot you plan to deploy.
General patterns of human manipulation come from the first, and everything specific to your application comes from the second. This distinction is critical with EgoDex.
Even if it is a valuable source of egocentric data, it doesn’t automatically become the right training data for every robot. A human demonstration recorded on Apple Vision Pro is still different from the data collected around your robot’s own hand, sensors, workspace, and control system.
Custom collection closes that distance. Rather than stretching a general-purpose dataset to cover an application it was never built for, teams can record demonstrations that match what they are actually deploying.
In particular, teams can lean on data partners to handle the collection itself, which keeps their own effort on the model rather than on building a capture operation from scratch. If you are looking for a data partner, we at Objectways handle egocentric, teleoperation, and UMI gripper data collection built around specific robotics requirements, which is often what stands between broad training data and a robot that can do a particular job.
The lesson from EgoDex has little to do with its size. It was built with robotics in mind, and that shows up in the hand tracking, the task selection, and the language descriptions attached to every episode. Those three choices are what turn a recording of a person into something a robot can learn from.
EgoDex also shows why no single public dataset covers every robotics need. Robots differ in their hands, their sensors, their environments, and the jobs they are built to do. A public dataset gives a model its pretraining foundation, and custom demonstrations supply everything specific to the robot going into production.
Looking for data built around your robotics project? Reach out to Objectways to learn more about our data collection services.
EgoDex is an egocentric robotics dataset designed to capture how people perform everyday tabletop tasks from a first-person perspective. Collected using Apple Vision Pro, it combines video with detailed 3D tracking of the head, body, and hands, as well as natural-language descriptions.
See how a LeRobot dataset stores robot demonstrations, how it compares with older formats, and what can get lost in the conversion to LeRobot.
Learn what dexterous manipulation means, why timing and contact data are key, and how multimodal datasets train reliable robot-hand policies
