When a robotic hand turns a key, rotates a screwdriver, or shifts an object between its fingers, it is constantly adjusting its grip. Each finger has to coordinate with the others, changing position and pressure as the object moves, slips, or changes shape. That continuous control is what makes dexterous manipulation different from simply picking up and moving an object.
A basic gripper can open, close, and release. A dexterous robot hand has to keep adjusting its contact with an object throughout the task. To do that reliably, it needs more than visual information.
A camera can show where the hand and object are, but it can’t tell the robot exactly how much force it is applying, when an object starts to slip, or how contact changes. That is where tactile sensing in robotics becomes crucial.
Robotic training data needs to capture these signals alongside video, joint movements, and other robot data, then keep them synchronized. Even a small timing error can pair the wrong force or tactile reading with a visual frame, teaching the robot to respond at the wrong moment. A grasp may look correct in the training data but fail when the robot encounters the same task in the real world.
As humanoid robot hands and other multi-fingered systems become more capable, the quality of the data used to train them is more essential than ever.
In this article, we’ll look at what dexterous manipulation data needs to capture, how it is labeled, why timing matters, and what to look for when choosing a data partner. Let’s get started!
Before we learn about dexterous data collection, let’s take a step back and understand what dexterous manipulation refers to in physical AI systems.
Dexterous manipulation is often confused with the ordinary pick-and-place task of a gripper. A simple gripper closes on an object, moves it somewhere else, and releases it. It performs a mostly binary action: grasp or release. However, dexterity starts when the robot has to keep controlling the object while manipulating it.
That can include rotating an object inside the hand, changing finger contact points, regrasping it without putting it down, handling soft or flexible objects, or using a tool while adjusting its angle. Tasks such as turning a key, folding fabric, or adjusting a screwdriver are much more complex than simply transporting an object.
However, dexterous manipulation doesn’t require a five-fingered humanoid robot hand. For instance, the Allegro Hand, a widely used research platform, has four fingers and 16 degrees of freedom, making it capable of complex in-hand manipulation. In a recent study, researchers equipped an Allegro Hand with tactile sensors and trained it to rotate objects about arbitrary axes, even when the palm faced down and gravity worked against the grasp.
The following four properties make dexterous manipulation tasks difficult:
Each of these properties creates a data challenge as well as a control challenge. Training a dexterous manipulation policy requires synchronized records of force, joint positions, tactile events, contact changes, and the object’s response throughout the task.
A dataset with only video and a high-level task label may look complete, but it leaves out many of the signals the policy needs to learn how to manipulate objects reliably.
Training a dexterous robot is tricky because the robot needs to make continuous, high-speed decisions while dealing with uncertainty about both the object and its own physical behavior.
Here’s a closer look at what makes dexterous manipulation so challenging:
These difficulties compound when a robot meets an object it has never handled before. For example, a dexterous manipulation controller tested on an unfamiliar duck-shaped object dropped it in 56% of trials, against 23% for a shape it had trained on. The duck was curved and smooth, so small differences in surface friction change the task, and friction is something a camera can’t read before contact.
A dexterous manipulation demonstration is more than a video of a robot hand performing a task. It is a multimodal recording that combines arm and hand joint states, the SE(3) end-effector pose, force and torque measurements, tactile pressure, wrist-camera video, and third-person scene video. The key is that all of these signals must be synchronized so they describe what was happening at the same moment.
That synchronization is one of the hardest parts of collecting dexterity data. Cameras, tactile sensors, and robot control systems often operate at different rates, so the data pipeline has to bring them onto a shared timeline. Each video frame needs to match the corresponding robot state, force reading, tactile signal, and action.
Even a small synchronization error can corrupt training. A drift of only a few milliseconds can attach the wrong force and torque measurements to the wrong visual frame, causing the policy to learn a relationship that never actually occurred.
The result can be unstable grasps, incorrect contact responses, and execution failures. This is also why dexterity data costs more than perception data to produce, since the expense sits in synchronization and validation rather than in capturing footage.
An interesting example comes from the 2026 RoboTacDex dataset, which contains 6,000 demonstrations collected on a humanoid robot with tactile dexterous hands. The tactile sensors publish readings at 100 Hz while the visual data is recorded at 30 Hz, so the researchers built a synchronization system to keep the different data streams aligned to the millisecond.
The researchers also tested what the tactile data added to the demonstrations. When a policy learned to unscrew a bottle cap with and without touch, the overall success rate changed very little.
But the types of failures were different. Without tactile data, the hand often spun without engaging the cap. With it, the hand engaged the cap but struggled to adjust its grip. The tactile signals captured contact information that the cameras couldn’t see.
So, how do you collect dexterous manipulation data? Several methods are available, each offering a different balance of accuracy, diversity, cost, and scalability.
Here’s an overview of some of the ways the data can be collected:
A well-known example is OpenAI Dactyl, which trained a 24-DOF Shadow Hand for dexterous manipulation entirely in simulation, without human demonstrations. Domain randomization helped the policy transfer to a physical robot, but performance still dropped.
In a block reorientation task, it completed a median of 50 consecutive rotations in simulation before failing, compared with 13 on the physical robot. This shows how difficult it can be to fully close the gap between simulated and real-world manipulation.
Just like the data itself, annotation and labeling for dexterous manipulation need to capture more than what happens in a video frame. The goal is to describe what the robot is doing, when it happens, and how its physical interaction with the object changes over time.
A typical annotation pipeline begins with grasp-type classification, using categories such as power, pinch, lateral, and precision grasps. These labels help organize how a robot grasps an object, but broad grasp categories don’t tell the whole story. A policy trained only on coarse labels may fail when an object is slightly rotated, heavier than expected, or has a different surface texture.
The next layer is contact-phase segmentation. Demonstrations are divided into stages such as reach, grasp, manipulate, and release so the learning system can model the transitions between them. These transitions are often more informative than the final successful grasp because they capture how contact is established, adjusted, and terminated.
High-quality dexterity datasets also preserve slip onset and failure events. Instead of discarding unsuccessful attempts, datasets can label why an object slipped, rotated unexpectedly, or was dropped. This gives the robot examples of what to do when things go wrong.
Getting those labels right depends on capturing the moment they refer to. For example, a 2025 slip detection study first recorded slips by slowly pulling the object out of the hand. The model validated well and failed in practice because real slips involve an abrupt change in force that the training setup didn’t capture.
Finally, dexterity annotation must align force vectors, torque measurements, tactile signals, joint angles, and video frames with the correct timestamps. This makes it a time-series synchronization problem, not just a frame-by-frame labeling task. That’s why quality checks focus on keeping contact events, force changes, and sensor streams synchronized throughout the demonstration.
Getting all of this right can be complex, especially when working with multiple data streams and different types of robotic systems. Working with experienced robotics data specialists can make the process easier and help ensure that the right information is captured, labeled, and validated from the start.
At Objectways, we support robotics and physical AI teams with data collection, annotation, and quality assurance across workflows including teleoperation, UMI gripper data, and egocentric demonstrations. Our work involves 3D point cloud annotation, contact-phase segmentation, and temporal quality checks that help keep complex robotics data accurate and aligned.
Let’s say you’re collecting or working with dexterous manipulation data. What should you pay attention to? Beyond the number of demonstrations, the way the data is captured, labeled, and delivered can have a major impact on how useful it is for training.
Here are four factors to keep in mind:
These details may seem small compared with the size of a dataset, but they can have a major impact on its usefulness. A large collection of demonstrations is only valuable if the underlying data is accurate, complete, and properly aligned.
The most expensive mistakes in a dexterous manipulation program often happen before large-scale collection begins. A small issue in the early stages can quickly become a problem across thousands of demonstrations.
Synchronization is a good place to start. If a pipeline has a timing error, that error gets repeated across every episode. A few milliseconds of drift may be easy to fix during a pilot, but once thousands of demonstrations have been collected, the data may need to be recorded again because the alignment cannot be repaired afterward.
With timing in place, the next step is to capture what happens throughout the task, not just whether it succeeds. Labeling contact phases such as reach, contact, adjustment, manipulation, and release shows a policy how the task unfolded and gives it more information than a simple success label.
The same principle applies to failures. Slips, drops, and recovery attempts show a policy how to respond when something goes wrong. Removing these episodes leaves the robot with little experience handling unexpected changes.
These decisions also affect the cost of a data collection project. Recording demonstrations and creating synchronized multimodal labels are separate engineering tasks with different costs, so annotation needs to be planned and budgeted separately.
The takeaway is simple. Two demonstrations can look almost identical on video but have very different value as training data. That difference often comes down to the choices made before recording begins.
Robot hands are becoming remarkably capable, but the data used to train them still has some catching up to do. For example, multi-fingered hands with tactile sensing are commercially available, and policies can already rotate objects and recover from slipping grasps. The constraint is increasingly the data.
A useful dexterity demonstration needs synchronized joint states, force and torque readings, tactile signals, and video. Contact phases need to be labeled, and failures need to stay in the data. Get those details wrong, and collecting more demonstrations won’t fix the problem because errors introduced during recording can’t always be repaired afterward.
Working on a dexterous manipulation project? Objectways supports robotics and physical AI teams with multimodal data collection, teleoperation and UMI gripper workflows, temporal annotation, and synchronization-aware labeling for contact-rich tasks. Reach out to learn more.
It is a robot with multi-jointed hands built to handle objects the way people do, using several fingers together along with touch feedback. The Shadow Dexterous Hand and Inspire’s RH56DFQ are two commercial examples.
See how a LeRobot dataset stores robot demonstrations, how it compares with older formats, and what can get lost in the conversion to LeRobot.
Explore what makes the EgoDex dataset useful for robotics, from hand pose and task design to language annotations, and how to evaluate egocentric datasets.
