What We Learned Building a SLAM Robotics Pipeline

You might think training a robot to do a task is all about training the AI model integrated with it. However, that’s not really the case. Effort goes in long before model training. 

It starts when you capture a human demonstration and convert it into structured data that a robot can learn from. This process depends on knowing exactly where the cameras were positioned throughout the whole recording. 

Without accurate camera localization and trajectories, the data can’t be transformed into usable robot training data. In other words, the data is just a video file.

We, at Objectways, encountered this challenge while building a SLAM (Simultaneous Localization and Mapping) pipeline for bimanual robot demonstrations (two-handed demonstrations). This makes it possible for the robot to learn coordinated two-handed manipulation.

An Example of a Bimanual Robot (Source)

Our initial approach relied on ORB-SLAM3, one of the most widely used visual SLAM systems in robotics. While it performed well in some recordings, it failed almost entirely in others. 

For example, in a laundry-folding demonstration, the primary camera was localized in only 6% of frames, rendering the recording unusable. After replacing its hand-crafted feature detector with learned alternatives (neural network-based feature detectors and matchers), that same recording achieved 100% trajectory coverage, with every frame successfully localized.

In this article, we’ll walk through how we built our multi-camera SLAM robotics pipeline, the approaches we tested before identifying the root cause, and how learned feature detection made the pipeline reliable. We’ll also see how the trajectories can calibrate the cameras into a shared 3D coordinate system for accurate robot training. Let’s get started!

Why Camera Trajectories Are Key in SLAM Robotics

When people think about robot demonstrations, they often picture a human or a human-controlled robot performing a task while cameras record the scene. But that video is only one part of the dataset. 

To learn from those demonstrations, a robot also needs to know where each camera was located and how it moved throughout the video recording. And for the robot to do that, every frame must be linked to an accurate camera position and orientation so observations from different viewpoints can be combined within the same 3D coordinate system.

Simultaneous Localization and Mapping, or SLAM, exists for exactly this reason. A SLAM system estimates the trajectory of a moving camera while building a map of its surroundings. Those trajectories later became the foundation for multi-camera calibration, allowing observations from different viewpoints to be aligned into a shared coordinate system.

The quality of these camera trajectories has a direct impact on robot learning. For instance, if tracking fails or large parts of a trajectory are missing, calibration becomes unreliable, making it difficult or impossible to reconstruct the demonstration accurately in 3D.

Simply put, building a reliable SLAM robotics pipeline is the first and most important step toward producing high-quality demonstration datasets.

Building Our Multi-Camera SLAM Robotics Pipeline

Now that we have a better understanding of why robot demonstrations need accurate camera trajectories, let’s take a closer look at how we built our SLAM robotics pipeline for bimanual robot demonstrations. 

Our goal was to build a pipeline that could reliably turn human demonstrations into data that a robot can learn from. So, we used a bimanual recording rig with three GoPro cameras and two robot grippers.

The cameras were labeled Primary, Left, and Right. The Primary camera captured a wider view of the scene, while the Left and Right cameras sat close to the grippers to record manipulation detail from either side. All three were tilted slightly outwards. Later in the pipeline, we would calibrate the three views into a single multi-camera system.

Since the robot learns from the movement of the robot grippers, we first needed to know where each camera was and how it moved throughout the recording. That made camera tracking the foundation of the entire SLAM robotics pipeline.

We chose visual SLAM (ORB-SLAM3) because it can estimate camera motion using only video, without requiring external motion-capture systems or additional tracking hardware. This made it a practical choice for recording demonstrations where the cameras move naturally with the ‘operator’. 

By the end of the process, we had only video data, but we quickly saw that we needed more than just video files. The pipeline had to produce complete camera trajectories, the position and orientation of each camera for every frame, and the transformations linking the three cameras together. 

Once we had that information, observations from all three viewpoints could be combined into a single shared 3D coordinate system. This could help us create structured data that could be used for calibration, data annotation, and robot learning.

Where ORB-SLAM3 Broke Down on Real Demonstrations

We started with ORB-SLAM3, one of the most widely used visual SLAM systems in robotics. It works by detecting and matching visual features in video frames, then using those matches to estimate the camera’s motion and build a map of the environment.

The problem was that our demonstrations were much less controlled than the benchmark scenarios where ORB-SLAM3 often performs well. Real-world manipulation tasks (such as our experiment) introduced motion blur, changing viewpoints, reflective surfaces, and repetitive textures. Under such conditions, the system struggled to find reliable visual features. 

This led to a frustrating pattern of results. For example, the same demonstration could work perfectly on one camera and fail almost completely on another. 

Two demonstrations showed that pattern clearly. When the operator folded bedsheets on a bed, the Left and Right cameras reached 100% coverage while the Primary camera managed 6%. 

Also, when the operator cooked noodles at a counter, the Primary camera reached 100% while Left fell to 33% and Right to 1.8%. The failure moved between cameras depending on the task, even though the rig and the software were identical in both recordings.

Initial Results From Our Bimanual Demonstration Experiments

Such inconsistent results were a major hurdle to overcome. We couldn’t predict which camera would succeed or fail for a given recording, which made the SLAM robotics pipeline unreliable for production use. 

Since accurate trajectories are essential for calibration and robot learning, an unpredictable SLAM pipeline meant the demonstrations couldn’t be trusted as training data.

Before replacing the system entirely, we needed to understand why ORB-SLAM3 was failing on our recordings and whether the problem could be solved within the existing SLAM robotics pipeline. So we tried making some changes. 

What We Tried to Fix Our Visual SLAM Tracking

Here is a quick look at what we tried before changing the pipeline:

• ArUco-based Tracking: ArUco markers, printed black-and-white squares used for camera pose estimation, were attached to the gripper cubes. But the cameras rarely kept them in view long enough for reliable tracking.
• Parameter Tuning: We increased the number of detected features, lowered thresholds to accept weaker corners, and used more permissive tracking settings. The gain was marginal, nowhere near what we needed.
• Map Reuse and Localization: We attempted to localize recordings against previously generated maps, including maps from other cameras and earlier sessions, and switched between localization and mapping modes. None of these approaches produced stable trajectories.
• Processing Every Frame: Fast motion appeared to contribute to tracking loss, so frame sampling was removed, and every frame was processed. The change meant no motion was skipped between frames, but it didn’t resolve the underlying failure mode.
• Hidden SLAM Robotics Pipeline Issue: Some map files belonged to different recording sessions, causing ORB-SLAM3 to attempt localization against the wrong environment, which failed until we remapped the scene and re-ran SLAM.

The bottleneck wasn’t pose optimization or parameter tuning. It was the system’s ability to recognize reliable visual features in our real-world demonstrations.

Replacing ORB Features With SuperPoint Keypoint Detection

ORB-SLAM3 relies on ORB features. ORB, or Oriented FAST and Rotated BRIEF, is a fast, lightweight computer vision algorithm that finds and describes keypoints in images.

The features are detected using manually designed rules that look for corners and other distinctive image patterns. These rules work well in many controlled or benchmark environments, but aren’t reliable for real-world demonstrations. 

Instead of replacing the entire SLAM system, we changed only the feature detection (ORB) and matching front end. We replaced it with SuperPoint and SuperGlue. 

SuperPoint is a neural network-based model that detects and describes keypoints, while SuperGlue matches those keypoints between frames. Because these models learn visual patterns from large image datasets, they are much more robust to changes in lighting, viewpoint, and image quality than hand-crafted ORB features.

After making this change, the results improved dramatically. Across the Bedroom Folding, Noodles, and other demonstrations, all three cameras achieved 100% trajectory coverage.

Final Results of Our Bimanual Demonstration Experiments

We also wanted to make sure the results were genuinely usable, not just numerically better. So, we visualized the Left, Primary, and Right camera trajectories (using ReRun) and checked them for discontinuities, impossible jumps, and other tracking errors. 

The reconstructed motion was smooth, physically plausible, and reliable enough for downstream calibration and robot-learning workflows.

3D Visualization of Camera Trajectories (Created Using ReRun)

From Reliable Tracking to Multi-Camera Calibration

Getting reliable trajectories was only the first step. The Left, Right, and Primary cameras could each track their own motion, but they were still working in separate coordinate systems. To combine all three views into one 3D representation, we needed extrinsic calibration, which tells us exactly where each camera is positioned and how it is oriented relative to the others.

We used the ArUco marker cubes attached to the grippers for this calibration. When the same marker was visible in two camera views at the same time, we could use those shared observations to calculate the relationship between the cameras.

Some frames didn’t contain enough overlap. To handle those cases, we added a bridging method. If two cameras couldn’t see the same marker, the third camera and its known trajectory were used to estimate the missing relationship. The script automatically chose the best method available for each frame, so calibration could continue even when some observations were missing.

The final pipeline produced three outputs; they are: 

• Calibration JSON File: It stores the camera transforms and scale information needed to align all cameras in a shared 3D space.
• Aligned Trajectory Files: They provide synchronized trajectories for the Left, Right, and Primary cameras for visualization and analysis.
• Confidence Metrics: They indicate how many usable observations were available for each camera pair. A low count doesn’t mean low accuracy – it usually means the task involved little movement or rotation, so fewer frames contributed new geometric information.

With these outputs, any point seen in one camera could be converted into the coordinate system of the other cameras, giving us a single shared 3D view of the entire bimanual demonstration.

Lessons From Real World Robot Demonstrations

Here are some of the lessons that we learned along the way while working on this SLAM robotics pipeline: 

• Recording Quality Matters: Reflective surfaces, repetitive textures, and rapid camera motion often caused tracking problems. A slow sweep of the room before starting usually produced more stable maps and better localization.
• Validate Trajectories Visually: Visualizing the trajectories in ReRun made it much easier to spot unrealistic jumps, discontinuities, and other tracking errors that summary metrics could hide.
• Treat Calibration as a Separate Engineering Task: Calibration isn’t just a small post-processing step. It has its own failure modes, depends heavily on camera overlap, and needs its own validation process to produce reliable results.
• Build Quality Checks Into The Pipeline: We learned to verify trajectories, confirm that map files belonged to the correct recording session, and inspect calibration outputs carefully. Real-world demonstrations consistently exposed edge cases that benchmark datasets rarely captured.

The biggest takeaway for us was that reliable robot learning starts with reliable spatial data.

Reliable Spatial Data Supports Every Layer Above It

At Objectways, this is exactly the foundation we focus on when building robotics and physical AI data pipelines. We support teams with demonstration capture, calibration, annotation, validation, and quality assurance that turn raw recordings into usable robot-learning datasets.

Why Better SLAM Robotics Means Better Robot Learning

This project taught us that most of the work in robot learning happens before any model is trained. A robot policy is only as good as the data behind it, and incomplete trajectories, poor calibration, or misaligned coordinate systems can make a demonstration unreliable.

For us, the real challenge was turning raw multi-camera recordings into spatial data we could trust. That meant generating accurate trajectories, calibrating the cameras correctly, and adding quality checks to catch issues such as incorrect map files or broken trajectories.

Reliable tracking, calibration, annotation, and validation are what produce cleaner datasets and more consistent demonstrations, and that is the layer we focus on when building robotics and physical AI pipelines.

If you’re building a robotics or physical AI system, Objectways can support your demonstration capture, calibration, annotation, and data-validation workflows. Contact our team to learn more.