NVlabs/BundleSDF Neural 6-DoF Tracking and 3D Reconstruction

NVlabs/BundleSDF: Neural 6-DoF Tracking and 3D Reconstruction

Reconstructing unknown objects from monocular RGBD video presents a fundamental challenge in computer vision and robotics: how do you track and model an object you've never seen before, with no prior 3D model, no texture assumptions, and no information about what's manipulating it? This is the exact problem that NVlabs/BundleSDF addresses. Developed by NVIDIA Labs and published at CVPR 2023, BundleSDF offers a near real-time method for 6-DoF tracking of arbitrary rigid objects while simultaneously performing neural 3D reconstruction. Whether you're building robotic manipulation pipelines, AR/VR applications, or 3D scanning tools, understanding how NVlabs/BundleSDF works—and where it fits in your stack—can save you from reinventing solutions to these hard problems.

What is NVlabs/BundleSDF?

NVlabs/BundleSDF is a Python^{↗ Bright Coding Blog}-based open-source implementation of the CVPR 2023 paper "BundleSDF: Neural 6-DoF Tracking and 3D Reconstruction of Unknown Objects." The repository is maintained by NVIDIA Labs and has accumulated 1,397 stars and 164 forks as of its last commit on May 1, 2026. It falls squarely into the category of neural reconstruction and tracking systems for computer vision, with particular relevance to robotics, AR/VR, and 3D perception research.

The core premise is ambitious: given only a monocular RGBD video sequence and a single segmentation mask in the first frame, the system must track the object's 6-DoF pose through the entire sequence while building a coherent 3D representation of its geometry and appearance. No additional object information is required. No assumptions are made about the interaction agent. The object can be largely textureless, experience partial or full occlusion, undergo large pose changes, and exhibit specular highlights.

What makes this relevant now is the convergence of affordable depth sensors, neural field representations, and real-time optimization techniques. Traditional SLAM and reconstruction pipelines often fail on textureless objects or require extensive priors. BundleSDF's approach—concurrent neural object field learning with pose graph optimization—represents a meaningful architectural departure that addresses these failure modes directly.

Key Features

Neural Object Field with Pose Graph Optimization. The system's central innovation is a Neural Object Field learned concurrently with a pose graph optimization process. This dual optimization allows robust information accumulation into a consistent 3D representation that captures both geometry and appearance, rather than treating tracking and reconstruction as separate, sequential stages.

Dynamic Memory Frame Pool. BundleSDF maintains a dynamic pool of posed memory frames that automatically facilitates communication between the tracking and reconstruction threads. This design avoids the rigid keyframe selection problems that plague many SLAM systems when objects undergo rapid motion or occlusion.

Single-Frame Segmentation Requirement. Unlike methods needing per-frame annotations or 3D models, BundleSDF only requires object segmentation in the first frame. This dramatically reduces annotation burden for new objects and enables true "unknown object" operation.

Handling of Adverse Conditions. The system explicitly targets challenging scenarios: large pose changes, partial and full occlusion, untextured surfaces, and specular highlights. These are precisely the conditions where classical feature-based methods and many learning-based alternatives fail.

Near Real-Time Performance. While exact throughput depends on hardware, the method is described as "near real-time," making it viable for interactive applications rather than purely offline batch processing.

Validated on Multiple Datasets. Results are demonstrated on HO3D, YCBInEOAT, and BEHAVE datasets, with claims of significant outperformance over existing approaches on these benchmarks.

Use Cases

Robotic Manipulation and Grasping. In unstructured environments, robots frequently encounter novel objects without pre-existing models. BundleSDF enables online pose tracking and shape estimation, supporting grasp planning and manipulation strategies for previously unseen items. The lack of interaction agent assumptions makes it suitable for human-robot collaborative settings.

AR/VR Object Digitization. For augmented reality applications requiring real-time object insertion or interaction, BundleSDF provides the necessary pose estimates and 3D geometry from commodity RGBD sensors. The single-segmentation requirement aligns well with user-driven "scan this object" workflows.

Industrial Inspection and Reverse Engineering. Manufacturing and quality control scenarios often involve scanning parts with varying surface properties—including untextured or specular metal surfaces. BundleSDF's explicit handling of these conditions makes it applicable where photogrammetry or structured light scanning might struggle.

Research Prototyping in Neural Fields. For researchers investigating neural radiance fields (NeRFs) and their variants, BundleSDF offers a concrete, working system that combines neural representations with traditional geometric optimization. It serves as both a baseline and a component for larger pipelines.

Dataset Generation for Training. The tracked poses and reconstructed meshes can serve as ground truth or pseudo-ground truth for training downstream perception models, particularly in domains where real annotated data is scarce.

Installation & Setup

BundleSDF provides Docker^{↗ Bright Coding Blog}-based setup, which is strongly recommended given the dependency complexity. Follow these steps precisely:

Step 1: Build the Docker image. Navigate to the docker directory and build:

cd docker
docker build --network host -t nvcr.io/nvidian/bundlesdf .

The --network host flag ensures the build process can access external resources. This step only needs to be performed once but may take considerable time due to CUDA, PyTorch, and other heavy dependencies.

Step 2: Start the container and compile machine-dependent packages.

cd docker && bash run_container.sh

# Inside docker container, compile the packages which are machine dependent
bash build.sh

The run_container.sh script handles GPU passthrough and volume mounting. The build.sh step compiles CUDA extensions and other components that must match your specific GPU architecture.

Pretrained Model Downloads. Before running, obtain the required weights:

Download the XMem segmentation network weights and place at ./BundleTrack/XMem/saves/XMem-s012.pth
Download the LoFTR outdoor_ds.ckpt weights and place at ./BundleTrack/LoFTR/weights/outdoor_ds.ckpt

Dataset Setup (for HO3D evaluation). Download the augmented HO3D data and YCB-Video object models, then ensure this structure:

HO3D_v3
  ├── evaluation
  ├── models
  └── masks_XMem

Update HO3D_ROOT in BundleTrack/scripts/data_reader.py to your actual path.

Real Code Examples

BundleSDF's README provides explicit commands for three operational modes. Here are the documented examples with explanations.

Example 1: Joint Tracking and Reconstruction on Custom Data

# Run joint tracking and reconstruction
python run_custom.py --mode run_video \
  --video_dir /home/bowen/debug/2022-11-18-15-10-24_milk \
  --out_folder /home/bowen/debug/bundlesdf_2022-11-18-15-10-24_milk \
  --use_segmenter 1 \
  --use_gui 1 \
  --debug_level 2

This is the primary entry point. The --mode run_video flag activates the core tracking-reconstruction loop. --use_segmenter 1 enables automatic segmentation (requires XMem weights); set to 0 if providing pre-computed masks. --use_gui 1 launches a visualization window. --debug_level 2 produces verbose output for troubleshooting. Note the README's caveat: the demo assumes maximum relevant depth < 1 meter; modify BundleTrack/config_ho3d.yml line 16 if your scene differs.

Example 2: Global Refinement Post-Processing

# Run global refinement post-processing to refine the mesh
python run_custom.py --mode global_refine \
  --video_dir /home/bowen/debug/2022-11-18-15-10-24_milk \
  --out_folder /home/bowen/debug/bundlesdf_2022-11-18-15-10-24_milk

The global refinement step optimizes the entire trajectory and reconstruction jointly, reducing drift accumulated during sequential processing. This is run as a separate pass after the initial tracking completes.

Example 3: Optional Pose Visualization

# Draw oriented bounding box to visualize pose
python run_custom.py --mode draw_pose \
  --out_folder /home/bowen/debug/bundlesdf_2022-11-18-15-10-24_milk

This generates visualizations matching the paper's demo style, overlaying oriented bounding boxes on the input frames.

Example 4: HO3D Dataset Evaluation

# Run BundleSDF on HO3D evaluation sequence
python run_ho3d.py \
  --video_dirs /mnt/9a72c439-d0a7-45e8-8d20-d7a235d02763/DATASET/HO3D_v3/evaluation/SM1 \
  --out_dir /home/bowen/debug/ho3d_ours

# Benchmark against ground truth
python benchmark_ho3d.py \
  --video_dirs /mnt/9a72c439-d0a7-45e8-8d20-d7a235d02763/DATASET/HO3D_v3/evaluation/SM1 \
  --out_dir /home/bowen/debug/ho3d_ours

The HO3D pipeline separates inference (run_ho3d.py) from evaluation (benchmark_ho3d.py), enabling modular experimentation.

The README contains these four documented examples. No additional code samples are provided in the current documentation.

Advanced Usage & Best Practices

Custom Data Preparation. For your own RGBD sequences, structure inputs exactly as specified:

root
  ├──rgb/        # PNG files
  ├──depth/      # PNG, uint16, millimeters, matching rgb filenames
  ├──masks/      # PNG, 0=background, >0=foreground, matching rgb filenames
  └──cam_K.txt   # 3x3 intrinsic matrix, space-delimited

Depth must be uint16 in millimeters—floating-point meters or other conventions will silently produce incorrect geometry. The intrinsic matrix format is rigid; verify with a text editor before processing.

Depth Range Configuration. The default max_relevant_depth of 1 meter in config_ho3d.yml is optimized for close-range manipulation. For room-scale or larger objects, adjust this threshold to include your actual working volume without excessive background inclusion.

Online Segmentation Limitations. Due to licensing, XMem is not bundled. If you need automatic per-frame segmentation rather than pre-computed masks, you must integrate XMem separately and add a wrapper in segmentation_utils.py. For batch processing, pre-computing masks with XMem or another segmenter is often more efficient.

GPU Memory Management. Neural object fields and pose graph optimization are memory-intensive. If encountering OOM errors, reduce the memory frame pool size or process shorter subsequences before global refinement.

Reproducibility Note. The Docker environment is strongly recommended over native installation. CUDA version mismatches between host and container, or missing system libraries for OpenCV with GPU support, are common failure modes in native setups.

Comparison with Alternatives

Aspect	NVlabs/BundleSDF	Traditional SLAM (e.g., ORB-SLAM3)	Category-Level Methods (e.g., NOCS)
Object prior required	None (fully unknown)	None	Category shape priors
Texture assumption	None (handles untextured)	Requires features	Moderate texture needed
Segmentation need	First frame only	N/A (full scene)	Per-frame or first frame
Output	Pose + textured mesh	Sparse/dense map	Pose + coarse shape
Occlusion handling	Partial and full	Partial only	Partial
Real-time	Near real-time	Real-time	Varies
Specular highlights	Explicitly handled	Fails	Often fails

Traditional SLAM systems excel at camera tracking in textured environments but lack object-centric semantics and fail on textureless objects. Category-level methods like NOCS reduce unknown-object handling to category-specific shape spaces, which BundleSDF avoids entirely—at the cost of potentially slower convergence for complex shapes. The trade-off is generality versus speed: BundleSDF handles broader conditions but may require more compute per frame than optimized SLAM pipelines.

FAQ

What hardware is required? CUDA-capable GPU is essential. The Docker base suggests NVIDIA GPU with sufficient VRAM for neural field training; exact minimum is unspecified.

Is the license open source? The repository lists "Other" license. Review the actual license file before commercial use—it's not standard MIT/Apache.

Can I run without Docker? Possible but not recommended. The README only documents Docker setup; native installation requires resolving CUDA, PyTorch, and OpenCV GPU dependencies manually.

Does it work on Windows? The provided scripts are bash-based; Windows users should use WSL2 or adapt the Docker commands.

How accurate is the reconstruction? The paper claims significant outperformance on HO3D, YCBInEOAT, and BEHAVE benchmarks, but specific ATE/RPE numbers aren't in the README.

Can it track deformable objects? No—BundleSDF assumes rigid objects. Deformable tracking would require substantial architectural changes.

Is real-time operation guaranteed? "Near real-time" is the stated goal; actual throughput depends on GPU, object complexity, and sequence characteristics.

Conclusion

NVlabs/BundleSDF fills a genuine gap in the computer vision toolkit: robust 6-DoF tracking and neural 3D reconstruction of truly unknown, potentially textureless objects from single RGBD sequences. Its single-frame segmentation requirement, explicit handling of occlusion and specularities, and concurrent neural field optimization make it particularly valuable for robotics researchers, AR/VR developers, and anyone building perception pipelines for unstructured environments.

The system is not plug-and-play—it demands careful data preparation, Docker-based deployment, and attention to depth conventions. But for problems where classical SLAM fails and category-level priors don't exist, it offers capabilities that are difficult to replicate by combining simpler tools.

If your work involves object manipulation, online scanning, or neural field research, BundleSDF deserves evaluation in your stack. The implementation is available now at https://github.com/NVlabs/BundleSDF.