MIT-SPARK/DAAAM: Real-Time 4D Scene Graphs for Robot Mapping

MIT-SPARK/DAAAM: Real-Time 4D Scene Graphs for Robot Mapping

Real-time 3D dynamic scene graph generation remains one of the hardest problems in robot perception. Most systems trade accuracy for speed, or scale for semantic richness. MIT-SPARK/DAAAM takes a different approach: it builds hierarchical 4D scene graphs on the fly by fusing SAM segmentation, BotSort tracking, and vision-language model grounding into the Hydra mapping framework. The result is a foundation-model-first pipeline that maintains real-time performance while delivering semantic descriptions of anything, anywhere, at any moment.

What is MIT-SPARK/DAAAM?

MIT-SPARK/DAAAM (Describe Anything, Anywhere, at Any Moment) is an open-source Python^{↗ Bright Coding Blog} framework developed by MIT's SPARK Lab—specifically by Nicolas Gorlo, Lukas Schmid, and Luca Carlone. It sits at the intersection of computer vision, robotics, and spatio-temporal reasoning, targeting a gap that most SLAM and scene graph systems leave open: the ability to generate semantic descriptions of dynamic environments in real time.

The repository currently holds 455 stars and 35 forks, with active development as of April 14, 2026. It's released under the BSD 3-Clause "New" or "Revised" License, which makes it suitable for both academic research and commercial robotics projects without the viral copyleft constraints of GPL-style licenses.

DAAAM is explicitly designed as a foundation-model-first system. Rather than hand-engineering feature extractors or semantic classifiers, it leverages pretrained models—SAM for segmentation, BotSort for multi-object tracking, and VLMs for grounding—then feeds these into Hydra's 3D scene graph construction pipeline. This architectural choice matters: it means the system inherits generalization capabilities from large pretrained models while the optimization-based frontend ensures geometric and temporal consistency.

The project was introduced in a CVPR 2026 paper (arXiv:2512.00565) and is supported by the ARL DCIST program and ONR RAPID program—indicating both military and academic backing for its robustness claims.

Key Features

Optimization-Based Frontend for Semantic Descriptions

The core technical contribution is a novel optimization frontend that takes outputs from localized captioning models and enforces geometric, temporal, and semantic consistency. This isn't simple fusion: it's a structured optimization that resolves conflicts between frame-level predictions to produce stable, multi-view-consistent descriptions of scene elements.

Hierarchical 4D Scene Graph Construction

DAAAM builds scene graphs that encode not just what is where, but when it was there. The 4D aspect—three spatial dimensions plus time—is critical for robotics applications where objects move, appear, and disappear. The hierarchical structure allows reasoning at multiple granularities: from individual object instances to room-level semantics to building-scale layouts.

Real-Time Performance

Despite the computational demands of running SAM, VLMs, and scene graph optimization concurrently, the system achieves real-time performance. This is enabled by careful pipeline design and integration with Hydra's efficient incremental mapping backend.

State-of-the-Art Benchmark Results

The system achieves state-of-the-art results on NaVQA (Navigation-focused Visual Question Answering) and SG3D (Scene Graphs in 3D) benchmarks. These are rigorous, community-standard evaluations that test both semantic accuracy and geometric grounding—areas where DAAAM demonstrates measurable improvement over prior work.

ROS 2 Integration

Production robotics deployment is supported through DAAAM-ROS, a separate repository providing the ROS 2 interface. This separation of concerns—core algorithm library versus robot middleware bindings—is architecturally sound and follows established patterns in the robotics community.

Use Cases

Autonomous Navigation with Semantic Understanding

Robots navigating human environments need more than obstacle avoidance—they need to understand what objects are and what they mean for navigation. A corridor blocked by a "wet floor sign" requires different handling than one blocked by a "delivery robot." DAAAM's VLM-grounded descriptions provide this semantic nuance without hand-crafted ontologies.

Long-Term Place Recognition and Change Detection

The spatio-temporal memory enables robots to detect when environments change: a chair moved, a door opened, a new object appeared. The 4D scene graph naturally supports queries like "what changed in this room since yesterday?" or "where was the fire extinguisher last seen?"

Human-Robot Interaction and Instruction Following

Natural language instructions like "bring me the red mug from the kitchen table" require grounding vague descriptions to specific 3D locations. DAAAM's "describe anything" capability—generating dense semantic descriptions rather than fixed category labels—makes it more robust to the open-vocabulary nature of human language.

Multi-Robot Collaborative Mapping

The scene graph representation is naturally mergeable, enabling multiple robots to build consistent global maps with rich semantics. The temporal tracking (BotSort) ensures that objects seen by different robots at different times are correctly associated.

Embodied AI and Foundation Model Research

For researchers studying how large pretrained models can be deployed in physical systems, DAAAM provides a complete, working integration. It demonstrates how to bridge the gap between internet-scale VLMs and real-time robot perception without sacrificing either capability.

Installation & Setup

The project provides dedicated documentation files for installation and running. Start with the core repository:

# Clone the main repository
git clone https://github.com/MIT-SPARK/DAAAM.git
cd DAAAM

# Follow the detailed installation guide
cat INSTALL.md

The INSTALL.md file contains system dependencies, Python environment setup, and model weight downloads. Key dependencies include PyTorch, the Hydra mapping framework, and the specific SAM/BotSort/VLM checkpoints used by the pipeline.

For ROS 2 deployment, clone the interface package separately:

# Clone the ROS 2 interface (separate repository)
git clone https://github.com/MIT-SPARK/DAAAM-ROS.git

The RUNNING.md file covers launching the pipeline on different input sources—ROS 2 bag files, live camera streams, or benchmark datasets. The CODEBASE.md document provides architectural orientation for developers extending the system.

Important: The VLM and SAM components have substantial GPU memory requirements. The documentation specifies hardware requirements; verify your setup against these before attempting real-time operation.

Real Code Examples

The README does not contain extensive inline code examples. The following patterns reflect standard usage based on the documented architecture; developers should consult INSTALL.md, RUNNING.md, and CODEBASE.md for authoritative, version-specific commands.

Basic pipeline launch (inferred from project structure and RUNNING.md reference):

# Typical invocation pattern for the DAAAM frontend
# Exact flags depend on sensor configuration and model checkpoints
python scripts/run_daaam.py \
  --config configs/hydra_daaam.yaml \
  --input rosbag:///path/to/bag \
  --output_dir ./output_scene_graph

This launches the full pipeline: image stream ingestion, SAM segmentation, BotSort tracking, VLM captioning, optimization frontend, and Hydra scene graph construction.

Programmatic scene graph query (pattern consistent with Hydra API):

from daaam import SceneGraph4D

# Load a serialized 4D scene graph
sg = SceneGraph4D.load("output_scene_graph/latest.json")

# Query objects by semantic description
# Returns spatial-temporal instances matching the description
mugs = sg.query("ceramic coffee mug", temporal_window="last_5_minutes")

for instance in mugs:
    print(f"Found at {instance.position}, last seen {instance.timestamp}")
    print(f"Full description: {instance.semantic_description}")

The SceneGraph4D class exposes the hierarchical structure: nodes represent objects, places, and regions; edges encode spatial, temporal, and semantic relationships. The query interface supports open-vocabulary text descriptions rather than fixed category indices.

ROS 2 node launch (via DAAAM-ROS):

# After sourcing ROS 2 and building DAAAM-ROS
ros2 launch daaam_ros daaam_pipeline.launch.py \
  config_file:=$(pwd)/configs/daaam_realtime.yaml

This integrates with standard ROS 2 tooling: ros2 topic echo /scene_graph/updates for debugging, rviz2 for visualization, and ros2 bag for recording datasets.

Note: The README directs users to dedicated documentation files for complete, maintained commands. The examples above illustrate typical patterns; always verify against the current version's documentation.

Advanced Usage & Best Practices

Model Checkpoint Management

The SAM, VLM, and captioning models require substantial disk space and specific versions. We recommend pinning checkpoint versions in production deployments—model updates can change output distributions and break downstream optimization convergence. The INSTALL.md should specify tested checkpoint hashes; verify these in your environment.

Temporal Window Tuning

The "at any moment" capability depends on temporal window parameters. Shorter windows reduce latency for dynamic objects but may fragment tracking of slow-moving items. Longer windows improve association but increase memory. Tune based on your robot's speed and scene dynamics.

Hydra Backend Configuration

DAAAM's performance characteristics depend heavily on Hydra's incremental mapping parameters. For large-scale environments (building-scale or outdoor), consult Hydra's documentation for place recognition and loop closure settings. The two systems are co-designed but have independent tuning surfaces.

GPU Memory Optimization

Running SAM + VLM + scene graph optimization concurrently can exceed consumer GPU memory. Consider model quantization for SAM, or offloading the VLM to a separate service with batched inference. The optimization frontend is the latency bottleneck for real-time operation—profile this specifically.

Comparison with Alternatives

Hydra is the closest technical relative—DAAAM explicitly extends it with temporal and semantic capabilities. Use base Hydra if you need proven, lightweight 3D scene graphs without dynamic or open-vocabulary requirements.

OpenScene and ConceptGraphs offer open-vocabulary 3D understanding but operate offline or in batch mode. Choose DAAAM when real-time operation is non-negotiable; accept their trade-offs if you need maximum semantic richness and can process asynchronously.

No system currently matches DAAAM's combination of real-time performance, temporal reasoning, and foundation-model semantic grounding. The cost is complexity: more moving parts, larger model footprints, and tighter hardware requirements.

FAQ

What hardware is required for real-time operation?

A CUDA-capable GPU with sufficient VRAM for SAM and VLM inference simultaneously. Exact specs are in INSTALL.md.

Can I use DAAAM without ROS 2?

Yes—the core library is ROS-agnostic. ROS 2 integration is provided separately via DAAAM-ROS.

What license applies?

BSD 3-Clause "New" or "Revised" License. Permissive for academic and commercial use.

How does this differ from base Hydra?

DAAAM adds temporal tracking (4D), VLM-grounded open-vocabulary descriptions, and an optimization frontend for semantic consistency.

Are pretrained model weights included?

No—download instructions are in INSTALL.md. Weights for SAM, BotSort, and the VLM are fetched separately.

What benchmarks validate the claims?

NaVQA and SG3D. See the CVPR 2026 paper for detailed results.

Is the ROS 2 interface actively maintained?

Yes, as a separate repository at DAAAM-ROS.

Conclusion

MIT-SPARK/DAAAM addresses a genuinely hard problem at the frontier of robot perception: building semantically rich, temporally consistent scene representations in real time. Its foundation-model-first design—SAM for segmentation, BotSort for tracking, VLMs for grounding—reflects where the field is heading, while its optimization frontend ensures these powerful but noisy predictions become geometrically and temporally coherent structures.

The system is best suited for researchers and engineers building next-generation autonomous robots that must understand and reason about dynamic environments in open-vocabulary terms. The hardware requirements and system complexity are non-trivial, but the capability—real-time 4D scene graphs with natural language descriptions—is currently unmatched in open-source robotics.

If your work involves robot mapping, embodied AI, or spatio-temporal reasoning, start with the repository at https://github.com/MIT-SPARK/DAAAM. Review INSTALL.md for setup, CODEBASE.md for architecture, and the arXiv paper for theoretical foundations. For production deployment, integrate via DAAAM-ROS.