Stop Labeling Thousands of Images! SAM 3 Segments Anything with Just Text

Stop Labeling Thousands of Images! SAM 3 Segments Anything with Just Text

What if you could point at any object in a video—living, manufactured, or even abstract—and simply whisper "that one" to your code? No bounding boxes. No painstaking polygon masks. No armies of labelers burning through your budget. Just text. Sounds like science fiction, right? Here's the gut punch: Meta just made it real. And if you're still hand-crafting COCO-format annotations in 2025, you're leaving insane productivity gains on the table.

The painful truth? Traditional computer vision pipelines are annotation prisons. Want to detect "a player in white" versus "a player in red"? Prepare for weeks of labeling, class rebalancing, and model retraining. Need to track that same player across 10,000 video frames? Multiply your pain by a thousand. Existing open-vocabulary detectors like OWLv2 crumble when concepts get nuanced. Specialized trackers demand pristine initialization masks. The gap between "I can describe it" and "my model can find it" has been a chasm swallowing ML engineering hours whole.

Enter SAM 3—the third evolution of Meta's Segment Anything Model that doesn't just segment anything, but understands concepts. Released by Meta Superintelligence Labs, this 848M-parameter foundation model accepts text prompts, visual exemplars, points, boxes, and masks to detect, segment, and track objects across images and videos. With a new SA-Co benchmark covering 270,000 unique concepts (50× more than existing benchmarks), SAM 3 achieves 75-80% of human performance on open-vocabulary segmentation. The annotation bottleneck? Obliterated. The future of computer vision just arrived—and it speaks your language, literally.

What is SAM 3?

SAM 3 (Segment Anything Model 3) is Meta's unified foundation model for promptable segmentation in images and videos, officially released through the facebookresearch/sam3 repository. Building upon SAM 2's video tracking capabilities, SAM 3 introduces a paradigm shift: exhaustive segmentation of open-vocabulary concepts specified by text phrases or visual exemplars. This isn't incremental improvement—it's a fundamental reimagining of how machines understand visual semantics.

The model emerges from Meta Superintelligence Labs' massive research collective, led by core contributors Nicolas Carion, Laura Gustafson, Yuan-Ting Hu, and others, with project direction from Piotr Dollár, Nikhila Ravi, Kate Saenko, Pengchuan Zhang, and Christoph Feichtenhofer. Their breakthrough rests on two pillars: an automated data engine that annotated 4 million unique concepts (creating the largest high-quality open-vocabulary segmentation dataset ever), and architectural innovations including a presence token for fine-grained text discrimination and a decoupled detector–tracker design that eliminates task interference.

Why is SAM 3 trending now? Three forces converge. First, the release of SAM 3.1 Object Multiplex (March 27, 2026) introduces shared-memory joint multi-object tracking—dramatically faster without accuracy loss. Second, the SA-Co benchmark exposes how pitifully existing models handle real-world concept diversity: OWLv2 achieves 24.6 cgF1 on SA-Co/Gold versus SAM 3's 54.1 cgF1, while human performance sits at 72.8. Third, the practical implications are immediate—developers can now build vision systems that respond to natural language without retraining, a capability previously locked behind proprietary APIs or impossible engineering sprints.

The architecture itself tells the story: a shared vision encoder feeds both a DETR-based detector (conditioned on text, geometry, and image exemplars) and a SAM 2-derived transformer tracker. This dual design means SAM 3 doesn't just find objects via text—it remembers them across video frames with interactive refinement. For ML engineers, this collapses what previously required separate detection, segmentation, and tracking pipelines into a single, queryable model.

Key Features That Change Everything

Open-Vocabulary Text Segmentation at Scale. SAM 3's defining superpower is handling "a vastly larger set of open-vocabulary prompts" than any prior work. We're not talking about 1,000 ImageNet classes—this model reasons about 270,000 unique concepts in the SA-Co benchmark. The presence token architecture specifically addresses the nightmare scenario of similar text prompts: distinguishing "a player in white" from "a player in red" with precision that previous models simply couldn't muster.

Unified Image-Video Architecture. One model, two modalities. SAM 3's detector–tracker decoupling means image segmentation and video tracking share representations but minimize interference. The tracker inherits SAM 2's proven encoder-decoder structure, while the detector adds text-conditioning through DETR-style queries. For deployment, this means consistent APIs and unified checkpoints rather than fragile multi-model orchestration.

Exhaustive Instance Segmentation. Unlike detection models that return bounding boxes, SAM 3 outputs pixel-precise masks for every instance matching your text prompt. Need all "scratched brake rotors" in a quality inspection video? Every single one, individually masked, with tracking IDs across frames. This is segmentation as database query—semantic precision with instance completeness.

Interactive Refinement Pipeline. SAM 3 doesn't demand perfect prompts. Start with text, then refine with points, boxes, or additional masks. The video predictor supports arbitrary frame indexing for prompt injection, enabling correction and guidance at any temporal point. For production systems, this means human-in-the-loop workflows that converge faster than any pure-automation approach.

Production-Ready Engineering. CUDA 12.6 support, PyTorch 2.7+ compatibility, Flash Attention 3 integration for faster inference, and batched inference notebooks for throughput optimization. The SAM 3.1 checkpoints on Hugging Face represent actively maintained, performance-improved model weights—not abandonware.

Use Cases Where SAM 3 Destroys the Competition

Autonomous Vehicle Perception with Natural Language Queries. Imagine a safety operator typing "pedestrian with stroller crossing against signal" into a monitoring dashboard, and SAM 3 instantly highlights every matching instance across 8 camera feeds with persistent tracking. No pre-trained class for "stroller." No bounding box annotations from 10,000 hours of driving. Just the description, understood and executed. The decoupled tracker maintains IDs as objects move between camera views, solving the multi-camera handoff problem that plagues conventional tracking systems.

Medical Imaging Analysis Without Domain Retraining. Radiologists describe findings in precise, non-standardized language: "ill-defined ground-glass opacity in the left upper lobe periphery." Traditional systems require curated datasets and fine-tuning for each anatomical variation. SAM 3's open-vocabulary capability means the same model segments based on clinical descriptions immediately, with interactive point refinement for boundary precision. The presence token's discrimination power handles subtle differentiators like "peripheral" versus "central" without architectural changes.

Content Moderation at Scale. Video platforms need to identify policy-violating content that evolves faster than classifier training cycles. "Dangerous stunt involving moving vehicle" or "unauthorized trademark on counterfeit merchandise"—these concepts resist fixed taxonomies. SAM 3 enables moderation teams to query emergent violation patterns in natural language, segment all matching instances exhaustively, and track their persistence across video segments. The batched inference capabilities process thousands of frames efficiently.

Robotics and Embodied AI. A warehouse robot receiving the instruction "pick up the dented blue bin behind the pallet jack" needs to parse spatial relationships, object attributes, and visual conditions simultaneously. SAM 3's multi-modal prompting (text + box + point) grounds language in visual perception with the precision required for physical interaction. The video tracking ensures the robot maintains object lock even as viewpoints shift during approach.

Scientific Research and Biodiversity Monitoring. Ecologists studying "juvenile male elephant seals with fresh propeller scars" face impossible annotation requirements for rare, visually complex phenomena. SAM 3 transforms descriptive field notes directly into segmentation queries across drone footage archives, enabling population-scale analysis without prior model training on each specific condition.

Step-by-Step Installation & Setup Guide

SAM 3 demands modern infrastructure—don't expect this to run on your 2019 laptop. Here's the precise path from zero to inference.

Prerequisites

• Python^{↗ Bright Coding Blog} 3.12 or higher (non-negotiable; type hints and asyncio patterns depend on recent versions)
• PyTorch 2.7 or higher with CUDA support
• CUDA-compatible GPU with CUDA 12.6 or higher (the model's 848M parameters and attention mechanisms require substantial VRAM)

Environment Setup

Create and activate an isolated Conda environment to avoid dependency conflicts with existing PyTorch installations:

# Create fresh environment with Python 3.12
conda create -n sam3 python=3.12
conda deactivate  # Ensure clean state
conda activate sam3

PyTorch Installation

Install PyTorch with CUDA 12.8 support (the README specifies this version for optimal compatibility):

pip install torch==2.10.0 torchvision --index-url https://download.pytorch.org/whl/cu128

Repository Installation

Clone the official repository and install in editable mode for development flexibility:

git clone https://github.com/facebookresearch/sam3.git
cd sam3
pip install -e .

Optional Dependencies

For Jupyter notebook examples (essential for learning the API):

pip install -e ".[notebooks]"

For training and development workflows:

pip install -e ".[train,dev]"

For maximum inference speed, install Flash Attention 3 and optimized kernels:

pip install einops ninja && pip install flash-attn-3 --no-deps --index-url https://download.pytorch.org/whl/cu128
pip install git+https://github.com/ronghanghu/cc_torch.git

Critical: Model Access Authentication

Before any inference, request access to SAM 3 checkpoints on the Hugging Face repository. Once approved, authenticate your environment:

# Install huggingface-hub if not present
pip install huggingface-hub

# Authenticate with your access token
huggingface-cli login
# Or: hf auth login (depending on hub version)

Without this step, checkpoint downloads will fail with authentication errors that look like repository access issues.

Launching Examples

# Start with the core image predictor example
jupyter notebook examples/sam3_image_predictor_example.ipynb

REAL Code Examples from the Repository

The SAM 3 repository provides clean, production-ready APIs that abstract the complex detector-tracker architecture. Here are the exact patterns from the official README, explained with the technical depth you need to deploy confidently.

Image Segmentation with Text Prompts

This is SAM 3's signature capability—natural language to pixel-precise masks in five lines of effective code:

import torch
from PIL import Image
from sam3.model_builder import build_sam3_image_model
from sam3.model.sam3_image_processor import Sam3Processor

# Initialize the 848M-parameter image model
# This loads the detector branch with text-conditioned DETR queries
model = build_sam3_image_model()

# Processor handles image preprocessing, tokenization, and output decoding
processor = Sam3Processor(model)

# Load target image—standard PIL Image, any format
image = Image.open("<YOUR_IMAGE_PATH.jpg>")

# Set image establishes the visual feature cache for this inference session
# The shared vision encoder computes multi-scale features once, reused for all prompts
inference_state = processor.set_image(image)

# The magic: text prompt drives segmentation without any bounding box or mask hint
# The presence token in the architecture resolves ambiguities in natural language
output = processor.set_text_prompt(
    state=inference_state, 
    prompt="<YOUR_TEXT_PROMPT>"  # e.g., "a player in white jersey"
)

# Extract structured outputs: binary masks, axis-aligned boxes, confidence scores
# Masks are full-resolution, ready for downstream processing or visualization
masks, boxes, scores = output["masks"], output["boxes"], output["scores"]

What's happening under the hood? The build_sam3_image_model() instantiates the detector branch—DETR architecture with text encoding via the presence token mechanism. Sam3Processor manages the interface: image normalization, vision encoder forward pass, and prompt encoding. The critical insight is that set_image computes and caches visual features, making subsequent text prompts (or geometric prompts) computationally cheap. The presence token specifically addresses the "player in white" vs "player in red" discrimination problem by encoding fine-grained attribute relationships in the text-to-visual query space.

Video Segmentation with Text Initialization

Video processing leverages the full detector-tracker architecture, with text prompts establishing initial object identities that persist across frames:

from sam3.model_builder import build_sam3_video_predictor

# Video predictor instantiates both detector and tracker with shared encoder
video_predictor = build_sam3_video_predictor()

# Supports JPEG frame folders or MP4 files directly
video_path = "<YOUR_VIDEO_PATH>"

# Step 1: Initialize tracking session
# The predictor allocates per-video state for temporal consistency
response = video_predictor.handle_request(
    request=dict(
        type="start_session",
        resource_path=video_path,
    )
)
# response contains session_id for subsequent operations

# Step 2: Add text prompt at arbitrary frame index
# This triggers detection on that frame, then tracker initialization
response = video_predictor.handle_request(
    request=dict(
        type="add_prompt",
        session_id=response["session_id"],
        frame_index=0,  # Can be any frame; not restricted to start
        text="<YOUR_TEXT_PROMPT>",  # e.g., "scratched brake rotor"
    )
)

# outputs contain masks, boxes, scores, and track IDs for this frame
# Tracker propagates these identities forward and backward in time
output = response["outputs"]

Why this API design matters: The handle_request pattern with explicit session management enables multi-object, multi-prompt, temporally-flexible workflows. Unlike rigid video APIs that demand frame-by-frame processing or single initialization, SAM 3 allows prompt injection at any temporal point. The frame_index=0 is illustrative—you could correct tracking drift at frame 10,000 with a refined text prompt. The decoupled detector-tracker means text prompts invoke the detector (expensive, one-time per prompt), while frame-to-frame propagation uses the lightweight tracker. This architectural separation is what enables SAM 3.1's shared-memory multi-object acceleration.

Batched Inference for Production Throughput

For processing large image collections, the repository includes sam3_image_batched_inference.ipynb demonstrating how to amortize model loading across many images and exploit GPU parallelism. The pattern extends the basic image API with tensor batching and asynchronous preprocessing—essential for any production deployment where single-image latency would bottleneck throughput.

Advanced Usage & Best Practices

Prompt Engineering for Precision. SAM 3's text understanding is remarkable but not magical. Use specific, visually-grounded descriptions: "red circular sign with white horizontal bar" outperforms "stop sign" in ambiguous contexts. Leverage the interactive refinement—start broad, then add point prompts to correct boundaries. The presence token helps with attribute discrimination, but explicit language reduces ambiguity.

Memory Management for Video Sessions. Each start_session allocates GPU memory for temporal feature caches. Explicitly close sessions when done, or reuse session IDs for sequential video processing. For long videos, consider chunking into overlapping segments and merging track IDs post-hoc—SAM 3's tracker maintains local consistency exceptionally well.

Checkpoint Versioning. SAM 3.1 checkpoints represent meaningful improvements; always verify you're using the latest by running git pull and reinstalling when updates are announced. The Hugging Face repository hosts versioned weights—pin to specific revisions for reproducible production deployments.

Flash Attention for Latency-Critical Paths. The optional Flash Attention 3 installation isn't cosmetic—it reduces memory bandwidth pressure on the transformer layers, yielding 20-40% throughput improvements on modern GPUs (A100, H100). For real-time video applications, this is often the difference between meeting and missing frame-rate requirements.

SA-Co Evaluation for Domain Validation. Before deploying SAM 3 on a new visual domain, sample evaluation on the relevant SA-Co split (Gold for high-precision needs, Silver for diversity, VEval for video) establishes realistic performance expectations. The benchmark's 270K concepts provide better transfer estimation than narrow academic datasets.

Comparison with Alternatives

The verdict: SAM 2 remains excellent for geometric-prompt segmentation but cannot understand text. OWLv2 and DINO-X handle text-to-detection but lack pixel masks and video tracking. Gemini 2.5 is a generalist with poor segmentation precision (13.0 cgF1 vs SAM 3's 54.1). SAM 3 uniquely combines text comprehension, instance segmentation, and temporal tracking in an open, reproducible, production-optimized package.

FAQ

What hardware do I need to run SAM 3? A CUDA-compatible GPU with 12.6+ support is required. For the full 848M model, 16GB+ VRAM is recommended; smaller configurations or CPU offloading may work with reduced batch sizes. The Flash Attention optimizations particularly benefit Ampere and Hopper architectures.

How does SAM 3 handle negative prompts (objects that don't exist)? The SA-Co benchmark explicitly includes negative prompts with no matching objects. SAM 3 returns empty mask tensors and near-zero confidence scores for these cases, enabling reliable existence detection without separate classification heads.

Can I fine-tune SAM 3 on my domain? Yes—pip install -e ".[train,dev]" installs training dependencies. The repository supports fine-tuning both the detector and tracker components, though the pre-trained open-vocabulary capabilities often reduce or eliminate domain-specific training needs.

Is SAM 3.1 backward compatible with SAM 3 checkpoints? No—you need the latest code (git pull and reinstall) to use SAM 3.1 checkpoints. The model architecture improvements, particularly the shared-memory multi-object tracking, require updated implementation code.

How does the presence token actually work? The presence token is an architectural innovation that encodes fine-grained attribute relationships in the text-to-visual query space. For prompts like "player in white," it strengthens the "white" attribute binding against the "player" category, reducing confusion with similar descriptions. This is learned automatically from the 4M-concept dataset.

What's the latency for real-time video applications? With Flash Attention 3 and optimized kernels, SAM 3.1 achieves significant speedups for multi-object tracking. Exact latency depends on resolution, object count, and GPU, but the shared-memory design specifically targets real-time feasibility that SAM 3's baseline couldn't guarantee.

Can SAM 3 replace my entire detection-segmentation-tracking pipeline? For open-vocabulary scenarios, often yes. For fixed, small taxonomies with extreme latency requirements, optimized specialist models may still win. Evaluate on SA-Co or your specific domain to determine the tradeoff.

Conclusion

SAM 3 isn't an incremental upgrade—it's a fundamental reframing of what's possible in computer vision. The leap from geometric prompts to natural language understanding, at scale, with video tracking, collapses weeks of annotation and pipeline engineering into minutes of prompt iteration. The numbers don't lie: 54.1 cgF1 on SA-Co/Gold where competitors struggle below 25, 270K concepts understood, 4 million concepts learned during training.

For ML engineers, this means shipping vision features that previously required dedicated research teams. For product teams, it means responding to user needs with descriptive queries instead of training sprints. For researchers, it means asking questions of visual data that were previously unaskable.

The repository at facebookresearch/sam3 is actively maintained, with SAM 3.1 representing ongoing investment in performance and capability. The example notebooks provide immediate hands-on experience; the SA-Co benchmarks enable rigorous evaluation. If you're building anything involving visual understanding in 2025, SAM 3 deserves your immediate attention—and your git clone.

Stop annotating. Start describing. Segment anything with concepts.