NotaGen: Why Composers Are Ditching Traditional Tools for AI Sheet Music

NotaGen: Why Composers Are Ditching Traditional Tools for AI Sheet Music

What if your next symphony could be composed by artificial intelligence? Not generated audio you can't edit—but actual, readable, publishable sheet music that real musicians can perform from. For centuries, composers wrestled with blank staves and erasers. Today, a revolutionary open-source project is changing everything.

Meet NotaGen, the breakthrough symbolic music generation model that's making traditional composition tools look obsolete. While most AI music tools pump out uneditable audio files, NotaGen generates ABC notation that converts to professional MusicXML scores. We're talking about a system trained on 1.6 million musical pieces, fine-tuned on nearly 9,000 classical compositions, and optimized through a novel reinforcement learning technique called CLaMP-DPO—all without needing a single human annotation.

The secret? NotaGen treats music like language. By applying Large Language Model training paradigms to symbolic music, researchers have cracked something that eluded the field for years: generating music with genuine musicality. The kind that respects period style, composer voice, and proper instrumentation. If you're a developer, musician, or researcher in computational creativity, ignoring this tool means missing the biggest shift in algorithmic composition since MIDI itself.

Ready to see how it works? Let's dive deep.

What is NotaGen?

NotaGen is a symbolic music generation model developed by researchers including Yashan Wang, Shangda Wu, and colleagues. Published in February 2025 with arXiv paper 2502.18008, it represents a paradigm shift in how we approach computational music composition.

Unlike audio-generation models like MusicLM or Jukebox that produce waveforms, NotaGen operates in the symbolic domain—generating structured notation that can be rendered as sheet music, edited in notation software, and performed by human musicians. This distinction matters enormously for working composers, educators, and publishers who need interoperable, editable output.

The project's core innovation lies in its three-stage training pipeline borrowed from LLM success stories:

1. Massive pre-training on 1.6 million diverse musical pieces to learn general musical grammar
2. Style-specific fine-tuning on ~9,000 classical works using structured prompts
3. Reinforcement learning optimization via CLaMP-DPO for quality refinement

What's driving NotaGen's rapid adoption? Three converging trends: the maturation of transformer architectures for non-text sequences, growing demand for controllable AI creativity tools, and frustration with black-box audio generators. The GitHub repository has become a hub for developers building next-generation music applications, with community contributions including ComfyUI nodes and Google Colab notebooks.

The project also released NotaGen-X, an enhanced version inspired by DeepSeek-R1's training methodology, featuring improved post-training stages and more reasonable instrument ranges. This iterative improvement culture signals a project with serious long-term momentum.

Key Features That Set NotaGen Apart

NotaGen isn't just another music AI. Its technical architecture reveals sophisticated engineering decisions that solve real problems in symbolic generation:

Dual-Decoder Hierarchical Architecture

NotaGen employs a patch-level decoder (processing musical segments) and a character-level decoder (handling fine-grained notation details). This hierarchical design mirrors how musicians actually think—grasping phrase structure before noting individual pitches. The small variant uses 12 patch layers with 3 character layers; the large scales to 20 and 6 respectively, with 1,280-dimensional hidden states.

Period-Composer-Instrumentation Conditioning

The fine-tuning stage introduces structured semantic prompts that control generation with unprecedented precision. Want a Baroque piece for solo harpsichord? A Romantic string quartet in Brahms's style? NotaGen accepts these as compositional directives, not afterthoughts. This solves the controllability crisis plaguing generative models.

CLaMP-DPO: Reward-Free Reinforcement Learning

Here's where things get genuinely innovative. Traditional RL for music requires human preference datasets or hand-crafted reward functions—both expensive and biased. CLaMP-DPO leverages the pre-trained CLaMP 2 music-text alignment model to automatically score generated pieces against stylistic references, then applies Direct Preference Optimization to improve the policy. No human annotators. No predefined rules. Just musical quality emerging from statistical alignment.

Multiple Model Scales for Different Needs

This scalability lets researchers prototype with small models and deploy production systems with large variants.

Rich Ecosystem Integration

From Hugging Face Spaces to ComfyUI nodes, from web demos to local Gradio interfaces, NotaGen meets users where they work. The automatic conversion pipeline handles ABC notation → MusicXML → audio rendering → PDF scores, eliminating format friction.

Real-World Use Cases Where NotaGen Shines

1. Educational Composition Tools

Music theory instructors need controllable examples for teaching. NotaGen can generate Bach-style chorales with specific voice-leading rules, or Romantic-era piano pieces demonstrating particular harmonic progressions. Students receive editable scores they can analyze in MuseScore or Sibelius—not passive audio files.

2. Game and Film Prototyping

Indie developers often need placeholder music matching specific moods and instrumentation. NotaGen's prompt system generates "Classical-Orchestral-Strings" or "Baroque-Harpsichord-Solo" pieces instantly, with notation ready for live musician replacement when budgets allow.

3. Style Transfer and Composer Emulation Research

Musicologists studying compositional style can fine-tune NotaGen on specific corpora—say, all known works by a lesser-studied Baroque composer—and generate hypothetical completions or variations. The Schubert lieder example in the repository demonstrates this workflow concretely.

4. Adaptive Music Systems

Interactive media needs music that responds to user input. Symbolic generation enables structural transformations (modulation, variation, recombination) impossible with audio models. NotaGen's ABC output feeds into real-time notation renderers for dynamic scores.

5. Accessibility and Assistive Technology

Visually impaired musicians relying on screen readers and Braille notation need structured symbolic output. Audio generators are useless here; NotaGen's MusicXML pipeline integrates with accessible notation workflows.

Step-by-Step Installation & Setup Guide

Getting NotaGen running requires careful environment management. Follow these exact steps from the official repository:

Environment Creation

# Create dedicated conda environment with Python↗ Bright Coding Blog 3.10
conda create --name notagen python=3.10
conda activate notagen

# Install PyTorch with CUDA 11.8 support
conda install pytorch==2.3.0 pytorch-cuda=11.8 -c pytorch -c nvidia

# Install Hugging Face utilities for model loading
pip install accelerate
pip install optimum

# Install remaining dependencies
pip install -r requirements.txt

Critical hardware note: NotaGen-large requires 24GB GPU memory for training and inference. For prototyping on consumer hardware, use NotaGen-small or medium variants, adjusting config.py accordingly. Local Gradio demos for NotaGen-X can run with 8GB GPU memory.

Model Weights Download

Pre-trained weights come in three scales from Hugging Face:

# Example: Download NotaGen-large pre-trained weights
wget https://huggingface.co/ElectricAlexis/NotaGen/blob/main/weights_notagen_pretrain_p_size_16_p_length_1024_p_layers_20_c_layers_6_h_size_1280_lr_0.0001_batch_4.pth

# Fine-tuned weights for conditional generation
wget https://huggingface.co/ElectricAlexis/NotaGen/blob/main/weights_notagen_pretrain-finetune_p_size_16_p_length_1024_p_layers_c_layers_6_20_h_size_1280_lr_1e-05_batch_1.pth

# RL-optimized weights (recommended for quality)
wget https://huggingface.co/ElectricAlexis/NotaGen/blob/main/weights_notagen_pretrain-finetune-RL3_beta_0.1_lambda_10_p_size_16_p_length_1024_p_layers_20_c_layers_6_h_size_1280_lr_1e-06_batch_1.pth

# NotaGen-X (latest improved version)
wget https://huggingface.co/ElectricAlexis/NotaGen/blob/main/weights_notagenx_p_size_16_p_length_1024_p_layers_20_h_size_1280.pth

Quick Demo Launch

For immediate experimentation, use the online Gradio demo or run locally:

# Navigate to gradio directory and follow README
cd gradio/
# See gradio/README.md for local deployment instructions

Google Colab users can use this community notebook contributed by @deeplearn-art for instant access with public Gradio links.

REAL Code Examples from the Repository

Let's examine actual implementation patterns from NotaGen's codebase, with detailed explanations of what each section accomplishes.

Example 1: Fine-Tuning Configuration Setup

Before training, you must configure data paths and model parameters. This Python configuration from finetune/config.py shows the critical setup:

# Configuration for the data
# Point to your preprocessed training data index
DATA_TRAIN_INDEX_PATH = "../data/schubert_augmented_train.jsonl" 
# Point to evaluation data for validation metrics
DATA_EVAL_INDEX_PATH  = "../data/schubert_augmented_eval.jsonl"

# Load pre-trained NotaGen-large weights as starting point
# This path must match your downloaded checkpoint location
PRETRAINED_PATH = "../pretrain/weights_notagen_pretrain_p_size_16_p_length_1024_p_layers_20_c_layers_6_h_size_1280_lr_0.0001_batch_4.pth"

# Experiment tag becomes part of output checkpoint names
# Use descriptive tags to track different fine-tuning experiments
EXP_TAG = "schubert"

What's happening here? The .jsonl index files contain metadata mapping musical pieces to their processed ABC representations. The PRETRAINED_PATH loads NotaGen-large's general musical knowledge, which gets specialized toward Schubert's lieder style through fine-tuning. The EXP_TAG ensures you don't overwrite checkpoints when running multiple experiments.

Example 2: Launching Fine-Tuning Training

With configuration set, execute training with this command:

# Navigate to fine-tuning directory
cd finetune/

# Restrict to single GPU (modify for multi-GPU if available)
# train-gen.py reads config.py automatically
CUDA_VISIBLE_DEVICES=0 python train-gen.py

Training dynamics: The script loads your pre-trained weights, initializes the optimizer with learning rates from config.py, and iterates through your indexed dataset. Validation runs periodically using DATA_EVAL_INDEX_PATH to prevent overfitting. Expect several hours for convergence on Schubert-sized datasets with NotaGen-large.

Example 3: CLaMP-DPO Inference for Preference Data Generation

The reinforcement learning pipeline begins with generating candidate pieces. This configuration from inference/config.py controls the sampling process:

# Path to your fine-tuned model weights
INFERENCE_WEIGHTS_PATH = '../finetune/weights_notagen_schubert_p_size_16_p_length_1024_p_layers_20_c_layers_6_h_size_1280_lr_1e-05_batch_1.pth'

# Number of candidate pieces to generate for preference ranking
# Higher numbers improve DPO training stability but increase compute cost
NUM_SAMPLES = 1000

Execute generation with:

cd inference/
python inference.py

Output structure explained: The script creates an output/ directory with two critical subdirectories:

• original/: Raw ABC notation straight from the model
• interleaved/: Post-processed with rest measure completion for CLaMP 2 compatibility

Each subdirectory contains model-specific folders named with sampling parameters (k_9_p_0.9_temp_1.2 indicates top-k=9, top-p=0.9, temperature=1.2). This systematic naming prevents confusion when comparing generation strategies.

Example 4: Feature Extraction for Automatic Scoring

CLaMP-DPO's magic depends on extracting semantic features from both ground truth and generated music. Modify clamp2/extract_clamp2.py:

# Source: your ground truth ABC files in interleaved format
input_dir = '../data/schubert_interleaved'
# Destination: computed feature vectors for similarity scoring
output_dir = 'feature/schubert_interleaved'

Then run extraction:

cd clamp2/
python extract_clamp2.py

Behind the scenes: CLaMP 2 processes musical notation through a specialized encoder, producing embedding vectors where cosine similarity measures stylistic alignment. This replaces human judgment with a pre-trained aesthetic model trained on millions of music-text pairs.

Example 5: Preference Data Construction for DPO

The critical step that enables reward-free RL—building preference pairs from automatically scored generations:

# Paths to previously extracted features
gt_feature_folder = '../clamp2/feature/schubert_interleaved'
output_feature_folder = '../clamp2/feature/weights_notagen_schubert_p_size_16_p_length_1024_p_layers_20_c_layers_6_h_size_1280_lr_1e-05_batch_1_k_9_p_0.9_temp_1.2'

# Original ABC files for the chosen/rejected pieces
output_original_abc_folder = '../output/original/weights_notagen_schubert_p_size_16_p_length_1024_p_layers_20_c_layers_6_h_size_1280_lr_1e-05_batch_1_k_9_p_0.9_temp_1.2'
output_interleaved_abc_folder = '../output/interleaved/weights_notagen_schubert_p_size_16_p_length_1024_p_layers_20_c_layers_6_h_size_1280_lr_1e-05_batch_1_k_9_p_0.9_temp_1.2'

# Output preference dataset for DPO training
data_index_path = 'schubert_RL1.json'  # First RL iteration
data_select_portion = 0.1  # Top/bottom 10% become chosen/rejected

Quality filtering rules (automatically applied):

• Duration alignment check: Excludes pieces with rhythmic inconsistencies
• Plagiarism guard: Removes pieces with >0.95 Levenshtein similarity to training data
• Instrument grouping validation: Ensures staves for identical instruments cluster together

These safeguards prevent degenerate outputs from contaminating the preference dataset, a common failure mode in naive RL implementations.

Advanced Usage & Best Practices

Memory Optimization for Large Models

NotaGen-large's 516M parameters demand strategic GPU management. Use accelerate's DeepSpeed integration for model parallelism, or quantize to int8 using optimum for inference. The 8GB Gradio demo achieves this through careful attention caching and gradient checkpointing.

Prompt Engineering for Period-Composer-Instrumentation

The fine-tuned model expects structured prompts. Experiment with:

• "Baroque-Bach-Organ" for contrapuntal complexity
• "Romantic-Brahms-Orchestral" for lush orchestral textures
• "Classical-Mozart-Piano-Solo" for clarity and balance

Invalid instrumentations (piano concerto with string quartet forces) may produce incoherent results—the model learned orchestration rules during fine-tuning.

Iterative RL Improvement

The published results show dramatic CLaMP 2 score improvements across DPO iterations:

Each iteration uses the previous RL checkpoint as its starting point. Budget at least 3 iterations for production-quality output.

Custom Dataset Preparation

For new composer styles, follow the data preprocessing pipeline exactly. ABC notation must be interleaved (multi-voice synchronized) for CLaMP 2 compatibility. The Schubert example dataset provides the canonical format reference.

Comparison with Alternatives

Why NotaGen wins: It's the only open system generating professionally usable sheet music with fine-grained stylistic control and automated quality optimization. MIDI-based alternatives require tedious cleanup; audio generators are fundamentally uneditable for musicians.

FAQ: Common Developer Questions

Q: Can I run NotaGen without a GPU? A: Inference on CPU is theoretically possible but impractically slow. The small model might generate short pieces in acceptable time. Use the Hugging Face Space demo for zero-hardware experimentation.

Q: What musical periods does NotaGen support? A: The fine-tuned model covers Baroque, Classical, Romantic, and Early Modern periods with specific composer conditioning. NotaGen-X improved instrument range合理性 for more authentic orchestration.

Q: How do I add a completely new composer style? A: See GitHub issue #18 for detailed instructions. You'll need ~100+ pieces in ABC format, preprocessing through the data pipeline, and full three-stage training.

Q: Is generated music copyright-safe? A: The plagiarism filter (ld_sim > 0.95 exclusion) provides strong protection against direct copying. However, consult legal counsel for commercial use—the model learned from copyrighted training data, and generated outputs may receive thin copyright protection in some jurisdictions.

Q: Can NotaGen generate jazz, pop, or electronic music? A: The released models specialize in Western classical traditions. The pre-training data (1.6M pieces) likely includes diverse genres, but fine-tuning and RL optimization focused on classical. Retraining on jazz/pop corpora is an active research direction.

Q: What's the difference between NotaGen and NotaGen-X? A: NotaGen-X adds a post-training stage between pre-training and fine-tuning, removes problematic key augmentation, and iterates RL with refreshed data. It's strictly superior and recommended for new projects.

Q: How does CLaMP-DPO compare to RLHF used in ChatGPT? A: CLaMP-DPO eliminates the need for human preference annotators entirely. Instead of paying humans to rank music, it uses the pre-trained CLaMP 2 model's embeddings for automatic scoring—then applies the same DPO algorithm for policy optimization. Cheaper, more scalable, and arguably more consistent.

Conclusion: The Future of Algorithmic Composition is Symbolic

NotaGen represents something rare in AI music: a system designed for musicians, not just around them. By committing to symbolic output, preserving editability, and solving the reward engineering problem through CLaMP-DPO, its creators built a tool that integrates into real creative workflows rather than replacing them.

The technical achievement is substantial—1.6M pieces pre-trained, novel hierarchical decoders, automatic quality optimization without human labels. But the practical impact matters more. Composers can now prototype faster. Educators can generate unlimited examples. Researchers can explore style spaces impossible to navigate manually.

My assessment? NotaGen is the most significant open-source music AI since Google's Magenta project, with far greater practical utility. The NotaGen-X improvements show active, thoughtful development. If you're building tools for creative professionals, symbolic generation with structured conditioning is the architecture to bet on.

Don't just read about it—generate your first piece today. Visit the official repository, try the web demo, or launch the Hugging Face Space for instant experimentation. The blank staff page is dead. Long live intelligent composition.

Ready to explore NotaGen's capabilities? Star the repository, join the discussion in GitHub issues, and share what you create. The future of music is open source.