going-doer/Paper2Code: Automating ML Paper-to-Code with Multi-Agent LLMs

going-doer/Paper2Code: Automating ML Paper-to-Code with Multi-Agent LLMs

Reproducing machine learning research remains one of the most time-intensive tasks in the field. A developer or researcher typically spends hours—sometimes days—poring over dense academic papers, extracting architectural details, and translating mathematical notation into working code. The gap between published research and executable implementation creates friction that slows validation, blocks reproducibility efforts, and stalls downstream innovation. going-doer/Paper2Code addresses this directly with PaperCoder, a multi-agent LLM system designed to automate code generation from scientific papers in machine learning. Rather than treating paper-to-code as a single translation step, PaperCoder decomposes the problem into specialized planning, analysis, and generation phases—each handled by dedicated agents. With 4,799 GitHub stars, 669 forks, and acceptance at ICLR 2026, this Apache 2.0-licensed Python^{↗ Bright Coding Blog} project has gained substantial traction among developers who need faithful, high-quality implementations without manual extraction.

What is going-doer/Paper2Code?

going-doer/Paper2Code is an open-source research artifact and benchmark suite developed by Minju Seo^{↗ Bright Coding Blog}, Jinheon Baek, Seongyun Lee, and Sung Ju Hwang (with Baek and Hwang as equal advisors). It was published at the International Conference on Learning Representations (ICLR) 2026, with the full paper available on arXiv.

At its core, the project introduces PaperCoder: a multi-agent LLM system that transforms ML research papers into complete code repositories. Unlike single-prompt code generation tools, PaperCoder follows a three-stage pipeline:

1. Planning: An agent decomposes the paper's methodology into implementable components
2. Analysis: Specialized agents extract architectural details, hyperparameters, and training procedures
3. Code Generation: Agents produce the actual implementation with proper module structure

This decomposition mirrors how experienced researchers actually approach reproduction—first understanding the paper's structure, then drilling into specifics, then writing code. The system supports both PDF-based inputs (via JSON conversion) and LaTeX source directly, giving flexibility depending on what's available.

The project also includes Paper2Code Benchmark Datasets (hosted on Hugging Face) and evaluation infrastructure for measuring generation quality. The last commit was March 25, 2026, indicating active maintenance.

Key Features

Multi-Agent Architecture with Role Specialization PaperCoder's design assigns distinct responsibilities to different agents rather than overloading a single model. The planning agent establishes implementation scope, analysis agents extract technical specifics from paper sections, and coding agents generate actual Python modules. This separation reduces context window pressure and improves coherence for long, complex papers.

Dual Input Format Support The system accepts both PDF-derived JSON (via the s2orc-doc2json pipeline) and raw LaTeX source. PDF conversion uses Grobid for structured extraction, while LaTeX mode skips conversion entirely—useful when authors release source files. The README explicitly notes that their experiments used PDF-to-JSON conversion for all papers.

Flexible Backend Options Developers can run PaperCoder with OpenAI's API (o3-mini estimated at $0.50–$0.70 per paper) or open-source models via vLLM, with deepseek-ai/DeepSeek-Coder-V2-Lite-Instruct as the default. This dual-path design accommodates both commercial users needing reliability and researchers preferring local, auditable inference.

Structured Output Artifacts Generation produces organized directory trees with planning_artifacts, analyzing_artifacts, coding_artifacts, and a final _repo directory containing the executable implementation. This traceability helps debug failures and audit agent decisions.

Model-Based Evaluation Framework The included evaluation system supports both reference-free critique (scoring implementation correctness against paper claims) and reference-based comparison against author-released code. Both use o3-mini-high to generate 1–5 correctness scores averaged over 8 samples, with detailed token cost reporting.

Benchmark Dataset on Hugging Face The Paper2Code benchmark enables standardized comparison across methods, with dataset descriptions in data/paper2code and full methodological details in Section 4.1 of the paper.

Use Cases

Research Reproduction for Verification When a new paper claims state-of-the-art results, ML engineers can use PaperCoder to rapidly generate a baseline implementation for independent verification. The multi-agent approach catches architectural details that single-pass generation might miss, such as specific normalization placements or custom loss terms.

Educational Code Generation Students and self-directed learners working through seminal papers (the repository includes "Attention Is All You Need" as a running example) can generate reference implementations to study alongside the text. The artifact directories reveal how the system interprets each paper section, providing pedagogical value beyond the final code.

Benchmarking New Generation Methods The Paper2Code benchmark and evaluation framework serve as a standardized testbed for researchers developing improved code generation techniques. The reference-based evaluation mode enables direct comparison against ground-truth author implementations.

Accelerating Prototype Development ML practitioners exploring methods from adjacent subfields can use PaperCoder to bootstrap implementations without domain-specific expertise. The planning stage's decomposition helps identify which components require custom development versus off-the-shelf alternatives.

Repository Auditing and Gap Analysis The reference-free evaluation mode enables quality assessment of existing or generated repositories against paper claims, useful for maintainers verifying that implementations haven't drifted from described methods.

Installation & Setup

The project requires Python and recommends a virtual environment. Dependencies are modular based on your backend choice.

Basic Environment Setup

# Recommended: create and activate a virtual environment first
# python -m venv venv && source venv/bin/activate

# For OpenAI API usage
pip install openai

# For open-source models via vLLM
pip install vllm

# Or install all dependencies at once
pip install -r requirements.txt

Note: The README emphasizes using the latest openai package for o3-mini support. If vLLM installation fails, consult the official vLLM repository for platform-specific guidance.

PDF-to-JSON Conversion (Optional)

If using PDF input without LaTeX source:

# Clone the conversion toolchain
git clone https://github.com/allenai/s2orc-doc2json.git

# Start the Grobid processing service
cd ./s2orc-doc2json/grobid-0.7.3
./gradlew run

# In another terminal, convert your PDF
mkdir -p ./s2orc-doc2json/output_dir/paper_coder
python ./s2orc-doc2json/doc2json/grobid2json/process_pdf.py \
    -i ${PDF_PATH} \
    -t ./s2orc-doc2json/temp_dir/ \
    -o ./s2orc-doc2json/output_dir/paper_coder

This produces a structured JSON representation that PaperCoder consumes. If you have LaTeX source, skip this step entirely.

Real Code Examples

The repository provides complete shell-based workflows. Below are the exact commands from the README, with explanatory context.

Running with OpenAI API (PDF Input)

# Set your API key
export OPENAI_API_KEY="<OPENAI_API_KEY>"

# Execute the default example (Attention Is All You Need)
cd scripts
bash run.sh

This runs the full PaperCoder pipeline on the included Transformer example. The run.sh script orchestrates the three-stage agent pipeline and writes outputs to the outputs/ directory. Estimated cost: $0.50–$0.70 using o3-mini.

Running with OpenAI API (LaTeX Input)

export OPENAI_API_KEY="<OPENAI_API_KEY>"

cd scripts
bash run_latex.sh

Identical pipeline, but reads .tex source directly—faster when available, avoiding PDF conversion overhead and potential extraction errors.

Running with vLLM (Open-Source Models)

# PDF input with local inference
cd scripts
bash run_llm.sh

# LaTeX input with local inference
bash run_latex_llm.sh

The default model is deepseek-ai/DeepSeek-Coder-V2-Lite-Instruct. Local execution eliminates API costs but requires sufficient GPU memory for inference.

Output Structure

outputs
├── Transformer
│   ├── analyzing_artifacts      # Extracted technical details
│   ├── coding_artifacts         # Generated code modules
│   └── planning_artifacts       # Implementation decomposition
└── Transformer_repo             # Final executable repository

The artifact directories enable debugging and auditing; the _repo directory contains the runnable implementation.

Evaluation Example (Reference-Based)

# Install evaluation dependency
pip install tiktoken
export OPENAI_API_KEY="<OPENAI_API_KEY>"

# Run evaluation comparing generated vs. ground-truth code
cd codes/
python eval.py \
    --paper_name Transformer \
    --pdf_json_path ../examples/Transformer_cleaned.json \
    --data_dir ../data \
    --output_dir ../outputs/Transformer \
    --target_repo_dir ../outputs/Transformer_repo \
    --gold_repo_dir ../examples/Transformer_gold_repo \
    --eval_result_dir ../results \
    --eval_type ref_based \
    --generated_n 8 \
    --papercoder

This produces a detailed score report including correctness ratings, validity counts, and token cost breakdowns.

Advanced Usage & Best Practices

Input Quality Determines Output Quality The README notes that their experiments used PDF-to-JSON conversion for all papers, implying this is the tested path. However, LaTeX source avoids extraction artifacts from Grobid—if you have access to .tex files, prefer run_latex.sh or run_latex_llm.sh for potentially cleaner results.

Cost Monitoring for API Mode The evaluation script reports exact token costs (e.g., $0.16451380 for a reference-based run). For batch processing, wrap the execution in cost-tracking scripts—the per-paper o3-mini estimate of $0.50–$0.70 scales linearly with paper complexity.

Local Inference for Sensitive Research For unpublished or proprietary methods, vLLM mode keeps all computation local. The default DeepSeek-Coder-V2-Lite-Instruct provides a balance of capability and resource requirements, though the README doesn't specify minimum GPU memory—plan for testing with your hardware.

Artifact Inspection for Debugging When PaperCoder produces unexpected implementations, examine planning_artifacts first: misalignment at the planning stage propagates through analysis and coding. The modular output design enables targeted intervention without rerunning the full pipeline.

Benchmark Integration for Method Development Researchers improving upon PaperCoder should use the Hugging Face dataset for standardized training and evaluation. The reference-based evaluation mode with gold_repo_dir enables fair comparison against author implementations.

Comparison with Alternatives

Trade-offs to consider: PaperCoder's multi-agent design adds latency versus single-prompt generation but improves coherence for complex papers. The requirement for PDF conversion (when LaTeX is unavailable) introduces a dependency on Grobid's extraction quality. For papers with existing author code, direct use remains preferable; PaperCoder fills the gap where code is unpublished or incomplete.

FAQ

What does going-doer/Paper2Code cost to run? Using OpenAI's o3-mini: approximately $0.50–$0.70 per paper. Evaluation adds roughly $0.16 per comparison run. vLLM mode eliminates API costs but requires local GPU resources.

Can I use my own research papers? Yes. Modify the environment variables in the run scripts to point to your paper's PDF or LaTeX source, then execute the appropriate run*.sh script.

What license covers the code? Apache License 2.0—permissive for commercial and research use with attribution.

How accurate are the generated implementations? The paper reports outperforming strong baselines on Paper2Code and PaperBench benchmarks. The evaluation framework produces 1–5 scores averaged over 8 samples; example output shows 4.5/5 for the Transformer reference-based evaluation.

Do I need Grobid for all inputs? No—only for PDF inputs without LaTeX source. If you have .tex files, use run_latex.sh or run_latex_llm.sh directly.

What Python version is required? The README doesn't specify; the primary language is Python, and standard virtual environment practice applies.

Is the benchmark dataset publicly available? Yes, on Hugging Face as iaminju/paper2code.

Conclusion

going-doer/Paper2Code offers a genuinely useful approach to a persistent ML infrastructure problem: bridging the gap between published research and executable code. Its multi-agent architecture—planning, analysis, generation—reflects how skilled practitioners actually work, while the dual backend support (OpenAI API or local vLLM) accommodates diverse operational constraints.

The tool is best suited for researchers verifying claims, engineers exploring methods from adjacent fields, and benchmark developers needing standardized evaluation infrastructure. It's not a replacement for reading papers or understanding methods, but it dramatically accelerates the mechanical translation from mathematical description to running code.

With 4,799 stars, active maintenance through March 2026, and peer-reviewed validation at ICLR, the project has demonstrated both technical credibility and community traction. The Apache 2.0 license removes adoption friction for commercial and academic use.

Ready to automate your next paper reproduction? Explore the full repository, benchmark datasets, and paper at https://github.com/going-doer/Paper2Code.