SOTA Tracker

The definitive open-source database of State-of-the-Art AI models.

Auto-updated daily from LMArena, Artificial Analysis, and HuggingFace.

Why This Exists

AI models are released weekly. Keeping track is impossible. This project:

1. Curates authoritative data - LMArena Elo rankings, manual curation for video/image/audio models
2. Updates daily via GitHub Actions
3. Exports to JSON/CSV/SQLite - Use in your own projects
4. Provides multiple interfaces - Static files, REST API, or MCP server

Quick Start: Use the Data

Option 1: Download JSON/CSV

# Latest data (updated daily)
curl -O https://raw.githubusercontent.com/romancircus/sota-tracker-mcp/main/data/sota_export.json
curl -O https://raw.githubusercontent.com/romancircus/sota-tracker-mcp/main/data/sota_export.csv

Option 2: Clone and Query Locally

git clone https://github.com/romancircus/sota-tracker-mcp.git
cd sota-tracker-mcp

# Query with sqlite3
sqlite3 data/sota.db "SELECT name, sota_rank FROM models WHERE category='llm_api' ORDER BY sota_rank LIMIT 10"

# List forbidden/outdated models
sqlite3 data/sota.db "SELECT name, reason, replacement FROM forbidden"

Option 3: Use with Claude Code (Recommended)

The recommended approach for Claude Code users is static file embedding (lower token cost than MCP):

# Set up daily auto-update of CLAUDE.md
cp scripts/update_sota_claude_md.py ~/scripts/

# Enable systemd timer (runs at 6 AM daily)
systemctl --user enable --now sota-update.timer

# Or run manually
python ~/scripts/update_sota_claude_md.py --update

This embeds a compact SOTA summary directly in your ~/.claude/CLAUDE.md file.

Option 4: REST API

# Start the API server
uvicorn rest_api:app --host 0.0.0.0 --port 8000

# Query endpoints
curl "http://localhost:8000/api/v1/models?category=llm_api"
curl "http://localhost:8000/api/v1/forbidden"
curl "http://localhost:8000/api/v1/models/FLUX.1-dev/freshness"

Option 5: MCP Server (Optional)

MCP support is available but disabled by default (higher token cost). To enable:

# Edit .mcp.json to add the server config
cat > .mcp.json << 'EOF'
{
  "mcpServers": {
    "sota-tracker": {
      "command": "python",
      "args": ["server.py"]
    }
  }
}
EOF

Data Sources

| Source | Data | Update Frequency | |--------|------|------------------| | LMArena | LLM Elo rankings (6M+ human votes) | Daily | | Artificial Analysis | LLM benchmarks, pricing, speed | Daily | | HuggingFace | Model downloads, trending | Daily | | Manual curation | Video, Image, Audio, Video2Audio models | As needed |

Categories

| Category | Description | Top Models (Feb 2026) | |----------|-------------|----------------------| | llm_api | Cloud LLM APIs | Gemini 3 Pro, Grok 4.1, Claude Opus 4.5 | | llm_local | Local LLMs (GGUF) | Qwen3, Llama 3.3, DeepSeek-V3 | | llm_coding | Code-focused LLMs | Qwen3-Coder, DeepSeek-V3 | | image_gen | Image generation | Z-Image-Turbo, FLUX.2-dev, Qwen-Image | | video | Video generation | LTX-2, Wan 2.2, HunyuanVideo 1.5 | | video2audio | Video-to-audio (foley) | MMAudio V2 Large | | tts | Text-to-speech | ChatterboxTTS, F5-TTS | | stt | Speech-to-text | Whisper Large v3 | | embeddings | Vector embeddings | BGE-M3 |

REST API Endpoints

| Endpoint | Description | |----------|-------------| | GET /api/v1/models?category=X | Get SOTA for a category | | GET /api/v1/models/:name/freshness | Check if model is current or outdated | | GET /api/v1/forbidden | List outdated models to avoid | | GET /api/v1/compare?model_a=X&model_b=Y | Compare two models | | GET /api/v1/recent?days=30 | Models released in past N days | | GET /api/v1/recommend?task=chat | Get recommendation for a task | | GET /health | Health check |

Run Your Own Scraper

# Install dependencies
pip install -r requirements.txt
pip install playwright
playwright install chromium

# Run all scrapers
python scrapers/run_all.py --export

# Output:
# data/sota_export.json
# data/sota_export.csv
# data/lmarena_latest.json

GitHub Actions (Auto-Update)

This repo uses GitHub Actions to:

• Daily: Scrape all sources, update database, commit changes
• Weekly: Create a tagged release with JSON/CSV exports

To enable on your fork:

1. Fork this repo
2. Go to Settings → Actions → Enable workflows
3. Data will auto-update daily at 6 AM UTC

File Structure

sota-tracker-mcp/
├── server.py                    # MCP server (optional)
├── rest_api.py                  # REST API server
├── init_db.py                   # Database initialization + seeding
├── requirements.txt             # Dependencies
├── data/
│   ├── sota.db                  # SQLite database
│   ├── sota_export.json         # Full JSON export
│   ├── sota_export.csv          # CSV export
│   └── forbidden.json           # Outdated models list
├── scrapers/
│   ├── lmarena.py               # LMArena scraper (Playwright)
│   ├── artificial_analysis.py   # AA scraper (Playwright)
│   └── run_all.py               # Unified runner
├── fetchers/
│   ├── huggingface.py           # HuggingFace API
│   └── cache_manager.py         # Smart caching
└── .github/workflows/
    └── daily-scrape.yml         # GitHub Actions workflow

Contributing

Found a model that's missing or incorrectly ranked?

1. For manual additions: Edit init_db.py and submit a PR
2. For scraper improvements: Edit files in scrapers/
3. For new data sources: Add a new scraper and update run_all.py

See CONTRIBUTING.md for full developer setup and PR process.

OpenCode / Agents.md Integration

The repo now supports updating agents.md files for OpenCode agents:

# Update your agents.md with latest SOTA data
python update_agents_md.py

# Minimal version (top 1 model per category, lightweight)
python update_agents_md.py --minimal

# Custom categories and limit
python update_agents_md.py --categories llm_local image_gen --limit 3

# Force refresh from sources first
python update_agents_md.py --refresh

Automation

Add to your cron or systemd timer for daily updates:

# ~: crontab -e
@daily python ~/Apps/sota-tracker-mcp/update_agents_md.py

Or systemd:

# ~/.config/systemd/user/sota-update.service
[Unit]
Description=Update SOTA models for agents
After=network.target

[Service]
ExecStart=%h/Apps/sota-tracker-mcp/update_agents_md.py

[Install]
WantedBy=default.target

# ~/.config/systemd/user/sota-update.timer
[Unit]
Description=Daily SOTA data update
OnCalendar=daily
AccuracySec=1h

[Install]
WantedBy=timers.target

# Enable
systemctl --user enable --now sota-update.timer

See CONTRIBUTING.md for full setup guide

Data Attribution & Legal

This project aggregates publicly available benchmark data from third-party sources. We do not claim ownership of rankings, Elo scores, or benchmark results.

Data Sources (Used With Permission)

| Source | Data | Permission | |--------|------|------------| | LMArena | Chatbot Arena Elo rankings | robots.txt: Allow: / | | Artificial Analysis | LLM quality benchmarks | robots.txt: Allow: / (explicitly allows AI crawlers) | | HuggingFace | Model metadata, downloads | Public API | | Open LLM Leaderboard | Open-source LLM benchmarks | CC-BY license |

Disclaimer

...