Use Case
Crawler is a powerful AI agent skill that extends your assistant with new capabilities. Web crawling and scraping reference β robots.txt protocol, Scrapy framework, anti-bot detection, headless browsers, and legal considerations This guide walks you through installing Crawler, configuring it for your setup, and running your first commands β so you can start getting value in minutes.
Install Crawler: npx clawhub@latest --dir ~/.claude/skills install crawler
Restart your AI client (Claude Code, Cursor, Gemini CLI, or OpenClaw)
Type a natural language request related to data & analytics to trigger Crawler
Review the output and refine your prompt for better results
Combine Crawler with other skills to build multi-step workflows
Copy these prompts and use them with your AI agent after installing Crawler
Help me get started with Crawler
What can Crawler do for my data & analytics workflow?
Show me an example of using Crawler
Select your agent
Option 1: Install via CLI (recommended)
Recommended (no pre-install needed)
npx clawhub@latest --dir ~/.claude/skills install crawlerOr via clawhub CLI (if already installed)
clawhub --dir ~/.claude/skills install crawlerβ οΈ Requires Node.js 18+. No Node? Use Option 2 below to download the ZIP instead. Install Node.js β
Option 2: Manual install (no Node required)
Download the ZIP, extract it, and place the folder at the path below. Restart your agent to activate.
Install path
~/.claude/skills/crawler/π‘Extract and place the folder at the path above, then restart your agent.