--- name: stealth-browser description: Anti-bot browser automation using Camoufox and Nodriver. Bypasses Cloudflare Turnstile, Datadome, and aggressive anti-bot on sites like Airbnb and Yelp. Use when standard Playwright/Selenium gets blocked. metadata: openclaw: emoji: "🥷" requires: bins: ["distrobox"] env: [] ---

Stealth Browser Skill 🥷

Anti-bot browser automation that bypasses Cloudflare Turnstile, Datadome, and aggressive fingerprinting.

When to Use

• Standard Playwright/Selenium gets blocked
• Site shows Cloudflare challenge or "checking your browser"
• Need to scrape Airbnb, Yelp, or similar protected sites
• playwright-stealth isn't working anymore

Tool Selection

| Target Difficulty | Tool | When to Use | |------------------|------|-------------| | Browser | Camoufox | All protected sites - Cloudflare, Datadome, Yelp, Airbnb | | API Only | curl_cffi | No browser needed, just TLS spoofing |

Quick Start

All scripts run in pybox distrobox for isolation.

⚠️ Use python3.14 explicitly - pybox may have multiple Python versions with different packages installed.

1. Setup (First Time)

# Install tools in pybox (use python3.14)
distrobox-enter pybox -- python3.14 -m pip install camoufox curl_cffi

# Camoufox browser downloads automatically on first run (~700MB Firefox fork)

2. Fetch a Protected Page

Browser (Camoufox):

distrobox-enter pybox -- python3.14 scripts/camoufox-fetch.py "https://example.com" --headless

API only (curl_cffi):

distrobox-enter pybox -- python3.14 scripts/curl-api.py "https://api.example.com/endpoint"

Architecture

┌─────────────────────────────────────────────────────────┐
│                     OpenClaw Agent                       │
├─────────────────────────────────────────────────────────┤
│  distrobox-enter pybox -- python3.14 scripts/xxx.py         │
├─────────────────────────────────────────────────────────┤
│                      pybox Container                     │
│         ┌─────────────┐  ┌─────────────┐               │
│         │  Camoufox   │  │  curl_cffi  │               │
│         │  (Firefox)  │  │  (TLS spoof)│               │
│         └─────────────┘  └─────────────┘               │
└─────────────────────────────────────────────────────────┘

Tool Details

Camoufox

• What: Custom Firefox build with C++ level stealth patches
• Pros: Best fingerprint evasion, passes Turnstile automatically
• Cons: ~700MB download, Firefox-based
• Best for: All protected sites - Cloudflare, Datadome, Yelp, Airbnb

curl_cffi

• What: Python HTTP client with browser TLS fingerprint spoofing
• Pros: No browser overhead, very fast
• Cons: No JS execution, API endpoints only
• Best for: Known API endpoints, mobile app reverse engineering

Critical: Proxy Requirements

Datacenter IPs (AWS, DigitalOcean) = INSTANT BLOCK on Airbnb/Yelp

You MUST use residential or mobile proxies:

# Example proxy config
proxy = "http://user:[email protected]:8080"

See references/proxy-setup.md for proxy configuration.

Behavioral Tips

Sites like Airbnb/Yelp use behavioral analysis. To avoid detection:

1. Warm up: Don't hit target URL directly. Visit homepage first, scroll, click around.
2. Mouse movements: Inject random mouse movements (Camoufox handles this).
3. Timing: Add random delays (2-5s between actions), not fixed intervals.
4. Session stickiness: Use same proxy IP for 10-30 min sessions, don't rotate every request.

Headless Mode Warning

⚠️ Old --headless flag is DETECTED. Options:

1. New Headless: Use headless="new" (Chrome 109+)
2. Xvfb: Run headed browser in virtual display
3. Headed: Just run headed if you can (most reliable)

# Xvfb approach (Linux)
Xvfb :99 -screen 0 1920x1080x24 &
export DISPLAY=:99
python scripts/camoufox-fetch.py "https://example.com"

Troubleshooting

| Problem | Solution | |---------|----------| | "Access Denied" immediately | Use residential proxy | | Cloudflare challenge loops | Try Camoufox instead of Nodriver | | Browser crashes in pybox | Install missing deps: sudo dnf install gtk3 libXt | | TLS fingerprint blocked | Use curl_cffi with impersonate="chrome120" | | Turnstile checkbox appears | Add mouse movement, increase wait time | | ModuleNotFoundError: camoufox | Use python3.14 not python or python3 | | greenlet segfault (exit 139) | Python version mismatch - use python3.14 explicitly | | libstdc++.so.6 errors | NixOS lib path issue - use python3.14 in pybox |

Python Version Issues (NixOS/pybox)

The pybox container may have multiple Python versions with separate site-packages:

# Check which Python has camoufox
distrobox-enter pybox -- python3.14 -c "import camoufox; print('OK')"

# Wrong (may use different Python)
distrobox-enter pybox -- python3.14 scripts/camoufox-session.py ...

# Correct (explicit version)
distrobox-enter pybox -- python3.14 scripts/camoufox-session.py ...

If you get segfaults or import errors, always use python3.14 explicitly.

Examples

Scrape Airbnb Listing

distrobox-enter pybox -- python3.14 scripts/camoufox-fetch.py \
  "https://www.airbnb.com/rooms/12345" \
  --headless --wait 10 \
  --screenshot airbnb.png

Scrape Yelp Business

distrobox-enter pybox -- python3.14 scripts/camoufox-fetch.py \
  "https://www.yelp.com/biz/some-restaurant" \
  --headless --wait 8 \
  --output yelp.html

API Scraping with TLS Spoofing

distrobox-enter pybox -- python3.14 scripts/curl-api.py \
  "https://api.yelp.com/v3/businesses/search?term=coffee&location=SF" \
  --headers '{"Authorization": "Bearer xxx"}'

Session Management

Persistent sessions allow reusing authenticated state across runs without re-logging in.

Quick Start

# 1. Login interactively (headed browser opens)
distrobox-enter pybox -- python3.14 scripts/camoufox-session.py \
  --profile airbnb --login "https://www.airbnb.com/account-settings"

# Complete login in browser, then press Enter to save session

# 2. Reuse session in headless mode
distrobox-enter pybox -- python3.14 scripts/camoufox-session.py \
  --profile airbnb --headless "https://www.airbnb.com/trips"

# 3. Check session status
distrobox-enter pybox -- python3.14 scripts/camoufox-session.py \
  --profile airbnb --status "https://www.airbnb.com"

Flags

| Flag | Description | |------|-------------| | --profile NAME | Named profile for session storage (required) | | --login | Interactive login mode - opens headed browser | | --headless | Use saved session in headless mode | | --status | Check if session appears valid | | --export-cookies FILE | Export cookies to JSON for backup | | --import-cookies FILE | Import cookies from JSON file |

Storage

• Location: ~/.stealth-browser/profiles//
• Permissions: Directory 700, files 600
• Profile names: Letters, numbers, _, - only (1-63 chars)

Cookie Handling

• Save: All cookies from all domains stored in browser profile
• Restore: Only cookies matching target URL domain are used
• SSO: If redirected to Google/auth domain, re-authenticate once and profile updates

Login Wall Detection

The script detects session expiry using multiple signals:

1. HTTP status: 401, 403
2. URL patterns: /login, /signin, /auth
3. Title patterns: "login", "sign in", etc.
4. Content keywords: "captcha", "verify", "authenticate"
5. Form detection: Password input fields

If detected during --headless mode, you'll see:

🔒 Login wall signals: url-path, password-form

Re-run with --login to refresh the session.

Remote Login (SSH)

Since --login requires a visible browser, you need display forwarding:

...