grub 🕷 v1.0.0

Agentic web crawler with bot-bypass and mesh P2P.

Crawl any URL — even Cloudflare, Incapsula, or DataDome.
Returns clean markdown. Ghost Protocol vision-fallback for hard-blocked pages.
Camoufox stealth · MCP tooling · Bounded LLM agent loop · Policy gates · Replayable traces.

Try it View API Quick Start Health GitHub

—

Status

—

Tools

Camoufox

Engine

MCP

Transport

Capabilities

Crawl

Playwright + Camoufox stealth. Cookie injection, residential proxy support, automatic Cloudflare/Incapsula challenge solver.

Ghost Protocol

When a page is blocked, screenshot it and extract content via vision LLM. Fallback survives most anti-bot detection.

Markdown

Rust-backed extraction (grub_md) returning clean markdown. Body/visible char counts, content-quality classification.

Agent (Mode B)

Bounded LLM loop: plan → tool call → observe. Anthropic, OpenAI, Ollama providers. Policy gates on every dispatch.

MCP Server

Streamable-HTTP MCP at /mcp/. Exposes grub_crawl, grub_screenshot, grub_diagnose to any MCP client.

Mesh P2P

Optional peer-to-peer mesh — crawl jobs route to the healthiest peer. Zero-trust HMAC + replay-protected nonces.

API

Method	Endpoint	Description
POST	`/api/crawl`	Crawl URL → HTML + markdown + screenshot
POST	`/api/markdown`	Crawl URL → markdown only (faster)
POST	`/api/batch`	Crawl multiple URLs in one request
POST	`/api/jobs/crawl`	Async job — queue and poll for result
POST	`/api/agent/run`	Run bounded LLM agent on a task
POST	`/api/agent/ghost`	Ghost Protocol — vision extract a blocked URL
GET	`/mcp/`	MCP streamable-http transport (tools)
GET	`/health`	Status, version, tool count

All /api/* endpoints require Authorization: Bearer <ahp_token>. Get a token from your dashboard.

Try it

Quick start

# Crawl → markdown
curl -X POST https://grub.nuts.services/api/markdown \
  -H "Authorization: Bearer ahp_yourtoken" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://example.com"}'

# With Camoufox warmup (for Cloudflare-protected sites)
curl -X POST https://grub.nuts.services/api/markdown \
  -H "Authorization: Bearer ahp_yourtoken" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://target.com", "options": {"warmup": true}}'

# Inject pre-solved cookies to bypass challenges
curl -X POST https://grub.nuts.services/api/markdown \
  -H "Authorization: Bearer ahp_yourtoken" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://target.com", "options": {"cookies": {"visid_incap_X": "..."}}}'

# Add as MCP server in Claude Code
claude mcp add grubcrawler \
  --transport http \
  --header "Authorization: Bearer ahp_yourtoken" \
  https://grub.nuts.services/mcp/

Docs Health MCP GitHub