Build a docs corpus
Crawl a whole documentation site and push clean Markdown into your vector store. One async job, hundreds of pages.
Markdown extraction for AI agents
HTML to Markdown for RAG pipelines
Convert any URL into clean, LLM-ready Markdown. Nav, footers, and ads stripped. Headings, links, and code blocks preserved. Conservative extraction tuned for vector search and agent context.
Works with your agent stack
Use cases
Ship RAG that reads the live web
Gain clean context from public docs, changelogs, and marketing sites. Build corpora, cut token spend, and stop agents from citing boilerplate with the same /markdown endpoint. See API docs.
Cut inference cost
Strip nav, footers, and ads before embedding. Typical docs pages drop from ~12k tokens to ~900 for the same facts.
Improve agent answers
Models stop citing cookie banners and sidebar labels when context only contains the article body.
Technical approach
Clean extraction built for LLMs
CaptureWeb Markdown is purpose-built for agent context, not repurposed from raw HTML dumps. We preserve document structure and strip everything the model cannot use.
Main content extraction
Readability-style DOM reduction keeps headings, paragraphs, lists, and tables. Nav, footer, and aside blocks disappear.
Links preserved
Inline links keep their targets so agents can cite sources and follow up on references without a second scrape.
Smart render chain
Static fetch when HTML is enough. Headless browser when JavaScript must run. Stealth when bot protection blocks naive requests.
Integration
Automated Markdown extraction via API
REST API
POST any URL to /v1/markdown. Get a JSON envelope with markdown, title, word_count, and token_estimate.
View docs →MCP server
Connect Claude, Cursor, or Codex in minutes. Your agent calls markdown as a tool without custom HTTP wiring.
Read agents.md →Site crawl
Walk a docs tree or blog archive asynchronously. Render each URL once, convert to Markdown, index on your schedule.
Crawl docs →FAQs
RAG extraction FAQs
Common questions about Markdown extraction for agents and vector pipelines.
View docs001Is Markdown better than HTML for LLM input?
For reading, summarizing, and retrieval: yes. HTML carries presentation detail models cannot use. Keep HTML when you need exact DOM structure or visual replay via /content.
002Do JavaScript-heavy pages work?
Yes. If a static fetch returns an empty shell, CaptureWeb escalates to a full browser render automatically. You send the same request either way.
003What about pages behind login?
Pass a session cookie or authenticated context when your use case requires it. Public URL extraction is the default path for RAG over docs, marketing sites, and changelogs.
004Do repeat URLs cost credits?
No. Same URL and parameters hit cache at 0 credits. Agents that re-read pricing, changelogs, or docs every session do not burn quota twice.
005Can I get Markdown and a screenshot in one call?
Yes. POST to /v1/snapshot for screenshot, PDF, HTML, and Markdown from a single browser navigation (2 credits).
006How does this compare to scraping HTML myself?
Hand-rolled tag strippers break on every new site template. CaptureWeb handles JS rendering, bot protection, and boilerplate removal server-side so your pipeline stays focused on chunking and embedding.
Explore more
Web capture for every workflow
Data extraction
Sales & outreach
Screenshot the prospect site and pull Markdown for the model writing the email.
Learn more →Capture at scaleDirectories & marketplaces
Thumbnail every listing. Crawl categories until the full catalog is covered.
Learn more →Quality & complianceCompliance & archival
PNG, PDF, and HTML from one load. Provably the same moment.
Learn more →Start extracting Markdown today
Clean context for your agents, lower token bills, and MCP-ready on day one.