A browser and a language model can look at the same URL and effectively see two different things.
A browser sees a rendered interface: navigation, cookie banners, buttons, ads, sidebars, images, scripts, interactive components, and eventually the text a human came to read. A language model sees whatever representation we decide to give it.
That distinction matters when the URL is going into a RAG pipeline.
Open the developer tools on any major news or documentation site and look at the raw HTML. A typical article page runs between 300KB and 800KB of markup. The article text itself is usually between 2KB and 10KB. The ratio of markup to content is consistently between 10:1 and 40:1 depending on how heavily templated the site is. When you pass raw HTML to a language model, you are passing all of it, and most pipelines treat this as an acceptable default.
Perceive is the endpoint we built to fix that. You give it a URL. It returns clean Markdown. This post is about what happens in between and why we made the engineering decisions we did.
Why raw HTML is a poor RAG input
The token waste is real but it is not the worst problem. Three failure modes compound each other.
Token waste. A blog post with 800 words of real content can run to 6,000–12,000 tokens as raw HTML once you include navigation, scripts, inline styles, and layout markup. The same content in Markdown is often 900–1,200 tokens. That is not just a cost issue. It is context window space that cannot go to content.
Embedding contamination. Embedding models are trained predominantly on natural language. When you embed a chunk containing <div class="sidebar-widget__title">Related Articles</div> alongside the article content, the vector is pulled toward the markup semantics rather than the content
semantics. The embedding does not cleanly represent the article; it represents a mixture of the article and the site's component naming conventions. Retrieval degrades as a result: chunks that should be semantically similar rank below chunks that happen to share markup patterns.
Chunking breakdown. Standard splitters are designed around prose or structured Markdown. HTML does not have semantic boundaries at the text level. A </div> tag is not a paragraph break. Splitting HTML by character count produces chunks with no meaningful coherence. The wrong content gets embedded together and the right content gets split apart.
The Perceive pipeline
Perceive processes every URL through four discrete stages.
Fetch. An HTTP request with appropriate headers, redirect handling, timeout management, and response validation. Non-200 responses are surfaced as errors rather than silently returning empty content. This stage handles rate limiting and robots.txt compliance.
Render. For JavaScript-rendered pages (SPAs, documentation sites built on frameworks like Docusaurus or GitBook, any page where content is injected into the DOM after initial load) a real browser render runs before extraction. The render uses a multi-engine fallback: a fast TLS fingerprint pass first, escalating to headless Chrome when the page needs JavaScript, and once more to a stealth-hardened render when anti-bot protection is detected. Cookie banners are dismissed, the page is scrolled to trigger lazy-loaded content, and images are given time to load.
One detail worth noting: the render step waits for network idle rather than just DOM ready. Modern SPAs often make secondary API calls after the initial render to populate content. Waiting for DOM ready captures the shell. Waiting for network idle captures the content. The cost is latency. The trade-off is necessary.
Extract. This is the hardest stage. The system needs to identify what is content and what is chrome. Navigation is not a semantic HTML tag; sites implement it with <nav>, <header>, <div class="nav-wrapper">, <ul class="menu">, and dozens of other patterns. The extraction layer uses a combination of element position in the DOM, text-to-link ratio within elements, text density relative to the page body, and known structural patterns to infer content boundaries. It is heuristic, not rule-based, which means it fails in specific ways rather than universally.
Structure. Extracted content is converted to Markdown. Headings map to ATX heading levels based on their HTML hierarchy. Lists preserve nesting. Tables become Markdown tables. Code blocks retain language hints where the site uses syntax highlighting. Inline links are preserved. Images become alt text references where available.
The only_main_content parameter
Passing only_main_content: true runs an additional stripping pass. The conceptual goal is to answer one question: if a human came to this URL to read the page, what information would they actually care about?
What gets removed: navigation elements, site headers and footers, cookie consent banners, sidebar content, related article blocks, and social sharing widgets.
What stays: the page title, the main content body, all heading structure within that body, lists, tables, code blocks, and inline links that appear within the content.
The engineering challenge is that the boundary between content and chrome is not always clear. An article with a "Further reading" section at the bottom is content. A dynamically injected related articles widget is chrome. The system uses element injection patterns and DOM position relative to the identified content body to distinguish between the two. On well-structured sites it works cleanly. On unusual layouts it occasionally over-strips and when stripping would remove too much real content, the full page is returned instead with a warning. That is why only_main_content is optional.
Why Markdown
Plain text loses structure. If a documentation page has four heading levels, three nested lists, and two tables, flattening it to plain text collapses the hierarchy that makes chunking semantically coherent.
Consider the difference in chunking behaviour. A RecursiveCharacterTextSplitter splitting on \n## produces chunks that correspond to sections. The same splitter on plain text produces chunks that correspond to character counts. Those are very different things to embed and retrieve against.
JSON requires an upfront schema decision and adds parsing overhead for the consumer. For a RAG pipeline where content goes directly into an embedding call or a LangChain Document, that parsing step is overhead without benefit.
Markdown preserves heading hierarchy, list nesting, table structure, and code block language identifiers. It is readable without a parser. It chunks predictably. Embedding models handle it better than raw HTML because it reads closer to natural language with light structural annotation.
One production detail that matters: we enforce consistent Markdown flavour across all output. ATX headings with a space after the hash. Fenced code blocks with language identifiers. GFM-style tables. If your chunking strategy relies on splitting at heading boundaries, inconsistency in heading format breaks it silently. Consistent output means your downstream logic behaves the same regardless of which site the content came from.
Edge cases
JavaScript-heavy SPAs. The multi-engine render handles the majority of these. The failure case is client-side routing where navigating to a URL does not trigger a full page load, only a state change. Some SPAs require specific interaction patterns to surface content that Perceive does not replicate.
Paywalled content. Perceive fetches what is publicly accessible. If a page returns a login wall or a metered paywall modal, that is what extraction runs on. The output is the paywall content, not the article behind it. There is no mechanism to bypass authentication and there should not be.
Scroll-triggered and lazy-loaded content. Content that only enters the DOM on user scroll or interaction is frequently not captured. Infinite scroll feeds, image galleries that load on demand, and content inside collapsed accordions that require a click to expand all fall into this category.
Inconsistent site structures. The failure mode that comes up most in practice is heavily templated pages where the main content column is structurally indistinguishable from the sidebar. Sites that use the same component markup for editorial content and promotional content are the hardest case. The heuristics fail here more than anywhere else.
A real request
Using curl:
curl -X POST https://api.enconvert.com/v2/perceive \
-H "X-API-Key: sk_your_private_key" \
-H "Content-Type: application/json" \
-d '{
"url": "https://example.com/blog/post",
"outputs": ["markdown"],
"only_main_content": true
}'
The response is a JSON envelope. The Markdown comes back as a pre-signed URL that expires after 15 minutes, alongside the signals worth checking before trusting the content:
{"operation_id":"per_3f9a2c1b8e7d4a6f90b1c2d3e4f5a6b7","status":"completed","url":"https://example.com/blog/post","url_final":"https://example.com/blog/post","content_hash":"9f2b8c1a...d4e5","render_quality":0.93,"cache_hit":false,"outputs":{"markdown":{"url":"https://spaces.example.com/...signed...","object_key":"env/files/4127/v2-perceive/per_3f9a..._markdown.md","size_bytes":8421,"content_type":"text/markdown; charset=utf-8","expires_in":900}},"extraction_tier":"heuristic","duration_ms":6230,"warnings":[]}Or in Python, fetching the Markdown from its signed URL and passing it into a LangChain Document:
import requests
from langchain_core.documents import Document
from langchain_text_splitters import RecursiveCharacterTextSplitter
BASE = "https://api.enconvert.com"
HEADERS = {"X-API-Key": "sk_your_private_key"}
result = requests.post(
f"{BASE}/v2/perceive",
headers=HEADERS,
json={
"url": "https://example.com/blog/post",
"outputs": ["markdown"],
"only_main_content": True,
},
timeout=120,
).json()
// Below 0.40 means a failed render: anti-bot page, login wall, soft 404, empty shell.
if result["render_quality"] < 0.40:
raise RuntimeError(f"unreliable render for {result['url_final']}")
// The signed URL is good for 15 minutes.
markdown = requests.get(result["outputs"]["markdown"]["url"], timeout=60).text
doc = Document(
page_content=markdown,
metadata={
"source": result["url_final"],
"content_hash": result["content_hash"],
"render_quality": result["render_quality"],
},
)
// Heading-aware chunks, which is the whole reason for asking for Markdown.
splitter = RecursiveCharacterTextSplitter(
separators=["\n## ", "\n### ", "\n\n", "\n", ""],
chunk_size=1500,
chunk_overlap=150,
)
chunks = splitter.split_documents([doc])
If you want to skip the signed URL step and receive the Markdown bytes directly in the response body, add "direct_download": true to the request. The response will be the raw Markdown file rather than a JSON envelope:
curl -X POST https://api.enconvert.com/v2/perceive \
-H "X-API-Key: sk_your_private_key" \
-H "Content-Type: application/json" \
-d '{
"url": "https://example.com/blog/post",
"outputs": ["markdown"],
"direct_download": true
}' \
-o post.md
The metadata that would have been in the envelope rides in response headers instead: X-Operation-Id, X-Render-Quality, X-Content-Hash, X-Cache-Hit, X-Source-Status-Code, and X-Warnings-Count. The response structure is consistent across Perceive, Ingest, and Convert. Error handling and retry logic transfer without modification across all three endpoints.
One honest limitation
Content boundary detection fails on sites that do not have a clear content region. If a page is built as a grid of equally weighted content blocks with no structural differentiation between editorial content and navigational content, the extraction heuristics make a guess. Sometimes that guess is wrong. The output contains more chrome than it should, or misses content that sits in a region the system classified as navigation.
There is no clean fix at the extraction layer without site-specific rules, which do not scale. The current workaround is to check the render_quality score returned in the response. Scores below 0.40 indicate a failed or unreliable render: an anti-bot page, a login wall, or an empty shell. For sources you know are structurally ambiguous, a post-extraction review pass or a second model call to strip non-content is more reliable than trusting the heuristic output.
We are looking at surfacing a confidence signal more explicitly in a future response version.
Free tier is 500 ops per month, no credit card required. If you are building RAG pipelines and the ingestion layer is where you are losing time, try it here and let us know what breaks. Feedback from engineers running this at scale is the most useful input we have.