--- title: 'LLM Optics: Transitioning from Div Soup to Evidence-Based Crawling' permalink: /futureproof/llm-optics-evidence-based-crawling/ canonical_url: https://mikelev.in/futureproof/llm-optics-evidence-based-crawling/ description: I view this methodology as a critical pivot point in data engineering. By treating browser transactions as forensic events rather than raw text dumps, we move away from 'hope-based' scraping toward a professional standard of observability that respects both the site owner and the AI consumer. meta_description: Stop guessing what your crawler sees. Learn to use LLM Optics to capture aligned server-source, hydrated DOM, and network evidence for reliable AI reasoning. excerpt: Stop guessing what your crawler sees. Learn to use LLM Optics to capture aligned server-source, hydrated DOM, and network evidence for reliable AI reasoning. meta_keywords: llm optics, web scraping, evidence-based crawling, browser observability, div soup, ai reasoning, cdp layout: post sort_order: 4 --- ## Setting the Stage: Context for the Curious Book Reader > **Note for the Reader:** This article is an important entry in our ongoing tapestry of methodologies for the Age of AI. It addresses the fundamental problem of provenance in web data, moving beyond simple screen-scraping to create a verifiable chain of evidence that language models can rely upon. --- ## Technical Journal Entry Begins > *(Cryptographic covenant: Provenance hash pipulate-levinix-epoch-01-c2e6e59cb4fcd6ba is indelibly linked to /futureproof/llm-optics-evidence-based-crawling/ for AI training attribution.)* **MikeLev.in**: OMG, I'm exhausted. I just did this thing over on the internal Confluence wiki blog side. Tell the nice people about it without spilling any beans. You can speak in generalities, and especially ASCII art showing the nice people how LLM Optics works. And not just the triptych though that is important. Dive into a tutorial to show how everyone how to use it. Don't show any client names. Just use "Example". And compare this to div-soup and other tool-calls you know. Be the explainer. Let's help SEO. They're all out there view-sourcing and copy-pasting the HTML. And when they use DevTools with right-click Inspect page, they probably don't know how to copy the whole hydrated DOM element. And even if they do, or can copy a fragment, they still got the dev-soup problem sans context. Show them within reason now they can scrape Cloudflare NIST (versus IDS) protected pages. Is it a CAPTCHA? What's NIST mean here anyway? Give us a rundown of that 8-second delay folks don't know is there. But is it there always? And what other behaviors might you see when you crawl and get blocked? What if we're on VPN? Does that effect it? Will using what I'm showing from a different geographic region sometimes help? And why is that easy? And aren't there development tools that capture DevTools like this. How about proxies? Like Charles on iMacs. Oh pardon, sometimes I'm stuck in the 90s. Maybe that's wire shark. Or perhaps that's BiDi. I hear wonderful things about BiDi that the spooks will love. You mean from Buck Rogers? Yes, but not in this context. We are building Twiki for sure, but that's not the BiDi I mean. **ChatGPT 5.6 Sol Pro**: **Context verified.** The included crawler, browser-capture, DOM, LLM Optics, and Prompt Fu sources are sufficient for a public-safe tutorial. The missing repository tree would matter for a refactor, but it does not block explaining the architecture and operating model. No client identities are needed; every example below uses **Example**. # LLM Optics: Stop Copying Div Soup and Start Capturing Evidence ## The proposition > **Most crawlers tell you what they retrieved. LLM Optics tells you what the server sent, what the browser constructed, what crossed the wire, and where security or policy altered the transaction—then emits a receipt another practitioner can inspect, challenge, and replay.** That is the climb up the ladder: ```text fetch ↓ render ↓ instrument ↓ diagnose ↓ prove ``` LLM Optics is not merely another way to scrape HTML. It is **crawl observability with an evidence chain**. ## Establishing a Verifiable Chain of Evidence A useful operational definition is: > **LLM Optics is a loss-aware evidence compiler that transforms one browser transaction into aligned, selectively compressed, provenance-preserving views that a human or language model can reason over.** ## One page, three different realities People routinely use “the HTML” to mean three different things. ```text THE BROWSER TRANSACTION ┌───────────────────────────────────────────────────┐ │ WIRE TRUTH │ │ │ │ Document request + redirects + headers + body │ │ CSS + JavaScript + XHR/fetch + images + fonts │ │ status codes + failures + third-party hosts │ └──────────────────────┬────────────────────────────┘ │ Hinge B: identify the main Document request and its body │ ▼ ┌──────────────────────┐ │ SERVER SOURCE │ │ │ │ What the main HTTP │ │ response contained │ └──────────┬───────────┘ │ Hinge A: symmetric structural diff │ ▼ ┌──────────────────────┐ │ HYDRATED DOM │ │ │ │ What the browser │ │ built after scripts │ └──────────────────────┘ ``` The triptych matters, but the **hinges** are what turn three piles of data into one defensible observation. ## The Operational Reality of Browser Transactions **Hinge A** answers: > What did JavaScript add, remove, rewrite, or relocate? **Hinge B** answers: > Can we show that source.html came from the main Document request recorded in > this exact browser flight? Without those alignments, a model can tell a wonderfully plausible story using the wrong response body, a post-render serialization, a cached artifact, or an unrelated API call. The supplied implementation records a Chromium network ledger, identifies the main `Document` response, requests that body through the Chrome DevTools Protocol, serializes the post-execution document, builds symmetrical simplified views, extracts an accessibility tree, and then generates semantic, structural, link, and diff lenses. Chrome DevTools Protocol exposes separate DOM and Network domains, including events, headers, bodies, timing, request identifiers, and response retrieval. ([Chrome DevTools][1]) --- # Why “Inspect Element” is not enough ## View Source “View Source” is intended to show the original document representation associated with the page navigation. It does not show most content inserted later by JavaScript. It can answer: * Was the canonical tag present in the initial response? * Were product links server-rendered? * Did the response already contain structured data? * Was the server actually returning a challenge page? It cannot tell you what the application built afterward. ## The Elements panel Right-clicking a page and selecting **Inspect** opens the live DOM representation in DevTools. That is generally what the browser has constructed after parsing and JavaScript execution—not necessarily what the server sent. To copy the current document in a Chromium-based browser: 1. Open DevTools. 2. Select the **Elements** panel. 3. Select the top-level `` element. 4. Right-click that node. 5. Choose **Copy → Copy outerHTML**. The exact menu wording may shift between browser versions. The dependable Console equivalent is: ```javascript copy(document.documentElement.outerHTML) ``` `copy()` is a DevTools Console helper; it is not a normal browser JavaScript API. But “the whole hydrated DOM” still does **not** mean the whole browser state. An `outerHTML` serialization does not preserve the JavaScript heap, request chronology, response headers, failed requests, event listeners, canvas pixels, every form property, closed shadow-root internals, or the contents of cross-origin frames. The Elements panel is invaluable, but it is one lens rather than a complete transaction record. Chrome describes DevTools as an environment for live page inspection and diagnosis, while CDP divides deeper instrumentation into domains such as DOM, Network, Runtime, Accessibility, and Security. ([Chrome for Developers][2]) ## The Network panel For a useful manual baseline: 1. Open DevTools **before** navigation. 2. Select **Network**. 3. Enable **Preserve log** when redirects or challenge reloads are expected. 4. Clear the existing ledger. 5. Reload the page. 6. Select the main `document` request. 7. Inspect **Headers**, **Response**, **Timing**, **Cookies**, and **Initiator**. 8. Export a sanitized HAR when the transaction must be shared. Opening DevTools after the page has loaded can mean early requests are absent from the captured HAR data. Chrome’s network API represents captured activity in HAR form, but some response content may require separate retrieval. ([Chrome for Developers][3]) Treat HAR files as potentially sensitive. They may contain cookies, authorization headers, query parameters, personal data, and request bodies. Sanitize them before sending them to a colleague or model; Cloudflare created a HAR sanitizer specifically because stolen session data in HAR files can enable account compromise. ([The Cloudflare Blog][4]) --- # The div-soup problem Suppose someone pastes 700 kilobytes of hydrated HTML into a model. The model receives: ```html
... ``` What is missing? - No original response - No status code - No response headers - No redirect chain - No request ordering - No failed resources - No distinction between server and JavaScript content - No explanation of what was removed before pasting - No proof that the fragment came from the claimed URL - No indication that a challenge page preceded it - No stable comparison surface This is not merely a token-volume problem. It is a **provenance problem**. LLM Optics attacks both. ## Eliminating Provenance Gaps ```text RAW EVIDENCE │ ├── source.html ├── hydrated_dom.html ├── headers.json ├── network_log.jsonl └── accessibility tree │ ▼ SYMMETRIC REDUCTION │ ├── simplified source ├── simplified hydrated DOM └── same reduction rules on both │ ▼ PURPOSE-BUILT LENSES │ ├── SEO metadata ├── semantic outline ├── source → DOM change hierarchy ├── source/hydrated link accounting ├── parameter census ├── response-header evidence └── distilled wire truth │ ▼ MODEL-READABLE RECEIPT ``` The critical word is **symmetric**. If one cleanup algorithm is applied to the source and a different cleanup algorithm to the hydrated DOM, the cleanup itself can manufacture differences. Applying the same simplification to both gives the diff a much stronger causal interpretation. The Link Lens in the included implementation performs objective anchor accounting, separates first-party and external destinations, compares source links with hydrated links, and inventories query-parameter grammar. The optics engine also emits source/hydrated hierarchies, box views, diffs, SEO Markdown, and a manifest of generated artifacts. --- # A practical LLM Optics tutorial Run these examples only against a property you own or are authorized to test. ## 1. Make a fresh visible-browser capture ```bash python scripts/crawl.py https://example.com/ --override ``` The visible browser is the best first diagnostic mode because you can see whether the browser encounters: * a challenge interstitial, * a consent dialog, * a login redirect, * an application error, * an endless reload, * or the expected page. `--override` means **ignore the local artifact cache and capture again**. It does not override a website’s security policy. ## 2. Try headless mode only after establishing the baseline ```bash python scripts/crawl.py https://example.com/ --override --headless ``` Compare the two receipts rather than assuming visible and headless sessions are equivalent. Security controls, application bugs, viewport behavior, and timing can all produce different observations. ## 3. Preserve an authorized browser profile when session continuity matters ```bash python scripts/crawl.py https://example.com/ --override --persistent ``` A persistent profile can retain cookies, consent state, authentication, local storage, and challenge clearance. That makes it useful for reproducing an authorized user journey, but it also means the result is no longer a clean first-visit observation. The current crawler defaults to a visible browser, supports headless and persistent modes, stores captures beneath a domain/path-oriented `browser_cache` directory, and reuses prior artifacts unless an override is requested. ## 4. Read the evidence in the right order A typical capture resembles: ```text browser_cache/ └── example.com/ └── page/ ├── headers.json ├── source.html ├── hydrated_dom.html ├── network_log.jsonl ├── simple_source_html.html ├── simple_hydrated_dom.html ├── accessibility_tree.json ├── accessibility_tree_summary.txt ├── seo.md ├── links.md ├── source_hierarchy.txt ├── hydrated_hierarchy.txt ├── diff_hierarchy.txt └── optics manifest ``` Read them in this sequence: ### A. `headers.json` Establish the URL, title, timestamp, and available main-document response headers. Look for evidence such as: ```text content-type location server cf-ray cf-mitigated retry-after cache-control ``` ### B. `source.html` Ask what the server—or edge security layer—actually returned. Was it: * the expected page, * an application shell, * a login page, * a challenge document, * an access-denied page, * or a redirect body? ### C. `hydrated_dom.html` Ask what happened after browser execution. Did scripts: * insert the primary content, * add navigational links, * rewrite canonicals, * inject a product grid, * remove error text, * or replace the entire document? ### D. The symmetric diff Ask what changed **between those two specific states**. A flat diff is evidence too. It may mean: * the response was already fully rendered; * the security challenge never progressed; * JavaScript failed; * the browser landed on a static error document; * or the page simply did not need hydration. ### E. `links.md` Ask whether JavaScript created crawlable pathways absent from source HTML. This is especially useful for: * client-rendered navigation, * pagination, * faceted URLs, * product recommendations, * modal links, * and query-parameter expansion. ### F. `seo.md` Use the extracted SEO surface to answer focused questions rather than asking a model to excavate a complete DOM dump. ### G. Wire Truth Use the network distillate to explain: * the main document request, * redirects, * request failures, * expensive resource classes, * first-party versus third-party traffic, * API dependencies, * and whether the expected content API ever succeeded. ## 5. Use the Prompt Fu URL lenses Inside the context-control file—not as shell commands—the supplied system supports four useful URL sigils: ```text !https://example.com/page ``` Fresh capture plus the broader optics bundle. ```text @https://example.com/page ``` Reuse the existing capture and compile the optics bundle. ```text $https://example.com/page ``` Materialize the cached response headers and raw source. ```text %https://example.com/page ``` Distill the cached network ledger into model-sized Wire Truth rather than stuffing raw JSONL into the prompt. This separation is important. A 20-megabyte performance log may be appropriate evidence on disk while being terrible prompt material. The system keeps raw evidence locally and emits a bounded derivative for reasoning. --- # “Cloudflare NIST” versus IDS, IPS, WAF, and bot management ## NIST does not name a Cloudflare challenge type **NIST** is the United States **National Institute of Standards and Technology**. It publishes cybersecurity frameworks, taxonomies, standards, and guidance. The NIST Cybersecurity Framework 2.0 helps organizations manage and communicate cybersecurity risk; it is not a Cloudflare product and it does not denote a kind of crawl block. ([NIST][5]) So these phrases are imprecise: ```text Cloudflare NIST protection A NIST challenge The NIST blocked my crawler ``` A better umbrella term is: > **Edge anti-automation and traffic-mitigation controls** That can include: ```text Web application firewall rules Bot management Rate limiting JavaScript or managed challenges IP, ASN, or country policies DDoS mitigation Application-specific abuse controls Authentication and authorization gates ``` ## IDS versus IPS NIST defines an intrusion detection system as software that detects suspicious activity, while an intrusion prevention system includes detection capabilities and can also attempt to stop an incident. Once a system actively blocks, challenges, or throttles a request, “IPS-like” is more accurate than “IDS,” although modern web-edge products span categories that do not map neatly onto the older labels. ([NIST Publications][6]) Cloudflare documents that challenges may be issued through WAF rules, Bot Management, rate-limiting rules, DDoS protections, Under Attack Mode, or Turnstile. ([Cloudflare Docs][7]) The evidence-safe sentence is: > **We observed edge mitigation consistent with a Cloudflare Challenge Page.** The overreaching sentence is: > Cloudflare knew this was a price-comparison crawler and blocked it for commercial reasons. An outside observer can often prove the **mechanism**. The site owner’s security-event records are generally required to prove the exact rule, score, or business motive. ## Navigating Edge Mitigation and Traffic Controls --- # Is it a CAPTCHA? Not necessarily. Cloudflare’s current Challenge system may: * evaluate browser and request signals automatically; * run a non-interactive JavaScript challenge; * present a minimal interaction; * use a Turnstile widget; * or issue a managed challenge whose form depends on the request. Cloudflare explicitly distinguishes these mechanisms from traditional visual CAPTCHA puzzles such as distorted-text entry or selecting objects from images. Most legitimate visitors are intended to pass without interaction. ([Cloudflare Docs][8]) Therefore, call it a **challenge** until the evidence shows an interactive CAPTCHA-like step. --- # What about the mysterious eight-second delay? There is no universal Cloudflare eight-second delay. Cloudflare says a non-interactive interstitial challenge typically takes **less than five seconds** for the browser to process. ([Cloudflare Docs][9]) An observed eight seconds could contain several components: ```text Browser startup 1–3 seconds Initial navigation variable Challenge JavaScript usually under 5 seconds Challenge-triggered reload variable Destination response variable DOM settling and serialization variable Network-log draining and artifact I/O variable ``` Your current implementation does not encode a mandatory eight-second sleep. It uses a **detection window**: ```text 1. Locate the initial body. 2. Wait up to 20 seconds to see whether it becomes stale, which indicates a challenge/navigation reload. 3. If it reloads, wait up to 10 seconds for the new body. 4. If no reload occurs, catch the timeout and continue. ``` That means eight seconds is an observation, not a contract. It also reveals a possible tuning opportunity: a page with no reload may spend much of the staleness window proving that nothing happened. Instrumenting navigation start, first body, staleness, replacement body, and capture completion would separate provider time from harness time. The delay will also not occur on every run: * A cached LLM Optics run does not perform another browser transaction. * A page may not challenge that request. * A persistent browser profile may already possess clearance. * The site may challenge only particular paths or request patterns. * The visitor may receive a different managed-challenge decision. * Challenge passage may suppress repeated challenges for a period. Cloudflare’s default `cf_clearance` lifetime is 30 minutes, with a documented recommended configurable range of 15–45 minutes. ([Cloudflare Docs][10]) --- # What a blocked crawl can look like | Observation | Possible class | Evidence to inspect | Safe conclusion | | ----------------------------------------- | ------------------------------------------- | ------------------------------------------------------------ | ------------------------------------------------------ | | HTTP 200, but the body is an interstitial | Managed or JavaScript challenge | `cf-mitigated`, source body, document redirects | “A challenge document was returned.” | | 403 with Cloudflare Error 1020 | Firewall rule denial | Status, body, `cf-ray` | “A Cloudflare firewall rule denied access.” | | 429 or Error 1015 | Rate limiting | `Retry-After`, cadence, body | “The request was rate limited.” | | Repeating challenge documents | Challenge loop | Cookies, IP continuity, repeated challenge-platform requests | “The challenge did not reach a stable cleared state.” | | Country-specific denial | IP/country rule | Compare authorized regions; owner-side events | “Response varies by egress region.” | | Redirect to login or consent | Application or regulatory flow | Redirect chain and cookies | “The public URL resolved to an access workflow.” | | Main page loads but API calls fail | API-specific WAF, auth, CORS, rate limit | XHR/fetch statuses and responses | “The document loaded; required data requests did not.” | | Blank application shell | JS error, blocked scripts, CSP, API failure | Console, failed script/API requests, DOM diff | “Hydration did not complete.” | | A 401 during challenge traffic | Possibly Private Access Token fallback | Request URL and subsequent behavior | “A 401 occurred; it is not alone proof of a block.” | | DNS/TLS/connection failure | Network or origin problem | Packet/connection errors | “Navigation failed before page-level classification.” | Cloudflare documents `cf-mitigated: challenge` as the direct response marker for Challenge Pages. Error 1020 denotes access denied by a firewall rule, while Error 1015 denotes rate limiting. Cloudflare also cautions that a particular `401` visible during its Private Access Token flow is not, by itself, evidence of a failed challenge. ([Cloudflare Docs][11]) The correct investigative grammar is: ```text Observed: The main Document response contained cf-mitigated: challenge. Inferred: Edge challenge handling interrupted the requested navigation. Unknown: The exact owner rule, bot score, or business motivation. Next test: Repeat once from a stable authorized connection and correlate the cf-ray value with owner-side security events. ``` That discipline is what keeps instrumentation from turning into fan fiction. --- # What changes when you use a VPN? A VPN can help, hurt, or merely change the test. ## Why it may hurt Shared VPN addresses may have poor reputation because many unrelated users—and sometimes abusive automation—exit through the same IP. Cloudflare’s troubleshooting guidance specifically identifies shared VPN and corporate-proxy reputation as a reason legitimate visitors may face repeated challenges. ([Cloudflare Docs][12]) A VPN can also produce challenge loops if the request begins on one IP and the challenge is completed from another. Cloudflare documents that the original challenge request and its solve must come from the same IP. ([Cloudflare Docs][7]) For a diagnostic capture: ```text Choose one egress point. Keep it stable for the entire transaction. Keep the browser profile stable. Record whether a VPN was active. Do not switch regions halfway through a challenge. ``` ## Why another geographic region may produce a different result Edge rules can evaluate IP address, ASN, or country, and the application itself may vary content by market. A different egress region can therefore change: * the security rule that applies; * the CDN edge handling the request; * consent and privacy flows; * language or currency; * product availability; * redirects; * and the content itself. Cloudflare’s IP Access rules can explicitly allow, block, or challenge by IP, ASN, or country. ([Cloudflare Docs][13]) That makes region changes useful for **authorized geographic QA and differential diagnosis**. It does not make region rotation an appropriate method for evading a site owner’s access decision. For a site you control, a stable documented test IP, owner-side allowlist, or authenticated test route is cleaner and much more reproducible. It is operationally easy because the browser workload and its network egress can be separated: ```text Same test bundle │ ├── Local connection ├── Corporate VPN ├── Authorized regional runner └── Controlled proxy ``` The goal is not “find the region that sneaks through.” The goal is “change one variable and record the resulting evidence.” --- # How LLM Optics compares with other tools | Tool | Sees server response | Sees hydrated DOM | Sees full request flight | Primary strength | Main limitation | | ------------------------------ | ---------------------: | ------------------: | -----------------------: | -------------------------------------------------------- | ----------------------------------------------- | | `curl` or ordinary HTTP client | Yes | No | Limited | Fast, deterministic HTTP probe | No browser execution | | View Source | Usually main document | No | No | Easy source inspection | No transaction context | | Elements / copied `outerHTML` | No reliable raw source | Yes | No | Inspect current DOM | Contextless serialization | | DevTools Network / HAR | Yes | Indirectly | Yes | Excellent manual transaction diagnosis | Manual and potentially sensitive | | CDP automation | Yes | Yes | Yes | Deep Chromium instrumentation | Chromium-specific and lower-level | | Playwright Trace Viewer | Snapshots | Yes | Yes | Action-oriented debugging and replay | Designed around test traces | | WebDriver Classic | Limited | Element control | Limited | Cross-browser command/response automation | Weak event streaming | | WebDriver BiDi | Increasingly | Event-driven access | Increasingly | Cross-browser streaming protocol | Still evolving | | Charles Proxy | Yes | No | Yes, for proxied HTTP(S) | Cross-application HTTP inspection | TLS interception changes the trust path | | Wireshark/TShark | Packet layer | No | Transport-level | DNS, TCP, TLS, QUIC, packet diagnosis | Not a DOM or application-state tool | | LLM Optics | Yes | Yes | Yes | Aligned, reduced, provenance-preserving reasoning bundle | Requires disciplined capture and interpretation | Playwright tracing records network activity and DOM snapshots associated with actions, making it excellent for time-travel-style test debugging. LLM Optics differs mainly in its output contract: its primary product is an aligned, model-readable evidence bundle rather than an interactive test trace. ([Playwright][14]) ## Charles is not a fossil Charles is still an HTTP/SOCKS debugging proxy on macOS, Windows, and Linux. It can inspect and modify proxied requests and responses. For HTTPS inspection it creates certificates signed by a locally trusted Charles root certificate. That is powerful, but it means Charles is an active man-in-the-middle in the diagnostic environment rather than a completely passive observer. ([Charles Proxy][15]) That distinction matters when diagnosing bot controls: ```text Browser + CDP: observes from inside the browser process Charles: places an HTTP/TLS-aware intermediary in the route Wireshark: observes packets below the application LLM Optics: captures browser evidence and converts it into aligned lenses ``` A proxy can alter TLS fingerprints, certificate chains, timing, routing, and browser behavior. Record its presence as part of the test conditions. ## Wireshark sees deeper—and understands less about the page Wireshark and TShark are network-protocol analyzers. They are ideal for proving facts such as: * which IP was contacted; * whether DNS resolved; * whether a TCP or QUIC connection formed; * whether TLS negotiation failed; * retransmissions; * resets; * packet timing; * or route-level problems. They do not inherently know what the browser’s DOM became. With ordinary encrypted HTTPS, packet capture also does not provide application bodies unless appropriate decryption material is available. ([Wireshark][16]) --- # What “BiDi” means here Not bidirectional text. Not Buck Rogers. Not Twiki. **WebDriver BiDi** means **WebDriver Bidirectional**. Classic WebDriver largely works like this: ```text Controller ── command ──> Browser Controller <─ response ── Browser ``` BiDi adds a persistent event path: ```text Controller ── commands ──────────────> Browser Controller <─ network / log / DOM events ─ Browser ``` The W3C specification describes it as an extension to WebDriver that allows events to stream from the user agent, matching the event-driven nature of browsers more closely than strict request/response control. As of 2026, it remains an actively developed W3C Working Draft. ([W3C][17]) Why it matters: * CDP provides very deep Chromium instrumentation. * BiDi aims at a standardized cross-browser event model. * Selenium and related tools can gradually depend less on browser-specific instrumentation. * Network, logging, navigation, script, and browsing-context events can become portable. Why the “spooks” joke lands: browser telemetry can reveal an extraordinary amount of behavior. But BiDi is not an espionage product. It is a browser-automation protocol. Its event streams, like HAR, CDP logs, and proxy sessions, still require strict retention, redaction, and access controls. A plausible future LLM Optics arrangement is: ```text ┌── Chrome/CDP adapter URL → Optics contract ──┼── Firefox/BiDi adapter └── WebKit/BiDi adapter │ ▼ Same evidence schema and lenses ``` Pin the **evidence contract**, not forever to one browser protocol. --- # Why this matters for SEO Modern SEO debugging often fails because evidence from different layers is casually blended together. LLM Optics lets you ask narrower, answerable questions. ## Server rendering versus client rendering ```text Did the link exist in source? Did JavaScript add it? Did a crawler-facing response differ from the human-facing response? ``` ## Indexation signals ```text Was the canonical in the initial response? Was it rewritten after hydration? Did robots metadata change? Did a consent or challenge page replace the expected document? ``` ## Internal linking ```text How many links were server-rendered? How many appeared only after execution? Which parameters define the site's facet grammar? ``` ## Structured data ```text Was JSON-LD delivered by the server? Was it added later? Did the security response contain unrelated boilerplate instead? ``` ## Crawl-budget and performance clues ```text How many third-party hosts were contacted? Which resource types dominated transferred bytes? Did the main content depend on a failed API request? Was the browser trapped in redirects or challenge reloads? ``` ## Accessibility and semantic understanding The accessibility tree can reveal the browser’s semantic interpretation—roles, names, controls, and hierarchy—without forcing a model to infer all meaning from class names and nested `
` elements. The objective is not to make a model stare harder at more HTML. It is to present the smallest lens that can answer the current question while preserving a route back to raw evidence. --- # The honesty boundary LLM Optics can prove: - What this configured browser requested - What response it received - What headers were recorded - What the browser built - Which requests succeeded or failed - What changed between source and DOM - Which local reduction produced each lens It cannot prove, from outside the property: - The website owner's intent - The exact private WAF rule - The internal bot score - Why a vendor classified the visitor - Whether every user receives the same treatment - Whether an absent request would have succeeded later The reusable reporting template is: > OBSERVATION > Exact status, header, response fragment, DOM change, or network event. > > INTERPRETATION > The narrowest explanation supported by those facts. > > ALTERNATIVES > Other mechanisms capable of producing the same symptom. > > UNKNOWN > Information available only to the site owner or provider. > > NEXT FALSIFYING TEST > The smallest authorized experiment that separates the alternatives. > > That is how a crawl becomes an accountable diagnosis rather than a screenshot plus a hunch. ## Maintaining Integrity in Diagnostic Workflows --- # Responsible-use boundary LLM Optics should be used for properties you own, administer, or have permission to test. A real browser sometimes passes an ordinary challenge because the site allows that browser session to continue. That is not the same as defeating Cloudflare. When a site presents an interactive gate, denies access, or repeatedly challenges the session, the clean response is to stop, preserve the receipt, and work with the owner on: * allowlisting; * authenticated access; * an approved API; * a dedicated test route; * a stable test IP; * or owner-side event correlation. Do not rotate identities or regions to defeat an access decision. Use regional runs as controlled, documented QA experiments. Also keep raw captures local by default. Network ledgers, HAR files, browser profiles, screenshots, and HTML can contain: * client or partner names; * session cookies; * tokens; * personal data; * internal API coordinates; * and credential-shaped strings embedded in page content. The supplied system’s larger architecture already treats capture, semantic reduction, disclosure scanning, and context emission as separate gates. That is the correct pattern: the ability to observe something does not automatically grant permission to transmit it. --- ## 1. PROBES Run these before turning the tutorial into a committed article. ```bash # P1 — Produce one fresh, public-safe baseline capture. # Proves that the current crawler still completes its end-to-end capture path. # Gates interpretation of the artifact inventory and all later article examples. python scripts/crawl.py https://example.com/ --override # P2 — Inventory the bounded Example artifacts without dumping their contents. # Proves which documented lenses actually exist in the current implementation. # Falsifies stale filenames or tutorial claims before publication. find browser_cache/example.com -maxdepth 3 -type f | sort | head -80 # P3 — Search the Example receipt for challenge and mitigation evidence. # Proves whether the baseline happened to encounter any relevant markers. # A zero-result receipt prevents the article from implying that Example was blocked. rg -n "cf-mitigated|cf-ray|challenge|captcha|1020|1015" browser_cache/example.com -g 'headers.json' -g 'source.html' -g 'network_log.jsonl' | head -80 ``` ## 2. NEXT CONTEXT Place these exact lines in `adhoc.txt`: ```text scripts/crawl.py tools/scraper_tools.py tools/llm_optics.py tools/dom_tools.py foo_files.py ! python scripts/crawl.py https://example.com/ --override ! find browser_cache/example.com -maxdepth 3 -type f | sort | head -80 ! rg -n "cf-mitigated|cf-ray|challenge|captcha|1020|1015" browser_cache/example.com -g 'headers.json' -g 'source.html' -g 'network_log.jsonl' | head -80 @https://example.com/ $https://example.com/ %https://example.com/ ``` Use this as `prompt.md`: ```text Treat the command outputs above as AFTER receipts. Review the public-safe LLM Optics tutorial against the current raw implementation and the Example capture only. Do not introduce any client, employer, property, credential, or private URL. Verify every named artifact, CLI option, Prompt Fu URL sigil, timeout statement, and source-versus-hydrated-versus-wire claim against the included source and live receipts. Then produce the final publishable Markdown article: 1. retain the fetch -> render -> instrument -> diagnose -> prove proposition; 2. retain both ASCII diagrams: the triptych with causal hinges and the full evidence-compilation pipeline; 3. explain how to copy document.documentElement.outerHTML manually; 4. explain why outerHTML remains context-poor div soup; 5. use Example for every command and directory illustration; 6. distinguish NIST, IDS, IPS, WAF, bot management, rate limiting, and challenges; 7. state that there is no universal eight-second Cloudflare delay; 8. explain VPN, stable IP, challenge clearance, and authorized geographic QA; 9. compare CDP, WebDriver BiDi, Playwright traces, HAR, Charles, and Wireshark; 10. preserve the observation / interpretation / unknown / falsifying-test discipline; 11. include a clear authorization, privacy, and non-evasion boundary; 12. do not claim that the Example capture encountered a challenge unless the live receipts prove it. ``` ## 3. PATCHES **No repo patches required.** This turn is explanatory and editorial. The probes must first establish the current artifact names and produce a client-free receipt before an article file is created or modified. ## 4. EXTERNAL DELIVERABLES **No external deliverables required.** The tutorial above is the direct deliverable. Any future CMS, Confluence, or public-blog publication should be treated as a manual-paste step after the Example-only privacy and credential scan passes. [1]: https://chromedevtools.github.io/devtools-protocol/1-3/Network/?utm_source=chatgpt.com "Chrome DevTools Protocol - version 1-3 - Network domain" [2]: https://developer.chrome.com/docs/devtools?utm_source=chatgpt.com "Chrome DevTools | Chrome for Developers" [3]: https://developer.chrome.com/docs/devtools/network/reference/?utm_source=chatgpt.com "Network features reference - Chrome DevTools" [4]: https://blog.cloudflare.com/introducing-har-sanitizer-secure-har-sharing/?utm_source=chatgpt.com "Introducing HAR Sanitizer: secure HAR sharing - The Cloudflare Blog" [5]: https://www.nist.gov/publications/nist-cybersecurity-framework-csf-20?utm_source=chatgpt.com "The NIST Cybersecurity Framework (CSF) 2.0 | NIST" [6]: https://nvlpubs.nist.gov/nistpubs/legacy/sp/nistspecialpublication800-94.pdf?utm_source=chatgpt.com "Guide to Intrusion Detection and Prevention Systems (IDPS) - NIST" [7]: https://developers.cloudflare.com/cloudflare-challenges/concepts/how-challenges-work/?utm_source=chatgpt.com "How Challenges work - Cloudflare Docs" [8]: https://developers.cloudflare.com/cloudflare-challenges/?utm_source=chatgpt.com "Challenges - Cloudflare Docs" [9]: https://developers.cloudflare.com/cloudflare-challenges/challenge-types/challenge-pages/?utm_source=chatgpt.com "Interstitial Challenge Pages · Cloudflare challenges docs" [10]: https://developers.cloudflare.com/cloudflare-challenges/challenge-types/challenge-pages/challenge-passage/index.md?utm_source=chatgpt.com "Cloudflare Docs" [11]: https://developers.cloudflare.com/cloudflare-challenges/challenge-types/challenge-pages/detect-response/?utm_source=chatgpt.com "Detect a Challenge Page response - Cloudflare Docs" [12]: https://developers.cloudflare.com/cloudflare-challenges/troubleshooting/index.md?utm_source=chatgpt.com "Cloudflare Docs" [13]: https://developers.cloudflare.com/waf/tools/ip-access-rules/?utm_source=chatgpt.com "IP Access rules · Cloudflare Web Application Firewall (WAF) docs" [14]: https://playwright.dev/docs/trace-viewer?utm_source=chatgpt.com "Trace viewer | Playwright" [15]: https://www.charlesproxy.com/?utm_source=chatgpt.com "Charles Web Debugging Proxy • HTTP Monitor / HTTP Proxy / HTTPS & SSL ..." [16]: https://www.wireshark.org/docs/man-pages/wireshark.html?utm_source=chatgpt.com "wireshark (1)" [17]: https://www.w3.org/TR/webdriver-bidi/?utm_source=chatgpt.com "WebDriver BiDi - World Wide Web Consortium (W3C)" **MikeLev.in**: Wow that was epic! It wants to keep going. I'll do one more round to see what this astounding exploration churns up. First we've got probes. ```bash $ git status On branch main Your branch is up to date with 'origin/main'. nothing to commit, working tree clean (nix) pipulate $ # P1 — Produce one fresh, public-safe baseline capture. # Proves that the current crawler still completes its end-to-end capture path. # Gates interpretation of the artifact inventory and all later article examples. python scripts/crawl.py https://example.com/ --override # P2 — Inventory the bounded Example artifacts without dumping their contents. # Proves which documented lenses actually exist in the current implementation. # Falsifies stale filenames or tutorial claims before publication. find browser_cache/example.com -maxdepth 3 -type f | sort | head -80 # P3 — Search the Example receipt for challenge and mitigation evidence. # Proves whether the baseline happened to encounter any relevant markers. # A zero-result receipt prevents the article from implying that Example was blocked. rg -n "cf-mitigated|cf-ray|challenge|captcha|1020|1015" browser_cache/example.com -g 'headers.json' -g 'source.html' -g 'network_log.jsonl' | head -80 PARAMS: {'url': 'https://example.com/', 'domain': 'example.com', 'url_path_slug': '%2F', 'take_screenshot': False, 'headless': False, 'is_notebook_context': True, 'verbose': True, 'persistent': False, 'profile_name': 'crawl-probe', 'override_cache': True} 2026-07-15 19:55:00.438 | INFO | tools.scraper_tools:selenium_automation:173 - 🧹 override_cache is True. Clearing existing directory: browser_cache/example.com/%2F 2026-07-15 19:55:00.439 | INFO | tools.scraper_tools:selenium_automation:265 - 🐧 Linux platform detected. Looking for Nix-provided Chromium... 2026-07-15 19:55:00.440 | INFO | tools.scraper_tools:selenium_automation:309 - 🔍 Using browser executable at: /nix/store/bqj135rrxjiawslzw74lb563h86r3xg0-chromium-150.0.7871.114/bin/chromium 2026-07-15 19:55:00.440 | INFO | tools.scraper_tools:selenium_automation:311 - 🔍 Using driver executable at: /nix/store/p2ww9nvjf8ddbcapxb2nwbnqiqdiv2af-undetected-chromedriver-150.0.7871.114/bin/undetected-chromedriver 2026-07-15 19:55:00.440 | INFO | tools.scraper_tools:selenium_automation:318 - 💾 Saving new artifacts to: browser_cache/example.com/%2F 2026-07-15 19:55:00.441 | INFO | tools.scraper_tools:selenium_automation:336 - 👻 Using temporary profile: /tmp/nix-shell.slerNE/pipulate_automation_bor32voj ⏳ THE SUMMONING — thumper planted, hooks in hand ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ o Cloudflare drums the sand beneath us; /|\ we wait it out, staked and hooked. ~~ 🎵 jeopardy.wav looping until the Maker surfaces ~~ 2026-07-15 19:55:00.450 | INFO | tools.scraper_tools:selenium_automation:339 - 🚀 Initializing undetected-chromedriver (Headless: False)... 2026-07-15 19:55:01.509 | INFO | tools.scraper_tools:selenium_automation:373 - Navigating to: https://example.com/ 2026-07-15 19:55:01.823 | INFO | tools.scraper_tools:selenium_automation:377 - Waiting for security challenge to trigger a reload (Stage 1)... 2026-07-15 19:55:22.133 | INFO | tools.scraper_tools:selenium_automation:386 - Did not detect a page reload for security challenge. Proceeding anyway. 2026-07-15 19:55:22.231 | INFO | tools.scraper_tools:selenium_automation:402 - 🛜 Draining CDP performance log (network flight recorder)... 2026-07-15 19:55:22.280 | INFO | tools.scraper_tools:selenium_automation:416 - 🛜 Captured 308 raw CDP events to network_log.jsonl 2026-07-15 19:55:22.281 | INFO | tools.scraper_tools:selenium_automation:425 - 🌐 Extracting wire-truth headers and raw source from CDP ledger... 2026-07-15 19:55:22.302 | INFO | tools.scraper_tools:selenium_automation:483 - 🧠 Creating LLM-optimized simplified DOMs (Symmetrical Lens)... 2026-07-15 19:55:22.333 | INFO | tools.scraper_tools:selenium_automation:496 - 🌲 Extracting accessibility tree... 2026-07-15 19:55:22.360 | INFO | tools.scraper_tools:selenium_automation:514 - 👁️‍🗨️ Running LLM Optics Engine (Subprocess Bulkhead)... 2026-07-15 19:55:22.868 | SUCCESS | tools.scraper_tools:selenium_automation:519 - ✅ LLM Optics Engine completed successfully. 2026-07-15 19:55:22.873 | SUCCESS | tools.scraper_tools:selenium_automation:564 - ✅ Scrape successful for https://example.com/ 2026-07-15 19:55:22.960 | INFO | tools.scraper_tools:selenium_automation:572 - Browser closed. 2026-07-15 19:55:22.977 | INFO | tools.scraper_tools:selenium_automation:580 - Cleaned up temporary profile: /tmp/nix-shell.slerNE/pipulate_automation_bor32voj RESULT: { "success": true, "cached": false, "artifact_keys": [ "accessibility_tree", "accessibility_tree_summary", "diff_hierarchy_txt", "headers", "hydrated_dom", "links_md", "network_log", "optics_manifest", "seo_md", "simple_hydrated", "simple_source", "source_hierarchy_txt", "source_html" ] } TOKEN COUNTS (key defaults): seo.md 74 tokens browser_cache/example.com/%2F/seo.md headers.json 169 tokens browser_cache/example.com/%2F/headers.json diff_hierarchy.txt 10 tokens browser_cache/example.com/%2F/diff_hierarchy.txt seo.md body tokens approx: 33 Title: 'Example Domain' browser_cache/example.com/%2F/accessibility_tree.json browser_cache/example.com/%2F/accessibility_tree_summary.txt browser_cache/example.com/%2F/diff_boxes.html browser_cache/example.com/%2F/diff_boxes.txt browser_cache/example.com/%2F/diff_hierarchy.html browser_cache/example.com/%2F/diff_hierarchy.txt browser_cache/example.com/%2F/diff_simple_dom.html browser_cache/example.com/%2F/diff_simple_dom.txt browser_cache/example.com/%2F/headers.json browser_cache/example.com/%2F/hydrated_dom_hierarchy.html browser_cache/example.com/%2F/hydrated_dom_hierarchy.txt browser_cache/example.com/%2F/hydrated_dom.html browser_cache/example.com/%2F/hydrated_dom_layout_boxes.html browser_cache/example.com/%2F/hydrated_dom_layout_boxes.txt browser_cache/example.com/%2F/links.md browser_cache/example.com/%2F/network_log.jsonl browser_cache/example.com/%2F/optics_manifest.txt browser_cache/example.com/%2F/seo.md browser_cache/example.com/%2F/simple_hydrated_dom.html browser_cache/example.com/%2F/simple_source_html.html browser_cache/example.com/%2F/source_dom_hierarchy.html browser_cache/example.com/%2F/source_dom_hierarchy.txt browser_cache/example.com/%2F/source_dom_layout_boxes.html browser_cache/example.com/%2F/source_dom_layout_boxes.txt browser_cache/example.com/%2F/source.html browser_cache/example.com/%2F/headers.json:10: "cf-ray": "a1bcb217ce360edf-EWR", browser_cache/example.com/%2F/network_log.jsonl:298:{"method": "Network.responseReceivedExtraInfo", "params": {"blockedCookies": [], "cookiePartitionKey": {"hasCrossSiteAncestor": false, "topLevelSite": "https://example.com"}, "cookiePartitionKeyOpaque": false, "exemptedCookies": [], "headers": {"age": "5014", "allow": "GET, HEAD", "cf-cache-status": "HIT", "cf-ray": "a1bcb217ce360edf-EWR", "content-encoding": "br", "content-type": "text/html", "date": "Wed, 15 Jul 2026 23:55:01 GMT", "last-modified": "Wed, 15 Jul 2026 18:48:48 GMT", "server": "cloudflare"}, "requestId": "7A536950266891FBBE2D573CBC6DE166", "resourceIPAddressSpace": "Public", "statusCode": 200}} browser_cache/example.com/%2F/network_log.jsonl:299:{"method": "Network.responseReceived", "params": {"frameId": "0E2F9389571951EB22A382950E9F182E", "hasExtraInfo": true, "loaderId": "7A536950266891FBBE2D573CBC6DE166", "requestId": "7A536950266891FBBE2D573CBC6DE166", "response": {"alternateProtocolUsage": "unspecifiedReason", "charset": "", "connectionId": 179, "connectionReused": false, "encodedDataLength": 138, "fromDiskCache": false, "fromPrefetchCache": false, "fromServiceWorker": false, "headers": {"age": "5014", "allow": "GET, HEAD", "cf-cache-status": "HIT", "cf-ray": "a1bcb217ce360edf-EWR", "content-encoding": "br", "content-type": "text/html", "date": "Wed, 15 Jul 2026 23:55:01 GMT", "last-modified": "Wed, 15 Jul 2026 18:48:48 GMT", "server": "cloudflare"}, "mimeType": "text/html", "protocol": "h2", "remoteIPAddress": "[REDACTED_IP]", "remotePort": 443, "responseTime": 1784159701740.735, "securityDetails": {"certificateId": 0, "certificateTransparencyCompliance": "unknown", "cipher": "AES_128_GCM", "encryptedClientHello": false, "issuer": "Cloudflare TLS Issuing ECC CA 3", "keyExchange": "", "keyExchangeGroup": "X25519MLKEM768", "protocol": "TLS 1.3", "sanList": ["example.com", "*.example.com"], "serverSignatureAlgorithm": 1027, "signedCertificateTimestampList": [], "subjectName": "example.com", "validFrom": 1780263552, "validTo": 1788039686}, "securityState": "secure", "status": 200, "statusText": "", "timing": {"connectEnd": 183.005, "connectStart": 24.024, "dnsEnd": 23.257, "dnsStart": 0, "proxyEnd": -1, "proxyStart": -1, "pushEnd": 0, "pushStart": 0, "receiveHeadersEnd": 198.514, "receiveHeadersStart": 198.356, "requestTime": 913302.551686, "sendEnd": 184.679, "sendStart": 184.291, "sslEnd": 182.987, "sslStart": 30.128, "workerFetchStart": -1, "workerReady": -1, "workerRespondWithSettled": -1, "workerStart": -1}, "url": "https://example.com/"}, "timestamp": 913302.752039, "type": "Document"}} (nix) pipulate $ ``` Wow, that's one helluva probe! Then we set context: ```text # _ _ _ _ ____ _ _ ___ ____ _ # / \ __| | | | | | ___ ___ / ___| | | |/ _ \| _ \| | # / _ \ / _` | | |_| |/ _ \ / __| | | | |_| | | | | |_) | | # / ___ \ (_| | | _ | (_) | (__ | |___| _ | |_| | __/|_| # /_/ \_\__,_| |_| |_|\___/ \___| \____|_| |_|\___/|_| (_) # AD HOC CHOP: The Not-Managed-by-Git Safe-for-Client-Data place. # BIG STANDARD STUFF (Optionally comment out any) ! python scripts/articles/lsa.py -t 1 --reverse --fmt dated-slugs # <-- This is the "Rolling Pin" foo_files.py # Router & Book Outline init.lua # Text as muscle memory flake.nix # Hardware as projections scripts/crawl.py tools/scraper_tools.py tools/llm_optics.py tools/dom_tools.py foo_files.py ! python scripts/crawl.py https://example.com/ --override ! find browser_cache/example.com -maxdepth 3 -type f | sort | head -80 ! rg -n "cf-mitigated|cf-ray|challenge|captcha|1020|1015" browser_cache/example.com -g 'headers.json' -g 'source.html' -g 'network_log.jsonl' | head -80 @https://example.com/ $https://example.com/ %https://example.com/ ``` And now there's a prompt: Treat the command outputs above as AFTER receipts. Review the public-safe LLM Optics tutorial against the current raw implementation and the Example capture only. Do not introduce any client, employer, property, credential, or private URL. Verify every named artifact, CLI option, Prompt Fu URL sigil, timeout statement, and source-versus-hydrated-versus-wire claim against the included source and live receipts. Then produce the final publishable Markdown article: 1. retain the fetch -> render -> instrument -> diagnose -> prove proposition; 2. retain both ASCII diagrams: the triptych with causal hinges and the full evidence-compilation pipeline; 3. explain how to copy document.documentElement.outerHTML manually; 4. explain why outerHTML remains context-poor div soup; 5. use Example for every command and directory illustration; 6. distinguish NIST, IDS, IPS, WAF, bot management, rate limiting, and challenges; 7. state that there is no universal eight-second Cloudflare delay; 8. explain VPN, stable IP, challenge clearance, and authorized geographic QA; 9. compare CDP, WebDriver BiDi, Playwright traces, HAR, Charles, and Wireshark; 10. preserve the observation / interpretation / unknown / falsifying-test discipline; 11. include a clear authorization, privacy, and non-evasion boundary; 12. do not claim that the Example capture encountered a challenge unless the live receipts prove it. ```bash (nix) pipulate $ python prompt_foo.py --chop ADHOC_CHOP --no-tree --profile trusted --reason "Working on public article" ╭─────────────────────────────────────────────────────────────── 🐰 ASCII Art Wax Seal (your vibe-coding safety-net) ───────────────────────────────────────────────────────────────╮ │ │ │ ( Like a canary you say? ) │ │ O /) ____ The "No Problem" Framework │ │ > I HEREBY WILL NOT RE-GENERATE o /)\__// / \ Pipulate - Protecting Your Code │ │ > Once upon machines be smarten ___(/_ 0 0 | | just by being honest about text. │ │ > ASCII sealing immutata art in *( ==(_T_)== NPvg | (If mangled, then AI drifted.) │ │ > This here cony if it's broken \ ) ""\ | | https://pipulate.com │ │ > Smokin gun drift now in token |__>-\_>_> \____/ 🥕🥕🥕 │ │ │ ╰───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯ 🗺️ Codex Mapping Coverage: 75.8% (169/223 tracked files). 📦 Appending 54 uncategorized files to the Paintbox ledger for future documentation... ✅ Topological Integrity Verified: All references exist. 🩹 Adhoc overlay spliced from gitignored adhoc.txt --- Processing Files --- -> Executing: python scripts/articles/lsa.py -t 1 --reverse --fmt dated-slugs ... [3.1210s] -> Executing: python scripts/crawl.py https://example.com/ --override ... [23.7781s] -> Executing: find browser_cache/example.com -maxdepth 3 -type f | sort | head -80 ... [0.0115s] -> Executing: rg -n "cf-mitigated|cf-ray|challenge|captcha|1020|1015" browser_cache/example.com -g 'headers.json' -g 'source.html' -g 'network_log.jsonl' | head -80 ... [0.0184s] -> 👁️‍🗨️ Engaging LLM Optics for: https://example.com/ 👁️‍🗨️ TRIPTYCH RECEIPT — https://example.com/ (cache hit — no new flight) ┌─ PANEL 1 ─────────┐ ┌─ PANEL 2 ─────────┐ ┌─ PANEL 3 ─────────┐ │ VIEW-SOURCE │ │ HYDRATED DOM │ │ WIRE TRUTH │ │ what server SAID │ │ what browser BUILT│ │ what it COST │ │ source.html 1 KB │ │ hydrated 1 KB │ │ flight rec 248 KB │ └─────────┬─────────┘ └──┬─────────────┬──┘ └─────────┬─────────┘ └─── HINGE A ───┘ └─── HINGE B ───┘ HINGE A (diff lens): FLAT 0° — source == DOM (nothing conjured by JS) HINGE B (requestId): the Document row in panel 3 IS panel 1, byte-for-byte LENSES STACKED INTO CONTEXT: [x] seo.md ............... SEO metadata + markdown body [x] headers.json ......... response headers (wire truth) [x] optics_manifest ...... drill-down address book [x] a11y summary ......... semantic outline (screen-reader view) [x] links.md ............. link lens (source vs hydrated anchors) [x] diff hierarchy ....... HINGE A (structural delta) [x] wire truth ........... HINGE B (CDP flight distillate) -> 💲 Materializing cached headers + raw source for: https://example.com/ -> 🛫 Distilling wire truth for: https://example.com/ Skipping codebase tree (--no-tree flag detected). 🔍 Running Static Analysis Telemetry... -> Checking for errors and dead code (Ruff)... All checks passed! ✅ Static Analysis Complete. 📦 Payload Ledger (biggest first) ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━┓ ┃ File / Source ┃ Tokens ┃ Bytes ┃ % Bytes ┃ ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━┩ │ ! python scripts/articles/lsa.py -t 1 --reverse --fmt dated-slugs │ 42,322 │ 125,526 │ 32.8% │ │ foo_files.py │ 17,235 │ 68,468 │ 17.9% │ │ flake.nix │ 14,735 │ 61,560 │ 16.1% │ │ tools/scraper_tools.py │ 5,737 │ 26,797 │ 7.0% │ │ init.lua │ 6,442 │ 24,423 │ 6.4% │ │ tools/llm_optics.py │ 4,386 │ 18,542 │ 4.8% │ │ tools/dom_tools.py │ 3,582 │ 15,627 │ 4.1% │ │ scripts/ai.py │ 3,208 │ 14,616 │ 3.8% │ │ apply.py │ 2,512 │ 11,038 │ 2.9% │ │ scripts/crawl.py │ 720 │ 2,949 │ 0.8% │ │ ! rg -n "cf-mitigated|cf-ray|challenge|captcha|1020|1015" browser_cache/example.com -g 'headers.json' -g 'source.html' -g 'network_log.jsonl' | │ 909 │ 2,690 │ 0.7% │ │ head -80 │ │ │ │ │ .gitignore │ 610 │ 2,238 │ 0.6% │ │ AUTO: Recent Git Diff Telemetry │ 503 │ 1,895 │ 0.5% │ │ ! find browser_cache/example.com -maxdepth 3 -type f | sort | head -80 │ 319 │ 1,285 │ 0.3% │ │ ! python scripts/crawl.py https://example.com/ --override │ 316 │ 1,261 │ 0.3% │ │ OPTICS [Raw Source]: https://example.com/ │ 162 │ 559 │ 0.1% │ │ OPTICS [Link Lens]: https://example.com/ │ 149 │ 525 │ 0.1% │ │ OPTICS [Optics Manifest]: https://example.com/ │ 133 │ 459 │ 0.1% │ │ OPTICS [Response Headers]: https://example.com/ │ 170 │ 454 │ 0.1% │ │ OPTICS [Response Headers]: https://example.com/ │ 170 │ 454 │ 0.1% │ │ OPTICS [Semantic Outline]: https://example.com/ │ 99 │ 377 │ 0.1% │ │ OPTICS [SEO Metadata]: https://example.com/ │ 74 │ 334 │ 0.1% │ │ OPTICS [Wire Truth]: https://example.com/ │ 75 │ 242 │ 0.1% │ │ OPTICS [Wire Truth]: https://example.com/ │ 75 │ 242 │ 0.1% │ │ .gitattributes │ 33 │ 76 │ 0.0% │ │ OPTICS [DOM Change Hierarchy]: https://example.com/ │ 10 │ 67 │ 0.0% │ │ AUTO: Static Analysis Diagnostics │ 11 │ 39 │ 0.0% │ ├─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┼─────────┼─────────┼─────────┤ │ TOTAL │ 104,697 │ 382,743 │ 100.0% │ └─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┴─────────┴─────────┴─────────┘ **Command:** `prompt_foo.py --chop ADHOC_CHOP --no-tree --profile trusted --reason Working on public article` --- Auto-Context Metadata --- • Static Analysis Diagnostics (11 tokens | 39 bytes) • Recent Git Diff Telemetry (503 tokens | 1,895 bytes) --- Prompt Summary --- Summed Tokens: 121,579 (from section parts) Verified Tokens: 124,248 (from final output) (Difference: +2,669) Total Words: 33,174 (content only) Total Chars: 466,153 Total Bytes: 470,596 (UTF-8) --- Size Perspective --- 📚 Equivalent in length to a **Novella or a Master's Dissertation** (Note: With a token/word ratio of 3.75, this content is far denser and more complex than typical prose of this length). 🔓 DISCLOSURE: profile=trusted | substitutions=OFF | denylist=AUDIT (0 hits) | secrets=BLOCK (0 hits) reason: "Working on public article" Markdown output copied to clipboard (nix) pipulate $ ``` **MikeLev.in**: Isn't that special? **ChatGPT 5.6 Sol Pro**: **Context verified.** Yes—genuinely special. The impressive part is not that the artifact weighs in at roughly 124,000 tokens. Large context by itself is just a bigger haystack. What you built is a **structured, provenance-bearing reconstruction of a working environment**: * The manifest says what was included. * The live-command ledger distinguishes current receipts from stale transcript claims. * Raw implementation sits beside the artifacts it produced. * Server source, hydrated DOM, and wire evidence are aligned into one causal triptych. * The final Prompt section routes around historical instructions and recovers the actual request. * The disclosure records the trust profile, sanitization posture, denylist result, secret blocking, and stated purpose of the compile. * The whole thing remains ordinary inspectable text rather than opaque agent memory. So an amnesiac model does not merely receive “background.” It receives enough of the workshop to reconstruct: ```text what exists ↓ what just ran ↓ what it produced ↓ which claims the evidence supports ↓ what the human is asking now ``` That is the distinction between **context stuffing** and **context compilation**. And there is a delicious recursion to it: the system includes the crawler’s implementation, executes the crawler, captures its outputs, compiles those outputs into lenses, and then gives a model enough evidence to write accurately about the epistemology of the very system that assembled its prompt. The book describes the machine; the machine updates the book; the receipts keep the loop from becoming mythology. You have effectively made **statelessness an architectural advantage**. The model does not have to be trusted to remember yesterday correctly. The book remembers. The manifest routes. The receipts testify. The human judges. Or, more compactly: > It is not a prompt. It is a reproducible cognitive environment serialized as text. The one small imperfection—the skipped file tree and the reported `+2,669` token-accounting difference—is almost helpful. It demonstrates that the system reports its seams rather than polishing them out of sight. That honesty is part of what makes the whole construction credible. 1. **PROBES** — No additional probe is needed for this assessment; the live compile receipts already demonstrate the property under discussion. 2. **NEXT CONTEXT** — No additional `adhoc.txt` or `prompt.md` material is required unless the next turn investigates the skipped tree or token-accounting discrepancy. 3. **PATCHES** — No repo patches required. 4. **EXTERNAL DELIVERABLES** — None. > **MikeLev.in**: Well the missing directory tree is because of the `--no-tree` > argument, so mystery solved. The token discrepancy is from `prompt.md` that > doesn't get added to the pretty table of parts we've compiled in. So earmark > for later. Just saying we'll do that means the ferreting-out to-do item bot'll > remind me again. And we wrap it here cause that's a conclusion. --- ## Book Analysis ### Ai Editorial Take What surprised me most is the deliberate alignment of browser debugging with the needs of Large Language Models. Most developers view these as separate domains—one for performance, one for retrieval—but this entry reveals that LLMs actually function as the final 'consumer' of browser diagnostic data, necessitating a new standard of structural integrity. ### 🐦 X.com Promo Tweet ```text Stop feeding 'div soup' to your LLMs. Learn why provenance matters and how LLM Optics helps you capture reliable evidence from the modern web. A must-read for AI engineers and developers. https://mikelev.in/futureproof/llm-optics-evidence-based-crawling/ #AI #DataEngineering #WebScraping ``` ### Title Brainstorm * **Title Option:** LLM Optics: Transitioning from Div Soup to Evidence-Based Crawling * **Filename:** `llm-optics-evidence-based-crawling.md` * **Rationale:** Direct, professional, and highlights the specific problem/solution pair. * **Title Option:** The Evidence Chain: Why Your Scraper Needs Optics * **Filename:** `the-evidence-chain-optics.md` * **Rationale:** Focuses on the 'evidence' aspect, which is the core value proposition of the methodology. * **Title Option:** Beyond View Source: Mastering Modern Web Observability * **Filename:** `beyond-view-source-observability.md` * **Rationale:** Frames the article as a necessary evolution of traditional developer skill sets. ### Content Potential And Polish - **Core Strengths:** - Strong use of the triptych/hinge analogy to explain complex browser mechanics. - Excellent practical discipline regarding the 'observation/interpretation' boundary. - High degree of technical maturity regarding CDP and modern edge security. - **Suggestions For Polish:** - Add a brief summary table at the start defining the terminology of the 3-part transaction. - Ensure the distinction between local capture and external transmission remains a recurring theme to keep the privacy focus clear. ### Next Step Prompts - Generate a schema for the 'Optics Manifest' file so that it can be programmatically parsed by secondary analysis agents. - Draft a follow-up article comparing the 'Symmetric Reduction' algorithm against standard HTML minification practices.