Tutorial
Audit a page for answer engines
Answer engines like ChatGPT, Perplexity, and Google's AI Overviews extract short passages, not whole pages. This is a manual, repeatable audit process using curl and a browser to check whether a page is actually extractable, in under twenty minutes per page.
- Foundational
- 10 minute read
- Last reviewed 2026-08-10
Ranking well in traditional search and being cited by an answer engine are related but not identical problems. Answer engines pull a specific passage to quote or summarize. If that passage is buried, rendered client-side only, or phrased vaguely, the page can rank fine and still never get cited.
This audit does not require paid tools. It uses curl, view-source, and a text editor. Run it on any page before and after a content or template change to confirm nothing broke.
Before you start
- Command line access with curl installed
- The live URL of the page being audited
- Access to the site's robots.txt file
Steps
Work through it in order
Step 1
Check what the server actually sends
Answer engine crawlers generally do not execute JavaScript the way a browser does. Fetch the raw HTML the same way a bot would and look for your core content in that response, not just in the rendered browser view.
Fetch raw server HTML curl -s -A "Mozilla/5.0" https://www.example.com/seo/technical-seo | grep -A 3 "<h1"Step 2
Confirm the answer sits in the first text node after the h1
If the raw HTML shows the h1 followed immediately by a hero image, a component wrapper with no text, or a navigation element, there is no extractable answer at the top of the page. Answer engines weight early content heavily because it mirrors how featured snippets are chosen.
The fix is a direct one to two sentence answer to the page's implied question, placed as plain text immediately after the h1, before any supporting narrative.
Confirm placement curl -s https://www.example.com/seo/technical-seo \ | grep -A 5 "<h1" \ | sed -E 's/<[^>]+>//g'Step 3
Check heading phrasing against real questions
Pull every heading on the page and check whether each one reads like a question someone would actually type or ask a voice assistant. 'Our Approach' extracts poorly. 'How long does a technical SEO audit take' extracts well because it matches the shape of a real query.
List all headings curl -s https://www.example.com/seo/technical-seo \ | grep -oE "<h[1-4][^>]*>.*?</h[1-4]>" \ | sed -E 's/<[^>]+>//g'Step 4
Verify entity clarity
Read the page as if you know nothing about the business. Within the first 100 words, can you tell who is offering the service, where they operate, and what the service actually is? Vague pages that lead with mission statements before naming the entity are harder for models to attribute correctly.
Step 5
Check internal links for descriptive anchor text
Pull all internal links and flag any using 'click here', 'learn more', or bare URLs as anchor text. Anchor text is a signal for what the destination page is about, used by both search crawlers and language models building an internal map of the site.
List internal links and anchor text curl -s https://www.example.com/seo/technical-seo \ | grep -oE "<a [^>]*href=\"/[^\"]*\"[^>]*>[^<]*</a>"Step 6
Confirm structured data is present and connected
Extract the JSON-LD block and check that it parses as valid JSON and that it actually describes this page's content, not a stale copy pasted from another template. See the schema markup tutorial for how these nodes should link together.
Pull JSON-LD curl -s https://www.example.com/seo/technical-seo \ | grep -A 200 'application/ld+json' \ | sed -n '1,50p'Step 7
Check robots.txt for AI crawler rules
Confirm you are not accidentally blocking the crawlers that power answer engines. Absence of a rule means allowed by default, but an explicit Disallow for these agents will remove the page from that engine's index entirely.
Check crawler access curl -s https://www.example.com/robots.txt | grep -A 2 -E "GPTBot|ClaudeBot|PerplexityBot"Step 8
Check for an llms.txt file at the domain root
llms.txt is an emerging convention giving language models a curated map of the most important pages, in plain markdown. It will not exist by default and is not required, but where present it should link to genuinely important pages, not the entire sitemap.
Check for llms.txt curl -s -o /dev/null -w "%{http_code}\n" https://www.example.com/llms.txt
Pitfalls
What goes wrong in practice
- Auditing the rendered page in a browser instead of the raw server response. This is the single most common mistake and produces a false pass.
- Writing an answer paragraph but placing it after three paragraphs of introduction, which pushes it below what most extraction windows consider 'early' content.
- Blocking GPTBot in robots.txt to reduce server load without realizing it also removes the page from ChatGPT's browsing and citation sources.
- Treating this as a one-time audit. Template changes and CMS updates regularly break server-rendered content without changing what a browser shows.
What is the fastest way to tell if a page is readable by answer engines?
Run curl against the live URL and read the raw HTML output. If the h1 and a clear one-sentence answer are both present as plain text near the top of that raw response, the page is extractable. If the raw response is mostly empty divs and script tags, it is not, regardless of how it looks in a browser.
- Test the server response, not the rendered page
- Answer must be plain text, near the top, in the raw HTML
- Check robots.txt for GPTBot, ClaudeBot, and PerplexityBot rules explicitly
Related
Where this fits
Rather have this done for you
We run these steps as part of a program, with the checks automated where they can be.