Practical Tips for Getting Found by AI Search

When a patient asks ChatGPT Search, Perplexity, Claude, or Google AI Overviews, "Who treats chronic migraine near me, and what does it cost?", will the engine cite your practice, and will it state accurate facts?

I recently prepared the website of an outpatient specialty medical practice for AI search retrieval. Most advice marketed as "Generative Engine Optimization" (GEO) amounts to cargo-cult SEO. The empirical research and platform documentation point to a simpler set of engineering requirements: explicit crawler access, unified structured data, direct factual copy, and CI tests that prevent machine-readable metadata from drifting out of sync with visible pages.

Note: The examples below use an anonymized practice ("Example Headache Clinic" at example.com).

1. What the Empirical Research Shows

The foundational study in this space is GEO: Generative Engine Optimization (Aggarwal et al., KDD 2024). The authors created a 10,000-query benchmark (GEO-bench), applied nine source-rewriting strategies, and evaluated each source's share of the generated response, weighted by citation rank.

Many marketing blogs circulate an unsourced table claiming exact gains (+40.1% for citations, +37.3% for statistics). The published paper's Table 1 (position-adjusted word count) shows different results. Lifts below are relative to the unoptimized baseline:

• Quotation Addition: +42.6% lift over baseline (27.8 vs. 19.5)
• Statistics Addition: +32.8% lift (25.9 vs. 19.5)
• Fluency Optimization: +28.7% lift (25.1 vs. 19.5)
• Cite Sources: +27.7% lift (24.9 vs. 19.5)
• Technical Terminology: +18.5% lift (23.1 vs. 19.5)
• Keyword Stuffing: -8.7% decline (17.8 vs. 19.5)

On Perplexity—a live search engine rather than an isolated lab setup—the top optimization strategies yielded visibility gains up to 37%.

Evaluate these findings with two technical constraints:

1. The authors conducted their primary benchmarks in 2023 against an engine powered by GPT-3.5. Modern search engines run more sophisticated multi-step retrieval and synthesis pipelines.
2. The benchmark measured relative word share in the generated response, not web traffic, click-through rates, or conversions.

The actionable takeaway remains clear: the engines in the study favored attributable, verifiable data points over marketing prose.

2. Google's Guidance: Generative Optimization Is Traditional SEO

Google's official documentation for AI features states plainly that AI Overviews and AI Mode require no bespoke markup or special optimizations beyond standard search indexing.

Google's guide to optimizing for generative AI lists common vendor tactics you can ignore in a dedicated "Mythbusting" section:

• Breaking content into small chunks for LLMs.
• Rewriting content just for AI systems.
• Creating llms.txt files or other "special" markup for Google Search.
• Adding special Schema.org markup for AI features.

Google instead points to standard search prerequisites:

1. Indexability: The page must be indexed and eligible for snippets (unblocked by nosnippet or noindex).
2. Text Parity: Structured data must match visible text on the page in substance.
3. Canonical Profiles: Google Business Profile and Merchant Center data must stay current.
4. Non-Commodity Content: The page should offer original, first-hand facts rather than summarized third-party prose.

For a professional service or medical practice, "non-commodity content" means concrete figures: exact consultation fees, specific visit lengths, accepted insurance, and verified board certifications published directly in the page text.

3. Crawler Management: Configuring robots.txt for AI Retrieval

Providers deploy distinct bots for model training, search indexing, and real-time user requests. Configure robots.txt based on the specific bot function:

Provider	Model Training	Search / Retrieval Indexing	User-Initiated Fetch
OpenAI	GPTBot	OAI-SearchBot	ChatGPT-User
Anthropic	ClaudeBot	Claude-SearchBot	Claude-User
Perplexity	(None; PerplexityBot does not train models)	PerplexityBot	Perplexity-User
Google	Google-Extended (token, not a crawler)	Googlebot (AI features run in Search)	Googlebot
Common Crawl	CCBot	(None)	(None)

Key operational rules:

• Blocking Google-Extended stops Google from using content to train Gemini, but it does not remove content from Google AI Overviews. Google AI Overviews uses the main Googlebot index.
• ChatGPT-User and Perplexity-User execute on-demand fetches when an end-user explicitly requests a URL or submits a live search. These fetchers may bypass robots.txt disallow directives because a human requested the page.
• OpenAI and Perplexity say robots.txt changes can take about 24 hours to take effect.

To maximize search visibility across all AI platforms without restriction:

User-agent: *
Allow: /

Sitemap: https://example.com/sitemap.xml

To permit AI search retrieval while blocking bulk model training:

User-agent: *
Allow: /

# Block bulk model training datasets
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: CCBot
Disallow: /

# Search bots (OAI-SearchBot, Claude-SearchBot, PerplexityBot) inherit the global Allow rule.

Sitemap: https://example.com/sitemap.xml

Never set User-agent: * with Disallow: / to prevent model training; doing so completely ejects the site from ChatGPT Search, Perplexity, and Claude's search index.

Inspect your CDN and WAF settings as well: aggressive bot-management rules and challenge pages can block AI search bots even when robots.txt permits access. Perplexity publishes its crawler IP ranges so you can allowlist them.

4. llms.txt: Operational Reality and Practical Scope

The llms.txt specification (introduced by Jeremy Howard, revised to v2 in August 2026) proposes serving a Markdown file at /llms.txt containing an H1 title, a summary blockquote, and an H2 list of Markdown-formatted resources. Version 2 supports path-scoped files (e.g., /docs/llms.txt) and .md mirrors of standard web pages.

Evaluate the protocol against production reality:

• Zero Search Impact: Google Search and Google AI Overviews completely ignore llms.txt.
• Near-Zero Crawler Traffic: An Ahrefs study of 137,000 domains (May 2026 server logs) found that 97% of published llms.txt files received zero requests. AI search retrieval crawlers made just 1.1% of the requests to the remaining files.
• Developer Agent Utility: Among AI bots, agents such as Claude Code fetched llms.txt most often (10.5% of requests), far more than AI search crawlers. Most requests came from SEO audit tools and other non-AI bots.

Publishing an llms.txt provides value only as an explicit context manifest for end-user agents pointed directly at your domain.

# Example Headache Clinic

> Example Headache Clinic provides outpatient headache medicine in Anytown, USA. Jane Example, MD leads the practice, holding subspecialty certification in headache medicine. The practice provides diagnostic evaluations and in-office treatments for chronic migraine and complex headache disorders.

## Services and Pricing
- [Services and Fee Schedule](https://example.com/services.md): Evaluation pricing, procedure costs, and HSA/FSA documentation.
- [New Patient Information](https://example.com/patients.md): Intake protocols, scheduling requirements, and cancellation policies.

## Physician Credentials
- [Dr. Jane Example](https://example.com/about.md): Medical training, board certifications, and clinical focus.

## Contact
- [Location and Contact](https://example.com/contact.md): Physical address, phone number, operating hours, and appointment requests.

Operational security requirement: Treat llms.txt as production source code. Put it under version control, restrict editing privileges, and keep it free of prompt-injection vectors.

5. Structured Data: Interconnected Entity Graphs via JSON-LD

Isolate structured data into a single, cohesive @graph block rendered in the HTML document <head>. Do not scatter disconnected JSON-LD snippets across multiple templates.

Establish deterministic @id URI references to explicitly link entities into a queryable graph:

// src/components/StructuredData.tsx
export function StructuredData() {
  const schema = {
    "@context": "https://schema.org",
    "@graph": [
      {
        "@type": ["MedicalClinic", "LocalBusiness"],
        "@id": "https://example.com/#organization",
        name: "Example Headache Clinic",
        url: "https://example.com/",
        telephone: "+1-555-010-0000",
        address: {
          "@type": "PostalAddress",
          streetAddress: "100 Main Street, Suite 200",
          addressLocality: "Anytown",
          addressRegion: "ST",
          postalCode: "00000",
          addressCountry: "US",
        },
        employee: { "@id": "https://example.com/#physician" },
        sameAs: [
          // Google Business Profile URL, LinkedIn company page, Wikidata entry if one exists
        ],
      },
      {
        "@type": ["Person", "Physician"],
        "@id": "https://example.com/#physician",
        name: "Jane Example, MD",
        jobTitle: "Neurologist, Headache Medicine",
        worksFor: { "@id": "https://example.com/#organization" },
        hasCredential: [
          {
            "@type": "EducationalOccupationalCredential",
            name: "Board certification in Neurology (ABPN)",
          },
          {
            "@type": "EducationalOccupationalCredential",
            name: "Subspecialty certification in Headache Medicine (UCNS)",
          },
        ],
        sameAs: [
          // NPI Registry entry, state medical board license lookup
        ],
      },
      {
        "@type": "WebSite",
        "@id": "https://example.com/#website",
        url: "https://example.com/",
        publisher: { "@id": "https://example.com/#organization" },
      },
    ],
  };

  return (
    <script
      type="application/ld+json"
      dangerouslySetInnerHTML={{ __html: JSON.stringify(schema) }}
    />
  );
}

Anchor entities using authoritative sameAs registries:

• Physicians: CMS NPI Registry, State Medical Licensing Boards.
• Financial Advisors: FINRA BrokerCheck.
• Attorneys: State Bar Directories.
• Businesses: Official Secretary of State Filings, Google Business Profile CID, Wikidata entries.

Every claim inside the structured data payload must match the visible text of the rendered page. Discrepancies between JSON-LD properties and visible DOM nodes make the markup ineligible for rich results and hand AI systems conflicting facts.

6. High-Density Copy: Answering User Queries Directly

Vague adjectives give a retrieval system nothing to extract. Place specific answers at the top of the content hierarchy.

Flawed: Subjective Marketing Copy

We provide world-class, caring migraine solutions and have been recognized as the best headache clinic in the area. Our team takes plenty of time with you. Contact us today to learn more about our affordable pricing!

Optimized: Verifiable Information Density

Example Headache Clinic is an outpatient headache medicine practice in Anytown, led by Jane Example, MD. New-patient diagnostic consultations require 60 to 90 minutes. Initial evaluations cost $X, follow-up appointments cost $Y, and in-office procedures (such as occipital nerve blocks) range from $A to $B. Dr. Example maintains board certification in Neurology through the American Board of Psychiatry and Neurology (ABPN) and subspecialty certification in Headache Medicine through the United Council for Neurologic Subspecialties (UCNS).

The optimized passage supplies five unambiguous predicates that retrieval models extract directly: practice type, location, appointment duration, transparent pricing, and governing credentialing bodies.

7. Automated Verification: Preventing Metadata Drift in CI

When facts reside across visible JSX/HTML, JSON-LD graphs, llms.txt, and .md mirrors, changes to one file cause data divergence. A price change in the page body that fails to update the schema can lead AI search engines to synthesize conflicting facts.

Prevent metadata drift by adding validation tests to your standard CI test suite.

Check 1: Route Integrity for Manifests

Validate that every URL referenced in llms.txt resolves to a valid route in the application router:

// src/geo/manifestLinks.test.ts
import { describe, it, expect } from "vitest";
import { readFileSync } from "node:fs";
import { routes } from "../routes";

const SITE_ORIGIN = "https://example.com";

function extractMarkdownLinks(content: string): string[] {
  const matches = content.matchAll(/\]\((https?:\/\/[^)\s]+)\)/g);
  return Array.from(matches, (match) => match[1]);
}

describe("llms.txt link integrity", () => {
  const manifestContent = readFileSync("public/llms.txt", "utf-8");
  const extractedLinks = extractMarkdownLinks(manifestContent);

  it("contains valid link definitions", () => {
    expect(extractedLinks.length).toBeGreaterThan(0);
  });

  it.each(extractedLinks)("verifies %s resolves to an active route", (link) => {
    const url = new URL(link);
    expect(url.origin).toBe(SITE_ORIGIN);

    // Normalize path by stripping .md extensions and trailing slashes
    const normalizedPath = url.pathname.replace(/\.md$/, "").replace(/\/$/, "") || "/";
    expect(routes).toContain(normalizedPath);
  });
});

Check 2: Schema-to-DOM Parity Testing

Write unit or integration tests (using Vitest or Playwright) asserting that every price, telephone number, address string, and credential defined in StructuredData.tsx appears in the rendered HTML output.

Check 3: Post-Deployment Smoke Verification

Add a deployment step that curls /robots.txt, /llms.txt, and sample .md mirrors to confirm HTTP 200 responses and proper Content-Type headers. Purge edge CDN caches on these files immediately following deployment to prevent stale text files from lingering in edge storage.

8. Measurement and Observability

Track AI visibility using direct, un-modeled data sources:

1. Google Search Console (Generative AI Report): Monitor how your pages perform in AI Overviews and AI Mode.
2. Edge Access Logs: Filter server and CDN logs for verified bot user-agents (OAI-SearchBot, Claude-SearchBot, PerplexityBot). Monitor request rates, 403 challenge occurrences, and crawled paths.
3. Deterministic Evaluation Sets: Execute a monthly test battery of 10–15 explicit patient search queries in ChatGPT Search, Perplexity, and Claude. Log whether the engine cites the domain, the linked URL, and whether the model reproduced the exact published figures.

Ignore third-party "GEO visibility scores." Proprietary ranking engines do not expose internal scoring models to external SEO audit platforms.

Summary of Engineering Rules

1. Unblock Search Retrieval: Permit OAI-SearchBot, Claude-SearchBot, and PerplexityBot in robots.txt. Block training bots (GPTBot, ClaudeBot) only if corporate policy demands it.
2. Unify Entity Graphs: Serve a single, fully connected JSON-LD @graph with deterministic @id references and authoritative sameAs verification URLs.
3. Lead with Facts: Publish exact fees, durations, credentials, and locations in plain text near the top of the page.
4. Publish llms.txt for Agents, Not Search: Use llms.txt to serve clean Markdown trees for coding and browsing agents; do not treat it as an organic ranking lever.
5. Lock Down Facts in CI: Enforce link integrity and schema-to-DOM parity through automated tests to eliminate metadata drift.

---

AI attribution: 2/4 — AI-drafted, human-edited. What this means

Jason Vertrees is the founder of Heavy Chain Engineering, which helps lower middle-market vertical SaaS companies and PE firms turn scattered AI usage into measurable delivery leverage — 85% faster feature velocity, six-to-eight-week projects shipped in days. If you want help building an AI-native engineering organization, book an AI Delivery Assessment or email jason.vertrees@gmail.com.