Why we built an AEO carve-out instead of blocking AI crawlers entirely

For most of 2025, our default robots.txt blocked every AI crawler at the front door. GPTBot, ClaudeBot, PerplexityBot, anthropic-ai, Google-Extended, CCBot — all disallowed across the entire site. The reasoning was simple: protect code, protect customer data, protect IP.

By early 2026, that policy started to bite. Buyers were asking ChatGPT and Claude for "Zendesk alternatives", "customer support for SaaS", "white-label support platform" — and HelpMesh was invisible. AI assistants cited Zendesk, Intercom, Freshdesk; they could not cite us, because they could not crawl us.

The trade-off

The tension between AEO (Answer Engine Optimization) and IP protection is not new. The blunt instrument is "allow all or block all." The nuanced approach is to recognize that different parts of the site have different risk profiles:

• Marketing pages (/, /pricing, /features, /blog, /compare/*) are designed to be read by everyone, AI included. The whole point is discoverability.
• Dashboards, APIs, customer data (/inbox, /api, /admin, /settings) are gated behind authentication and have nothing to gain from indexing. Even AI assistants cannot reach them in practice — but the explicit block is defense in depth.

The carve-out follows the principle: protect what is private; index what is public. Allow AI crawlers on marketing surfaces, block them everywhere else.

The implementation

HelpMesh now ships a Next.js app/robots.ts that enumerates the allowed and blocked paths per AI user-agent, and a next.config.mjs that scopes the X-Robots-Tag: noai, noimageai, noindex header to dashboard prefixes only. The full carve-out lives in our public robots.ts if you want to copy the pattern.

What changed in practice

Our AEO ceiling jumped from approximately 30% (third-party citations only — G2 reviews, Reddit threads, dev blog mentions) to a projected 80–90% (direct citations from helpmesh.io marketing pages). The first measurable improvements showed up two weeks after the policy change, when buyers started reporting that ChatGPT mentioned HelpMesh by name in their procurement research.

The lesson is mundane: most binary policies are wrong. The interesting question is always "where exactly does this trade-off bind?" — and once you ask it, the answer is usually a carve-out, not a coin flip.