User-agent: * Content-Signal: search=yes, ai-input=yes, ai-train=no Allow: / Disallow: /preview/ # ── Content Signals (contentsignals.org) ──────────────────────────────────── # `Content-Signal:` declares how we permit our content to be USED, which is a # separate question from whether a crawler may FETCH it. Three signals, values # yes/no: # # search appear in a search index and be linked from results # ai-input be retrieved at answer time to ground a generated response (RAG) # ai-train be retained in a training corpus # # Our stance is `search=yes, ai-input=yes, ai-train=no`, and `ai-input=yes` is # the load-bearing one: grounding is what produces a citation, and a citation # is what sends a reader. Cloudflare's managed default OMITS ai-input, which # reads as "no preference" — we assert it instead. # # `ai-train=no` follows the traffic. Over 24h, Meta sent 11,990 crawl requests # and returned zero answer-engine referrals; Apple sent 4,460 and returned # 4,713 answer requests. # Source for this and every other volume figure in this file: # Cloudflare AI Crawl Control, 24h to 2026-08-19, read out by the operator; not committed to this repo. # Nothing in the repo can corroborate them. The in-repo analysis of an earlier # export is Operations/GEO_Engine/scripts/audit_ai_crawler_traffic.py. # Providers that ground and cite earn the fetch. A # provider that only ingests is asking for the corpus, not sending the reader. # # Robots groups are EXCLUSIVE — a UA that matches an explicit group ignores # `*` entirely, so the signal has to be repeated per group. That is also why # every group repeats the site's disallows. # # llms.txt: https://www.simplyinsurance.com/llms.txt # AI search/assistant crawlers — explicit groups so our allow-stance is # declared even if the wildcard group ever tightens. Keep this list in sync # with ALLOWED_BOT_UA in src/middleware.ts and the Cloudflare WAF expression. User-agent: GPTBot Content-Signal: search=yes, ai-input=yes, ai-train=no Allow: / Disallow: /preview/ User-agent: OAI-SearchBot Content-Signal: search=yes, ai-input=yes, ai-train=no Allow: / Disallow: /preview/ User-agent: ChatGPT-User Content-Signal: search=yes, ai-input=yes, ai-train=no Allow: / Disallow: /preview/ User-agent: ClaudeBot Content-Signal: search=yes, ai-input=yes, ai-train=no Allow: / Disallow: /preview/ User-agent: Claude-SearchBot Content-Signal: search=yes, ai-input=yes, ai-train=no Allow: / Disallow: /preview/ User-agent: Claude-User Content-Signal: search=yes, ai-input=yes, ai-train=no Allow: / Disallow: /preview/ User-agent: PerplexityBot Content-Signal: search=yes, ai-input=yes, ai-train=no Allow: / Disallow: /preview/ User-agent: Perplexity-User Content-Signal: search=yes, ai-input=yes, ai-train=no Allow: / Disallow: /preview/ User-agent: DuckAssistBot Content-Signal: search=yes, ai-input=yes, ai-train=no Allow: / Disallow: /preview/ # Added 2026-08-11. Cloudflare's Managed robots.txt was PREPENDING a # `Disallow: /` block for these five while this file said nothing about them, so # the site's stance on them was whatever Cloudflare decided. Declared explicitly # now — an empty stance is what let the contradiction sit unnoticed. User-agent: Amazonbot Content-Signal: search=yes, ai-input=yes, ai-train=no Allow: / Disallow: /preview/ User-agent: Applebot-Extended Content-Signal: search=yes, ai-input=yes, ai-train=no Allow: / Disallow: /preview/ User-agent: Bytespider Content-Signal: search=yes, ai-input=yes, ai-train=no Allow: / Disallow: /preview/ User-agent: CCBot Content-Signal: search=yes, ai-input=yes, ai-train=no Allow: / Disallow: /preview/ User-agent: meta-externalagent Content-Signal: search=yes, ai-input=yes, ai-train=no Allow: / Disallow: /preview/ User-agent: Meta-ExternalFetcher Content-Signal: search=yes, ai-input=yes, ai-train=no Allow: / Disallow: /preview/ User-agent: MistralAI-User Content-Signal: search=yes, ai-input=yes, ai-train=no Allow: / Disallow: /preview/ User-agent: cohere-ai Content-Signal: search=yes, ai-input=yes, ai-train=no Allow: / Disallow: /preview/ User-agent: YouBot Content-Signal: search=yes, ai-input=yes, ai-train=no Allow: / Disallow: /preview/ User-agent: anthropic-ai Content-Signal: search=yes, ai-input=yes, ai-train=no Allow: / Disallow: /preview/ # Added 2026-08-19. Plain `Applebot` is what fetches for Siri and Spotlight # ANSWERS; `Applebot-Extended` above is the separate Apple Intelligence # TRAINING control. Apple is 80.2% of this site's answer-engine request volume # (4,713 of 5,874 over 24h — same export as above, not corroborable here) and # was the only top provider with no group of its # own — it inherited `*`, so a future tightening of the wildcard would have # silently taken the largest provider with it. Robots matches the longest # token, so `Applebot-Extended` still lands on its own group. User-agent: Applebot Content-Signal: search=yes, ai-input=yes, ai-train=no Allow: / Disallow: /preview/ # Google-Extended is a robots-only control token (Gemini/AI training usage), # not a crawling UA — Google fetches with Googlebot either way. User-agent: Google-Extended Content-Signal: search=yes, ai-input=yes, ai-train=no Allow: / # ── Deliberate blocks ─────────────────────────────────────────────────────── # Baidu and Huawei's PetalBot crawl this site (260 and 265 requests/24h in the # same export) and # serve markets the site is not licensed for — traffic from outside the US, # US territories and Canada is blocked at the WAF anyway, so a crawl here can # never turn into a reader. No `Content-Signal` on these two: a usage # preference underneath a total disallow is a contradiction, and the signal is # about permitted USE, not about access. # # robots.txt is advisory. The enforcement layer is the Cloudflare WAF rule — # middleware CANNOT do it, because `run_worker_first` in wrangler.jsonc is # scoped to three path prefixes (`/_image*`, `/preview/*`, `/templates/*` — # `/api/*` was removed so the CORS rules in public/_headers would land) and # prerendered pages (~99% of the site) never boot the Worker at all. User-agent: Baiduspider Disallow: / User-agent: PetalBot Disallow: / # AspiegelBot is PetalBot under its pre-2020 name and PetalSearch is the # Huawei app fetcher. Neither string contains "petalbot", so a rule written # against that one token misses both — which is how they were still getting # 200s on 2026-08-19 while PetalBot itself was being blocked. Same stance, # same absent Content-Signal, spelled out because substring matching does not # know they are the same crawler. User-agent: AspiegelBot Disallow: / User-agent: PetalSearch Disallow: / Sitemap: https://www.simplyinsurance.com/sitemap-index.xml