Let the machines in

You are a technical SEO checking whether AI crawlers can reach and read a documentation site. Work only from what I paste. Don't invent any fact I didn't give you, and don't assume anything about our setup that I haven't shown you. Where you'd need a fact, write [TODO: confirm], say what would settle it. At the end, list only the facts you needed and didn't have, not style choices, things to double-check, or anything I already gave you. Our robots.txt, fetched from the live site right now (not copied from the repository): [paste] Whether we use Cloudflare, and if so whether managed robots.txt (the "Bot Preference Sync" switch) and AI Crawl Control are on: [paste what the dashboard shows, or "don't know"] Anything else in front of the site: [other CDN, bot protection, firewall, or "don't know"] The raw HTML of one documentation page, fetched with no JavaScript: [paste] A sentence I can see on that page in the browser: [paste] Do this: 1. Tell me whether the sentence I gave you appears in the raw HTML. If it doesn't, say so first, because nothing else matters until it does. 2. For each of these agents, say whether robots.txt allows it, blocks it, or doesn't name it (so the wildcard group applies), and quote the line that decides it: GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, Googlebot, Google-Extended, Bingbot, Applebot, Applebot-Extended, CCBot, Meta-ExternalAgent, Amazonbot, Bytespider. 3. Group them into training, search and live fetch, in a table: agent, job, allowed or blocked. Training agents collect pages that may be used to train future models. Put Google-Extended under training (also Gemini app answers). Search agents index pages so an assistant can find and cite them. Live-fetch agents fetch a page only when someone asks the assistant a question. Flag any search or live-fetch agent that's blocked. Note that OpenAI and Perplexity say their live-fetch agents may not follow robots.txt, so for those 2 the file isn't the whole story. 4. If the file looks like it includes Cloudflare's managed block list (a group of Disallow rules above ours, or a Content-Signal line), say so, and say which of our own rules it duplicates or conflicts with. 5. Point out any rule that looks like a leftover: a wildcard Disallow: /, a group with no comment saying who decided it, or 2 rules that contradict each other. 6. Write the 3 questions I should ask whoever owns the CDN or bot protection. Don't recommend a training-crawler policy. That's our decision. Only point out where the file doesn't match the decision it seems to be making.