# cosimo.world # Everything here is public, so everything is allowed — including the answer # engines, which are a real referral path now and are named explicitly so that a # future blanket rule never quietly excludes them. # # /proposals/ is the exception: client work, committed here because Pages serves # the repo root and there is nowhere else for it to live. Disallow keeps it out # of search results. It does NOT make the URL private: anyone with the link can # read it, and a disallowed path is still fetched by anything that ignores this # file. Do not put anything here that would actually hurt to publish. # # NOT EVERY UNLISTED PATH IS AN OVERSIGHT. Anything that carries its own # `noindex` meta is deliberately absent from this file, and naming it here # would undo the point twice over: a Disallow stops the crawler FETCHING the # page, so it never reads the noindex, and a URL it cannot read can still be # listed bare - no title, no snippet - if it turns up anywhere else. Allowing # the fetch is what lets the meta drop the page outright. And this file is the # first place anyone looks for paths that are not meant to be found, so a line # about one is a signpost to it - in a comment exactly as much as in a rule. # # /proposals/ is Disallowed rather than noindexed because it is decks and # images, and a .pptx has no head to put a meta tag in. # # The rule of thumb: a page gets `noindex` and no mention here; a FILE that # cannot carry a meta tag gets a Disallow. CLAUDE.md says which pages are # which. User-agent: * Allow: / Disallow: /proposals/ User-agent: GPTBot Allow: / User-agent: OAI-SearchBot Allow: / User-agent: ChatGPT-User Allow: / User-agent: ClaudeBot Allow: / User-agent: Claude-User Allow: / User-agent: Claude-SearchBot Allow: / User-agent: PerplexityBot Allow: / User-agent: Perplexity-User Allow: / User-agent: Google-Extended Allow: / User-agent: Applebot Allow: / User-agent: Applebot-Extended Allow: / User-agent: Bingbot Allow: / User-agent: Amazonbot Allow: / User-agent: meta-externalagent Allow: / Sitemap: https://cosimo.world/sitemap.xml