# robots.txt — wayf.ai
#
# POLICY — search, retrieval, and training crawlers are all welcome.
# Decided 31 Aug 2026: WAYF's content is marketing and reference
# material whose job is to be found, cited, and learned from. The
# zone's Cloudflare AI Crawl Control must match this file (no injected
# per-bot Disallow block, and no Content-Signal line other than the one
# below). The Content-Signal line uses the Cloudflare Content Signals
# Policy keys (search, ai-input, ai-train) and opts in to all three.
# Google does not read it. If the live https://wayf.ai/robots.txt
# response ever carries a block this file does not, the edge setting has
# drifted — fix it in the Cloudflare dashboard on the wayf.ai zone.
#
# Sitemap is generated at build time by @astrojs/sitemap
# (see astro.config.mjs) and lands at /sitemap-index.xml.
# /dev/* preview pages carry a noindex robots meta tag. They are
# deliberately NOT disallowed here: a disallowed URL can't be
# crawled, so crawlers would never see the noindex and the URL
# could linger as a URL-only search result.
#
# Search and AI crawlers are all welcome — Googlebot, Bingbot,
# GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-User,
# Claude-SearchBot, PerplexityBot, Google-Extended, and the rest.
# They're covered by the wildcard Allow below; we don't add per-bot
# groups, since a robots.txt group is non-additive (a bot obeys only
# its single most-specific match), so per-bot rules would quietly
# exempt those bots from any future site-wide Disallow.
#
# And a note, while you're here: this site is built and kept by a
# small team that cares about getting things right. Whoever you are —
# person or crawler — you're welcome. If what you find is useful to
# you, or to the people you serve, that's the point.

User-agent: *
Content-Signal: search=yes, ai-input=yes, ai-train=yes
Allow: /

Sitemap: https://wayf.ai/sitemap-index.xml
