How we made wayf.ai readable by machines

A few weeks after we added a wildcard rule welcoming crawlers to wayf.ai, we fetched the live robots.txt.

The repository allowed every crawler. Cloudflare had inserted a managed block ahead of our file that denied training crawlers including ClaudeBot, GPTBot, and Google-Extended.

The public response was more restrictive than the source file. Retrieval bots including ChatGPT-User, OAI-SearchBot, Claude-User, Claude-SearchBot, and PerplexityBot remained allowed.

That finding changed how we think about machine readability. A crawler has to find a page, retrieve its substance, understand its structure and entities, and encounter claims that agree across the page and its machine-readable forms. Each link in that chain needs a production check.

We ran that audit on wayf.ai. Here is what changed, how we check it, and where the evidence stops. The audit kit near the end runs the same checks on your own site.

1. Discoverable: find and retrieve the page

The first test is a fetch:

curl -L https://wayf.ai/some-page/ | grep "a fact from the page"

A useful fact should be present in the response body. We prefer to ship the substance as HTML.

wayf.ai is built with Astro and produces static pages. Article bodies, headings, links, tables, and structured data are in the built HTML. Interactive code does not assemble the article a fetcher came to retrieve.

We check rendered output from a journal article, glossary entry, and service page.

One page, one durable address

Our Astro site URL drives canonical tags, Open Graph URLs, and the sitemap. When we moved the primary domain from wayfdigital.com to wayf.ai, we changed that value and preserved paths through permanent redirects.

Cloudflare Pages treated /old-path and /old-path/ as separate matches in our rules. A rename that covered one could leave the other serving an old page or returning a 404. We now test both forms.

We also found that a 120-rule redirects file stopped working near the end. Cloudflare documented a much higher limit, but production dropped rules after roughly the hundredth entry. Fifteen old /work/ URLs with trailing slashes returned 404s while their slash-less equivalents still redirected.

The repair replaced 70 static rules with two dynamic ones. After a deployment that changes redirects, we curl a rule near the bottom of the file. The deployed response decides whether the migration worked.

Give crawlers explicit maps

Astro generates the sitemap. One RSS feed covers the blog and journal and advertises itself in the page head. IndexNow gives participating search engines a direct notification when a meaningful URL changes. Submission sits in our release-review prompt, which sends the request when the change closes.

We also maintain /llms.txt: a curated map with a description beside each selected link. We update it when substantial content ships.

We do not treat llms.txt as a ranking lever. Google Search says it does not use these files, including for its generative features. Support elsewhere remains unclear. We keep ours because it is cheap to maintain and doubles as a compact description of the site.

Check crawler policy at the edge

Our repository’s robots.txt documents the intended policy and applies Allow: / to the wildcard group. Its comments name search, retrieval, and training crawlers separately.

In July 2026, the live response carried explicit disallow rules for training crawlers, inserted by Cloudflare AI Crawl Control. Those named groups overrode our wildcard allow. The same response left retrieval bots without a specific disallow. Reading public/robots.txt showed our broad intent. Fetching https://wayf.ai/robots.txt showed the policy each crawler received.

That check now belongs beside the source review:

curl -s https://wayf.ai/robots.txt

We ran that check again on 1 October 2026. The live file matched public/robots.txt line for line: one wildcard group allowing every crawler, and no managed block. The July response is why we keep comparing the two.

The same check applies to headers, redirects, canonicals, and rendered HTML. Infrastructure can change the public artifact after the build.

2. Understandable: identify the page and its entities

Semantic structure preserves the relationships in the copy.

Our article and glossary templates use heading hierarchies, lists for sequences, and real tables for comparisons. Strip the styling and read the outline: it should still explain the page.

This matters on glossary comparison pages. A <table> preserves the relationship between headers and values in the document itself.

A shared entity graph

The site layout emits an Organization node and a WebSite node in JSON-LD. The organization has a stable identifier:

https://wayf.ai/#organization

Page-level schemas reference that identifier. Blog and journal entries use BlogPosting; the glossary uses DefinedTerm and DefinedTermSet; service pages use Service and BreadcrumbList; role pages use JobPosting.

The company entity has one node. Page schemas point to it instead of restating it.

We validate rendered pages because the right object in source code can still produce broken output.

FAQ markup has to match the visible answers

Our glossary template takes FAQs from a frontmatter array. The same questions and answers render the visible accordion and the FAQPage JSON-LD, so there is no second copy for an editor to keep in step.

Blog and journal posts take their FAQPage JSON-LD from the same kind of array, but the visible FAQ is written in the article body, this one included. An edit to one needs the same edit to the other. The audit kit below looks for each FAQ question and answer in the page text and reports a finding when one is missing.

The shared FAQ component serializes its JSON-LD through safeJsonLd(), which escapes < so content cannot terminate the script element early.

Content fields have jobs

Our content collections are schema-validated. A glossary entry can carry a shortDef, FAQs, sources, and a longer body as separate values. shortDef holds the self-contained answer; the body handles boundaries, trade-offs, and examples.

Role pages preserve the published salary range in visible copy and emit the same range as baseSalary.

Typed fields give each value a job and let the build reject malformed content.

3. Trustworthy: keep claims honest across representations

Structured data creates another place to overstate reality. Unsupported values stay absent.

Our role pages provide the clearest example. JobPosting appears on individual role pages and stays off the careers index. We leave out directApply because the application continues on an external applicant-tracking system. We do not invent validThrough when a role has no confirmed expiry.

The roles were advertised as remote in the EU, but we did not have a definitive country list when the schema shipped. The collection leaves applicantLocationRequirements absent and emits a build warning. The roles remain ineligible for Google’s remote-job rich result until the real country list exists.

When a role is marked closed or passes validThrough, it leaves the careers index, renders a closed state, and stops emitting JobPosting. Its self-canonical URL stays live.

Trace claims back to their source

Machine-readable mistakes travel easily. Our editorial review treats provenance as part of the implementation.

Evidence-bearing glossary entries carry primary sources. Date-sensitive articles include a “Last verified” date. During a Drupal end-of-life update, we checked every date against Drupal’s documentation. We removed an expected 2028 date because Drupal had not published the release date needed to support it. The article now states the policy and labels the date implied by it as a floor.

A service-page review caught a different problem. Draft copy claimed “40% faster ops,” but the canonical Ingersoll Rand case study did not contain that figure. We cut it.

Every checkable claim needs a source URL, case study, repository file, or measured report.

Preserve the reason behind search decisions

Our migration guides and migration service pages had begun chasing the same query intent. We kept the guides informational and retitled 33 /migrate pages around service intent. The pull request records the before-and-after titles.

Both surfaces remain self-canonical. Their titles and cross-links make their jobs explicit.

Build checks support the editorial review. Content collections validate against their schemas. During the glossary depth pass, all 75 internal links were resolved against the content collection and checked for draft targets. The check was scoped to that change; it was not a site-wide guarantee.

4. Measurable: define the evidence

There is no clean metric for “AI readability.” We watch narrower signals.

Google Search Console is our first-party view of how queries meet pages. In the 24 August report, a query that appeared to contain a leaked assistant prompt placed one of our pages at position 3.8. The content-governance post also appeared for assistant-shaped queries at positions 3.6 to 9.3.

That shows machine-shaped language reaching our pages through Google Search. It does not identify the assistant, prove a fetch, or establish a citation.

We also monitor AI mentions in Prelio and disclose that it is our product. We treat it as a second lens on visibility.

The useful measurements attach to a specific change. We record a glossary entry’s starting queries and position, preserve old and new titles during an intent split, and retest broken URLs after a redirect repair.

For each change, we want four things:

1. The condition before the change.
2. The repository diff.
3. A test of the rendered or live result.
4. A defined measurement window.

Without that sequence, a feature inventory can look like an outcome report.

The order we would use on another site

If we were auditing another content-heavy site tomorrow, we would work in this order.

1. Fetch representative pages without executing JavaScript and confirm that their substantive content is present.
2. Check canonicals, legacy redirects, the sitemap, RSS, robots.txt, and any edge-layer modifications.
3. Read the heading outline and inspect whether comparisons, lists, and definitions carry semantic structure.
4. Establish stable entity identifiers and make page schemas reference them.
5. Generate visible FAQs and their structured representation from one source.
6. Separate extractable answers and machine values into typed fields where the content model benefits from them.
7. Omit unsupported schema properties and design expiry behavior before content becomes stale.
8. Trace statistics, dates, and company claims to canonical evidence.
9. Record a baseline and measurement window before expecting a result.

The first three steps expose whether the page survives retrieval and whether infrastructure changes what the application tried to publish.

Run the checks on your own site

Download the audit kit (ZIP, MIT licence). It is a small Node.js script that requests the URLs you name and checks what each response carries: status and redirects, the text in the HTML before any JavaScript runs, canonicals, indexing directives, robots.txt for search and training crawlers, JSON-LD, the sitemap and response headers.

Beyond the pages you name, it requests only what they point to: redirects, each site’s robots.txt and sitemaps, and a declared canonical. It does not crawl or need an account, sends no cookies and keeps no page bodies. Each request is capped at 10 seconds, 5 MB and 5 redirects, and anything over a cap is reported as unavailable. It needs Node.js 22 or later and one dependency, the parse5 HTML parser.

Run it

Unzip the file, enter examples/machine-readability-audit, and name a page or two from each template:

npm ci
node src/cli.mjs https://www.example.com/ https://www.example.com/pricing/

The report lands in results/audit.md. Most checks become pass or fail once you say what you expect: a sentence each page must carry, its canonical URL, whether it belongs in search, and the robots.txt your repository or CMS holds. audit.config.example.json shows the format. CHECKLIST.md covers the same audit as a list a person can fill in, including the checks a script cannot make.

Read the result

Each check ends in one of four results. There is no overall score.

Result	Meaning
Pass	The check ran and found what it looks for.
Finding	The check ran and found a problem worth fixing.
Manual review	The check ran; a person has to judge the result.
Unavailable	The check could not run: an error, a timeout, a refused address or a response over a limit. It never counts as a pass.

On 1 October 2026 we ran the kit against seven public wayf.ai and wayfdigital.com URLs, with expectations taken from our repository. Of 66 checks, 64 passed and two went to manual review. Five rows from that report:

Check	Result	Evidence
robots.txt against the repository file	Pass	The live file matches public/robots.txt exactly.
JSON-LD on this article	Pass	Four blocks parse. BlogPosting and FAQPage sit beside the shared Organization and WebSite nodes, and the FAQ markup matches the visible answers.
JSON-LD on a role page	Pass	JobPosting and BreadcrumbList, with the organization reference resolved on the page.
This URL without its trailing slash	Pass	One 308 redirect to the trailing-slash URL.
An old wayfdigital.com case-study URL	Manual review	Two 301s: the old domain to wayf.ai/work/tdm/, then on to /clients/tdm/.

The old domain redirects path for path, and the legacy /work/ rule on wayf.ai then sends the request on to the client page. Google’s crawlers follow up to ten redirects, so the page is reachable; one hop would save a request. The other manual review is the content check on the same URL, where we gave no expected sentence. The full report is in the download, in recorded-run/wayf-public.md.

Fix the issue

The kit’s edge-drift fixture reproduces the July condition: a managed block placed ahead of the site’s own robots.txt, modelled on the one Cloudflare documents. With the repository file as the source and a stated policy of allowing training crawlers, the report shows:

robots.txt against your source file: finding
  Only in the live file: "User-agent: GPTBot" (line 8), "Disallow: /" (line 9), …
Crawl permission: model training: finding
  GPTBot: blocked (own group, line 9: Disallow: /)
Crawl permission: search and user-requested fetches: pass
  Googlebot: allowed (* group, line 6: Allow: /)

The search crawlers stay allowed on the same file, as they did on wayf.ai in July. Our repository file was already correct; the block came from a Cloudflare setting, so the repair belonged in the Cloudflare dashboard. After a change like that, run the kit against the live URL again: the public response is the one that has to change.

What the results mean

Search and training are separate controls. Google says Google-Extended does not affect a site’s inclusion in Google Search, and OpenAI says a site can allow OAI-SearchBot for search while disallowing GPTBot for training. Blocking the training tokens leaves the search tokens as they were.

The kit reads robots.txt for each crawler token and sends its own User-Agent. A request that borrows a crawler’s name would still not show what that crawler receives: the operator’s crawler arrives from IP ranges it publishes, and a CDN or firewall can answer a look-alike differently. For Google’s own view of a page, use URL Inspection in Search Console.

Google renders JavaScript, though it says rendering is not guaranteed and not every bot runs scripts. The content check reports what a fetcher receives before any script runs.

Some controls sit outside the response. Since 31 August 2026, a Search Console setting decides whether Google uses a site’s content in AI Overviews and AI Mode, with inclusion as the default. CDN bot rules can block a crawler without touching robots.txt. The kit sees neither.

The kit does not look for llms.txt or special AI markup, because Google says its search features need neither. A pass means specific technical checks passed. Whether the content is accurate and useful needs a person, and ranking, AI citation and inclusion stay with each search engine and assistant.

How we tested it

The kit ships with twenty local fixtures: a passing baseline, and nineteen that each break one thing.

• content that only JavaScript adds
• a wrong canonical, two conflicting canonicals, and a canonical a tracking image pushes into the body
• noindex in a meta tag, and in a response header
• malformed JSON-LD
• a redirect loop, a temporary redirect and a redirect to a private address
• a page robots.txt blocks, and a page missing from the sitemap
• a timeout, a server error and a response over the size limit
• a robots.txt that answers 503, 404 or an HTML page, or carries a managed block

On 1 October 2026 all 195 checks across the fixtures matched their expected result, and the 70 tests passed on Node 22 and 24. The robots.txt tests use Google’s published matching examples and the group rules of RFC 9309. On 1,080 decisions, the matcher agreed with the separate robots-parser library every time. Five deliberate bugs each broke a test. The receipts are in recorded-run/ in the download.

What this work cannot promise

A machine-readable site can be retrieved and interpreted more reliably. Citation remains outside our control.

llms.txt may remain unread by the systems we care about. A crawler can respect a restrictive policy set by its operator. Search and assistant products can choose another source. Strong markup cannot rescue a thin answer, an unsupported claim, or a page that misses the user’s intent.

Our own implementation still has open work. The edge policy that contradicted the repository in July no longer does: on 1 October 2026 the live robots.txt matched the repository file. The canonical decision between our comparison blog posts and their /compare-cms twins was made only as this article went out — the blog versions now point their canonicals at the maintained comparisons, and the consolidation is still inside its measurement window.

Machine readability is a release discipline. The source, build, edge, live response, content model, and measurement need to agree.

On wayf.ai, the work came down to stable identifiers, honest omissions, shared data sources, typed fields, redirect tests, source checks, and live HTTP responses.

FAQ

1. How do I check whether AI systems can read my site?

   Fetch representative pages without executing JavaScript and confirm that their main facts, headings, links, and structured data appear in the returned HTML. Then inspect the live robots.txt, canonicals, redirects, sitemap, and response headers. This tests what a crawler receives, including changes made by a CDN or edge platform. The audit kit in this article runs those checks on URLs you choose.

2. Do I need an llms.txt file?

   No. Google says Google Search does not use llms.txt files, including for its AI features, and needs no special AI files or markup. Other services may read one, and it is inexpensive to maintain as a curated map of a site. Keep the sitemap, semantic HTML, internal links, and crawler policy accurate first.

3. Does structured data improve AI visibility?

   Structured data gives machines explicit information about a page and its entities. It improves consistency and can make a page eligible for search features. It does not guarantee rankings or citations. The visible page, structured data, and underlying evidence should describe the same reality.

Related reading

• Your CMS is a GEO decision
• Structuring content for LLM retrieval
• GEO / AEO glossary entry
• llms.txt glossary entry
• CMS replatforming services