Static implementation guide
Robots Boundaries for Test Origins
Separate crawler policy by hostname so the gateway is discoverable while the directly exposed origin remains blocked.
Last updated 2026-08-24
Apply policy at the host root
Robots rules belong to the hostname serving them. The public gateway publishes a root robots file that allows crawling and points to its sitemap. The origin publishes a separate root file that disallows every path. Keeping those files outside the rewritten feature prefix makes their ownership unambiguous and allows each hostname to state its own role.
A global noindex response header on the origin is intentionally avoided. Response headers can survive a proxy or rewrite, which could accidentally attach noindex to the public gateway response. The root robots boundary and public canonical provide a safer test arrangement for an origin that must remain directly reachable.
Inspect both hosts after deployment
Fetch each robots file directly after production deployment and record the response. Confirm the gateway file allows the site and contains only the gateway sitemap URL. Confirm the origin file contains a complete disallow rule. A preview deployment or authentication page is not an acceptable substitute because it changes the rewrite behavior being tested.
Robots directives are crawl guidance rather than access control. Do not place sensitive content at the origin, and do not rely on the file to secure it. This project contains only public test material and uses the rule to prevent an alternate hostname from competing with the chosen canonical.