# See https://www.robotstxt.org/robotstxt.html for what this file can say.
#
# Everything the archive vouches for is listed in the sitemap at the bottom.
# Nothing below is private: what is closed off is only pointless to crawl, being
# either the same reports under a different sort or an endpoint that is not a
# page at all.
#
# All crawlers are welcome, the ones answering questions on someone else's site
# included: an archive is worth more cited than hidden.

User-agent: *
Allow: /

# The same preference, in the form a machine can act on rather than read.
# See https://contentsignals.org/.
Content-Signal: ai-train=yes, search=yes, ai-input=yes

# The search box's own output. Thin, endless, and never linked to from a page.
Disallow: /*?q=
Disallow: /*&q=

# Stacked facets, which only a crawler builds: ten players crossed with an
# event is not a page anyone asked for, and the markup's noindex cannot stop
# the fetch that discovers it. A comma appears in no other address here, so
# this closes the combinatorial space without touching a single real page.
Disallow: /*,

# Not pages: the health check the platform polls, where the sign-up form posts,
# and the links the morning email carries, each addressed to one reader.
# Following one changes nothing, since leaving the roll takes a POST no crawler
# makes, but a page written for a single address is worth nothing to anyone else.
Disallow: /up
Disallow: /subscribers
Disallow: /unsubscribe/

# A sort variant used to be blocked here too. It should not have been: it already
# answers a canonical pointing at the unsorted page, and blocking a URL means the
# canonical is never fetched and so never read, which leaves the address eligible
# to be indexed on its link alone with nothing to say where it belongs. The rule
# in the markup does the job the rule in this file was preventing.
#
# Filtered views (/factions/necrons?format=singles and the like) are deliberately
# left open for the same reason. A shallow stack is a real page and says
# index,follow; a stacked, multi-valued or player-crossed one says noindex,follow
# instead. Either way the instruction lives in the markup, because a URL blocked
# here is never fetched, and an instruction that is never read is not one.

# The archive written out for a reader without a screen: what it holds, the words
# it uses, and every page worth knowing about.
# https://librariummilitaris.com/llms.txt
# https://librariummilitaris.com/llms-full.txt

Sitemap: https://librariummilitaris.com/sitemap.xml