Many tilde.club members have placed robots.txt files in their
~/public_html/ directories, unaware that they have no effect, because
crawlers consult robots.txt only at domain root[1]. Therefore, I have
written a script for the consoliation of the individual users's robots.txt files into a unified server-level robots.txt.
This script is now installed and reguarly executed on the tilde.club
machine, and everyone is welcome to verify whether their personal
robots.txt is properly incorporated:
<
http://tilde.club/robots.txt>
My latest version of the script is always available at:
<
http://tilde.club/~ant/pub/botcons>
Bug requests and feature reports are welcome in the form of diff
patches, English prose (or verse, for that matter) to my e-mail
(ant [at] tilde.club) or in the #club channel on Newnet IRC[2].
1. <
https://www.rfc-editor.org/rfc/rfc9309.html#section-2.3>:
The rules MUST be accessible in a file named "/robots.txt" (all
lowercase) in the top-level path of the service.
2. <
https://newnet.net/>
--- Synchronet 3.20a-Linux NewsLink 1.2