tildeverse
/
tilde.news
mirror of https://github.com/tildeverse/lobsters
5
3
Fork 3
Code Issues 2 Releases Wiki Activity
tilde.news/public/robots.txt

9 lines
122 B
Plaintext
Raw Permalink Blame History

User-agent: *
Crawl-delay: 1
Disallow: /search
Disallow: /page/
Disallow: /comments/page/
User-agent: GPTBot
Disallow: /
Reference in New Issue View Git Blame Copy Permalink