tildeverse
/
tilde.news
mirror of https://github.com/tildeverse/lobsters
5
3
Fork 3
Code Issues 2 Releases Wiki Activity
tilde.news/public/robots.txt

6 lines
90 B
Plaintext
Raw Blame History

User-agent: *
Crawl-delay: 1
Disallow: /search
Disallow: /page/
Disallow: /comments/page/
Reference in New Issue View Git Blame Copy Permalink