tilde.news/public/robots.txt

mirror of https://github.com/tildeverse/lobsters
User-agent: *
Crawl-delay: 1
Disallow: /search
Disallow: /page/
Disallow: /comments/page/

User-agent: GPTBot
Disallow: /
block robots from /search 2015-01-11 14:19:10 +00:00			`User-agent: *`
ask polite bots to slow down 2020-11-29 16:24:57 +00:00			`Crawl-delay: 1`
block robots from /search 2015-01-11 14:19:10 +00:00			`Disallow: /search`
robots: tell spiders not to crawl pages 2017-06-05 14:51:29 +00:00			`Disallow: /page/`
			`Disallow: /comments/page/`
robots.txt: disallow GPT scraper 2023-08-12 13:44:21 +00:00
			`User-agent: GPTBot`
			`Disallow: /`