mirror of
https://github.com/tildeverse/lobsters
synced 2024-06-22 16:27:05 +00:00
9 lines
122 B
Plaintext
9 lines
122 B
Plaintext
User-agent: *
|
|
Crawl-delay: 1
|
|
Disallow: /search
|
|
Disallow: /page/
|
|
Disallow: /comments/page/
|
|
|
|
User-agent: GPTBot
|
|
Disallow: /
|