tildeverse
/
tilde.news
mirror of https://github.com/tildeverse/lobsters
5
3
Fork 3
Code Issues 2 Releases Wiki Activity

robots.txt: disallow GPT scraper

This commit is contained in:
Peter Bhat Harkins 2023-08-12 08:44:21 -05:00
parent 26a7e87087
commit 1d924073c4
1 changed files with 3 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

3
public/robots.txt
View File

@ -3,3 +3,6 @@ Crawl-delay: 1
Disallow: /search
Disallow: /page/
Disallow: /comments/page/
User-agent: GPTBot
Disallow: /