robots.txt: disallow GPT scraper
This commit is contained in:
parent
26a7e87087
commit
1d924073c4
|
@ -3,3 +3,6 @@ Crawl-delay: 1
|
|||
Disallow: /search
|
||||
Disallow: /page/
|
||||
Disallow: /comments/page/
|
||||
|
||||
User-agent: GPTBot
|
||||
Disallow: /
|
||||
|
|
Loading…
Reference in New Issue