robots.txt: disallow GPT scraper
This commit is contained in:
parent
26a7e87087
commit
1d924073c4
|
@ -3,3 +3,6 @@ Crawl-delay: 1
|
||||||
Disallow: /search
|
Disallow: /search
|
||||||
Disallow: /page/
|
Disallow: /page/
|
||||||
Disallow: /comments/page/
|
Disallow: /comments/page/
|
||||||
|
|
||||||
|
User-agent: GPTBot
|
||||||
|
Disallow: /
|
||||||
|
|
Loading…
Reference in New Issue