Peter Bhat Harkins
|
1d924073c4
|
robots.txt: disallow GPT scraper
|
2023-08-24 02:53:07 -05:00 |
|
Peter Bhat Harkins
|
13072e05ae
|
ask polite bots to slow down
|
2020-11-29 10:24:57 -06:00 |
joshua stein
|
51f03832f2
|
robots: tell spiders not to crawl pages
|
2017-06-05 11:18:43 -05:00 |
|
joshua stein
|
148a63fd61
|
let spiders crawl all over us
|
2017-01-05 15:44:58 -06:00 |
|
joshua stein
|
ce8eb45008
|
block robots from /search
|
2015-01-11 12:59:41 -06:00 |
|
joshua stein
|
b1e9e359a3
|
block all spiders by default, allow major ones
|
2013-06-30 00:54:02 -05:00 |
|
joshua stein
|
dc71bbbd98
|
block archivers
|
2012-10-24 11:53:43 -05:00 |
|
joshua stein
|
aef2ccefba
|
empty robots for now, let everyone in
|
2012-08-24 11:42:23 -05:00 |
|
joshua stein
|
093747b796
|
initial work on conversion from php tree
|
2012-06-16 20:15:46 -05:00 |