mirror of https://github.com/tildeclub/site
Add robots.txt
This commit adds robots.txt entries to address web crawlers' accesses to user xwindows' web space. This patch is provided to Tilde.club under the terms of GNU General Public License, any version published by the Free Software Foundation.
This commit is contained in:
parent
f818c4551a
commit
3948a9b8c4
|
@ -0,0 +1,91 @@
|
|||
User-Agent: MojeekBot
|
||||
Allow: /~xwindows/
|
||||
|
||||
User-Agent: Qwantify
|
||||
Allow: /~xwindows/
|
||||
|
||||
User-Agent: Wibybot
|
||||
Allow: /~xwindows/
|
||||
|
||||
User-Agent: search.marginalia.nu
|
||||
Allow: /~xwindows/
|
||||
|
||||
User-Agent: SearchMySiteBot
|
||||
Allow: /~xwindows/
|
||||
|
||||
User-Agent: Duckduckbot
|
||||
Allow: /~xwindows/
|
||||
|
||||
User-Agent: ia_archiver
|
||||
Allow: /~xwindows/
|
||||
|
||||
User-Agent: Googlebot
|
||||
Allow: /~xwindows/$
|
||||
Allow: /~xwindows/index.html$
|
||||
Disallow: /~xwindows/
|
||||
|
||||
User-Agent: bingbot
|
||||
Allow: /~xwindows/$
|
||||
Allow: /~xwindows/index.html$
|
||||
Disallow: /~xwindows/
|
||||
|
||||
User-Agent: YandexBot
|
||||
Allow: /~xwindows/$
|
||||
Allow: /~xwindows/index.html$
|
||||
Disallow: /~xwindows/
|
||||
|
||||
User-Agent: YandexFavicons
|
||||
Disallow: /~xwindows/
|
||||
|
||||
User-Agent: MegaIndex.ru
|
||||
Disallow: /~xwindows/
|
||||
|
||||
User-Agent: Amazonbot
|
||||
Disallow: /~xwindows/
|
||||
|
||||
User-Agent: Linespider
|
||||
Disallow: /~xwindows/
|
||||
|
||||
User-Agent: Bytespider
|
||||
Disallow: /~xwindows/
|
||||
|
||||
User-Agent: CCBot
|
||||
Disallow: /~xwindows/
|
||||
|
||||
User-Agent: Neevabot
|
||||
Disallow: /~xwindows/
|
||||
|
||||
User-Agent: PetalBot
|
||||
Disallow: /~xwindows/
|
||||
|
||||
User-Agent: SemrushBot
|
||||
Disallow: /~xwindows/
|
||||
|
||||
User-Agent: AhrefsBot
|
||||
Disallow: /~xwindows/
|
||||
|
||||
User-Agent: DataForSeoBot
|
||||
Disallow: /~xwindows/
|
||||
|
||||
User-Agent: dotbot
|
||||
Disallow: /~xwindows/
|
||||
|
||||
User-Agent: Barkrowler
|
||||
Disallow: /~xwindows/
|
||||
|
||||
User-Agent: MJ12bot
|
||||
Disallow: /~xwindows/
|
||||
|
||||
User-Agent: BuiltWith
|
||||
Disallow: /~xwindows/
|
||||
|
||||
User-Agent: webprosbot
|
||||
Disallow: /~xwindows/
|
||||
|
||||
User-Agent: Dataprovider
|
||||
Disallow: /~xwindows/
|
||||
|
||||
User-Agent: *
|
||||
Allow: /~xwindows/$
|
||||
Allow: /~xwindows/index.html$
|
||||
Disallow: /~xwindows/
|
Loading…
Reference in New Issue