31 lines
1.0 KiB
Plaintext
31 lines
1.0 KiB
Plaintext
# Robots.txt borrowed from videolan's
|
|
# This file needs some attention
|
|
|
|
# Do not crawl CVS and .svn directories
|
|
User-agent: *
|
|
Disallow: /
|
|
|
|
# "This robot collects content from the Internet for the sole purpose of
|
|
# helping educational institutions prevent plagiarism. [...] we compare
|
|
# student papers against the content we find on the Internet to see if we
|
|
# can find similarities." (http://www.turnitin.com/robot/crawlerinfo.html)
|
|
# --> fuck off.
|
|
User-Agent: TurnitinBot
|
|
Disallow: /
|
|
|
|
# "NameProtect engages in crawling activity in search of a wide range of
|
|
# brand and other intellectual property violations that may be of interest
|
|
# to our clients." (http://www.nameprotect.com/botinfo.html)
|
|
# --> fuck off.
|
|
User-Agent: NPBot
|
|
Disallow: /
|
|
|
|
# "iThenticate® is a new service we have developed to combat the piracy
|
|
# of intellectual property and ensure the originality of written work for#
|
|
# publishers, non-profit agencies, corporations, and newspapers."
|
|
# (http://www.slysearch.com/)
|
|
# --> fuck off.
|
|
User-Agent: SlySearch
|
|
Disallow: /
|
|
|