Сайт http://www.robotstxt.org/robotstxt.html посвящен robots.txt На нем можно найти There are two important considerations when using /robots.txt: robots can ignore your /robots.txt. Especially malware robots that scan the web for security vulnerabilities, and email address harvesters used by spammers will pay no attention. the /robots.txt file is a publicly available file. Anyone can see what sections of your server you don’t want robots to use. Так же на странице http://www.robotstxt.org/faq/ignore.html можно найти ответ на вопрос почему робот игнорирует robots.txt
Дискуссии пользователя
Сайт http://www.robotstxt.org/robotstxt.html посвящен robots.txt На нем можно найти There are two important considerations when using /robots.txt: robots can ignore your /robots.txt. Especially malware robots that scan the web for security vulnerabilities, and email address harvesters used by spammers will pay no attention. the /robots.txt file is a publicly available file. Anyone can see what sections of your server you don’t want robots to use. Так же на странице http://www.robotstxt.org/faq/ignore.html можно найти ответ на вопрос почему робот игнорирует robots.txt