robots.txt

Every website should have a robots.txt file.  Some bots hit sites so often that they slow down performance, other bots simply aren't desirable.  robots.txt files can also be used to communicate sitemap location and limit request rate.  It's important that the correct robots.txt file is served on development servers though, and that file is usually much different than your production robots.txt file.  Here's a quick .htaccess snippet you can use to make that happen:

每个网站都应该有一个robots.txt文件。 有些漫游器经常访问网站，从而降低性能，而其他漫游器则根本不受欢迎。 robots.txt文件也可以用于传达站点地图位置和限制请求率。 不过，请务必在开发服务器上提供正确的robots.txt文件，并且该文件通常与您的生产robots.txt文件有很大不同。 这是一个快速的.htaccess代码段，您可以使用它来实现此目的：

RewriteCond %{HTTP_HOST} devdomainRewriteRule ^robots.txt$ robots-go-away.txt [L]

The robots-go-away.txt file most likely directs robots not to index anything, unless you want your dev server to be indexed for some reason (hint:  you really don't want this).

robots-go-away.txt文件很可能会指示机器人不对任何内容进行索引，除非您出于某种原因希望对开发服务器进行索引(提示：您确实不希望这样做)。

翻译自:

robots.txt