robots.txt sometimes disallowing all?
От | Josh Kupershmidt |
---|---|
Тема | robots.txt sometimes disallowing all? |
Дата | |
Msg-id | CAK3UJRGeUk+hAiaMVLnWT4echkwzL=-17CpCjhpwNycT0r=7Yg@mail.gmail.com обсуждение исходный текст |
Ответы |
Re: robots.txt sometimes disallowing all?
|
Список | pgsql-www |
I noticed an unusual search result shown as the top result by Google (search query "POSTGRESQL DROP TRIGGER", first result for me leads to www.postgresql.org/docs/8.3/static/sql-droptrigger.html ). The title of the result is somehow "英語 - PostgreSQL", and below that title reads: "A description for this result is not available because of this site's robots.txt – learn more." Sure enough, when I checked http://www.postgresql.org/robots.txt in Chrome on OS X, I see: User-agent: * Disallow: / though when I check in other browsers (Safari, wget), I see a more reasonable robots.txt: === User-agent: * Disallow: /admin/ Disallow: /account/ Disallow: /docs/devel/ Disallow: /list/ Disallow: /search/ Disallow: /message-id/raw/ Disallow: /message-id/flat/ Sitemap: http://www.postgresql.org/sitemap.xml === Is it intentional that we're serving up that first robots.txt to (apparently) Googlebot and Chrome? Josh
В списке pgsql-www по дате отправления: