Re: no mailing list hits in google

Hi,

On 2019-08-28 10:26:35 -0700, Andres Freund wrote:
> On August 28, 2019 9:22:44 AM PDT, Merlin Moncure <mmoncure@gmail.com> wrote:
> >Hackers,
> >[apologies if this is the incorrect list or is already discussed
> >material]
> 
> Probably should be on the -www list. Redirecting. Please trim in future replies.
> 
> >I've noticed that mailing list discussions in -hackers and other
> >mailing lists appear to not be indexed by google -- at all.
> 
> I noticed that there's fewer and fewer hits too. Pretty annoying. I have an online archive I can search, but that's
notsomething everyone should have to do.
 
> 
> I think it's because robots.txt tells search engines to ignore the lists. Quite hard to understand how that's a good
idea.
> 
> https://www.postgresql.org/robots.txt
> 
> User-agent: *
> Disallow: /admin/
> Disallow: /account/
> Disallow: /docs/devel/
> Disallow: /list/
> Disallow: /search/
> Disallow: /message-id/raw/
> Disallow: /message-id/flat/
> 
> Sitemap: https://www.postgresql.org/sitemap.xml 
> 
> Without /list, there's no links to the individual messages. So there needs to be another external reference for a
searchengine to arrive at individual messages.
 
> 
> Andres
> -- 
> Sent from my Android device with K-9 Mail. Please excuse my brevity.

For reasons that I do not understand, the previous mail had a broken
html part, making the above message invisible for people viewing the
html part.

Greetings,

Andres Freund