Re: No hits when search devel documentation

Hi Magnus,

On Tue, Jul 5, 2016 at 8:57 PM, Magnus Hagander <magnus@hagander.net> wrote:
> On Tue, Jun 21, 2016 at 10:10 AM, Amit Langote <amitlangote09@gmail.com>
> wrote:
>>
>> Hi,
>>
>> This is my first message to -www.
>>
>> When I try to search using the "Search Documentation" form on the
>> website, I get no hits from
>> https://www.postgresql.org/docs/devel/static/
>>
>> The same is not true for, say https://www.postgresql.org/docs/9.6/static/
>>
>> Is that intentional (keeping devel stuff off of searched documents, that
>> is)?
>
>
> That looks like an unintentional side-effect of
> https://git.postgresql.org/gitweb/?p=pgweb.git;a=commit;h=4920f460aab95063cdbf26d96e23a7a24a5e5e36
>
> Today, the indexing for the main website is driven off the same sitemap.xml
> that we feed Google. And we did at some point in the past (quite long ago)
> decide we didn't want /devel/ to be included in the Google search hits.

I see, thanks for explaining.

> We probably want to include them in our own, at least when you are
> explicitly searching in that subsection. That will require some code changes
> to make it happy with that though - we'd have to send out a separate
> sitemap.xml to our own search engine.

It's just that I was used to getting hits from devel using the site's
search form that suddenly stopped, but I see why now.  Might be better
to fix that if was unintentional anyway.

> Or is there a change in the consensus that we should not have Google return
> search hits of that?

It might be ok to have Google return pages from current or older.

Thanks,
Amit