Обсуждение: Fragments in tsearch2 headline

Hi,

(I first posted it via google groups and realised that I have to be
subscribed; now posting directly)

I searched the list but couldn't find  anyone raising the issue (or it
might simply be my way of using the tool).

I'd like to search through some text documents for words and generate
headlines. The search works fine but, if the words are far apart in
the document, the headline only highlights one of the words. Enlarging
the headline  with Min/MaxWords is not an option as I have limited
space for displaying it.

Is there an easy way to generate a headline from separate fragments
containing the search words and maybe separated by "..."? An option is
to generate separate headlines and concatenate them before displaying
(with a problem when the words are in the same fragment). Another
option would be to somehow get the found words position in the
document (anyone knows how?) and generate the headline myself.

Any other ideas?

Thanks.

--
Catalin

"Catalin Marinas" <catalin.marinas@gmail.com> writes:
> Is there an easy way to generate a headline from separate fragments
> containing the search words and maybe separated by "..."?

Hmm, the documentation for ts_headline claims it does this already:

    <function>ts_headline</function> accepts a document along
    with a query, and returns one or more ellipsis-separated excerpts from
    the document in which terms from the query are highlighted.

However, a quick look at the code suggests this is a lie --- I see no
evidence whatever that there's any smarts for putting in ellipses.

Oleg, Teodor, is there something I missed here?  Or do we need to change
the documentation?

            regards, tom lane

On Sat, 27 Oct 2007, Tom Lane wrote:

> "Catalin Marinas" <catalin.marinas@gmail.com> writes:
>> Is there an easy way to generate a headline from separate fragments
>> containing the search words and maybe separated by "..."?
>
> Hmm, the documentation for ts_headline claims it does this already:
>
>    <function>ts_headline</function> accepts a document along
>    with a query, and returns one or more ellipsis-separated excerpts from
>    the document in which terms from the query are highlighted.
>
> However, a quick look at the code suggests this is a lie --- I see no
> evidence whatever that there's any smarts for putting in ellipses.
>
> Oleg, Teodor, is there something I missed here?  Or do we need to change
> the documentation?

Probably documentation is not correct here. 'ellipsis-separated' should be
treated as a general wording. Default highlighting is <b>..</b> as it
stated below in docs.

postgres=# select ts_headline('this is a highlighted text','highlight'::tsquery, 'StartSel=...,StopSel=...')
postgres-# ;
            ts_headline
----------------------------------
  this is a ...highlighted... text


     Regards,
         Oleg
_____________________________________________________________
Oleg Bartunov, Research Scientist, Head of AstroNet (www.astronet.ru),
Sternberg Astronomical Institute, Moscow University, Russia
Internet: oleg@sai.msu.su, http://www.sai.msu.su/~megera/
phone: +007(495)939-16-83, +007(495)939-23-83

On 28/10/2007, Oleg Bartunov <oleg@sai.msu.su> wrote:
> On Sat, 27 Oct 2007, Tom Lane wrote:
>
> > "Catalin Marinas" <catalin.marinas@gmail.com> writes:
> >> Is there an easy way to generate a headline from separate fragments
> >> containing the search words and maybe separated by "..."?
> >
> > Hmm, the documentation for ts_headline claims it does this already:
[...]
> > However, a quick look at the code suggests this is a lie --- I see no
> > evidence whatever that there's any smarts for putting in ellipses.
>
> Probably documentation is not correct here. 'ellipsis-separated' should be
> treated as a general wording. Default highlighting is <b>..</b> as it
> stated below in docs.

It seems that I'll have to implement the headline outside the query
(Python, in my case). I would use to_tsvector and to_tsquery to
generate the lexemes and the work position, add them to a hash table
and use the position of the matching lexemes to generate the headline.

I could also highlight the full text and generate the headline I want
based on it but if I limit the number of excerpts, it gets complicated
to avoid the same lexeme being shown in all excerpts. Is a lexeme
always a substring of the corresponding token (so that I can use
simple regexp)?

Any other ideas?

Thanks.

--
Catalin

Catalin,

what is your need ? What's wrong with this ?

postgres=# select ts_headline('1 2 3 4 5 3 4 abc abc 2 3 xyz','2'::tsquery, 'StartSel=...,StopSel=...')
;
                 ts_headline
-------------------------------------------
  1 ...2... 3 4 5 3 4 abc abc ...2... 3 xyz


Oleg
On Tue, 30 Oct 2007, Catalin Marinas wrote:

> On 28/10/2007, Oleg Bartunov <oleg@sai.msu.su> wrote:
>> On Sat, 27 Oct 2007, Tom Lane wrote:
>>
>>> "Catalin Marinas" <catalin.marinas@gmail.com> writes:
>>>> Is there an easy way to generate a headline from separate fragments
>>>> containing the search words and maybe separated by "..."?
>>>
>>> Hmm, the documentation for ts_headline claims it does this already:
> [...]
>>> However, a quick look at the code suggests this is a lie --- I see no
>>> evidence whatever that there's any smarts for putting in ellipses.
>>
>> Probably documentation is not correct here. 'ellipsis-separated' should be
>> treated as a general wording. Default highlighting is <b>..</b> as it
>> stated below in docs.
>
> It seems that I'll have to implement the headline outside the query
> (Python, in my case). I would use to_tsvector and to_tsquery to
> generate the lexemes and the work position, add them to a hash table
> and use the position of the matching lexemes to generate the headline.
>
> I could also highlight the full text and generate the headline I want
> based on it but if I limit the number of excerpts, it gets complicated
> to avoid the same lexeme being shown in all excerpts. Is a lexeme
> always a substring of the corresponding token (so that I can use
> simple regexp)?
>
> Any other ideas?
>
> Thanks.
>
>

     Regards,
         Oleg
_____________________________________________________________
Oleg Bartunov, Research Scientist, Head of AstroNet (www.astronet.ru),
Sternberg Astronomical Institute, Moscow University, Russia
Internet: oleg@sai.msu.su, http://www.sai.msu.su/~megera/
phone: +007(495)939-16-83, +007(495)939-23-83

Oleg Bartunov wrote:
> Catalin,
>
> what is your need ? What's wrong with this ?
>
> postgres=# select ts_headline('1 2 3 4 5 3 4 abc abc 2 3
> xyz','2'::tsquery, 'StartSel=...,StopSel=...')
> ;
>                 ts_headline
> -------------------------------------------
>  1 ...2... 3 4 5 3 4 abc abc ...2... 3 xyz

I think he want's something like: "1 2 3 ... abc 2 3 ..."

A few characters of context around each match and then ... between. Kind
of like grep -C.

--
   Richard Huxton
   Archonet Ltd

On 30/10/2007, Richard Huxton <dev@archonet.com> wrote:
> Oleg Bartunov wrote:
> > Catalin,
> >
> > what is your need ? What's wrong with this ?
> >
> > postgres=# select ts_headline('1 2 3 4 5 3 4 abc abc 2 3
> > xyz','2'::tsquery, 'StartSel=...,StopSel=...')
> > ;
> >                 ts_headline
> > -------------------------------------------
> >  1 ...2... 3 4 5 3 4 abc abc ...2... 3 xyz
>
> I think he want's something like: "1 2 3 ... abc 2 3 ..."
>
> A few characters of context around each match and then ... between. Kind
> of like grep -C.

That's pretty much correct (with the difference that I'd like context
of words rather than lines as in "grep" and StartSel=<b>,
StopSel=</b>).

Since the text I want a headline for might be pretty long (tens of
lines), I'd like to only show the excerpts around the matching words.
Similar to the above example:

select ts_headline('1 2 3 4 5 3 4 abc x y z 2 3', '2 & abc'::tsquery);

should give:

'1 <b>2</b> 3 4 ... 3 4 <b>abc</b> x y'

Currently, if you limit the maximum words so that 'abc' is too far, it
only highlights the first match.

Many of the search engines (including google) show the headline this
way. I think Lucene can do this as well but I've never used it to be
sure.

--
Catalin

On Tue, 30 Oct 2007, Catalin Marinas wrote:

> On 30/10/2007, Richard Huxton <dev@archonet.com> wrote:
>> Oleg Bartunov wrote:
>>> Catalin,
>>>
>>> what is your need ? What's wrong with this ?
>>>
>>> postgres=# select ts_headline('1 2 3 4 5 3 4 abc abc 2 3
>>> xyz','2'::tsquery, 'StartSel=...,StopSel=...')
>>> ;
>>>                 ts_headline
>>> -------------------------------------------
>>>  1 ...2... 3 4 5 3 4 abc abc ...2... 3 xyz
>>
>> I think he want's something like: "1 2 3 ... abc 2 3 ..."
>>
>> A few characters of context around each match and then ... between. Kind
>> of like grep -C.
>
> That's pretty much correct (with the difference that I'd like context
> of words rather than lines as in "grep" and StartSel=<b>,
> StopSel=</b>).
>
> Since the text I want a headline for might be pretty long (tens of
> lines), I'd like to only show the excerpts around the matching words.
> Similar to the above example:
>
> select ts_headline('1 2 3 4 5 3 4 abc x y z 2 3', '2 & abc'::tsquery);
>
> should give:
>
> '1 <b>2</b> 3 4 ... 3 4 <b>abc</b> x y'
>
> Currently, if you limit the maximum words so that 'abc' is too far, it
> only highlights the first match.

ok, then you have to formalize many things - how long should be excerpts,
how much excerpts to show, etc. In tsearch2 we have get_covers() function,
which produces all excerpts like:

=# select get_covers(to_tsvector('1 2 3 4 5 3 4 abc x y z 2 3'), '2&3'::tsquery);
                    get_covers
------------------------------------------------
  1 {1 2 3 }1 4 5 {2 3 4 abc x y z {3 2 }2 3 }3
(1 row)

Once you formalize your requirements, you can look on it and adapt to your
needs (and share with people). I think it could be nice contrib module.


>
> Many of the search engines (including google) show the headline this
> way. I think Lucene can do this as well but I've never used it to be
> sure.
>
>

     Regards,
         Oleg
_____________________________________________________________
Oleg Bartunov, Research Scientist, Head of AstroNet (www.astronet.ru),
Sternberg Astronomical Institute, Moscow University, Russia
Internet: oleg@sai.msu.su, http://www.sai.msu.su/~megera/
phone: +007(495)939-16-83, +007(495)939-23-83

This is a nice idea and seems easy to implement. I will try to write
it down and send a patch to the mailing list.

I was also working to add support for phrase search. Currently to
check for phrase you have to match the entire document. It will be
better if a filter like are_words_consecutive(tsvector *t, tsquery *q)
can be added to reduce the number of matching documents before we
actually do the phrase search. Do you think this will improve the
performance of phrase search?  If so I will like to write this
function and send a patch.

-Sushant.

On 10/30/07, Oleg Bartunov <oleg@sai.msu.su> wrote:
> On Tue, 30 Oct 2007, Catalin Marinas wrote:
>
> > On 30/10/2007, Richard Huxton <dev@archonet.com> wrote:
> >> Oleg Bartunov wrote:
> >>> Catalin,
> >>>
> >>> what is your need ? What's wrong with this ?
> >>>
> >>> postgres=# select ts_headline('1 2 3 4 5 3 4 abc abc 2 3
> >>> xyz','2'::tsquery, 'StartSel=...,StopSel=...')
> >>> ;
> >>>                 ts_headline
> >>> -------------------------------------------
> >>>  1 ...2... 3 4 5 3 4 abc abc ...2... 3 xyz
> >>
> >> I think he want's something like: "1 2 3 ... abc 2 3 ..."
> >>
> >> A few characters of context around each match and then ... between. Kind
> >> of like grep -C.
> >
> > That's pretty much correct (with the difference that I'd like context
> > of words rather than lines as in "grep" and StartSel=<b>,
> > StopSel=</b>).
> >
> > Since the text I want a headline for might be pretty long (tens of
> > lines), I'd like to only show the excerpts around the matching words.
> > Similar to the above example:
> >
> > select ts_headline('1 2 3 4 5 3 4 abc x y z 2 3', '2 & abc'::tsquery);
> >
> > should give:
> >
> > '1 <b>2</b> 3 4 ... 3 4 <b>abc</b> x y'
> >
> > Currently, if you limit the maximum words so that 'abc' is too far, it
> > only highlights the first match.
>
> ok, then you have to formalize many things - how long should be excerpts,
> how much excerpts to show, etc. In tsearch2 we have get_covers() function,
> which produces all excerpts like:
>
> =# select get_covers(to_tsvector('1 2 3 4 5 3 4 abc x y z 2 3'),
> '2&3'::tsquery);
>                     get_covers
> ------------------------------------------------
>   1 {1 2 3 }1 4 5 {2 3 4 abc x y z {3 2 }2 3 }3
> (1 row)
>
> Once you formalize your requirements, you can look on it and adapt to your
> needs (and share with people). I think it could be nice contrib module.
>
>
> >
> > Many of the search engines (including google) show the headline this
> > way. I think Lucene can do this as well but I've never used it to be
> > sure.
> >
> >
>
>      Regards,
>          Oleg
> _____________________________________________________________
> Oleg Bartunov, Research Scientist, Head of AstroNet (www.astronet.ru),
> Sternberg Astronomical Institute, Moscow University, Russia
> Internet: oleg@sai.msu.su, http://www.sai.msu.su/~megera/
> phone: +007(495)939-16-83, +007(495)939-23-83
>
> ---------------------------(end of broadcast)---------------------------
> TIP 6: explain analyze is your friend
>

> On 10/30/07, Oleg Bartunov <oleg@sai.msu.su> wrote:
>> ... In tsearch2 we have get_covers() function,
>> which produces all excerpts like:

I had not realized till just now that the 8.3 core version of tsearch
omitted any material feature of contrib/tsearch2.  Why was get_covers()
left out?

            regards, tom lane

On Tue, 30 Oct 2007, Tom Lane wrote:

>> On 10/30/07, Oleg Bartunov <oleg@sai.msu.su> wrote:
>>> ... In tsearch2 we have get_covers() function,
>>> which produces all excerpts like:
>
> I had not realized till just now that the 8.3 core version of tsearch
> omitted any material feature of contrib/tsearch2.  Why was get_covers()
> left out?

That time we considered it as developers function useful only for debugging.


     Regards,
         Oleg
_____________________________________________________________
Oleg Bartunov, Research Scientist, Head of AstroNet (www.astronet.ru),
Sternberg Astronomical Institute, Moscow University, Russia
Internet: oleg@sai.msu.su, http://www.sai.msu.su/~megera/
phone: +007(495)939-16-83, +007(495)939-23-83

On 30/10/2007, Oleg Bartunov <oleg@sai.msu.su> wrote:
> ok, then you have to formalize many things - how long should be excerpts,
> how much excerpts to show, etc. In tsearch2 we have get_covers() function,
> which produces all excerpts like:
>
> =# select get_covers(to_tsvector('1 2 3 4 5 3 4 abc x y z 2 3'), '2&3'::tsquery);
>                     get_covers
> ------------------------------------------------
>   1 {1 2 3 }1 4 5 {2 3 4 abc x y z {3 2 }2 3 }3
> (1 row)

This function generates the lexemes, so cannot be used directly, but
it is probably a good starting point.

> Once you formalize your requirements, you can look on it and adapt to your
> needs (and share with people). I think it could be nice contrib module.

It seems that Sushant already wants to implement this function. He
would probably be faster than me :-) (I'm relatively new to db stuff).
Since I mainly rely on whatever a web hosting company provides, I'll
probably stick with a Python implementation outside the SQL query.

Thanks for your answers.

--
Catalin

I wrote a headline generation function for my app and I have attached
the patch (against the cvs head). It generates multiple contexts in
which the query appears. Essentially, it uses the cover function to
generate all covers, chooses smallest covers and stretches each
selected cover according to the chosen parameters. I think ideally
changes should be made to prsd_headline function but I couldn't
understand that segment of code well.

The sql interface is

headline_with_fragments(text parser, tsvector docvector, text doc,
tsquery queryin, int4 maxcoverSize, int4 mincoverSize, int4 maxWords)
 RETURNS text

This will generate headline that contain maxWords and each cover
stretched to maxcoverSize. It will not add any fragment with less than
mincoverSize.
I am running my app with maxcoverSize = 20, mincoverSize = 5, maxWords = 40.
So it shows roughly two fragments per query.

If Teoder or Oleg want to add this to main branch, I will be happy to
clean it up and test it better.

-Sushant.




On Oct 31, 2007 6:26 PM, Catalin Marinas <catalin.marinas@gmail.com> wrote:
> On 30/10/2007, Oleg Bartunov <oleg@sai.msu.su> wrote:
> > ok, then you have to formalize many things - how long should be excerpts,
> > how much excerpts to show, etc. In tsearch2 we have get_covers() function,
> > which produces all excerpts like:
> >
> > =# select get_covers(to_tsvector('1 2 3 4 5 3 4 abc x y z 2 3'), '2&3'::tsquery);
> >                     get_covers
> > ------------------------------------------------
> >   1 {1 2 3 }1 4 5 {2 3 4 abc x y z {3 2 }2 3 }3
> > (1 row)
>
> This function generates the lexemes, so cannot be used directly, but
> it is probably a good starting point.
>
> > Once you formalize your requirements, you can look on it and adapt to your
> > needs (and share with people). I think it could be nice contrib module.
>
> It seems that Sushant already wants to implement this function. He
> would probably be faster than me :-) (I'm relatively new to db stuff).
> Since I mainly rely on whatever a web hosting company provides, I'll
> probably stick with a Python implementation outside the SQL query.
>
> Thanks for your answers.
>
> --
> Catalin
>
> ---------------------------(end of broadcast)---------------------------
>
> TIP 5: don't forget to increase your free space map settings
>

This has been saved for the 8.4 release:

    http://momjian.postgresql.org/cgi-bin/pgpatches_hold

---------------------------------------------------------------------------

Sushant Sinha wrote:
> I wrote a headline generation function for my app and I have attached
> the patch (against the cvs head). It generates multiple contexts in
> which the query appears. Essentially, it uses the cover function to
> generate all covers, chooses smallest covers and stretches each
> selected cover according to the chosen parameters. I think ideally
> changes should be made to prsd_headline function but I couldn't
> understand that segment of code well.
>
> The sql interface is
>
> headline_with_fragments(text parser, tsvector docvector, text doc,
> tsquery queryin, int4 maxcoverSize, int4 mincoverSize, int4 maxWords)
>  RETURNS text
>
> This will generate headline that contain maxWords and each cover
> stretched to maxcoverSize. It will not add any fragment with less than
> mincoverSize.
> I am running my app with maxcoverSize = 20, mincoverSize = 5, maxWords = 40.
> So it shows roughly two fragments per query.
>
> If Teoder or Oleg want to add this to main branch, I will be happy to
> clean it up and test it better.
>
> -Sushant.
>
>
>
>
> On Oct 31, 2007 6:26 PM, Catalin Marinas <catalin.marinas@gmail.com> wrote:
> > On 30/10/2007, Oleg Bartunov <oleg@sai.msu.su> wrote:
> > > ok, then you have to formalize many things - how long should be excerpts,
> > > how much excerpts to show, etc. In tsearch2 we have get_covers() function,
> > > which produces all excerpts like:
> > >
> > > =# select get_covers(to_tsvector('1 2 3 4 5 3 4 abc x y z 2 3'), '2&3'::tsquery);
> > >                     get_covers
> > > ------------------------------------------------
> > >   1 {1 2 3 }1 4 5 {2 3 4 abc x y z {3 2 }2 3 }3
> > > (1 row)
> >
> > This function generates the lexemes, so cannot be used directly, but
> > it is probably a good starting point.
> >
> > > Once you formalize your requirements, you can look on it and adapt to your
> > > needs (and share with people). I think it could be nice contrib module.
> >
> > It seems that Sushant already wants to implement this function. He
> > would probably be faster than me :-) (I'm relatively new to db stuff).
> > Since I mainly rely on whatever a web hosting company provides, I'll
> > probably stick with a Python implementation outside the SQL query.
> >
> > Thanks for your answers.
> >
> > --
> > Catalin
> >
> > ---------------------------(end of broadcast)---------------------------
> >
> > TIP 5: don't forget to increase your free space map settings
> >

[ Attachment, skipping... ]

>
> ---------------------------(end of broadcast)---------------------------
> TIP 6: explain analyze is your friend

--
  Bruce Momjian  <bruce@momjian.us>        http://momjian.us
  EnterpriseDB                             http://postgres.enterprisedb.com

  + If your life is a hard drive, Christ can be your backup. +

Teodor, Oleg, do we want this?

    http://archives.postgresql.org/pgsql-general/2007-11/msg00508.php

---------------------------------------------------------------------------

Sushant Sinha wrote:
> I wrote a headline generation function for my app and I have attached
> the patch (against the cvs head). It generates multiple contexts in
> which the query appears. Essentially, it uses the cover function to
> generate all covers, chooses smallest covers and stretches each
> selected cover according to the chosen parameters. I think ideally
> changes should be made to prsd_headline function but I couldn't
> understand that segment of code well.
>
> The sql interface is
>
> headline_with_fragments(text parser, tsvector docvector, text doc,
> tsquery queryin, int4 maxcoverSize, int4 mincoverSize, int4 maxWords)
>  RETURNS text
>
> This will generate headline that contain maxWords and each cover
> stretched to maxcoverSize. It will not add any fragment with less than
> mincoverSize.
> I am running my app with maxcoverSize = 20, mincoverSize = 5, maxWords = 40.
> So it shows roughly two fragments per query.
>
> If Teoder or Oleg want to add this to main branch, I will be happy to
> clean it up and test it better.
>
> -Sushant.
>
>
>
>
> On Oct 31, 2007 6:26 PM, Catalin Marinas <catalin.marinas@gmail.com> wrote:
> > On 30/10/2007, Oleg Bartunov <oleg@sai.msu.su> wrote:
> > > ok, then you have to formalize many things - how long should be excerpts,
> > > how much excerpts to show, etc. In tsearch2 we have get_covers() function,
> > > which produces all excerpts like:
> > >
> > > =# select get_covers(to_tsvector('1 2 3 4 5 3 4 abc x y z 2 3'), '2&3'::tsquery);
> > >                     get_covers
> > > ------------------------------------------------
> > >   1 {1 2 3 }1 4 5 {2 3 4 abc x y z {3 2 }2 3 }3
> > > (1 row)
> >
> > This function generates the lexemes, so cannot be used directly, but
> > it is probably a good starting point.
> >
> > > Once you formalize your requirements, you can look on it and adapt to your
> > > needs (and share with people). I think it could be nice contrib module.
> >
> > It seems that Sushant already wants to implement this function. He
> > would probably be faster than me :-) (I'm relatively new to db stuff).
> > Since I mainly rely on whatever a web hosting company provides, I'll
> > probably stick with a Python implementation outside the SQL query.
> >
> > Thanks for your answers.
> >
> > --
> > Catalin
> >
> > ---------------------------(end of broadcast)---------------------------
> >
> > TIP 5: don't forget to increase your free space map settings
> >

[ Attachment, skipping... ]

>
> ---------------------------(end of broadcast)---------------------------
> TIP 6: explain analyze is your friend

--
  Bruce Momjian  <bruce@momjian.us>        http://momjian.us
  EnterpriseDB                             http://postgres.enterprisedb.com

  + If your life is a hard drive, Christ can be your backup. +

> Teodor, Oleg, do we want this?
>     http://archives.postgresql.org/pgsql-general/2007-11/msg00508.php

I suppose, we want it. But there are a questions/issues:
- Is it needed to introduce new function? may be it will be better to add option
to existing headline function. I'd like to keep current layout: ts_headline
provides some common interface to headline generation. Finding and marking
fragments is deal of parser's headline method and generation of exact pieces of
text is made by ts_headline.
- Covers may be overlapped. So, overlapped fragments will be looked odd.


In any case, the patch was developed for contrib version of tsearch.
--
Teodor Sigaev                                   E-mail: teodor@sigaev.ru
                                                    WWW: http://www.sigaev.ru/

Ah I missed this email. I agree with Teodor that this is not the best
way to implement this functionality. At the time I was in a bit of hurry
to have something better than the default one and just hacked this. And
if we want to have this functionality across languages and parsers it
will be better to be implemented in the general framework.

The patch takes into account the corner case of overlap. Here is the
code for that
// start check
if (!startHL && *currentpos >= startpos)
       startHL = 1;

The headline generation will not start until currentpos has gone past
startpos.

You can also check how this headline function is working at my website
indiankanoon.com. Some example queries are murder, freedom of speech,
freedom of press etc.

Should I develop the patch for the current cvs head of postgres?

Thanks,
-Sushant.



On Mon, 2008-03-17 at 22:00 +0300, Teodor Sigaev wrote:
> > Teodor, Oleg, do we want this?
> >     http://archives.postgresql.org/pgsql-general/2007-11/msg00508.php
>
> I suppose, we want it. But there are a questions/issues:
> - Is it needed to introduce new function? may be it will be better to add option
> to existing headline function. I'd like to keep current layout: ts_headline
> provides some common interface to headline generation. Finding and marking
> fragments is deal of parser's headline method and generation of exact pieces of
> text is made by ts_headline.
> - Covers may be overlapped. So, overlapped fragments will be looked odd.
>
>
> In any case, the patch was developed for contrib version of tsearch.

> The patch takes into account the corner case of overlap. Here is the
> code for that
> // start check
> if (!startHL && *currentpos >= startpos)
>        startHL = 1;
>
> The headline generation will not start until currentpos has gone past
> startpos.
Ok

>
> You can also check how this headline function is working at my website
> indiankanoon.com. Some example queries are murder, freedom of speech,
> freedom of press etc.
Looks good.

> Should I develop the patch for the current cvs head of postgres?

I'd like to commit your patch, but if it should be:
  - for current HEAD
  - as extension of existing ts_headline.

--
Teodor Sigaev                                   E-mail: teodor@sigaev.ru
                                                    WWW: http://www.sigaev.ru/

Where are we on this?

---------------------------------------------------------------------------

Teodor Sigaev wrote:
> > The patch takes into account the corner case of overlap. Here is the
> > code for that
> > // start check
> > if (!startHL && *currentpos >= startpos)
> >        startHL = 1;
> >
> > The headline generation will not start until currentpos has gone past
> > startpos.
> Ok
>
> >
> > You can also check how this headline function is working at my website
> > indiankanoon.com. Some example queries are murder, freedom of speech,
> > freedom of press etc.
> Looks good.
>
> > Should I develop the patch for the current cvs head of postgres?
>
> I'd like to commit your patch, but if it should be:
>   - for current HEAD
>   - as extension of existing ts_headline.
>
> --
> Teodor Sigaev                                   E-mail: teodor@sigaev.ru
>                                                     WWW: http://www.sigaev.ru/
>
> --
> Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
> To make changes to your subscription:
> http://www.postgresql.org/mailpref/pgsql-general

--
  Bruce Momjian  <bruce@momjian.us>        http://momjian.us
  EnterpriseDB                             http://enterprisedb.com

  + If your life is a hard drive, Christ can be your backup. +

Hi,

I've ported the patch of Sushant Sinha for fragmented headlines to pg8.3.1
(http://archives.postgresql.org/pgsql-general/2007-11/msg00508.php)

W.r.t, http://archives.postgresql.org/pgsql-general/2008-03/msg00806.php
I can continue the work until this becomes an acceptable patch for pg.

Pierre-yves.

Thanks Pierre for porting this! I just tested this for my application
and it works. There was a small bug in that startHL has to be
initialized to 0 for each chosen cover. I fixed that and attached the
new patch.

Teodor did not want a separate function. He wanted it as an extension to
ts_headline. One way to do this will be to invoke it only when options
like MaxCoverSize is used. It will be slightly ugly though.

It still seems to have bugs. I will try to clean that up.

-Sushant.


On Thu, 2008-05-22 at 13:31 +0200, Pierre-Yves Strub wrote:
> Hi,
>
> I've ported the patch of Sushant Sinha for fragmented headlines to pg8.3.1
> (http://archives.postgresql.org/pgsql-general/2007-11/msg00508.php)
>
> W.r.t, http://archives.postgresql.org/pgsql-general/2008-03/msg00806.php
> I can continue the work until this becomes an acceptable patch for pg.
>
> Pierre-yves.

On Fri, May 23, 2008 at 7:10 AM, Sushant Sinha <sushant354@gmail.com> wrote:
> Teodor did not want a separate function. He wanted it as an extension to
> ts_headline. One way to do this will be to invoke it only when options
> like MaxCoverSize is used. It will be slightly ugly though.

What I understand is that Teodor wants hlparsetext to be used instead
of hlparsetext_with_cover. This is not too hard to do if hlparsetext
gives the positions of lexems. I started writting the patch, but I'm
stucked with the function LexizeExec which I do not totally understand
(... and is not well documents too :) )

[moved to -hackers, because talk is about implementation details]

> I've ported the patch of Sushant Sinha for fragmented headlines to pg8.3.1
> (http://archives.postgresql.org/pgsql-general/2007-11/msg00508.php)
Thank you.

1 > diff -Nrub postgresql-8.3.1-orig/contrib/tsearch2/tsearch2.c
now contrib/tsearch2 is compatibility layer for old applications - they don't
know about new features. So, this part isn't needed.

2 solution to compile function (ts_headline_with_fragments)  into core, but
using it only from contrib module looks very odd. So, new feature can be used
only with compatibility layer for old release :)

3 headline_with_fragments() is hardcoded to use default parser, but what will be
in case when configuration uses another parser? For example, for japanese language.

4 I would prefer the signature ts_headline( [regconfig,] text, tsquery [,text] )
and function should accept 'NumFragments=>N' for default parser. Another parsers
may use another options.

5 it just doesn't work correctly, because new code doesn't care of parser
specific type of lexemes.
contrib_regression=# select headline_with_fragments('english', 'wow asd-wow
wow', 'asd', '');
      headline_with_fragments
----------------------------------
  ...wow asd-wow<b>asd</b>-wow wow
(1 row)


So, I incline to use existing framework/infrastructure although it may be a
subject to change.

Some description:
1 ts_headline defines a correct parser to use
2 it calls hlparsetext to split text into structure suitable for both goals:
find the best fragment(s) and concatenate that fragment(s) back to the text
representation
3 it calls parser specific method   prsheadline which works with preparsed text
(parse was done in hlparsetext). Method should mark a needed
words/parts/lexemes etc.
4 ts_headline glues fragments into text and returns that.

We need a parser's headline method because only parser knows all about its lexemes.


--
Teodor Sigaev                                   E-mail: teodor@sigaev.ru
                                                    WWW: http://www.sigaev.ru/