Re: multi terabyte fulltext searching

Benjamin,

as one of the author of tsearch2 I'd like to know more about your setup.
tsearch2 in 8.2 has GIN index support, which scales much better than old
GiST index.

Oleg

On Wed, 21 Mar 2007, Benjamin Arai wrote:

> Hi,
>
> I have been struggling with getting fulltext searching for very large
> databases.  I can fulltext index 10s if gigs without any problem but when I
> start geting to hundreds of gigs it becomes slow.  My current system is a
> quad core with 8GB of memory.  I have the resource to throw more hardware at
> it but realistically it is not cost effective to buy a system with 128GB of
> memory.  Is there any solutions that people have come up with for indexing
> very large text databases?
>
> Essentially I have several terabytes of text that I need to index.  Each
> record is about 5 paragraphs of text.  I am currently using TSearch2
> (stemming and etc) and getting sub-optimal results.  Queries take more than a
> second to execute.  Has anybody implemented such a database using multiple
> systems or some special add-on to TSearch2 to make things faster?  I want to
> do something like partitioning the data into multiple systems and merging the
> ranked results at some master node.  Is something like this possible for
> PostgreSQL or must it be a software solution?
>
> Benjamin
>
> ---------------------------(end of broadcast)---------------------------
> TIP 9: In versions below 8.0, the planner will ignore your desire to
>     choose an index scan if your joining column's datatypes do not
>     match

     Regards,
         Oleg
_____________________________________________________________
Oleg Bartunov, Research Scientist, Head of AstroNet (www.astronet.ru),
Sternberg Astronomical Institute, Moscow University, Russia
Internet: oleg@sai.msu.su, http://www.sai.msu.su/~megera/
phone: +007(495)939-16-83, +007(495)939-23-83