Re: Sphinx indexing problem

Joshua Tolley wrote:
<blockquote
 cite="mid:AANLkTinR6Fo2M3gAuduV78sqYqPEOw0RX_5-BdbeMdZS@mail.gmail.com"
 type="cite">

    Is there anything I can do to prevent the API from attempting to put the
entire query result in memory?



Use a cursor, and fetch chunks of the result set one at a time.

http://www.postgresql.org/docs/current/interactive/sql-declare.html

--
Joshua Tolley / eggyknap
End Point Corporation




I would have done so,  had I written the application. Unfortunately,
the application was written by somebody else. Putting the entire result
set in memory is a bad idea and Postgres client should be changed,
probably by adding some configuration options, like maximum memory that
the client is allowed to consume and a "swap file". These options
should be configurable per user, not system-wide. As I have said in my
post, I do have a solution for my immediate problem but this slows
things down:

[root@medo etc]# ../bin/indexer
--all
  Sphinx 0.9.9-release (r2117)
  Copyright (c) 2001-2009, Andrew Aksyonoff

  using config file
'/usr/local/etc/sphinx.conf'...
  indexing index 'test1'...
  collected 12757000 docs, 21313.6 MB


It's still running, from slightly before I sent my post.
--
Mladen Gogala
Sr. Oracle DBA
1500 Broadway
New York, NY 10036
(212) 329-5251
www.vmsinfo.com