Re: [HACKERS] sort on huge table

Tatsuo Ishii <t-ishii@sra.co.jp> writes:
>> OK, committed.  I have tested this code using a small RELSEG_SIZE,
>> and it seems to work, but I don't have the spare disk space to try
>> a full-scale test with > 4Gb of data.  Anyone care to try it?

> I will test it with my 2GB table. Creating 4GB would probably be
> possible, but I don't have enough sort space for that:-)

OK.  I am working on reducing the space requirement, but it would be
nice to test the bottom-level multi-temp-file code before layering
more stuff on top of it.  Anyone else have a whole bunch of free
disk space they could try a big sort with?

> I ran my previous test on 6.5.2, not on current. I hope current is
> stable enough to perform my testing.

It seems reasonably stable here, though I'm not doing much except
testing... main problem is you'll need to initdb, which means importing
your large dataset...

> Talking about the -S, I did use the default since setting -S seems to
> consume too much memory. For example, if I set it to 128MB, backend
> process grows over 512MB and it was killed due to swap space was run
> out. Maybe 4x law can be also applicated to -S?

If the code is working correctly then -S should be obeyed ---
approximately, anyway, since psort.c only counts the actual tuple data;
it doesn't know anything about AllocSet overhead &etc.  But it looked
to me like there might be some plain old memory leaks in psort.c, which
could account for actual usage being much more than intended.  I am
going to work on cleaning up psort.c after I finish building
infrastructure for it.
        regards, tom lane