memory problems in copying large table to STDOUT
От | Martin Weinberg |
---|---|
Тема | memory problems in copying large table to STDOUT |
Дата | |
Msg-id | 199910091504.LAA26487@osprey.astro.umass.edu обсуждение исходный текст |
Ответы |
Re: [HACKERS] memory problems in copying large table to STDOUT
|
Список | pgsql-hackers |
Folks, I've been struggling to copy a large table (200 million records, 60GB) to tape using: psql -qc "copy psc to STDOUT;" Winter99 | dd of=/dev/st0 bs=32k After processing about 10 million records (this varies), I get: FATAL 1: Memory exhausted in AllocSetAlloc() (The tape drive is a DLT 7000, and the tape is not filled at this point). There is no evidence that the backend has really exhausted avail memory (I have 256MB but 1GB swap and the postgres user and database user both have unlimited memory usage). This is 6.5.1 on Linux 2.2.11 (w/Debian 2.1) on a dual 450Mhz Xeon box with a 128GB software Raid0 array. I've set SHMEM to 128MB am using "-B 12288 -S 8192". I've been trying to figure this out for a few weeks but can't seem to get this table copied to tape. Can one of the developers offer a suggestion? BTW, this is a large astronomical database which will eventually grow to about 500 million records. Besides this tape problem, pgsql is now working nicely for our application. I've been following the mysql thread. You folks may want to add "works with databases over 2GB" to your plus column. With thoughtful indexing, one can retrieve queries of <100000 records in 1 to 15 minutes which competes nicely with our main data server, a bunch of Sun Enterprise 5000 and 6000s running Informix. Of course, many people using this large system simultaneously, but our goal for this project is to recommend an alternative hardware/software solution to the astronomical community for <$10K. --M =========================================================================== Martin Weinberg Phone: (413) 545-3821 Dept. of Physics and Astronomy FAX: (413) 545-2117/0648 530 Graduate Research Tower weinberg@astro.umass.edu University of Massachusetts http://www.astro.umass.edu/~weinberg/ Amherst, MA 01003-4525
В списке pgsql-hackers по дате отправления: