Re: [HACKERS] memory problems in copying large table to STDOUT

Tom Lane wrote on Sat, 09 Oct 1999 11:54:34 EDT
>Martin Weinberg <weinberg@osprey.astro.umass.edu> writes:
>> I've been struggling to copy a large table (200 million
>> records, 60GB) to tape using:
>>    psql -qc "copy psc to STDOUT;" Winter99 | dd of=/dev/st0 bs=32k
>> After processing about 10 million records (this varies), I
>> get:
>>   FATAL 1:  Memory exhausted in AllocSetAlloc()
>
>Hmm.  What is the exact declaration of the table?
>
>The only explanation I can think of offhand is that the output
>conversion function for one of the column types is leaking memory...
>copy.c itself looks to be pretty careful not to.
>

The table def is:

CREATE TABLE psc (hemis        text,date        date,scan        int2,id        int4,ra        float4,dec
float4,glon       float4,glat        float4,err_maj        float4,err_min        float4,err_ang        int2,xscan
float4,cnf_flag    text,j_m        float4,h_m        float4,k_m        float4,j_msig        float4,h_msig
float4,k_msig       float4,j_m_psf        float4,h_m_psf        float4,k_m_psf        float4,j_psfchi
float4,h_psfchi   float4,k_psfchi    float4,j_skyval    float4,h_skyval    float4,k_skyval    float4,j_blend
int2,h_blend       int2,k_blend        int2,j_m_stdap    float4,h_m_stdap    float4,k_m_stdap    float4,j_msig_stdap
float4,h_msig_stdap   float4,k_msig_stdap    float4,j_prob_pers    float4,h_prob_pers    float4,k_prob_pers
float4,j_prg_flg   text,h_prg_flg    text,k_prg_flg    text,j_mrg_flg    text,h_mrg_flg    text,k_mrg_flg
text,j_pix_flg   text,h_pix_flg    text,k_pix_flg    text,j_cal        float4,h_cal        float4,k_cal
float4,gal_contam   int2,id_opt        text,dist_opt    float4,b_m_opt        float4,r_m_opt        float4,j_h
float4,h_k       float4,j_k        float4,dup_src        int2,use_src        int2,ext_key_1    int4
 
);

Thanks for taking a look,

--Martin

===========================================================================

Martin Weinberg                      Phone: (413) 545-3821
Dept. of Physics and Astronomy       FAX:   (413) 545-2117/0648
530 Graduate Research Tower         weinberg@astro.umass.edu
University of Massachusetts         http://www.astro.umass.edu/~weinberg/
Amherst, MA  01003-4525