TEXT column > 1Gb
От | Joe Carlson |
---|---|
Тема | TEXT column > 1Gb |
Дата | |
Msg-id | 80025ECD-44A6-454F-A4F9-784474B84952@lbl.gov обсуждение исходный текст |
Ответы |
Re: TEXT column > 1Gb
Re: TEXT column > 1Gb |
Список | pgsql-general |
Hello, I’ve recently encountered the issue of trying to insert more than 1 Gb into a TEXT column. While the docs say TEXT is unlimitedlength, I had been unaware of the 1Gb buffer size limitations. We can debate whether or not saving something this big in a single column is a good idea (spoiler: it isn’t. But not my designand, in fairness, was not anticipated when the schema was designed.), I’d like to implement something that is not amajor disruption and try to keep the mods on the server side. My first idea is to have a chunked associated table (in pseudocode) CREATE TABLE associated(key_id integer references main_table(key_id), chunk integer, text_start integer, text_end integer,text_chunk TEXT); And define functions for inserting and selecting by dividing into 1Mb chunks CREATE FUNCTION insertText(INTEGER,TEXT) RETURNS INTEGER AS $$ DECLARE chunk INTEGER := 0; key_id ALIAS for $1; the_text ALIAS for $2; text_chunk TEXT; BEGIN LOOP text_chunk := substr(the_text,chunk*1000000,1000000); IF length(text_chunk) = 0 THEN EXIT; END IF; INSERT INTO associated(key_id,chunk,text_start,text_end,text_chunk) VALUES (key_id,chunk,chunk*1000000,(chunk*1000000+length(text_chunk)),text_chunk); chunk := chunk + 1; END LOOP; RETURN chunk; END; $$ LANGUAGE plpgsql; This apparently runs into the same issues of buffers size: I get an ‘invalid message length’ in the log file and the insertfails. I can see from adding notices in the code that I never enter the LOOP; I assume having function arguments >1Gb is also a bad thing. I’d like to continue to keep the modifications on the server size. And I’d like to believe someone else has had this problembefore. Any suggestions other than have the client do the chunking? Can I use a different language binding and getaround the argument length limitations? Thanks
В списке pgsql-general по дате отправления: