Re: Make COPY format extendable: Extract COPY TO format implementations
| От | Sutou Kouhei |
|---|---|
| Тема | Re: Make COPY format extendable: Extract COPY TO format implementations |
| Дата | |
| Msg-id | 20251003.160650.355943655010493993.kou@clear-code.com обсуждение исходный текст |
| Ответ на | Re: Make COPY format extendable: Extract COPY TO format implementations (Masahiko Sawada <sawada.mshk@gmail.com>) |
| Список | pgsql-hackers |
Hi, In <CAD21AoADXWgdizS0mV5w8wdfftDRsm8sUtNW=CzYYS1OhjFD2A@mail.gmail.com> "Re: Make COPY format extendable: Extract COPY TO format implementations" on Mon, 15 Sep 2025 10:00:18 -0700, Masahiko Sawada <sawada.mshk@gmail.com> wrote: >> > I think we can use a local variable of CopyFormatOptions and memcpy it >> > to the opts of the returned cstate. >> >> It'll work too. Can we proceed this proposal with this >> approach? Should we collect more opinions before we proceed? >> If so, Could you add key people in this area to Cc to hear >> their opinions? > > Since we don't have a single decision-maker, we should proceed through > consensus-building and careful evaluation of each approach. I see that > several senior hackers are already included in this thread, which is > excellent. Since you and I, who have been involved in these > discussions, agreed with this approach, I believe we can proceed with > this direction. If anyone proposes alternative solutions that we find > more compelling, we might have to change the approach. OK. There is no objection for now. How about the attached patch? The patch uses the approach only for CopyToStateData. If this looks good, I can do it for CopyFromStateData too. This patch splits CopyToStateData to * CopyToStateData * CopyToStateInternalData * CopyToStateBuiltinData structs. This is based on the category described in https://www.postgresql.org/message-id/flat/CAD21AoBa0Wm3C2H12jaqkvLidP2zEhsC%2Bgf%3D3w7XiA4LQnvx0g%40mail.gmail.com#85cb988b0bec243d1e8dce699e02e009 : > 1. fields used only the core (not by format callback) > 2. fields used by both the core and format callbacks > 3. built-in format specific fields (mostly for text and csv) CopyToStateInternalData is for 1. CopyToStateData is for 2. CopyToStateBuiltinData is for 3. This patch adds CopyToRoutine::CopyToGetStateSize() that returns size of state struct for the routine. For example, Built-in formats use sizeof(CopyToStateBuiltinData) for it. BeginCopyTo() allocates sizeof(CopyToStateInternalData) + CopyToGetStateSize() size continuous memory and uses the front part as CopyToStateInternalData and the back part as CopyToStateData/CopyToStateBuilinData. Thanks, -- kou From 20539ad10512ef45785c4bd70b93f94eec1125ba Mon Sep 17 00:00:00 2001 From: Sutou Kouhei <kou@clear-code.com> Date: Fri, 3 Oct 2025 15:23:01 +0900 Subject: [PATCH] Split CopyToStateData to CopyToState{,Internal,Builtin}Data --- src/backend/commands/copyto.c | 282 ++++++++++++++++++--------------- src/include/commands/copyapi.h | 53 +++++++ 2 files changed, 211 insertions(+), 124 deletions(-) diff --git a/src/backend/commands/copyto.c b/src/backend/commands/copyto.c index 67b94b91cae..30298c0df0c 100644 --- a/src/backend/commands/copyto.c +++ b/src/backend/commands/copyto.c @@ -37,19 +37,36 @@ #include "utils/snapmgr.h" /* - * Represents the different dest cases we need to worry about at - * the bottom level + * This struct contains the state variables used internally. All COPY TO + * routines including built-in format routines should not use this. */ -typedef enum CopyDest +typedef struct CopyToStateInternalData { - COPY_FILE, /* to file (or a piped program) */ - COPY_FRONTEND, /* to frontend */ - COPY_CALLBACK, /* to callback function */ -} CopyDest; + /* format-specific routines */ + const CopyToRoutine *routine; + + /* parameters from the COPY command */ + QueryDesc *queryDesc; /* executable query to copy from */ + char *filename; /* filename, or NULL for STDOUT */ + bool is_program; /* is 'filename' a program to popen? */ + + Node *whereClause; /* WHERE condition (or NULL) */ + + /* + * Working state + */ + MemoryContext copycontext; /* per-copy execution context */ + MemoryContext rowcontext; /* per-row evaluation context */ +} CopyToStateInternalData; +typedef struct CopyToStateInternalData *CopyToStateInternal; + +#define CopyToStateInternalGetState(cstate_internal) \ + ((CopyToState) (((char *) cstate_internal) + sizeof(CopyToStateInternalData))) +#define CopyToStateGetInternal(cstate) \ + ((CopyToStateInternal) (((char *) cstate) - sizeof(CopyToStateInternalData))) /* - * This struct contains all the state variables used throughout a COPY TO - * operation. + * This struct contains the state variables used by built-in format routines. * * Multi-byte encodings: all supported client-side encodings encode multi-byte * characters by having the first byte's high bit set. Subsequent bytes of the @@ -62,40 +79,16 @@ typedef enum CopyDest * invoke pg_encoding_mblen() to skip over them. encoding_embeds_ascii is true * when we have to do it the hard way. */ -typedef struct CopyToStateData +typedef struct CopyToStateBuiltinData { - /* format-specific routines */ - const CopyToRoutine *routine; + CopyToStateData parent; /* low-level state data */ - CopyDest copy_dest; /* type of copy source/destination */ - FILE *copy_file; /* used if copy_dest == COPY_FILE */ - StringInfo fe_msgbuf; /* used for all dests during COPY TO */ - int file_encoding; /* file or remote side's character encoding */ bool need_transcoding; /* file encoding diff from server? */ bool encoding_embeds_ascii; /* ASCII can be non-first byte? */ - - /* parameters from the COPY command */ - Relation rel; /* relation to copy to */ - QueryDesc *queryDesc; /* executable query to copy from */ - List *attnumlist; /* integer list of attnums to copy */ - char *filename; /* filename, or NULL for STDOUT */ - bool is_program; /* is 'filename' a program to popen? */ - copy_data_dest_cb data_dest_cb; /* function for writing data */ - - CopyFormatOptions opts; - Node *whereClause; /* WHERE condition (or NULL) */ - - /* - * Working state - */ - MemoryContext copycontext; /* per-copy execution context */ - - FmgrInfo *out_functions; /* lookup info for output functions */ - MemoryContext rowcontext; /* per-row evaluation context */ - uint64 bytes_processed; /* number of bytes processed so far */ -} CopyToStateData; +} CopyToStateBuiltinData; +typedef struct CopyToStateBuiltinData *CopyToStateBuiltin; /* DestReceiver for COPY (query) TO */ typedef struct @@ -118,6 +111,7 @@ static void CopyAttributeOutCSV(CopyToState cstate, const char *string, bool use_quote); /* built-in format-specific routines */ +static size_t CopyToBuiltinGetStateSize(void); static void CopyToTextLikeStart(CopyToState cstate, TupleDesc tupDesc); static void CopyToTextLikeOutFunc(CopyToState cstate, Oid atttypid, FmgrInfo *finfo); static void CopyToTextOneRow(CopyToState cstate, TupleTableSlot *slot); @@ -150,6 +144,7 @@ static void CopySendInt16(CopyToState cstate, int16 val); /* text format */ static const CopyToRoutine CopyToRoutineText = { + .CopyToGetStateSize = CopyToBuiltinGetStateSize, .CopyToStart = CopyToTextLikeStart, .CopyToOutFunc = CopyToTextLikeOutFunc, .CopyToOneRow = CopyToTextOneRow, @@ -158,6 +153,7 @@ static const CopyToRoutine CopyToRoutineText = { /* CSV format */ static const CopyToRoutine CopyToRoutineCSV = { + .CopyToGetStateSize = CopyToBuiltinGetStateSize, .CopyToStart = CopyToTextLikeStart, .CopyToOutFunc = CopyToTextLikeOutFunc, .CopyToOneRow = CopyToCSVOneRow, @@ -166,6 +162,7 @@ static const CopyToRoutine CopyToRoutineCSV = { /* binary format */ static const CopyToRoutine CopyToRoutineBinary = { + .CopyToGetStateSize = CopyToBuiltinGetStateSize, .CopyToStart = CopyToBinaryStart, .CopyToOutFunc = CopyToBinaryOutFunc, .CopyToOneRow = CopyToBinaryOneRow, @@ -185,18 +182,27 @@ CopyToGetRoutine(const CopyFormatOptions *opts) return &CopyToRoutineText; } +/* Implementation of the allocate callback for all built-in formats */ +static size_t +CopyToBuiltinGetStateSize(void) +{ + return sizeof(CopyToStateBuiltinData); +} + /* Implementation of the start callback for text and CSV formats */ static void CopyToTextLikeStart(CopyToState cstate, TupleDesc tupDesc) { + CopyToStateBuiltin cstate_builtin = (CopyToStateBuiltin) cstate; + /* * For non-binary copy, we need to convert null_print to file encoding, * because it will be sent directly with CopySendString. */ - if (cstate->need_transcoding) + if (cstate_builtin->need_transcoding) cstate->opts.null_print_client = pg_server_to_any(cstate->opts.null_print, cstate->opts.null_print_len, - cstate->file_encoding); + cstate_builtin->file_encoding); /* if a header has been requested send the line */ if (cstate->opts.header_line == COPY_HEADER_TRUE) @@ -401,7 +407,7 @@ SendCopyBegin(CopyToState cstate) for (i = 0; i < natts; i++) pq_sendint16(&buf, format); /* per-column formats */ pq_endmessage(&buf); - cstate->copy_dest = COPY_FRONTEND; + cstate->copy_dest = COPY_DEST_FRONTEND; } static void @@ -444,16 +450,17 @@ CopySendChar(CopyToState cstate, char c) static void CopySendEndOfRow(CopyToState cstate) { + CopyToStateInternal cstate_internal = CopyToStateGetInternal(cstate); StringInfo fe_msgbuf = cstate->fe_msgbuf; switch (cstate->copy_dest) { - case COPY_FILE: + case COPY_DEST_FILE: if (fwrite(fe_msgbuf->data, fe_msgbuf->len, 1, cstate->copy_file) != 1 || ferror(cstate->copy_file)) { - if (cstate->is_program) + if (cstate_internal->is_program) { if (errno == EPIPE) { @@ -482,11 +489,11 @@ CopySendEndOfRow(CopyToState cstate) errmsg("could not write to COPY file: %m"))); } break; - case COPY_FRONTEND: + case COPY_DEST_FRONTEND: /* Dump the accumulated row as one CopyData message */ (void) pq_putmessage(PqMsg_CopyData, fe_msgbuf->data, fe_msgbuf->len); break; - case COPY_CALLBACK: + case COPY_DEST_CALLBACK: cstate->data_dest_cb(fe_msgbuf->data, fe_msgbuf->len); break; } @@ -507,7 +514,7 @@ CopySendTextLikeEndOfRow(CopyToState cstate) { switch (cstate->copy_dest) { - case COPY_FILE: + case COPY_DEST_FILE: /* Default line termination depends on platform */ #ifndef WIN32 CopySendChar(cstate, '\n'); @@ -515,7 +522,7 @@ CopySendTextLikeEndOfRow(CopyToState cstate) CopySendString(cstate, "\r\n"); #endif break; - case COPY_FRONTEND: + case COPY_DEST_FRONTEND: /* The FE/BE protocol uses \n as newline for all platforms */ CopySendChar(cstate, '\n'); break; @@ -561,9 +568,10 @@ CopySendInt16(CopyToState cstate, int16 val) static void ClosePipeToProgram(CopyToState cstate) { + CopyToStateInternal cstate_internal = CopyToStateGetInternal(cstate); int pclose_rc; - Assert(cstate->is_program); + Assert(cstate_internal->is_program); pclose_rc = ClosePipeStream(cstate->copy_file); if (pclose_rc == -1) @@ -575,7 +583,7 @@ ClosePipeToProgram(CopyToState cstate) ereport(ERROR, (errcode(ERRCODE_EXTERNAL_ROUTINE_EXCEPTION), errmsg("program \"%s\" failed", - cstate->filename), + cstate_internal->filename), errdetail_internal("%s", wait_result_to_str(pclose_rc)))); } } @@ -586,23 +594,25 @@ ClosePipeToProgram(CopyToState cstate) static void EndCopy(CopyToState cstate) { - if (cstate->is_program) + CopyToStateInternal cstate_internal = CopyToStateGetInternal(cstate); + + if (cstate_internal->is_program) { ClosePipeToProgram(cstate); } else { - if (cstate->filename != NULL && FreeFile(cstate->copy_file)) + if (cstate_internal->filename != NULL && FreeFile(cstate->copy_file)) ereport(ERROR, (errcode_for_file_access(), errmsg("could not close file \"%s\": %m", - cstate->filename))); + cstate_internal->filename))); } pgstat_progress_end_command(); - MemoryContextDelete(cstate->copycontext); - pfree(cstate); + MemoryContextDelete(cstate_internal->copycontext); + pfree(CopyToStateGetInternal(cstate)); } /* @@ -630,11 +640,16 @@ BeginCopyTo(ParseState *pstate, List *attnamelist, List *options) { + CopyToStateInternal cstate_internal; CopyToState cstate; + CopyToStateBuiltin cstate_builtin; bool pipe = (filename == NULL && data_dest_cb == NULL); TupleDesc tupDesc; int num_phys_attrs; + MemoryContext copycontext; MemoryContext oldcontext; + CopyFormatOptions opts = {0}; + const CopyToRoutine *routine; const int progress_cols[] = { PROGRESS_COPY_COMMAND, PROGRESS_COPY_TYPE @@ -686,24 +701,34 @@ BeginCopyTo(ParseState *pstate, } - /* Allocate workspace and zero all fields */ - cstate = (CopyToStateData *) palloc0(sizeof(CopyToStateData)); - /* * We allocate everything used by a cstate in a new memory context. This * avoids memory leaks during repeated use of COPY in a query. */ - cstate->copycontext = AllocSetContextCreate(CurrentMemoryContext, - "COPY", - ALLOCSET_DEFAULT_SIZES); + copycontext = AllocSetContextCreate(CurrentMemoryContext, + "COPY", + ALLOCSET_DEFAULT_SIZES); - oldcontext = MemoryContextSwitchTo(cstate->copycontext); + oldcontext = MemoryContextSwitchTo(copycontext); /* Extract options from the statement node tree */ - ProcessCopyOptions(pstate, &cstate->opts, false /* is_from */ , options); + ProcessCopyOptions(pstate, &opts, false /* is_from */ , options); - /* Set format routine */ - cstate->routine = CopyToGetRoutine(&cstate->opts); + /* Get format routine */ + routine = CopyToGetRoutine(&opts); + + /* Allocate workspace and set known values */ + MemoryContextSwitchTo(oldcontext); + cstate_internal = (CopyToStateInternal) palloc0(sizeof(CopyToStateInternal) + routine->CopyToGetStateSize()); + MemoryContextSwitchTo(copycontext); + cstate = CopyToStateInternalGetState(cstate_internal); + if (routine == &CopyToRoutineText || routine == &CopyToRoutineCSV || routine == &CopyToRoutineBinary) + cstate_builtin = (CopyToStateBuiltin) cstate; + else + cstate_builtin = NULL; + cstate_internal->copycontext = copycontext; + cstate->opts = opts; + cstate_internal->routine = routine; /* Process the source/target relation or query */ if (rel) @@ -835,19 +860,19 @@ BeginCopyTo(ParseState *pstate, ((DR_copy *) dest)->cstate = cstate; /* Create a QueryDesc requesting no output */ - cstate->queryDesc = CreateQueryDesc(plan, pstate->p_sourcetext, - GetActiveSnapshot(), - InvalidSnapshot, - dest, NULL, NULL, 0); + cstate_internal->queryDesc = CreateQueryDesc(plan, pstate->p_sourcetext, + GetActiveSnapshot(), + InvalidSnapshot, + dest, NULL, NULL, 0); /* * Call ExecutorStart to prepare the plan for execution. * * ExecutorStart computes a result tupdesc for us */ - ExecutorStart(cstate->queryDesc, 0); + ExecutorStart(cstate_internal->queryDesc, 0); - tupDesc = cstate->queryDesc->tupDesc; + tupDesc = cstate_internal->queryDesc->tupDesc; } /* Generate or convert list of attributes to process */ @@ -883,31 +908,34 @@ BeginCopyTo(ParseState *pstate, } } - /* Use client encoding when ENCODING option is not specified. */ - if (cstate->opts.file_encoding < 0) - cstate->file_encoding = pg_get_client_encoding(); - else - cstate->file_encoding = cstate->opts.file_encoding; + if (cstate_builtin) + { + /* Use client encoding when ENCODING option is not specified. */ + if (cstate->opts.file_encoding < 0) + cstate_builtin->file_encoding = pg_get_client_encoding(); + else + cstate_builtin->file_encoding = cstate->opts.file_encoding; - /* - * Set up encoding conversion info if the file and server encodings differ - * (see also pg_server_to_any). - */ - if (cstate->file_encoding == GetDatabaseEncoding() || - cstate->file_encoding == PG_SQL_ASCII) - cstate->need_transcoding = false; - else - cstate->need_transcoding = true; + /* + * Set up encoding conversion info if the file and server encodings + * differ (see also pg_server_to_any). + */ + if (cstate_builtin->file_encoding == GetDatabaseEncoding() || + cstate_builtin->file_encoding == PG_SQL_ASCII) + cstate_builtin->need_transcoding = false; + else + cstate_builtin->need_transcoding = true; - /* See Multibyte encoding comment above */ - cstate->encoding_embeds_ascii = PG_ENCODING_IS_CLIENT_ONLY(cstate->file_encoding); + /* See Multibyte encoding comment above */ + cstate_builtin->encoding_embeds_ascii = PG_ENCODING_IS_CLIENT_ONLY(cstate_builtin->file_encoding); + } - cstate->copy_dest = COPY_FILE; /* default */ + cstate->copy_dest = COPY_DEST_FILE; /* default */ if (data_dest_cb) { progress_vals[1] = PROGRESS_COPY_TYPE_CALLBACK; - cstate->copy_dest = COPY_CALLBACK; + cstate->copy_dest = COPY_DEST_CALLBACK; cstate->data_dest_cb = data_dest_cb; } else if (pipe) @@ -920,18 +948,18 @@ BeginCopyTo(ParseState *pstate, } else { - cstate->filename = pstrdup(filename); - cstate->is_program = is_program; + cstate_internal->filename = pstrdup(filename); + cstate_internal->is_program = is_program; if (is_program) { progress_vals[1] = PROGRESS_COPY_TYPE_PROGRAM; - cstate->copy_file = OpenPipeStream(cstate->filename, PG_BINARY_W); + cstate->copy_file = OpenPipeStream(cstate_internal->filename, PG_BINARY_W); if (cstate->copy_file == NULL) ereport(ERROR, (errcode_for_file_access(), errmsg("could not execute command \"%s\": %m", - cstate->filename))); + cstate_internal->filename))); } else { @@ -952,7 +980,7 @@ BeginCopyTo(ParseState *pstate, oumask = umask(S_IWGRP | S_IWOTH); PG_TRY(); { - cstate->copy_file = AllocateFile(cstate->filename, PG_BINARY_W); + cstate->copy_file = AllocateFile(cstate_internal->filename, PG_BINARY_W); } PG_FINALLY(); { @@ -967,7 +995,7 @@ BeginCopyTo(ParseState *pstate, ereport(ERROR, (errcode_for_file_access(), errmsg("could not open file \"%s\" for writing: %m", - cstate->filename), + cstate_internal->filename), (save_errno == ENOENT || save_errno == EACCES) ? errhint("COPY TO instructs the PostgreSQL server process to write a file. " "You may want a client-side facility such as psql's \\copy.") : 0)); @@ -977,12 +1005,12 @@ BeginCopyTo(ParseState *pstate, ereport(ERROR, (errcode_for_file_access(), errmsg("could not stat file \"%s\": %m", - cstate->filename))); + cstate_internal->filename))); if (S_ISDIR(st.st_mode)) ereport(ERROR, (errcode(ERRCODE_WRONG_OBJECT_TYPE), - errmsg("\"%s\" is a directory", cstate->filename))); + errmsg("\"%s\" is a directory", cstate_internal->filename))); } } @@ -1004,12 +1032,14 @@ BeginCopyTo(ParseState *pstate, void EndCopyTo(CopyToState cstate) { - if (cstate->queryDesc != NULL) + CopyToStateInternal cstate_internal = CopyToStateGetInternal(cstate); + + if (cstate_internal->queryDesc != NULL) { /* Close down the query and free resources. */ - ExecutorFinish(cstate->queryDesc); - ExecutorEnd(cstate->queryDesc); - FreeQueryDesc(cstate->queryDesc); + ExecutorFinish(cstate_internal->queryDesc); + ExecutorEnd(cstate_internal->queryDesc); + FreeQueryDesc(cstate_internal->queryDesc); PopActiveSnapshot(); } @@ -1025,7 +1055,8 @@ EndCopyTo(CopyToState cstate) uint64 DoCopyTo(CopyToState cstate) { - bool pipe = (cstate->filename == NULL && cstate->data_dest_cb == NULL); + CopyToStateInternal cstate_internal = CopyToStateGetInternal(cstate); + bool pipe = (cstate_internal->filename == NULL && cstate->data_dest_cb == NULL); bool fe_copy = (pipe && whereToSendOutput == DestRemote); TupleDesc tupDesc; int num_phys_attrs; @@ -1038,7 +1069,7 @@ DoCopyTo(CopyToState cstate) if (cstate->rel) tupDesc = RelationGetDescr(cstate->rel); else - tupDesc = cstate->queryDesc->tupDesc; + tupDesc = cstate_internal->queryDesc->tupDesc; num_phys_attrs = tupDesc->natts; cstate->opts.null_print_client = cstate->opts.null_print; /* default */ @@ -1052,8 +1083,8 @@ DoCopyTo(CopyToState cstate) int attnum = lfirst_int(cur); Form_pg_attribute attr = TupleDescAttr(tupDesc, attnum - 1); - cstate->routine->CopyToOutFunc(cstate, attr->atttypid, - &cstate->out_functions[attnum - 1]); + cstate_internal->routine->CopyToOutFunc(cstate, attr->atttypid, + &cstate->out_functions[attnum - 1]); } /* @@ -1062,11 +1093,11 @@ DoCopyTo(CopyToState cstate) * datatype output routines, and should be faster than retail pfree's * anyway. (We don't need a whole econtext as CopyFrom does.) */ - cstate->rowcontext = AllocSetContextCreate(CurrentMemoryContext, - "COPY TO", - ALLOCSET_DEFAULT_SIZES); + cstate_internal->rowcontext = AllocSetContextCreate(CurrentMemoryContext, + "COPY TO", + ALLOCSET_DEFAULT_SIZES); - cstate->routine->CopyToStart(cstate, tupDesc); + cstate_internal->routine->CopyToStart(cstate, tupDesc); if (cstate->rel) { @@ -1101,13 +1132,13 @@ DoCopyTo(CopyToState cstate) else { /* run the plan --- the dest receiver will send tuples */ - ExecutorRun(cstate->queryDesc, ForwardScanDirection, 0); - processed = ((DR_copy *) cstate->queryDesc->dest)->processed; + ExecutorRun(cstate_internal->queryDesc, ForwardScanDirection, 0); + processed = ((DR_copy *) cstate_internal->queryDesc->dest)->processed; } - cstate->routine->CopyToEnd(cstate); + cstate_internal->routine->CopyToEnd(cstate); - MemoryContextDelete(cstate->rowcontext); + MemoryContextDelete(cstate_internal->rowcontext); if (fe_copy) SendCopyEnd(cstate); @@ -1121,15 +1152,16 @@ DoCopyTo(CopyToState cstate) static inline void CopyOneRowTo(CopyToState cstate, TupleTableSlot *slot) { + CopyToStateInternal cstate_internal = CopyToStateGetInternal(cstate); MemoryContext oldcontext; - MemoryContextReset(cstate->rowcontext); - oldcontext = MemoryContextSwitchTo(cstate->rowcontext); + MemoryContextReset(cstate_internal->rowcontext); + oldcontext = MemoryContextSwitchTo(cstate_internal->rowcontext); /* Make sure the tuple is fully deconstructed */ slot_getallattrs(slot); - cstate->routine->CopyToOneRow(cstate, slot); + cstate_internal->routine->CopyToOneRow(cstate, slot); MemoryContextSwitchTo(oldcontext); } @@ -1146,13 +1178,14 @@ CopyOneRowTo(CopyToState cstate, TupleTableSlot *slot) static void CopyAttributeOutText(CopyToState cstate, const char *string) { + CopyToStateBuiltin cstate_builtin = (CopyToStateBuiltin) cstate; const char *ptr; const char *start; char c; char delimc = cstate->opts.delim[0]; - if (cstate->need_transcoding) - ptr = pg_server_to_any(string, strlen(string), cstate->file_encoding); + if (cstate_builtin->need_transcoding) + ptr = pg_server_to_any(string, strlen(string), cstate_builtin->file_encoding); else ptr = string; @@ -1170,7 +1203,7 @@ CopyAttributeOutText(CopyToState cstate, const char *string) * it's worth making two copies of it to get the IS_HIGHBIT_SET() test out * of the normal safe-encoding path. */ - if (cstate->encoding_embeds_ascii) + if (cstate_builtin->encoding_embeds_ascii) { start = ptr; while ((c = *ptr) != '\0') @@ -1225,7 +1258,7 @@ CopyAttributeOutText(CopyToState cstate, const char *string) start = ptr++; /* we include char in next run */ } else if (IS_HIGHBIT_SET(c)) - ptr += pg_encoding_mblen(cstate->file_encoding, ptr); + ptr += pg_encoding_mblen(cstate_builtin->file_encoding, ptr); else ptr++; } @@ -1300,6 +1333,7 @@ static void CopyAttributeOutCSV(CopyToState cstate, const char *string, bool use_quote) { + CopyToStateBuiltin cstate_builtin = (CopyToStateBuiltin) cstate; const char *ptr; const char *start; char c; @@ -1312,8 +1346,8 @@ CopyAttributeOutCSV(CopyToState cstate, const char *string, if (!use_quote && strcmp(string, cstate->opts.null_print) == 0) use_quote = true; - if (cstate->need_transcoding) - ptr = pg_server_to_any(string, strlen(string), cstate->file_encoding); + if (cstate_builtin->need_transcoding) + ptr = pg_server_to_any(string, strlen(string), cstate_builtin->file_encoding); else ptr = string; @@ -1342,8 +1376,8 @@ CopyAttributeOutCSV(CopyToState cstate, const char *string, use_quote = true; break; } - if (IS_HIGHBIT_SET(c) && cstate->encoding_embeds_ascii) - tptr += pg_encoding_mblen(cstate->file_encoding, tptr); + if (IS_HIGHBIT_SET(c) && cstate_builtin->encoding_embeds_ascii) + tptr += pg_encoding_mblen(cstate_builtin->file_encoding, tptr); else tptr++; } @@ -1366,8 +1400,8 @@ CopyAttributeOutCSV(CopyToState cstate, const char *string, CopySendChar(cstate, escapec); start = ptr; /* we include char in next run */ } - if (IS_HIGHBIT_SET(c) && cstate->encoding_embeds_ascii) - ptr += pg_encoding_mblen(cstate->file_encoding, ptr); + if (IS_HIGHBIT_SET(c) && cstate_builtin->encoding_embeds_ascii) + ptr += pg_encoding_mblen(cstate_builtin->file_encoding, ptr); else ptr++; } diff --git a/src/include/commands/copyapi.h b/src/include/commands/copyapi.h index 2a2d2f9876b..7c536c74a18 100644 --- a/src/include/commands/copyapi.h +++ b/src/include/commands/copyapi.h @@ -16,12 +16,65 @@ #include "commands/copy.h" +/* + * Represents the different dest cases we need to worry about at + * the bottom level + */ +typedef enum CopyDest +{ + COPY_DEST_FILE, /* to file (or a piped program) */ + COPY_DEST_FRONTEND, /* to frontend */ + COPY_DEST_CALLBACK, /* to callback function */ +} CopyDest; + +/* + * This struct contains the state variables used by PostgreSQL, built-in + * format routines and custom format routines. + */ +typedef struct CopyToStateData +{ + /* parameters from the COPY command */ + Relation rel; /* relation to copy to */ + List *attnumlist; /* integer list of attnums to copy */ + copy_data_dest_cb data_dest_cb; /* function for writing data */ + CopyFormatOptions opts; + FmgrInfo *out_functions; /* lookup info for output functions */ + + /* low-level state data */ + CopyDest copy_dest; /* type of copy source/destination */ + FILE *copy_file; /* used if copy_dest == COPY_FILE */ + StringInfo fe_msgbuf; /* used for all dests during COPY TO */ + + /* + * Working state + */ + uint64 bytes_processed; /* number of bytes processed so far */ +} CopyToStateData; + /* * API structure for a COPY TO format implementation. Note this must be * allocated in a server-lifetime manner, typically as a static const struct. */ typedef struct CopyToRoutine { + /* + * Return state size for this routine. + * + * If this routine uses CopyToStateData as-is, `return + * sizeof(CopyToStateData)` can be used. + * + * If this routine needs additional data than CopyToStateData, a new + * struct based on CopyToStateData can be used something like: + * + * typedef struct MyCopyToStateDate { + * struct CopyToStateData parent; + * int define_additional_members_here; + * } MyCopyToStateData; + * + * In the case, this callback returns `sizeof(MyCopyToStateData)`. + */ + size_t (*CopyToGetStateSize) (void); + /* * Set output function information. This callback is called once at the * beginning of COPY TO. -- 2.51.0
В списке pgsql-hackers по дате отправления: