Обсуждение: pgsql: Redesign initialization of partition routing structures
Redesign initialization of partition routing structures This speeds up write operations (INSERT, UPDATE, DELETE, COPY, as well as the future MERGE) on partitioned tables. This changes the setup for tuple routing so that it does far less work during the initial setup and pushes more work out to when partitions receive tuples. PartitionDispatchData structs for sub-partitioned tables are only created when a tuple gets routed through it. The possibly large arrays in the PartitionTupleRouting struct have largely been removed. The partitions[] array remains but now never contains any NULL gaps. Previously the NULLs had to be skipped during ExecCleanupTupleRouting(), which could add a large overhead to the cleanup when the number of partitions was large. The partitions[] array is allocated small to start with and only enlarged when we route tuples to enough partitions that it runs out of space. This allows us to keep simple single-row partition INSERTs running quickly. Redesign The arrays in PartitionTupleRouting which stored the tuple translation maps have now been removed. These have been moved out into a PartitionRoutingInfo struct which is an additional field in ResultRelInfo. The find_all_inheritors() call still remains by far the slowest part of ExecSetupPartitionTupleRouting(). This commit just removes the other slow parts. In passing also rename the tuple translation maps from being ParentToChild and ChildToParent to being RootToPartition and PartitionToRoot. The old names mislead you into thinking that a partition of some sub-partitioned table would translate to the rowtype of the sub-partitioned table rather than the root partitioned table. Authors: David Rowley and Amit Langote, heavily revised by Álvaro Herrera Testing help from Jesper Pedersen and Kato Sho. Discussion: https://postgr.es/m/CAKJS1f_1RJyFquuCKRFHTdcXqoPX-PYqAd7nz=GVBwvGh4a6xA@mail.gmail.com Branch ------ master Details ------- https://git.postgresql.org/pg/commitdiff/3f2393edefa5ef2b6970a5a2fa2c7e9c55cc10cf Modified Files -------------- src/backend/commands/copy.c | 86 +-- src/backend/executor/execMain.c | 2 +- src/backend/executor/execPartition.c | 957 ++++++++++++++++++--------------- src/backend/executor/nodeModifyTable.c | 165 ++---- src/backend/optimizer/prep/prepunion.c | 3 - src/backend/utils/cache/partcache.c | 16 +- src/include/catalog/partition.h | 6 +- src/include/executor/execPartition.h | 105 +--- src/include/nodes/execnodes.h | 11 +- 9 files changed, 637 insertions(+), 714 deletions(-)
Alvaro Herrera <alvherre@alvh.no-ip.org> writes:
> Redesign initialization of partition routing structures
Some of the buildfarm doesn't like this:
ccache gcc -std=c99 -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Wendif-labels
-Wmissing-format-attribute-Wformat-security -fno-strict-aliasing -fwrapv -g -O2 -I../../../src/include -isysroot
/Developer/SDKs/MacOSX10.6.sdk-DCOPY_PARSE_PLAN_TREES -DRAW_EXPRESSION_COVERAGE_TEST -c -o functioncmds.o
functioncmds.c
execPartition.c:96: error: redefinition of typedef 'PartitionTupleRouting'
../../../src/include/executor/execPartition.h:23: error: previous declaration of 'PartitionTupleRouting' was here
make[3]: *** [execPartition.o] Error 1
regards, tom lane
On 2018-Nov-16, Tom Lane wrote:
> Alvaro Herrera <alvherre@alvh.no-ip.org> writes:
> > Redesign initialization of partition routing structures
>
> Some of the buildfarm doesn't like this:
>
> ccache gcc -std=c99 -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Wendif-labels
-Wmissing-format-attribute-Wformat-security -fno-strict-aliasing -fwrapv -g -O2 -I../../../src/include -isysroot
/Developer/SDKs/MacOSX10.6.sdk-DCOPY_PARSE_PLAN_TREES -DRAW_EXPRESSION_COVERAGE_TEST -c -o functioncmds.o
functioncmds.c
> execPartition.c:96: error: redefinition of typedef 'PartitionTupleRouting'
> ../../../src/include/executor/execPartition.h:23: error: previous declaration of 'PartitionTupleRouting' was here
> make[3]: *** [execPartition.o] Error 1
Thanks, I just noticed it and will push this in a minute. Since I
cannot reproduce the error, it's a blind fix, but seems correct.
diff --git a/src/backend/executor/execPartition.c b/src/backend/executor/execPartition.c
index e3cb4fb1be..ec5628c9c2 100644
--- a/src/backend/executor/execPartition.c
+++ b/src/backend/executor/execPartition.c
@@ -82,7 +82,7 @@
* Memory context used to allocate subsidiary structs.
*-----------------------
*/
-typedef struct PartitionTupleRouting
+struct PartitionTupleRouting
{
Relation partition_root;
PartitionDispatch *partition_dispatch_info;
@@ -93,7 +93,7 @@ typedef struct PartitionTupleRouting
int max_partitions;
HTAB *subplan_resultrel_htab;
MemoryContext memcxt;
-} PartitionTupleRouting;
+};
/*-----------------------
* PartitionDispatch - information about one partitioned table in a partition
--
Álvaro Herrera https://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
Hi, On 2018-11-16 16:54:03 -0300, Alvaro Herrera wrote: > Thanks, I just noticed it and will push this in a minute. Since I > cannot reproduce the error, it's a blind fix, but seems correct. FWIW, if you use clang you can probably reproduce this using -Werror=typedef-redefinition Greetings, Andres Freund
Hello On 2018-Nov-16, Andres Freund wrote: > On 2018-11-16 16:54:03 -0300, Alvaro Herrera wrote: > > Thanks, I just noticed it and will push this in a minute. Since I > > cannot reproduce the error, it's a blind fix, but seems correct. > > FWIW, if you use clang you can probably reproduce this using > -Werror=typedef-redefinition Thanks ... Ran out of space in the /usr partition, so that'll have to wait until I make some time to reorganize. /me bangs head on desk -- Álvaro Herrera https://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services