Re: Memory-Bounded Hash Aggregation
От | Adam Lee |
---|---|
Тема | Re: Memory-Bounded Hash Aggregation |
Дата | |
Msg-id | 20200108071202.GA1511@mars.local обсуждение исходный текст |
Ответ на | Re: Memory-Bounded Hash Aggregation (Jeff Davis <pgsql@j-davis.com>) |
Список | pgsql-hackers |
Hi, Jeff I tried to use the logical tape APIs for hash agg spilling, based on your 1220 version. Turns out it doesn't make much of performance difference with the default 8K block size (might be my patch's problem), but the disk space (not I/O) would be saved a lot because I force the respilling to use the same LogicalTapeSet. Logtape APIs with default block size 8K: ``` postgres=# EXPLAIN ANALYZE SELECT avg(g) FROM generate_series(0,5000000) g GROUP BY g; QUERY PLAN -------------------------------------------------------------------------------------------------------------------------------------------- HashAggregate (cost=75000.02..75002.52 rows=200 width=36) (actual time=7701.706..24473.002 rows=5000001 loops=1) Group Key: g Memory Usage: 4096kB Batches: 516 Disk: 116921kB -> Function Scan on generate_series g (cost=0.00..50000.01 rows=5000001 width=4) (actual time=1611.829..3253.150 rows=5000001loops=1) Planning Time: 0.194 ms Execution Time: 25129.239 ms (6 rows) ``` Bare BufFile APIs: ``` postgres=# EXPLAIN ANALYZE SELECT avg(g) FROM generate_series(0,5000000) g GROUP BY g; QUERY PLAN -------------------------------------------------------------------------------------------------------------------------------------------- HashAggregate (cost=75000.02..75002.52 rows=200 width=36) (actual time=7339.835..24472.466 rows=5000001 loops=1) Group Key: g Memory Usage: 4096kB Batches: 516 Disk: 232773kB -> Function Scan on generate_series g (cost=0.00..50000.01 rows=5000001 width=4) (actual time=1580.057..3128.749 rows=5000001loops=1) Planning Time: 0.769 ms Execution Time: 26696.502 ms (6 rows) ``` Even though, I'm not sure which API is better, because we should avoid the respilling as much as we could in the planner, and hash join uses the bare BufFile. Attached my hacky and probably not robust diff for your reference. -- Adam Lee
Вложения
В списке pgsql-hackers по дате отправления: