XLog size reductions: Reduced XLog record header size for PG17
От | Matthias van de Meent |
---|---|
Тема | XLog size reductions: Reduced XLog record header size for PG17 |
Дата | |
Msg-id | CAEze2WgcW5SkPO_TGTqirRcCS_FN0xEZqUsXDE6pPz2TYzXQBA@mail.gmail.com обсуждение исходный текст |
Ответы |
Re: XLog size reductions: Reduced XLog record header size for PG17
Re: XLog size reductions: Reduced XLog record header size for PG17 |
Список | pgsql-hackers |
Hi, As I've mentioned earlier on-list [0][1] and off-list, I've been working on reducing the volume of WAL that we write. This is one intermediate step towards that effort. Attached is a patchset that reduces the storage overhead of each WAL record, with theoretical total savings in pgbench transactions somewhere between 4-5%, depending on platform and the locality of updates. These savings are achieved by reducing the amount of data stored, and by not emitting bytes that don't carry data relevant to each record's redo- and decode routines. Patches contained: 0001 is copied essentially verbatim from [1] and reduces overhead in the registered block's length field where possible. It is included to improve code commonality between varcoded integer fields. See [1] for more details. 0002 updates accesses to the rmgr-managed bits of xl_info with its own macro returning only rmgr-managed bits, and updates XLogRecGetInfo() to return only the xlog-managed bits. 0003 renames the rm_identify argument from 'info' to 'rmgrinfo'; and stops providing the xlog-managed bits to the function - rmgrs have no need to know the xlog internal info bits. 0004 continues on 0003 and moves the rmgr info bits into their own xl_rmgrinfo of type uint8, stored in the alignment hole in the XLogRecord struct. 0005 updates the code to only include a valid XID in the record when the rmgr actually needs to use that XID. 0006 implements a new, variable length, WAL record header format, previously discussed at [0] and [2]. This new WAL record header is a minimum of 14 bytes large, but generally will be 15 to 21 bytes in size, depending on the data contained, the type of record, and whether the record needs an XID. Notes: - The patchset includes [1] for its variable-length encoding of uint32, and this is included in the savings calculation. - Not all records include the backend's XID anymore. XLog API users must explicitly request the inclusion of XID in the record with the XLOG_INCLUDE_XID record flag. - XLog records are now always aligned to 8 bytes. This was needed to reduce the complexity of var-coding the record length on 32-bit systems. Savings on 32-bit systems still exist, but can be expected to be less impactful. - XLog length is now varlength encoded. No more records with <255 bytes of data storing 3 0-bytes - the length is now stored in 0, 1, 2 or 4 bytes. - RMGRs now get their own uint8 info/flags field. No more sharing bits with WAL infrastructure in xl_info. The byte is only stored if it is non-0, and otherwise omitted (indicated by flag bits in xl_info). Todo: - Check for any needed documentation / code comments updates - benchmark this Future work: - we could omit XLR_BLOCK_ID_DATA_[LONG,SHORT] if it is the only "block ID" in the record (such as checkpoint records, commit/rollback records, etc.). This would be indicated by a xl_info bit, and this would save 2-5 bytes per applicable record. - This patch inherits [1]'s property in which we can release the BKPBLOCK_HAS_DATA flag bit (its value is already implied by XLR_BLOCKID_SZCLASS), allowing us to use it for something else, like indicating varcoded RelFileLocator/BlockId. Kind regards, Matthias van de Meent Neon, Inc. [0] https://www.postgresql.org/message-id/flat/CAEze2Whf%3DfwAj7rosf6aDM9t%2B7MU1w-bJn28HFWYGkz%2Bics-hg%40mail.gmail.com [1] https://www.postgresql.org/message-id/flat/CAEze2WjuJqVeB6EUZ1z75_ittk54H6Lk7WtwRskEeGtZubr4bQ%40mail.gmail.com [2] https://www.postgresql.org/message-id/flat/CA+Tgmoaa9Yc9O-FP4vS_xTKf8Wgy8TzHpjnjN56_ShKE=jrP-Q@mail.gmail.com
Вложения
- v1-0001-Reduce-overhead-of-small-block-data-in-xlog-recor.patch
- v1-0004-Move-rmgr-info-bits-into-their-own-field-in-the-x.patch
- v1-0005-Stop-the-logging-of-XIDs-in-records-where-the-XID.patch
- v1-0003-Rename-rmgr_identify-info-bits-argument-to-rmgrin.patch
- v1-0002-Replace-accesses-to-rmgr-s-XLogRecord-info-bit-ac.patch
- v1-0006-Reformat-the-XLog-record-header.patch
В списке pgsql-hackers по дате отправления: