Just to confirm, we have 21 bits left for nonce in CTR? We have LSN (8 bytes), page-number (4 bytes) and counter (11 bits) in 16 bytes nonce space. Even though we have 21 bits left we cannot store relfilenode to the IV.
Fields like relfilenode, database, or tablespace could be added to the derived key, not the per-page IV. There's no space limitations as they are additional inputs into the HKDF (key derivation function).
Additional salt of any size, either some stored random value or something deterministic like the relfilenode/database/tablespace, can be added to the HKDF. There's separate issues with including those specific fields but it's not a size limitation.
For WAL encryption, before flushing WAL we encrypt whole 8k WAL page and then write only the encrypted data of the new WAL record using pg_pwrite() rather than write whole encrypted page. So each time we encrypt 8k WAL page we end up with encrypting different data with the same key+nonce but since we don't write to the disk other than space where we actually wrote WAL records it's not a problem. Is that right?
Ah, this is what I was referring to in my previous mail. I'm not familiar with how the writes happen yet (reading up...) but, yes, we would need to ensure that encrypted data is not written more than once (i.e. no writing of encrypt(zero) followed by writing of encrypt(non-zero) at the same spot).