Обсуждение: NUMA shared memory interleaving

Thanks to having pg_numa.c, we can now simply address problem#2 of
NUMA imbalance from [1] pages 11-14, by interleaving shm memory in
PG19 - patch attached. We do not need to call numa_set_localalloc() as
we only interleave shm segments, while local allocations stay the same
(well, "local" means relative to the CPU asking for private memory).
Below is result from legacy 4s32t64 Sandy Bridge EP box with low NUMA
(QPI) interconnect bandwidth to better illustrate the problem (it's
little edgecase, but some one may hit it):

Testcase:
    small SB (here it was 4GB*) that fully fits NUMA hugepage zone as
this was tested with hugepages=on

    $ cat seqconcurrscans.pgb
    \set num (:client_id % 8) + 1
    select sum(octet_length(filler)) from pgbench_accounts_:num;

    /usr/local/pgsql/bin/pg_ctl -D /db/data -l logfile restart
    /usr/local/pgsql/bin/psql  -c "select
pg_prewarm('pgbench_accounts_'||s) from generate_series(1, 8) s;"
#load all using current policy
    /usr/local/pgsql/bin/psql  -c "select * from
pg_shmem_allocations_numa where name = 'Buffer Blocks';"
    /usr/local/pgsql/bin/pgbench -c 64 -j 8 -P 1 -T 60 -f seqconcurrscans.pgb

on master and numa=off (default) and in previous versions:
         name      | numa_node |    size
    ---------------+-----------+------------
     Buffer Blocks |         0 |          0
     Buffer Blocks |         1 |          0
     Buffer Blocks |         2 | 4297064448
     Buffer Blocks |         3 |          0

    latency average = 1826.324 ms
    latency stddev = 665.567 ms
    tps = 34.708151 (without initial connection time)

on master and numa=on:
         name      | numa_node |    size
    ---------------+-----------+------------
     Buffer Blocks |         0 | 1073741824
     Buffer Blocks |         1 | 1073741824
     Buffer Blocks |         2 | 1075838976
     Buffer Blocks |         3 | 1073741824

    latency average = 1002.288 ms
    latency stddev = 214.392 ms
    tps = 63.344814 (without initial connection time)

Normal pgbench workloads tend to be not affected, as each backend
tends to touch just a small partition of shm (thanks to BAS
strategies). Some remaining questions are:
1. How to name this GUC (numa or numa_shm_interleave) ? I prefer the
first option, as we could potentially in future add more optimizations
behind that GUC.
2. Should we also interleave DSA/DSM for Parallel Query? (I'm not an
expert on DSA/DSM at all)
3. Should we fail to start if we numa=on on an unsupported platform?

* interesting tidbit to get reliable measurement: one needs to double
check that s_b (hugepage allocation) is smaller than per-NUMA zone
free hugepages (s_b fits static hugepage allocation within a single
zone). This shouldn't be a problem on 2 sockets (as most of the time
there, s_b is < 50% RAM anyway, well usually 26-30% with some stuff by
max_connections, it's higher than 25% but people usually sysctl
nr_hugepages=25%RAM) , but with >= 4 NUMA nodes (4 sockets or some
modern MCMs) kernel might start spilling the s_b (> 25%) to the other
NUMA node on it's own, so it's best to verify it using
pg_shmem_allocations_numa...

-J.

[1] - https://anarazel.de/talks/2024-10-23-pgconf-eu-numa-vs-postgresql/numa-vs-postgresql.pdf

On Wed, Apr 16, 2025 at 9:14 PM Jakub Wartak
<jakub.wartak@enterprisedb.com> wrote:
> 2. Should we also interleave DSA/DSM for Parallel Query? (I'm not an
> expert on DSA/DSM at all)

I have no answers but I have speculated for years about a very
specific case (without any idea where to begin due to lack of ... I
guess all this sort of stuff): in ExecParallelHashJoinNewBatch(),
workers split up and try to work on different batches on their own to
minimise contention, and when that's not possible (more workers than
batches, or finishing their existing work at different times and going
to help others), they just proceed in round-robin order.  A beginner
thought is: if you're going to help someone working on a hash table,
it would surely be best to have the CPUs and all the data on the same
NUMA node.  During loading, cache line ping pong would be cheaper, and
during probing, it *might* be easier to tune explicit memory prefetch
timing that way as it would look more like a single node system with a
fixed latency, IDK (I've shared patches for prefetching before that
showed pretty decent speedups, and the lack of that feature is
probably a bigger problem than any of this stuff, who knows...).
Another beginner thought is that the DSA allocator is a source of
contention during loading: the dumbest problem is that the chunks are
just too small, but it might also be interesting to look into per-node
pools.  Or something.   IDK, just some thoughts...

On Thu, Apr 17, 2025 at 1:58 AM Thomas Munro <thomas.munro@gmail.com> wrote:
> I have no answers but I have speculated for years about a very
> specific case (without any idea where to begin due to lack of ... I
> guess all this sort of stuff): in ExecParallelHashJoinNewBatch(),
> workers split up and try to work on different batches on their own to
> minimise contention, and when that's not possible (more workers than
> batches, or finishing their existing work at different times and going
> to help others), they just proceed in round-robin order.  A beginner
> thought is: if you're going to help someone working on a hash table,
> it would surely be best to have the CPUs and all the data on the same
> NUMA node.  During loading, cache line ping pong would be cheaper, and
> during probing, it *might* be easier to tune explicit memory prefetch
> timing that way as it would look more like a single node system with a
> fixed latency, IDK (I've shared patches for prefetching before that
> showed pretty decent speedups, and the lack of that feature is
> probably a bigger problem than any of this stuff, who knows...).
> Another beginner thought is that the DSA allocator is a source of
> contention during loading: the dumbest problem is that the chunks are
> just too small, but it might also be interesting to look into per-node
> pools.  Or something.   IDK, just some thoughts...

And BTW there are papers about that (but they mostly just remind me
that I have to reboot the prefetching patch long before that...), for
example:

https://15721.courses.cs.cmu.edu/spring2023/papers/11-hashjoins/lang-imdm2013.pdf

On Wed, Apr 16, 2025 at 5:14 AM Jakub Wartak
<jakub.wartak@enterprisedb.com> wrote:
> Normal pgbench workloads tend to be not affected, as each backend
> tends to touch just a small partition of shm (thanks to BAS
> strategies). Some remaining questions are:
> 1. How to name this GUC (numa or numa_shm_interleave) ? I prefer the
> first option, as we could potentially in future add more optimizations
> behind that GUC.

I wonder whether the GUC needs to support interleaving between a
designated set of nodes rather than only being able to do all nodes.
For example, suppose someone is pinning the processes to a certain set
of NUMA nodes; perhaps then they wouldn't want to use memory from
other nodes.

--
Robert Haas
EDB: http://www.enterprisedb.com

Hi,

On Wed, Apr 16, 2025 at 10:05:04AM -0400, Robert Haas wrote:
> On Wed, Apr 16, 2025 at 5:14 AM Jakub Wartak
> <jakub.wartak@enterprisedb.com> wrote:
> > Normal pgbench workloads tend to be not affected, as each backend
> > tends to touch just a small partition of shm (thanks to BAS
> > strategies). Some remaining questions are:
> > 1. How to name this GUC (numa or numa_shm_interleave) ? I prefer the
> > first option, as we could potentially in future add more optimizations
> > behind that GUC.
> 
> I wonder whether the GUC needs to support interleaving between a
> designated set of nodes rather than only being able to do all nodes.
> For example, suppose someone is pinning the processes to a certain set
> of NUMA nodes; perhaps then they wouldn't want to use memory from
> other nodes.

+1. That could be used for instances consolidation on the same host. One could
ensure that numa nodes are not shared across instances (cpu and memory resource
isolation per instance). Bonus point, adding Direct I/O into the game would
ensure that the OS page cache is not shared too.

Regards,

-- 
Bertrand Drouvot
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com

Hi,

On Thu, Apr 17, 2025 at 01:58:44AM +1200, Thomas Munro wrote:
> On Wed, Apr 16, 2025 at 9:14 PM Jakub Wartak
> <jakub.wartak@enterprisedb.com> wrote:
> > 2. Should we also interleave DSA/DSM for Parallel Query? (I'm not an
> > expert on DSA/DSM at all)
> 
> I have no answers but I have speculated for years about a very
> specific case (without any idea where to begin due to lack of ... I
> guess all this sort of stuff): in ExecParallelHashJoinNewBatch(),
> workers split up and try to work on different batches on their own to
> minimise contention, and when that's not possible (more workers than
> batches, or finishing their existing work at different times and going
> to help others), they just proceed in round-robin order.  A beginner
> thought is: if you're going to help someone working on a hash table,
> it would surely be best to have the CPUs and all the data on the same
> NUMA node.  During loading, cache line ping pong would be cheaper, and
> during probing, it *might* be easier to tune explicit memory prefetch
> timing that way as it would look more like a single node system with a
> fixed latency, IDK (I've shared patches for prefetching before that
> showed pretty decent speedups, and the lack of that feature is
> probably a bigger problem than any of this stuff, who knows...).
> Another beginner thought is that the DSA allocator is a source of
> contention during loading: the dumbest problem is that the chunks are
> just too small, but it might also be interesting to look into per-node
> pools.  Or something.   IDK, just some thoughts...

I'm also thinking that could be beneficial for parallel workers. I think the
ideal scenario would be to have the parallel workers spread across numa nodes and
accessing their "local" memory first (and help with "remote" memory access if
there is still more work to do "remotely").

Regards,

-- 
Bertrand Drouvot
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com

On Fri, Apr 18, 2025 at 7:43 PM Bertrand Drouvot
<bertranddrouvot.pg@gmail.com> wrote:
>
> Hi,
>
> On Wed, Apr 16, 2025 at 10:05:04AM -0400, Robert Haas wrote:
> > On Wed, Apr 16, 2025 at 5:14 AM Jakub Wartak
> > <jakub.wartak@enterprisedb.com> wrote:
> > > Normal pgbench workloads tend to be not affected, as each backend
> > > tends to touch just a small partition of shm (thanks to BAS
> > > strategies). Some remaining questions are:
> > > 1. How to name this GUC (numa or numa_shm_interleave) ? I prefer the
> > > first option, as we could potentially in future add more optimizations
> > > behind that GUC.
> >
> > I wonder whether the GUC needs to support interleaving between a
> > designated set of nodes rather than only being able to do all nodes.
> > For example, suppose someone is pinning the processes to a certain set
> > of NUMA nodes; perhaps then they wouldn't want to use memory from
> > other nodes.
>
> +1. That could be used for instances consolidation on the same host. One could
> ensure that numa nodes are not shared across instances (cpu and memory resource
> isolation per instance). Bonus point, adding Direct I/O into the game would
> ensure that the OS page cache is not shared too.

Hi, the attached patch has two changes:
1. It adds more modes and supports this 'consolidation' and
'isolation' scenario from above. Doc in patch briefly explains the
merit.
2. it adds trivial NUMA for PQ

The original initial test expanded on the very same machine
(4s32c128t, QPI interconnect):

numa='off'
    latency average = 1271.019 ms
    latency stddev = 245.061 ms
    tps = 49.683923 (without initial connection time)
    explanation(pcm-memory): 3 sockets doing ~46MB/s on RAM (almost
idle), 1 socket doing ~17GB/s (fully saturated because s_b ended up in
this scenario only on NUMA node)

numa='all'
    latency average = 702.622 ms
    latency stddev = 13.259 ms
    tps = 90.026526 (without initial connection time)
    explanation(pcm-memory): this forced to interleave s_b across 4
NUMA nodes and each socket gets equal part of workload (9.2 - 10GB/s)
totalling ~37GB/s of memory bandwidth

This gives a boost: 90/49.6=1.8x. The values for memory bandwidth are
combined read+write.

NUMA impact on the Parallel Query:
----------------------------------
with:
    with the most simplistic interleaving of s_b +
dynamic_shared_memory for PQ interleaved too :
    max_worker_processes=max_parallel_workers=max_parallel_workers_per_gather=64
    alter on 1 partition to force real 64 parallel seq scans
The query:
    select sum(octet_length(filler)) from pgbench_accounts;
launched 64 effective parallel workes launched for 64 partitions each
of 400MB  (25600MBs), All of that was fitting in the s_b (32GB), so
all fetched from s_b. All was hot, several first runs are not shown.

select sum(octet_length(filler)) from pgbench_accounts;

numa='off'
    Time: 1108.178 ms (00:01.108)
    Time: 1118.494 ms (00:01.118)
    Time: 1104.491 ms (00:01.104)
    Time: 1112.221 ms (00:01.112)
    Time: 1105.501 ms (00:01.106)
    avg: 1109 ms

    -- not interleaved, more like appended:
    postgres=# select * from pg_shmem_allocations_numa where name =
'Buffer Blocks';
         name      | numa_node |    size
    ---------------+-----------+------------
     Buffer Blocks |         0 | 9277800448
     Buffer Blocks |         1 | 7044333568
     Buffer Blocks |         2 | 9097445376
     Buffer Blocks |         3 | 8942256128

numa='all'
    Time: 1026.747 ms (00:01.027)
    Time: 1024.087 ms (00:01.024)
    Time: 1024.179 ms (00:01.024)
    Time: 1037.026 ms (00:01.037)
    avg: 1027 ms

        postgres=# select * from pg_shmem_allocations_numa where name
= 'Buffer Blocks';
         name      | numa_node |    size
    ---------------+-----------+------------
     Buffer Blocks |         0 | 8589934592
     Buffer Blocks |         1 | 8592031744
     Buffer Blocks |         2 | 8589934592
     Buffer Blocks |         3 | 8589934592

1109/1027=1.079x, not bad for such trivial change and the paper
referenced by Thomas also stated (`We can see an improvement by a
factor of more than three by just running
the non-NUMA-aware implementation on interleaved memor`), probably it
could be improved much further, but I'm not planning to work on this
more. So if anything:
- latency-wise: it would be best to place leader+all PQ workers close
to s_b, provided s_b fits NUMA shared/huge page memory there and you
won't need more CPU than there's on that NUMA node... (assuming e.g.
hosting 4 DBs on 4-sockets each on it's own, it would be best to pin
everything including shm, but PQ workers too)
- capacity/TPS-wise or s_b > NUMA: just interleave to maximize
bandwidth and get uniform CPU performance out of this

The patch supports e.g. numa='@1' which should fully isolate the
workload to just memory and CPUs on node #1.
As for the patch: I'm lost with our C headers policy :)

One of less obvious reasons (outside of better efficiency of
consolidation of multiple PostgreSQL cluster on single NUMA server),
why I've implemented '=' and '@' is that seems that CXL memory can be
attached as a CPU-less(!) NUMA node, thus Linux - depending on
sysctls/sysfs setup - could use it for automatic memory tiering and
the above provides configurable way to prevent allocation on such
(potential) systems - simply exclude such NUMA node via config for now
and we are covered I think. I have no access to real hardware, so I
haven't researched it further, but it looks like in the far future we
could probably indicate preferred NUMA memory nodes (think big s_b,
bigger than "CPU" RAM), and that kernel could transparently do NUMA
auto balancing/demotion for us (AKA Transparent Page Placement AKA
memory) or vice versa: use small s_b and do not use CXL node at all
and expect that VFS cache could be spilled there.
numa_weighted_interleave_memory() / MPOL_WEIGHTED_INTERLEAVE is not
yet supported in distros (although new libnuma has support for it), so
I have not included it in the patch, as it was too early.

BTW: DO NOT USE meson's --buildtype=debug as it somehow disables the
NUMA optimizations benefit, I've lost hours on it (probably -O0 is so
slow that it wasn't stressing interconnects enough). Default is
--buildtype=debugoptimized which is good. Also if testing performance,
check that HW that has proper realistic NUMA remote access distances
first, e.g. here my remote had remote access 2x or even 3x. Probably
this is worth only testing on multi-sockets which have really higher
latencies/throughput limitations, but reports from 1 socket MCMs CPUs
(with various Node-per-Socket BIOS settings) are welcome too.

kernel 6.14.7 was used with full isolation:
    cpupower frequency-set --governor performance
    cpupower idle-set -D0
    echo 1 > /sys/devices/system/cpu/intel_pstate/no_turbo
    echo never > /sys/kernel/mm/transparent_hugepage/enabled
    echo never > /sys/kernel/mm/transparent_hugepage/defrag

max_connections = '10000'
huge_pages = 'on'
wal_level = 'minimal'
wal_buffers = '1024MB'
max_wal_senders = '0'
shared_buffers = '4 GB'
autovacuum = 'off'
max_parallel_workers_per_gather = '0'
numa = 'all'
#numa = 'off'

[1] - https://lwn.net/Articles/897536/

On Fri, Apr 18, 2025 at 7:48 PM Bertrand Drouvot
<bertranddrouvot.pg@gmail.com> wrote:
>
> Hi,
>
> On Thu, Apr 17, 2025 at 01:58:44AM +1200, Thomas Munro wrote:
> > On Wed, Apr 16, 2025 at 9:14 PM Jakub Wartak
> > <jakub.wartak@enterprisedb.com> wrote:
> > > 2. Should we also interleave DSA/DSM for Parallel Query? (I'm not an
> > > expert on DSA/DSM at all)
> >
> > I have no answers but I have speculated for years about a very
> > specific case (without any idea where to begin due to lack of ... I
> > guess all this sort of stuff): in ExecParallelHashJoinNewBatch(),
> > workers split up and try to work on different batches on their own to
> > minimise contention, and when that's not possible (more workers than
> > batches, or finishing their existing work at different times and going
> > to help others), they just proceed in round-robin order.  A beginner
> > thought is: if you're going to help someone working on a hash table,
> > it would surely be best to have the CPUs and all the data on the same
> > NUMA node.  During loading, cache line ping pong would be cheaper, and
> > during probing, it *might* be easier to tune explicit memory prefetch
> > timing that way as it would look more like a single node system with a
> > fixed latency, IDK (I've shared patches for prefetching before that
> > showed pretty decent speedups, and the lack of that feature is
> > probably a bigger problem than any of this stuff, who knows...).
> > Another beginner thought is that the DSA allocator is a source of
> > contention during loading: the dumbest problem is that the chunks are
> > just too small, but it might also be interesting to look into per-node
> > pools.  Or something.   IDK, just some thoughts...
>
> I'm also thinking that could be beneficial for parallel workers. I think the
> ideal scenario would be to have the parallel workers spread across numa nodes and
> accessing their "local" memory first (and help with "remote" memory access if
> there is still more work to do "remotely").

Hi Bertrand, I've played with CPU pinning of PQ workers (via adjusting
postmaster pin), but I've got quite opposite results - please see
attached, especially "lat"ency against how the CPUs were assigned VS
NUMA/s_b when it was not interleaved. Not that I intend to spend a lot
of time researching PQ vs NUMA , but I've included interleaving of PQ
shm segments too in the v4 patch in the subthread nearby. Those
attached results here, were made some time ago with v1 of the patch
where PQ shm segment was not interleaved.

If anything it would be to hear if there are any sensible
production-like scenarios/workloads when dynamic_shared_memory should
be set to sysv or mmap (instead of default posix) ? Asking for Linux
only, I couldn't imagine anything (?)

-J.

Hi,

I agree we should improve the behavior on NUMA systems. But I'm not sure
this patch does far enough, or perhaps the approach seems a bit too
blunt, ignoring some interesting stuff.

AFAICS the patch essentially does the same thing as

   numactl --interleave=all

except that it only does that to shared memory, not to process private
memory (as if we called numa_set_localalloc). Which means it has some of
the problems people observe with --interleave=all.

In particular, this practically guarantees that (with 4K memory pages),
each buffer hits multiple NUMA nodes. Because with the first half will
do to node N, while the second half goes to node (N+1).

That doesn't seem great. It's likely better than a misbalanced system
with everything allocated on a single NUMA node, but I don't see how it
could be better than "balanced" properly warmed up system where the
buffers are not split like this.

But OK, admittedly this only happens for 4K memory pages, and serious
systems with a lot of memory are likely to use huge pages, which makes
this less of an issue (only the buffers crossing the page boundaries
might get split).


My bigger comment however is that the approach focuses on balancing the
nodes (i.e. ensuring each node gets a fair share of shared memory), and
is entirely oblivious to the internal structure of the shared memory.

* It interleaves the shared segment, but it has many pieces - shared
buffers are the largest but not the only one. Does it make sense to
interleave all the other pieces?

* Some of the pieces are tightly related. For example, we talk about
shared buffers as if it was one big array, but it actually is two arrays
- blocks and descriptors. Even if buffers don't get split between nodes
(thanks to huge pages), there's no guarantee the descriptor for the
buffer does not end on a different node.

* In fact, the descriptors are so much smaller that blocks that it's
practically guaranteed all descriptors will end up on a single node.


I could probably come up with a couple more similar items, but I think
you get the idea. I do think making Postgres NUMA-aware will require
figuring out how to distribute (or not distribute) different parts of
the shared memory, and do that explicitly. And do that in a way that
allows us to do other stuff in NUMA-aware way, e.g. have a separate
freelists and clocksweep for each NUMA node, etc.

That's something numa_interleave_memory simply can't do for us, and I
suppose it might also have other downsides on large instances. I mean,
doesn't it have to create a separate mapping for each memory page?
Wouldn't that be a bit inefficient/costly for big instances?


Of course, I'm not saying all this as a random passerby - I've been
working on a similar patch for a while, based on Andres' experimental
NUMA branch. It's far from complete/perfect, more of a PoC quality, but
I hope to share it on the mailing list sometime soon.


FWIW while I think the patch doesn't go far enough, there's one area
where I think it probably goes way too far - configurability. I agree
it's reasonable to allow running on a subset of nodes, e.g. to split the
system between multiple instances etc. But do we need to configure that
from Postgres? Aren't people likely to already use something like
containers or k8 anyway? I think we should just try to inherit this from
the environment, i.e. determine which nodes we're allowed to run, and
use that. Maybe we'll find we need to be smarter, but I think we caan
leave that for later.


regards

-- 
Tomas Vondra

Hi Tomas!

On Fri, Jun 27, 2025 at 6:41 PM Tomas Vondra <tomas@vondra.me> wrote:

> I agree we should improve the behavior on NUMA systems. But I'm not sure
> this patch does far enough, or perhaps the approach seems a bit too
> blunt, ignoring some interesting stuff.
>
> AFAICS the patch essentially does the same thing as
>
>    numactl --interleave=all
>
> except that it only does that to shared memory, not to process private
> memory (as if we called numa_set_localalloc). Which means it has some of
> the problems people observe with --interleave=all.
>
> In particular, this practically guarantees that (with 4K memory pages),
> each buffer hits multiple NUMA nodes. Because with the first half will
> do to node N, while the second half goes to node (N+1).
>
> That doesn't seem great. It's likely better than a misbalanced system
> with everything allocated on a single NUMA node, but I don't see how it
> could be better than "balanced" properly warmed up system where the
> buffers are not split like this.
>
> But OK, admittedly this only happens for 4K memory pages, and serious
> systems with a lot of memory are likely to use huge pages, which makes
> this less of an issue (only the buffers crossing the page boundaries
> might get split).
>
>
> My bigger comment however is that the approach focuses on balancing the
> nodes (i.e. ensuring each node gets a fair share of shared memory), and
> is entirely oblivious to the internal structure of the shared memory.
>
> * It interleaves the shared segment, but it has many pieces - shared
> buffers are the largest but not the only one. Does it make sense to
> interleave all the other pieces?
>
> * Some of the pieces are tightly related. For example, we talk about
> shared buffers as if it was one big array, but it actually is two arrays
> - blocks and descriptors. Even if buffers don't get split between nodes
> (thanks to huge pages), there's no guarantee the descriptor for the
> buffer does not end on a different node.
>
> * In fact, the descriptors are so much smaller that blocks that it's
> practically guaranteed all descriptors will end up on a single node.
>
>
> I could probably come up with a couple more similar items, but I think
> you get the idea. I do think making Postgres NUMA-aware will require
> figuring out how to distribute (or not distribute) different parts of
> the shared memory, and do that explicitly. And do that in a way that
> allows us to do other stuff in NUMA-aware way, e.g. have a separate
> freelists and clocksweep for each NUMA node, etc.

I do understand what you mean, but I'm *NOT* stating here that it
makes PG fully "NUMA-aware". I actually try to avoid doing so with
each sentence. This is only about the imbalance problem specifically.
I think we could build those follow-up optimizations as separate
patches in this or follow-up threads. If we would do it all in one
giant 0001 (without split) the very first question would be to
quantify the impact of each of those optimizations (for which we would
probably need more GUCs?). Here I'm just showing that the very first
baby step - interleaving - helps avoid interconnect saturation in some
cases too.

Anyway, even putting the fact that local mallocs() would be
interleaved, adjusting systemd startup scripts to just include
`numactl --interleave=all` sounds like some dirty hack not like proper
UX.

Also please note that:
* I do not have lot of time to dedicate towards it, yet I was kind of
always interested in researching that and wondering why we couldn't it
for such long time, therefore the previous observability work and now
$subject (note it is not claiming to be full blown NUMA awareness,
just some basic NUMA interleave as first [well, second?] step).
* I've raised this question in the first post "How to name this GUC
(numa or numa_shm_interleave) ?" I still have no idea, but `numa`,
simply looks better, and we could just add way more stuff to it over
time (in PG19 or future versions?). Does that sound good?

> That's something numa_interleave_memory simply can't do for us, and I
> suppose it might also have other downsides on large instances. I mean,
> doesn't it have to create a separate mapping for each memory page?
> Wouldn't that be a bit inefficient/costly for big instances?

No? Or what kind of mapping do you have in mind? I think our shared
memory on the kernel side is just a single VMA (contiguous memory
region), on which technically we execute mbind() (libnuma is just a
wrapper around it). I have not observed any kind of regressions,
actually quite the opposite. Not sure what you also mean by 'big
instances' (AFAIK 1-2TB shared_buffers might even fail to start).

> Of course, I'm not saying all this as a random passerby - I've been
> working on a similar patch for a while, based on Andres' experimental
> NUMA branch. It's far from complete/perfect, more of a PoC quality, but
> I hope to share it on the mailing list sometime soon.

Cool, I didn't know Andres's branch was public till now, I know he
referenced multiple issues in presentation (and hackathon!), but I
wanted to divide it and try to get something in at least partially,
step by step, to have at least something. I think we should
collaborate (not a lot of people interested in this?) and I can try to
offer my limited help if you attack those more advanced problems. I
think we could improve this by properly ensuring that by
over(allocating)/spreading/padding certain special regions (e.g.
better distribute ProcArray, but what about cache hits?) - we get more
juice, or do you want to start from scratch and re-design/re-think all
shm allocations case by case?

> FWIW while I think the patch doesn't go far enough, there's one area
> where I think it probably goes way too far - configurability. I agree
> it's reasonable to allow running on a subset of nodes, e.g. to split the
> system between multiple instances etc. But do we need to configure that
> from Postgres? Aren't people likely to already use something like
> containers or k8 anyway?
> I think we should just try to inherit this from
> the environment, i.e. determine which nodes we're allowed to run, and
> use that. Maybe we'll find we need to be smarter, but I think we caan
> leave that for later.

That's what "numa=all" is all about (take whatever is there in the
OS/namespace), but I do not know a better way than just let's say
numa_get_mems_allowed() being altered somehow by namespace/cgroups. I
think if one runs on k8/containers then it's quite limited/small
deployment and he wouldn't benefit from this at all (I struggle to
imagine the point of k8 pod using 2+ sockets), quite contrary: my
experience indicates that the biggest deployments are usually almost
baremetal? And it's way easier to get consistent results. Anyway as
You say, let's leave it for later. PG currently often is not CPU-aware
(i.e. is not even adjusting sizing of certain structs based on CPU
count), so making it NUMA-aware or cgroup/namespace-aware sounds
already like taking 2-3 steps ahead into future [I think we had
discussion at least one in LWLock partitionmanager /
FP_LOCK_SLOTS_PER_BACKEND where I've proposed to size certain
structures based on $VCPUs or I am misremembering this]

-J.

On Mon, Jun 30, 2025 at 12:55 PM Jakub Wartak
<jakub.wartak@enterprisedb.com> wrote:
[..]
> > FWIW while I think the patch doesn't go far enough, there's one area
> > where I think it probably goes way too far - configurability. I agree
> > it's reasonable to allow running on a subset of nodes, e.g. to split the
> > system between multiple instances etc. But do we need to configure that
> > from Postgres? Aren't people likely to already use something like
> > containers or k8 anyway?
> > I think we should just try to inherit this from
> > the environment, i.e. determine which nodes we're allowed to run, and
> > use that. Maybe we'll find we need to be smarter, but I think we caan
> > leave that for later.
>
> That's what "numa=all" is all about (take whatever is there in the
> OS/namespace)

My error, that should be: that's what "numa=AUTO" is all about (..)

-J.

On 6/30/25 12:55, Jakub Wartak wrote:
> Hi Tomas!
> 
> On Fri, Jun 27, 2025 at 6:41 PM Tomas Vondra <tomas@vondra.me> wrote:
> 
>> I agree we should improve the behavior on NUMA systems. But I'm not sure
>> this patch does far enough, or perhaps the approach seems a bit too
>> blunt, ignoring some interesting stuff.
>>
>> AFAICS the patch essentially does the same thing as
>>
>>    numactl --interleave=all
>>
>> except that it only does that to shared memory, not to process private
>> memory (as if we called numa_set_localalloc). Which means it has some of
>> the problems people observe with --interleave=all.
>>
>> In particular, this practically guarantees that (with 4K memory pages),
>> each buffer hits multiple NUMA nodes. Because with the first half will
>> do to node N, while the second half goes to node (N+1).
>>
>> That doesn't seem great. It's likely better than a misbalanced system
>> with everything allocated on a single NUMA node, but I don't see how it
>> could be better than "balanced" properly warmed up system where the
>> buffers are not split like this.
>>
>> But OK, admittedly this only happens for 4K memory pages, and serious
>> systems with a lot of memory are likely to use huge pages, which makes
>> this less of an issue (only the buffers crossing the page boundaries
>> might get split).
>>
>>
>> My bigger comment however is that the approach focuses on balancing the
>> nodes (i.e. ensuring each node gets a fair share of shared memory), and
>> is entirely oblivious to the internal structure of the shared memory.
>>
>> * It interleaves the shared segment, but it has many pieces - shared
>> buffers are the largest but not the only one. Does it make sense to
>> interleave all the other pieces?
>>
>> * Some of the pieces are tightly related. For example, we talk about
>> shared buffers as if it was one big array, but it actually is two arrays
>> - blocks and descriptors. Even if buffers don't get split between nodes
>> (thanks to huge pages), there's no guarantee the descriptor for the
>> buffer does not end on a different node.
>>
>> * In fact, the descriptors are so much smaller that blocks that it's
>> practically guaranteed all descriptors will end up on a single node.
>>
>>
>> I could probably come up with a couple more similar items, but I think
>> you get the idea. I do think making Postgres NUMA-aware will require
>> figuring out how to distribute (or not distribute) different parts of
>> the shared memory, and do that explicitly. And do that in a way that
>> allows us to do other stuff in NUMA-aware way, e.g. have a separate
>> freelists and clocksweep for each NUMA node, etc.
> 
> I do understand what you mean, but I'm *NOT* stating here that it
> makes PG fully "NUMA-aware". I actually try to avoid doing so with
> each sentence. This is only about the imbalance problem specifically.
> I think we could build those follow-up optimizations as separate
> patches in this or follow-up threads. If we would do it all in one
> giant 0001 (without split) the very first question would be to
> quantify the impact of each of those optimizations (for which we would
> probably need more GUCs?). Here I'm just showing that the very first
> baby step - interleaving - helps avoid interconnect saturation in some
> cases too.
> 
> Anyway, even putting the fact that local mallocs() would be
> interleaved, adjusting systemd startup scripts to just include
> `numactl --interleave=all` sounds like some dirty hack not like proper
> UX.
> 

I wasn't suggesting to do "numactl --interleave=all". My argument was
simply that doing numa_interleave_memory() has most of the same issues,
because it's oblivious to what's stored in the shared memory. Sure, the
fact that local memory is not interleaved too is an improvement.

But I just don't see how this could be 0001, followed by some later
improvements. ISTM the improvements would have to largely undo 0001
first, and it would be nontrivial if an optimization needs to do that
only for some part of the shared memory.

> Also please note that:
> * I do not have lot of time to dedicate towards it, yet I was kind of
> always interested in researching that and wondering why we couldn't it
> for such long time, therefore the previous observability work and now
> $subject (note it is not claiming to be full blown NUMA awareness,
> just some basic NUMA interleave as first [well, second?] step).

Sorry, I appreciate the time you spent working on these features. It
wasn't my intention to dunk on your patch. I'm afraid this is an example
of how reactions on -hackers are often focused on pointing out issues. I
apologize for that, I should have realized it earlier.

I certainly agree it'd be good to improve the NUMA support, otherwise I
wouldn't be messing with Andres' PoC patches myself.

> * I've raised this question in the first post "How to name this GUC
> (numa or numa_shm_interleave) ?" I still have no idea, but `numa`,
> simply looks better, and we could just add way more stuff to it over
> time (in PG19 or future versions?). Does that sound good?
> 

I'm not sure. In my WIP patch I have a bunch of numa_ GUCs, for
different parts of the shared memory. But that's mostly for development,
to allow easy experimentation. I don't have a clear idea what UX should
look like.

>> That's something numa_interleave_memory simply can't do for us, and I
>> suppose it might also have other downsides on large instances. I mean,
>> doesn't it have to create a separate mapping for each memory page?
>> Wouldn't that be a bit inefficient/costly for big instances?
> 
> No? Or what kind of mapping do you have in mind? I think our shared
> memory on the kernel side is just a single VMA (contiguous memory
> region), on which technically we execute mbind() (libnuma is just a
> wrapper around it). I have not observed any kind of regressions,
> actually quite the opposite. Not sure what you also mean by 'big
> instances' (AFAIK 1-2TB shared_buffers might even fail to start).
> 

Something as simple as giving a contiguous chunk of to each NUMA node.
Essentially 1/nodes goes to the first NUMA node, and so on. I haven't
looked into the details of how NUMA interleaving works, but from the
discussions I had about it, I understood it might be expensive. Not
sure, maybe that's wrong.

But the other reason for a simpler mapping is that it seems useful to be
able to easily calculate which NUMA node a buffer belongs to. Because
then you can do NUMA-aware freelists, clocksweep, etc.

>> Of course, I'm not saying all this as a random passerby - I've been
>> working on a similar patch for a while, based on Andres' experimental
>> NUMA branch. It's far from complete/perfect, more of a PoC quality, but
>> I hope to share it on the mailing list sometime soon.
> 
> Cool, I didn't know Andres's branch was public till now, I know he
> referenced multiple issues in presentation (and hackathon!), but I
> wanted to divide it and try to get something in at least partially,
> step by step, to have at least something. I think we should
> collaborate (not a lot of people interested in this?) and I can try to
> offer my limited help if you attack those more advanced problems. I
> think we could improve this by properly ensuring that by
> over(allocating)/spreading/padding certain special regions (e.g.
> better distribute ProcArray, but what about cache hits?) - we get more
> juice, or do you want to start from scratch and re-design/re-think all
> shm allocations case by case?
> 

+1 to collaboration, absolutely. I was actually planning to ping you
once I have something workable. I hope I'll be able to polish the WIP
patches a little bit and post them sometime this week.

>> FWIW while I think the patch doesn't go far enough, there's one area
>> where I think it probably goes way too far - configurability. I agree
>> it's reasonable to allow running on a subset of nodes, e.g. to split the
>> system between multiple instances etc. But do we need to configure that
>> from Postgres? Aren't people likely to already use something like
>> containers or k8 anyway?
>> I think we should just try to inherit this from
>> the environment, i.e. determine which nodes we're allowed to run, and
>> use that. Maybe we'll find we need to be smarter, but I think we caan
>> leave that for later.
> 
> That's what "numa=all" is all about (take whatever is there in the
> OS/namespace), but I do not know a better way than just let's say
> numa_get_mems_allowed() being altered somehow by namespace/cgroups. I
> think if one runs on k8/containers then it's quite limited/small
> deployment and he wouldn't benefit from this at all (I struggle to
> imagine the point of k8 pod using 2+ sockets), quite contrary: my
> experience indicates that the biggest deployments are usually almost
> baremetal? And it's way easier to get consistent results. Anyway as
> You say, let's leave it for later. PG currently often is not CPU-aware
> (i.e. is not even adjusting sizing of certain structs based on CPU
> count), so making it NUMA-aware or cgroup/namespace-aware sounds
> already like taking 2-3 steps ahead into future [I think we had
> discussion at least one in LWLock partitionmanager /
> FP_LOCK_SLOTS_PER_BACKEND where I've proposed to size certain
> structures based on $VCPUs or I am misremembering this]
> 

+1 to leave this for later, we can worry about this once we have it
working with the basic whole-system NUMA setups. I hope people doing
some of this would give us feedback what config they actually need.

regards

-- 
Tomas Vondra

On Mon, Jun 30, 2025 at 9:23 PM Tomas Vondra <tomas@vondra.me> wrote:
>
> I wasn't suggesting to do "numactl --interleave=all". My argument was
> simply that doing numa_interleave_memory() has most of the same issues,
> because it's oblivious to what's stored in the shared memory. Sure, the
> fact that local memory is not interleaved too is an improvement.

... and that's enough for me to start this ;)

> But I just don't see how this could be 0001, followed by some later
> improvements. ISTM the improvements would have to largely undo 0001
> first, and it would be nontrivial if an optimization needs to do that
> only for some part of the shared memory.

OK, maybe I'll back-off a bit to see Your ideas first. It seems you
are thinking about having multiple separate shared memory segments.

> I certainly agree it'd be good to improve the NUMA support, otherwise I
> wouldn't be messing with Andres' PoC patches myself.

Yup, cool, let's stick to that.

> > * I've raised this question in the first post "How to name this GUC
> > (numa or numa_shm_interleave) ?" I still have no idea, but `numa`,
> > simply looks better, and we could just add way more stuff to it over
> > time (in PG19 or future versions?). Does that sound good?
> >
>
> I'm not sure. In my WIP patch I have a bunch of numa_ GUCs, for
> different parts of the shared memory. But that's mostly for development,
> to allow easy experimentation.
[..]
> I don't have a clear idea what UX should look like.

Later (after research/experiments), I could still imagine sticking to
one big `numa` switch like it's today in v4-0001, but maybe with
additional 1-2 more `really_advanced_numa=stuff` (but not lots of
them, I would imagine e.g. that NUMA for analytics could be different
setup that NUMA for OLTP -- AKA do we want to optimize for
interconnect bandwidth or latency?).

> >> That's something numa_interleave_memory simply can't do for us, and I
> >> suppose it might also have other downsides on large instances. I mean,
> >> doesn't it have to create a separate mapping for each memory page?
> >> Wouldn't that be a bit inefficient/costly for big instances?
> >
> > No? Or what kind of mapping do you have in mind? I think our shared
> > memory on the kernel side is just a single VMA (contiguous memory
> > region), on which technically we execute mbind() (libnuma is just a
> > wrapper around it). I have not observed any kind of regressions,
> > actually quite the opposite. Not sure what you also mean by 'big
> > instances' (AFAIK 1-2TB shared_buffers might even fail to start).
> >
>
> Something as simple as giving a contiguous chunk of to each NUMA node.

That would actually be multiple separate VMAs/shared memory regions
(main, and specific ones for in case of NUMA - per structure) and
potentially - speculating here - slower fork()?

Related, the only complaint about memory allocated via
mmap(MAP_SHARED|MAP_HUGETLB) with NUMA, I have so far is that if the
per-zone HP free memory is too small (especially with HP=on), it
starts to spill over to the others nodes without interleaving and
without notification, you may have the same problem here unless it is
strict allocation.

> Essentially 1/nodes goes to the first NUMA node, and so on. I haven't
> looked into the details of how NUMA interleaving works, but from the
> discussions I had about it, I understood it might be expensive. Not
> sure, maybe that's wrong.

I would really like to hear the argument how NUMA interleaving is
expensive on the kernel side. It's literally bandwidth over latency. I
can buy the argument that e.g. by having dedicated mmap(MAP_SHARED)
for ProcArray[] (assuming we are HP=on/2MB), and by having a smaller
page size just for this stuct (sizeof() =~ 832b? so let's assume even
wasting 4kB per entry), could better enable kernel's NUMA
autobalancing to better relocate those necessary pages closer to the
active processes (warning: I'm making lots of assumptions, haven't
really checked memory access patterns for this struct). No idea how
bad it would be on the CPU caches too, though by making it so big here
in this theoretical context. But the easy counter argument also could
be: smaller page size = no HP available --> potentially making it
__swap-able__ and potentially causing worse dTLB hit-rates? ... and we
are just discussing a single shared memory entry and there are 73 :)

My take is that by doing it is - as an opt-in - basic interleaving is
safe, proven and and gives *more* predictable latency that without it
(of course as You mention we could do better with some allocation for
specific structures, but how do You know where CPU scheduler puts
stuff?) I think we would need to limit ourselves to just optimizing
the most crucial (hot) stuff, like ProcArray[], but probably doesn't
make sense for investigating structures like multixacts/substractions
in this attempt.

E.g. I could even imagine that we could boost standby's NUMA-awareness
too, just by putting most used memory (eg.g. XLOG) to the same NODE
that is used by startup/recovery and walreciever (by CPU pinning), not
sure is it worth the effort though in this attempt and the problem
would be: what to do with those low-level/optimized allocations after
pg_promote() to primary? In theory this quickly escalates to calling
interleave on that struct again, so maybe let's put it aside.

To sum up, my problem is that optimization possibilities are quite
endless, so we need to settle on something realistic, right?

> But the other reason for a simpler mapping is that it seems useful to be
> able to easily calculate which NUMA node a buffer belongs to. Because
> then you can do NUMA-aware freelists, clocksweep, etc.

Yay, sounds pretty advanced!

> +1 to collaboration, absolutely. I was actually planning to ping you
> once I have something workable. I hope I'll be able to polish the WIP
> patches a little bit and post them sometime this week.

Cool.

-J.

On 7/1/25 11:04, Jakub Wartak wrote:
> On Mon, Jun 30, 2025 at 9:23 PM Tomas Vondra <tomas@vondra.me> wrote:
>>
>> I wasn't suggesting to do "numactl --interleave=all". My argument was
>> simply that doing numa_interleave_memory() has most of the same issues,
>> because it's oblivious to what's stored in the shared memory. Sure, the
>> fact that local memory is not interleaved too is an improvement.
> 
> ... and that's enough for me to start this ;)
> 
>> But I just don't see how this could be 0001, followed by some later
>> improvements. ISTM the improvements would have to largely undo 0001
>> first, and it would be nontrivial if an optimization needs to do that
>> only for some part of the shared memory.
> 
> OK, maybe I'll back-off a bit to see Your ideas first. It seems you
> are thinking about having multiple separate shared memory segments.
> 
>> I certainly agree it'd be good to improve the NUMA support, otherwise I
>> wouldn't be messing with Andres' PoC patches myself.
> 
> Yup, cool, let's stick to that.
> 
>>> * I've raised this question in the first post "How to name this GUC
>>> (numa or numa_shm_interleave) ?" I still have no idea, but `numa`,
>>> simply looks better, and we could just add way more stuff to it over
>>> time (in PG19 or future versions?). Does that sound good?
>>>
>>
>> I'm not sure. In my WIP patch I have a bunch of numa_ GUCs, for
>> different parts of the shared memory. But that's mostly for development,
>> to allow easy experimentation.
> [..]
>> I don't have a clear idea what UX should look like.
> 
> Later (after research/experiments), I could still imagine sticking to
> one big `numa` switch like it's today in v4-0001, but maybe with
> additional 1-2 more `really_advanced_numa=stuff` (but not lots of
> them, I would imagine e.g. that NUMA for analytics could be different
> setup that NUMA for OLTP -- AKA do we want to optimize for
> interconnect bandwidth or latency?).
> 

Maybe. I have no clear idea yet, but I'd like to keep the number of new
GUCs as low as possible.

>>>> That's something numa_interleave_memory simply can't do for us, and I
>>>> suppose it might also have other downsides on large instances. I mean,
>>>> doesn't it have to create a separate mapping for each memory page?
>>>> Wouldn't that be a bit inefficient/costly for big instances?
>>>
>>> No? Or what kind of mapping do you have in mind? I think our shared
>>> memory on the kernel side is just a single VMA (contiguous memory
>>> region), on which technically we execute mbind() (libnuma is just a
>>> wrapper around it). I have not observed any kind of regressions,
>>> actually quite the opposite. Not sure what you also mean by 'big
>>> instances' (AFAIK 1-2TB shared_buffers might even fail to start).
>>>
>>
>> Something as simple as giving a contiguous chunk of to each NUMA node.
> 
> That would actually be multiple separate VMAs/shared memory regions
> (main, and specific ones for in case of NUMA - per structure) and
> potentially - speculating here - slower fork()?
> 

I may be confused about what you mean by VMA, but it certainly does not
require creating separate shared memory segments. Interleaving also does
not require that. You can move a certain range of memory to a particular
NUMA node, and that's it.

We may end up with separate shared memory segments for different parts
of the shared memory (instead of having a single segment like now), e.g.
to support dynamic shared_buffers resizing. But even with that we'd have
a shared memory segment for each part, split between NUMA nodes.

Well, we'd probably want separate segments, because for some parts it's
not great to have 2MB pages, because it's too coarse. And you can only
use huge pages for the whole segment.

> Related, the only complaint about memory allocated via
> mmap(MAP_SHARED|MAP_HUGETLB) with NUMA, I have so far is that if the
> per-zone HP free memory is too small (especially with HP=on), it
> starts to spill over to the others nodes without interleaving and
> without notification, you may have the same problem here unless it is
> strict allocation.
> 
>> Essentially 1/nodes goes to the first NUMA node, and so on. I haven't
>> looked into the details of how NUMA interleaving works, but from the
>> discussions I had about it, I understood it might be expensive. Not
>> sure, maybe that's wrong.
> 
> I would really like to hear the argument how NUMA interleaving is
> expensive on the kernel side. It's literally bandwidth over latency. I
> can buy the argument that e.g. by having dedicated mmap(MAP_SHARED)
> for ProcArray[] (assuming we are HP=on/2MB), and by having a smaller
> page size just for this stuct (sizeof() =~ 832b? so let's assume even
> wasting 4kB per entry), could better enable kernel's NUMA
> autobalancing to better relocate those necessary pages closer to the
> active processes (warning: I'm making lots of assumptions, haven't
> really checked memory access patterns for this struct). No idea how
> bad it would be on the CPU caches too, though by making it so big here
> in this theoretical context. But the easy counter argument also could
> be: smaller page size = no HP available --> potentially making it
> __swap-able__ and potentially causing worse dTLB hit-rates? ... and we
> are just discussing a single shared memory entry and there are 73 :)
> 

I admit I don't recall the exact details of why exactly interleaving
would be expensive on the kernel side. I've been told by smarter people
that might be the case, but I don't remember the exact explanation. And
maybe it isn't measurably more expensive ...

I've been focusing on the aspect that it makes certain things more
difficult, or even impossible ...

> My take is that by doing it is - as an opt-in - basic interleaving is
> safe, proven and and gives *more* predictable latency that without it
> (of course as You mention we could do better with some allocation for
> specific structures, but how do You know where CPU scheduler puts
> stuff?) I think we would need to limit ourselves to just optimizing
> the most crucial (hot) stuff, like ProcArray[], but probably doesn't
> make sense for investigating structures like multixacts/substractions
> in this attempt.
> 

My argument is that if we allocate the structs "well" then we can do
something smart later, like pick a PGPROC placed on the current NUMA
node (at connection time), and possibly even pin it to that NUMA node so
that it doesn't get migrated.

This means not just the PROPROC itself, but also stuff like fast-path
locking arrays (which are stored separately). And interleaving could
easily place them on a different NUMA node.

> E.g. I could even imagine that we could boost standby's NUMA-awareness
> too, just by putting most used memory (eg.g. XLOG) to the same NODE
> that is used by startup/recovery and walreciever (by CPU pinning), not
> sure is it worth the effort though in this attempt and the problem
> would be: what to do with those low-level/optimized allocations after
> pg_promote() to primary? In theory this quickly escalates to calling
> interleave on that struct again, so maybe let's put it aside.
> 
> To sum up, my problem is that optimization possibilities are quite
> endless, so we need to settle on something realistic, right?
> 

Perhaps. But maybe we should explore the possibilities first, before
just settling out on something at the very beginning. To make an
informed decision we need to know what the costs/benefits are, and we
need to understand what the "advanced" solution might look like, so that
we don't pick a design that makes that impossible.

>> But the other reason for a simpler mapping is that it seems useful to be
>> able to easily calculate which NUMA node a buffer belongs to. Because
>> then you can do NUMA-aware freelists, clocksweep, etc.
> 
> Yay, sounds pretty advanced!
> 
>> +1 to collaboration, absolutely. I was actually planning to ping you
>> once I have something workable. I hope I'll be able to polish the WIP
>> patches a little bit and post them sometime this week.
> 
> Cool.
> 

regards

-- 
Tomas Vondra

Hi Jakub,

FYI I've posted my experimental NUMA patch series here:

https://www.postgresql.org/message-id/099b9433-2855-4f1b-b421-d078a5d82017%40vondra.me

I've considered posting it to this thread, but it seemed sufficiently
different to start a new thread.

regards

-- 
Tomas Vondra