Re: Benchmarking a large server

> How many times was the kernel tested with this much memory, for example
> ? (never??)

This is actually *extremely* relevant.

Take a look at /proc/sys/vm/dirty_ratio and /proc/sys/vm/dirty_background_ratio if you have an older Linux system, or
/proc/sys/vm/dirty_bytes,and /proc/sys/vm/dirty_background_bytes with a newer one. 

On older systems for instance, those are set to 40 and 20 respectively (recent kernels cut these in half). That's
significantbecause ratio is the *percentage* of memory that can remain dirty before causing async, and background_ratio
tellsit when it should start writing in the background to avoid hitting that higher and much more disruptive number.
Thisis another source of IO that can be completely independent of the checkpoint spikes that long plagued PostgreSQL
versionsprior to 8.3. 

With that much memory (1TB!), that's over 100GB of dirty memory before it starts writing that out to disk even with the
newerconservative settings. We had to tweak and test for days to find good settings for these, and our servers only
have96GB of RAM. You also have to consider, as fast as the FusionIO drives are, they're still NVRAM, which has
write-amplificationissues. How fast do you think it can commit 100GB of dirty memory to disk? Even with a background
settingof 1%, that's 10GB on your system. 

That means you'd need to use a very new kernel so you can utilize the dirty_bytes and dirty_background_bytes settings
soyou can force those settings into more sane levels to avoid unpredictable several-minute long asyncs. I'm not sure
howmuch testing Linux sees on massive hardware like that, but that's just one hidden danger of not properly
benchmarkingthe server and just thinking 1TB of memory and caching the entire dataset is only an improvement. 

--
Shaun Thomas
Peak6 | 141 W. Jackson Blvd. | Suite 800 | Chicago, IL 60604
312-676-8870
sthomas@peak6.com

______________________________________________

See  http://www.peak6.com/email_disclaimer.php
for terms and conditions related to this email