Re: why does swap not recover?

On 3/26/10 4:57 PM, Richard Yen wrote:
> Hi everyone,
>
> We've recently encountered some swapping issues on our CentOS 64GB Nehalem machine, running postgres 8.4.2.
Unfortunately,I was foolish enough to set shared_buffers to 40GB.  I was wondering if anyone would have any insight
intowhy the swapping suddenly starts, but never recovers? 
>
> <img src="http://richyen.com/i/swap.png">
>
> Note, the machine has been up and running since mid-December 2009.  It was only a March 8 that this swapping began,
andit's never recovered. 
>
> If we look at dstat, we find the following:
>
> <img src="http://richyen.com/i/dstat.png">
>
> Note that it is constantly paging in, but never paging out.

This happens when you have too many processes using too much space to fit in real memory, but none of them are changing
theirmemory image.  If the system swaps a process in, but that process doesn't change anything in memory, then there
areno dirty pages and the kernel can just kick the process out of memory without writing anything back to the swap disk
--the data in the swap are still valid. 

It's a classic problem when processes are running round-robin. Say you have space for 100 processes, but you're running
101process.  When you get to the #101, #1 is the oldest so it swaps out.  Then #1 runs, and #2 is the oldest, so it
getskicked out.  Then #2 runs and kicks out #3 ... and so forth.  Going from 100 to 101 process brings the system
nearlyto a halt. 

Some operating systems try to use tricks to keep this from happening, but it's a hard problem to solve.

Craig