BUG #9635: Wal sender process is using 100% CPU

The following bug has been logged on the website:

Bug reference:      9635
Logged by:          Jamie Koceniak
Email address:      jkoceniak@mediamath.com
PostgreSQL version: 9.1.9
Operating system:   x86_64-unknown-linux-gnu (Debian 4.7.2-5) 64-bit
Description:

Periodically throughout the day, we keep seeing the wal sender process
utilize 100% of the CPU. We began noticing this after we added 2 new slave
servers, going from 2 to 4 slaves. See top results and I also included our
wal settings. Thanks!

top - 05:03:18 up 174 days,  4:51,  2 users,  load average: 5.57, 4.75,
3.16
Tasks: 387 total,   8 running, 379 sleeping,   0 stopped,   0 zombie
%Cpu(s): 29.3 us,  4.7 sy,  0.0 ni, 65.3 id,  0.4 wa,  0.0 hi,  0.4 si,  0.0
st
MiB Mem:    290797 total,   218532 used,    72264 free,      311 buffers
MiB Swap:     7812 total,        1 used,     7811 free,   206978 cached

  PID USER      PR  NI  VIRT  RES  SHR S  %CPU %MEM    TIME+  COMMAND

30244 postgres  20   0 8497m 5608 2820 R   100  0.0   1:44.72 postgres: wal
sen
14447 postgres  20   0 8497m 5596 2816 R   100  0.0   3:11.27 postgres: wal
sen
16075 postgres  20   0 8497m 5600 2820 R   100  0.0   3:32.32 postgres: wal
sen
 8177 postgres  20   0 8497m 5360 2820 S    36  0.0   0:03.35 postgres: wal
sen
 4920 postgres  20   0 9647m 9.3g 8.1g S     3  3.3   1097:40 postgres:
writer
 4923 postgres  20   0 68872 2072  788 S     3  0.0 511:01.76 postgres:
archive
 4921 postgres  20   0 8496m  18m  17m S     2  0.0 593:36.38 postgres: wal
wri
 7853 root      20   0 23432 1836 1176 R     1  0.0   0:00.44 top

 4916 postgres  20   0 8492m 229m 228m S     0  0.1 598:44.57
/usr/lib/postgres


Current Wal settings:

             name             |   setting   |
------------------------------+-------------+
 max_wal_senders              | 10          |
 wal_block_size               | 8192        |
 wal_buffers                  | 2048        |
 wal_keep_segments            | 5000        |
 wal_level                    | hot_standby |
 wal_receiver_status_interval | 10          |
 wal_segment_size             | 2048        |
 wal_sender_delay             | 1000        |
 wal_sync_method              | fdatasync   |
 wal_writer_delay             | 200         |