pg_dump being killed by oom killer

Oct 29 12:07:14 www1 kernel: OOM killed process 0 (pg_dump) vm:5269904kB, rss:5210224kB, swap:0kB

The vm has 16 cpu's and is using around 10%, when I run pg_dump it goes up by 3 / 5 % so not using alot of cpu usage. IT has 30gb of ram and 1gb of swap, the system uses around 2mb of swap and 15gb of ram, when pg_dump runs the swap doesn't move but the ram goes up by 3 / 4 gb before the process is killed.

The machine had only 20gb of ram so I've increased the ram usage and still get the same issue. if I run pg_dump on any other db it works fine, if we run a pg_dump script by table on the failing database it works fine.

The backup is going directly to a nas device which has 1.2 tb free. The vm has aroudn 40gb free as well.

Any ideas what this could be? it seems to be a memory issue but I can't seem to see what / why it's happening.

oh the DB is around 1.5 gb / 2gb in size


Thanks
Paul