Getting "timeout expired" error almost immediately (20-200ms)

I'm running 8.0.3 on Linux 2.6. Once my application starts to get a bit
of load, I start getting application exceptions with the "timeout
expired" string from postgresql. I think it is coming from
src/interfaces/libpq/fe-misc.c. There is an interesting comment in that
function (hasn't changed since 8.0.3 I believe):
/*
* pqWaitTimed: wait, but not past finish_time.
*
* If finish_time is exceeded then we return failure (EOF).  This is
like
* the response for a kernel exception because we don't want the caller
* to try to read/write in that case.
*
* finish_time = ((time_t) -1) disables the wait limit.
*/
int
pqWaitTimed(int forRead, int forWrite, PGconn *conn, time_t
finish_time)
{
        int                     result;

        result = pqSocketCheck(conn, forRead, forWrite, finish_time);

        if (result < 0)
                return EOF; /* errorMessage is already set */

        if (result == 0)
        {
                printfPQExpBuffer(&conn->errorMessage,
libpq_gettext("timeout expired\n"));
                return EOF;
        }

        return 0;
}

This is an example log message from my application code: (timeout after
203ms in this case)
06-01 02:13:21 XCP   12 INFO   dbcursor        :452 EXEC 203ms (203ms)
SELECT uid,optout FROM f_create_user(%s, %s, %s, %s, %s) ['support', 1,
None, False, False]; OperationalError: timeout expired [dbpool.py:43
execute()]

I just don't know how to get past this when I'm under a bit of load. Is
it a postgresql thing? A linux thing? What should I try next?