Too easy to log in as the "postgres" user?

I've noticed that if I just log in to my server, I don't su to root,
or become the postgres user, I can get straight into the database as
the postgres user merely with "psql -U postgres -h localhost".  My
user account isn't a member of the postgres group.

It appears I've not applied my security settings correctly.  What can
I do to prevent access this way?  I'd still want to be able to su to
the postgres user and log in that way, but not with the -U parameter
allowing access.

The pg_hba.conf is probably relevant here, so this is the setup:

# TYPE  DATABASE    USER        CIDR-ADDRESS          METHOD

# "local" is for Unix domain socket connections only
local   all         all                               trust
# IPv4 local connections:
host    all         all         127.0.0.1/32          trust
# IPv6 local connections:
host    all         all         ::1/128               trust

Thanks

Thom Brown
Crawley, UK