performance of foreign key constraints

I have a table that has about 20 foreign key constraints on it. I think
this is a bit excessive and am considering removing them ( they are all
related to the same table and I don't think there is much chance of any
integrity violations ). Would this improve performance or not?

thanks