slow insert speeds with bytea

Is there anything I can to increase insert speeds for bytea? Currently running postgres 9.6.15

I have a few tables without a bytea and a few with bytea. There is a large performance difference with inserts between the two. I'm inserting a byte[] that's usually less than 1MB on content. The content itself is actually just utf8 string data.

For the non-bytea table, inserts can be as high as 40k rows/sec, whereas the bytea table is closer to 4k/sec or less.

If this is just a limitation of postgres, then that's fine but the performance delta is so significant that i feel like i'm missing something