I can't disagree with your conclusion but I can offer a bit of perspective
of how the current situation came about.
Invalid indexes are in the same state they're in while a concurrent index
build is in progress. As far as other queries are concerned they're
effectively assuming the index build is still in progress and will still
eventually be completed.
They could maybe determine that's not the case but then that would be an
extra check for them to do in the normal case so not necessarily a win.
The real solution imho is to actually clean up failed index builds when a
build fails. That's what normal transactions do when they abort after all.
This was always the intention but looked like it was going to be a pain and
was put off (ie I was lazy). It's probably just several layers of
PG_TRY/PG_CATCH and closing the failed transactions and opening new ones.