Обсуждение: Make broken database start

Поиск
Список
Период
Сортировка

Make broken database start

От
Дмитрий Шалашов
Дата:
Hi everyone,

(sorry if this is a dup - I'm not sure if my previous message get through)

I need an advice in database recovery. Postgresql 9.2.9.
Last night we moved one of our databases to another server following this procedure: https://wiki.postgresql.org/wiki/Streaming_Replication
Replication, slave becomes master, pgbouncer restarts with new config, everyone happy.
Until soon we noticed strange errors. Turns out data was broken somewhere - we had unique indexes with duplicate records, we had missing chunks in toast tables, and the like.
Scale of the problems wasn't clear and I fixed some of the most obvious problems - recreated indices, cleared duplicates. Around 0:30 I started database-wide VACUUM, it started printing messages about fixing things. Everything seemed to work normally.
Around 3:30 database stopped responding to queries; it was noticed around 9:30 and I attempted to stop VACUUM (ctrl+c in psql command line). 30 minutes and nothing. I decided to restart postmaster. After another 30 minutes we switched back to old server. Already more than 6 hours passed and new server postmaster is still shutting down. And I would like to get it back to restore yesterday data.

At this point I could do with new server anything that will get database up and running sooner. kill -9, start, wait for recovery? Server reboot?

Any help will be greatly appreciated.

Best regards,
Dmitriy Shalashov

Чтобы сделать работу с сайтом удобнее, мы используем cookie и аналитический сервис «Яндекс.Метрика». Продолжая пользоваться сайтом, вы соглашаетесь с их использованием.