Обсуждение: [ADMIN] Database corrupted - PSQL 9.3 Ubuntu Server 14.04

Hi,
I have a big problem with one of my Postgresql server.
I started a standard package update and during the Postgresql update,
something happen that it make it fails. Since when, postgresql isn't
able to start.
In the log, here is what it is showed:

2016-12-15 08:40:03 EST FATAL:  "/var/lib/postgresql/9.3/main" is not a
valid data directory
2016-12-15 08:40:03 EST DETAIL:  File
"/var/lib/postgresql/9.3/main/PG_VERSION" is missing.

And if I go further, there is a reccuring message:

2016-12-14 22:01:14 EST FATAL:  could not open relation mapping file
"global/pg_filenode.map": No such file or directory
2016-12-14 22:01:15 EST ERROR:  could not open file "base/305852/11848":
No such file or directory
2016-12-14 22:01:15 EST CONTEXT:  writing block 0 of relation
base/305852/11848
2016-12-14 22:01:15 EST WARNING:  could not write block 0 of
base/305852/11848
2016-12-14 22:01:15 EST DETAIL:  Multiple failures --- write error might
be permanent.


I notice, when comparing with a similare server, that 3 db data in the
main/base folder are empty, I suppose it's the postgres, template0 et
template1 database.

I try pg_resetxlog on a copy of the data dir, the server came up but the
"postgres" user isn't a valide role...

I had a cron which make dump backup each day but there was un bug in the
script that make it right, so I know we should not rely on recovery, but
by now, It s really my only hope.

Could someone help me?

Sebastien


OS: Ubuntu Server 14.04 64bit
Postgresql: 9.3

On 15/12/2016 17:07, Sébastien Beausoleil wrote:
> Hi,
> I have a big problem with one of my Postgresql server.
> I started a standard package update and during the Postgresql update, something happen that it make it fails. Since
when,postgresql isn't able to start. 
> In the log, here is what it is showed:
What postgresql update ?
>
> 2016-12-15 08:40:03 EST FATAL:  "/var/lib/postgresql/9.3/main" is not a valid data directory
> 2016-12-15 08:40:03 EST DETAIL:  File "/var/lib/postgresql/9.3/main/PG_VERSION" is missing.
>
> And if I go further, there is a reccuring message:
>
> 2016-12-14 22:01:14 EST FATAL:  could not open relation mapping file "global/pg_filenode.map": No such file or
directory
> 2016-12-14 22:01:15 EST ERROR:  could not open file "base/305852/11848": No such file or directory
> 2016-12-14 22:01:15 EST CONTEXT:  writing block 0 of relation base/305852/11848
> 2016-12-14 22:01:15 EST WARNING:  could not write block 0 of base/305852/11848
> 2016-12-14 22:01:15 EST DETAIL:  Multiple failures --- write error might be permanent.
>
>
> I notice, when comparing with a similare server, that 3 db data in the main/base folder are empty, I suppose it's the
postgres,template0 et template1 database. 
That's pretty bad. Can you locate them anywhere else in your file system?
>
> I try pg_resetxlog on a copy of the data dir, the server came up but the "postgres" user isn't a valide role...
>
> I had a cron which make dump backup each day but there was un bug in the script that make it right, so I know we
shouldnot rely on recovery, but by now, It s really my only hope. 
Do you take any file system backup by pg_start_backup/pg_stop_backup ? or pg_basebackup? or WAL archiving?
>
> Could someone help me?
>
> Sebastien
>
>
> OS: Ubuntu Server 14.04 64bit
> Postgresql: 9.3
>
>
>
>
>


--
Achilleas Mantzios
IT DEV Lead
IT DEPT
Dynacom Tankers Mgmt

Thank you for answering me so fast Achilleas!


What postgresql update ?

    postgresql-9.3:amd64 (9.3.10-0ubuntu0.14.04, 9.3.15-0ubuntu0.14.04)

I have gone through this as well (the manual way) and didn't face any problem at all.

That's pretty bad. Can you locate them anywhere else in your file system?
   
    What you mean? I saw the folder but they are empty.

You might try smth like :
# locate PG_VERSION
and see if the ubuntu upgrade tried at least to salvage some data.

Do you take any file system backup by pg_start_backup/pg_stop_backup ? or pg_basebackup? or WAL archiving?

    I had setup un dump backup each day in the postgres cron (I thought it worked...), but it happens that a bug in the script make it not work...

Le 2016-12-15 à 10:31, Achilleas Mantzios a écrit :

On 15/12/2016 17:07, Sébastien Beausoleil wrote:

Hi,
I have a big problem with one of my Postgresql server.
I started a standard package update and during the Postgresql update, something happen that it make it fails. Since when, postgresql isn't able to start.
In the log, here is what it is showed:

What postgresql update ?

2016-12-15 08:40:03 EST FATAL:  "/var/lib/postgresql/9.3/main" is not a valid data directory
2016-12-15 08:40:03 EST DETAIL:  File "/var/lib/postgresql/9.3/main/PG_VERSION" is missing.

And if I go further, there is a reccuring message:

2016-12-14 22:01:14 EST FATAL:  could not open relation mapping file "global/pg_filenode.map": No such file or directory
2016-12-14 22:01:15 EST ERROR:  could not open file "base/305852/11848": No such file or directory
2016-12-14 22:01:15 EST CONTEXT:  writing block 0 of relation base/305852/11848
2016-12-14 22:01:15 EST WARNING:  could not write block 0 of base/305852/11848
2016-12-14 22:01:15 EST DETAIL:  Multiple failures --- write error might be permanent.


I notice, when comparing with a similare server, that 3 db data in the main/base folder are empty, I suppose it's the postgres, template0 et template1 database.

That's pretty bad. Can you locate them anywhere else in your file system?

I try pg_resetxlog on a copy of the data dir, the server came up but the "postgres" user isn't a valide role...

I had a cron which make dump backup each day but there was un bug in the script that make it right, so I know we should not rely on recovery, but by now, It s really my only hope.

Do you take any file system backup by pg_start_backup/pg_stop_backup ? or pg_basebackup? or WAL archiving?

Could someone help me?

Sebastien


OS: Ubuntu Server 14.04 64bit
Postgresql: 9.3

--
 

Sébastien Beausoleil Téléphone (Joliette):     450.559.NUBI (6824) Téléphone (Montréal):    514.447.4087 10, 7e avenue St-Ambroise-de-Kildare (Québec) J0K 1C0

-- 
Achilleas Mantzios
IT DEV Lead
IT DEPT
Dynacom Tankers Mgmt