Обсуждение: postgres server crash with "Segmentation fault"

 

---

The information transmitted herein is intended only for the person or entity to which it is addressed and may contain confidential, proprietary and/or privileged material. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon, this information by persons or entities other than the intended recipient is prohibited. If you received this in error, please contact the sender and delete the material from any computer.  

On Thu, 2025-10-16 at 11:07 +0000, Ishan Arunkumar Joshi wrote:
> We are using PG16.9 in Patroni Postgres setup in production. Last night we have face
> an issue [various data corruption errors]
>
> We had to drop the table “impacted_table” from database once the database is up.
>
> however we are not able to identify exact root cause behind “segmentation fault”
> error for this table and need expert advice to find the root case and also need
> suggestions to prevention steps.

By dropping the table, you have probably destroyed the evidence needed for that.
If you have a file system backup of the corrupted state, an expert might be able
to identify probable causes.

One prevention step would have been to run the latest minor release (currently 16.10).

Other than that, make sure that you have a good backup that is occasionally tested
and make sure that the backup is monitored (I have seen cases where the backup was a
daily pg_dump, and only when the corruption surfaced, people realized that the pg_dump
had been failing for the exact same reason...).

Yours,
Laurenz Albe

 

---

The information transmitted herein is intended only for the person or entity to which it is addressed and may contain confidential, proprietary and/or privileged material. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon, this information by persons or entities other than the intended recipient is prohibited. If you received this in error, please contact the sender and delete the material from any computer.  

Hi Laurenz,

>By dropping the table, you have probably destroyed the evidence needed for that.
>If you have a file system backup of the corrupted state, an expert might be able to identify probable causes.

>One prevention step would have been to run the latest minor release (currently 16.10).

>Other than that, make sure that you have a good backup that is occasionally tested and make sure that the backup is
monitored(I have seen cases >where the backup was a daily pg_dump, and only when the corruption surfaced, people
realizedthat the pg_dump had been failing for the exact same >reason...).
 

Thanks for your reply

We are using pgbackrest backup and incremental backup was taken it before. We have very large database (>25TB) and
currentlywe do not have another server having same capacity to restore. However I am suspecting the corruption on the
tablewas happen during some operation happening runtime which triggers the autovacuum and because the corruption happen
alreadyright before the restart, when autovacuum got lock to execute, it is getting segmentation fault and restart the
postgresservice.
 

I tried to execute select query on that same table but it failed with same error and hence I did not take risk of
executingpg_dump as it may restart db server and impact the production and because it is temp table populating for some
process,we took decision to drop the table and recreate.
 

We are still clueless how the table got corrupted and segmentation fault occurs.

I gone through old thread where it suggest to disable JIT but the same solution not worked  hence I am not considering
itas root cause and solution.
 

Thanks & Regards,
-------------------------
Ishan Joshi


-----Original Message-----
From: Laurenz Albe <laurenz.albe@cybertec.at>
Sent: Friday, October 17, 2025 12:20 PM
To: Ishan Arunkumar Joshi <ishan.joshi@netcracker.com>; pgsql-admin@postgresql.org
Subject: Re: postgres server crash with "Segmentation fault"

[External Email]
________________________________



On Thu, 2025-10-16 at 11:07 +0000, Ishan Arunkumar Joshi wrote:
> We are using PG16.9 in Patroni Postgres setup in production. Last
> night we have face an issue [various data corruption errors]
>
> We had to drop the table “impacted_table” from database once the database is up.
>
> however we are not able to identify exact root cause behind “segmentation fault”
> error for this table and need expert advice to find the root case and
> also need suggestions to prevention steps.

By dropping the table, you have probably destroyed the evidence needed for that.
If you have a file system backup of the corrupted state, an expert might be able to identify probable causes.

One prevention step would have been to run the latest minor release (currently 16.10).

Other than that, make sure that you have a good backup that is occasionally tested and make sure that the backup is
monitored(I have seen cases where the backup was a daily pg_dump, and only when the corruption surfaced, people
realizedthat the pg_dump had been failing for the exact same reason...).
 

Yours,
Laurenz Albe


________________________________
The information transmitted herein is intended only for the person or entity to which it is addressed and may contain
confidential,proprietary and/or privileged material. Any review, retransmission, dissemination or other use of, or
takingof any action in reliance upon, this information by persons or entities other than the intended recipient is
prohibited.If you received this in error, please contact the sender and delete the material from any computer.
 

Hi Team,

Do we have any more suggestions or way forward for further investigation. Please suggest the same.

Thanks & Regards,
-------------------------
Ishan Joshi

-----Original Message-----
From: Ishan Arunkumar Joshi
Sent: Friday, October 17, 2025 4:45 PM
To: 'Laurenz Albe' <laurenz.albe@cybertec.at>; pgsql-admin@postgresql.org
Subject: RE: postgres server crash with "Segmentation fault"

Hi Laurenz,

>By dropping the table, you have probably destroyed the evidence needed for that.
>If you have a file system backup of the corrupted state, an expert might be able to identify probable causes.

>One prevention step would have been to run the latest minor release (currently 16.10).

>Other than that, make sure that you have a good backup that is occasionally tested and make sure that the backup is
monitored(I have seen cases >where the backup was a daily pg_dump, and only when the corruption surfaced, people
realizedthat the pg_dump had been failing for the exact same >reason...).
 

Thanks for your reply

We are using pgbackrest backup and incremental backup was taken it before. We have very large database (>25TB) and
currentlywe do not have another server having same capacity to restore. However I am suspecting the corruption on the
tablewas happen during some operation happening runtime which triggers the autovacuum and because the corruption happen
alreadyright before the restart, when autovacuum got lock to execute, it is getting segmentation fault and restart the
postgresservice.
 

I tried to execute select query on that same table but it failed with same error and hence I did not take risk of
executingpg_dump as it may restart db server and impact the production and because it is temp table populating for some
process,we took decision to drop the table and recreate.
 

We are still clueless how the table got corrupted and segmentation fault occurs.

I gone through old thread where it suggest to disable JIT but the same solution not worked  hence I am not considering
itas root cause and solution.
 

Thanks & Regards,
-------------------------
Ishan Joshi


-----Original Message-----
From: Laurenz Albe <laurenz.albe@cybertec.at>
Sent: Friday, October 17, 2025 12:20 PM
To: Ishan Arunkumar Joshi <ishan.joshi@netcracker.com>; pgsql-admin@postgresql.org
Subject: Re: postgres server crash with "Segmentation fault"

[External Email]
________________________________



On Thu, 2025-10-16 at 11:07 +0000, Ishan Arunkumar Joshi wrote:
> We are using PG16.9 in Patroni Postgres setup in production. Last
> night we have face an issue [various data corruption errors]
>
> We had to drop the table “impacted_table” from database once the database is up.
>
> however we are not able to identify exact root cause behind “segmentation fault”
> error for this table and need expert advice to find the root case and
> also need suggestions to prevention steps.

By dropping the table, you have probably destroyed the evidence needed for that.
If you have a file system backup of the corrupted state, an expert might be able to identify probable causes.

One prevention step would have been to run the latest minor release (currently 16.10).

Other than that, make sure that you have a good backup that is occasionally tested and make sure that the backup is
monitored(I have seen cases where the backup was a daily pg_dump, and only when the corruption surfaced, people
realizedthat the pg_dump had been failing for the exact same reason...).
 

Yours,
Laurenz Albe


________________________________
The information transmitted herein is intended only for the person or entity to which it is addressed and may contain
confidential,proprietary and/or privileged material. Any review, retransmission, dissemination or other use of, or
takingof any action in reliance upon, this information by persons or entities other than the intended recipient is
prohibited.If you received this in error, please contact the sender and delete the material from any computer.