Обсуждение: Postgresql 9.0.1 installation error

Is your client psql compiled similarly to your server? IE, are you using the (correctly-) corresponding client, preferably the one built with the server?

That you can connect AOK from a different bin is the clue, I should think.

Lou Picciano

---

From: "Venkat Balaji" <venkat.balaji@verse.in>
To: pgsql-admin@postgresql.org
Sent: Monday, August 8, 2011 7:11:55 AM
Subject: [ADMIN] Postgresql 9.0.1 installation error

Hello,

This is my first post to the community.

We have been widely using PostgreSQL for our production environment.

We are pleased with the performance and features of PostgreSQL.

We have landed into a problem recently -

We had installed PostgreSQL9.0.1 for which i had executed the below commands -

./configure --with-openssl --prefix=/opt/Postgresql9.0.1/

make install

installed successfully

Created the cluster using initdb command successfully

When i said "./psql -p 5445", I am getting the below error

[postgres@warehouse1 bin]$ ./psql -p 5445

LOG:  unexpected EOF on client connection

Segmentation fault

Note : -

1. Port Number was specified in the postgresql.conf file

2. The same version of PostgreSQL was installed in the same server in a different location and is working fine without any issues -

3. Postgres process is running fine. When i am connecting from a different PG installer bin directory, 

    I am able to connect.

Can someone please help !!

I need PostgreSQL 9.0.1 with Open-SSL.

Thanks

Nag

Hi Craig,

Thank you very much for your reply !

Here is my reply -

1. We are using RHEL with the below version

   

Linux version 2.6.18-194.26.1.el5 (mockbuild@x86-004.build.bos.redhat.com) (gcc version 4.1.2 20080704 (Red Hat 4.1.2-48)) #1 SMP Fri Oct 29 14:21:16 EDT 2010

2. LD_LIBRARY_PATH is not set

3. Yes, this seems to be a libpq settings problem.

     Below is the GDB result -

    [postgres@warehouse1 bin]$ ./psql -p 5444

LOG:  unexpected EOF on client connection

Segmentation fault

[postgres@warehouse1 bin]$ gdb --args ./psql -p 5444

GNU gdb (GDB) Red Hat Enterprise Linux (7.0.1-23.el5_5.2)

Copyright (C) 2009 Free Software Foundation, Inc.

License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>

This is free software: you are free to change and redistribute it.

There is NO WARRANTY, to the extent permitted by law.  Type "show copying"

and "show warranty" for details.

This GDB was configured as "x86_64-redhat-linux-gnu".

For bug reporting instructions, please see:

<http://www.gnu.org/software/gdb/bugs/>...

Reading symbols from /opt/Postgres9.0.1/bin/psql...(no debugging symbols found)...done.

(gdb) run

Starting program: /opt/Postgres9.0.1/bin/psql -p 5444

[Thread debugging using libthread_db enabled]

Program received signal SIGSEGV, Segmentation fault.

0x00002aaaaaac2ecc in resetPQExpBuffer () from /opt/Postgres9.0.1/lib/libpq.so.5

Thanks

Nag

On Tue, Aug 9, 2011 at 8:22 AM, Craig Ringer <ringerc@ringerc.id.au> wrote:

On 8/08/2011 7:11 PM, Venkat Balaji wrote:

We had installed PostgreSQL9.0.1 for which i had executed the below
commands -

./configure --with-openssl --prefix=/opt/Postgresql9.0.1/
make install
installed successfully

OK, you're building Pg from source rather than using packages provided by EDB or your operating system. Fair enough, albeit somewhat unusual for many production setups.

When i said "./psql -p 5445", I am getting the below error

The most likely cause is that the dynamic linker is using a different library than what Pg was compiled against for libssl, libpg, or something like that.

You have neglected to mention the platform you're using or any of the other important data, so it's hard to give any more detailed instructions about how to track the issue down. It'd help if you mentioned what OS you were using, whether you uninstalled/deleted any prior versions of Pg, what your LD_LIBRARY_PATH is, what `ldd' reports when run on psql, etc.

[postgres@warehouse1 bin]$ ./psql -p 5445
LOG:  unexpected EOF on client connection
Segmentation fault

Most likely you have linkage problems, but if not it'd help if you'd get a backtrace of that client crash. Assuming you're on a machine with gdb:

gdb --args ./psql -p 5445
(gdb) run
.... blah blah
.... terminated with SIGSEGV (Segmentation fault)
(gdb) bt

   2. The same version of PostgreSQL was installed in the     same server in a

different location and is working fine without any issues -
3. Postgres process is running fine. When i am connecting from a
different PG installer bin directory,
    I am able to connect.

It's *EXTREMELY* likely to be an issue where you're linking to a different libpq possibly compiled with different options, then.

--
Craig Ringer

On 9/08/2011 2:31 PM, Venkat Balaji wrote:

Hi Craig,

Thank you very much for your reply !

Here is my reply -

1. We are using RHEL with the below version
Linux version 2.6.18-194.26.1.el5
(mockbuild@x86-004.build.bos.redhat.com

<mailto:mockbuild@x86-004.build.bos.redhat.com>) (gcc version 4.1.2

20080704 (Red Hat 4.1.2-48)) #1 SMP Fri Oct 29 14:21:16 EDT 2010

2. LD_LIBRARY_PATH is not set

OK. In that case, please show the output of:

ldd ./psql

run in exactly the same environment as you'd run ./psql and get a crash.

Program received signal SIGSEGV, Segmentation fault.
0x00002aaaaaac2ecc in resetPQExpBuffer () from
/opt/Postgres9.0.1/lib/libpq.so.5

OK, you haven't built with debugging. Try rebuilding with --enable-debug to configure, then re-running the gdb command. When you get a crash and the above message is printed out, you'll get a prompt like:

(gdb)

Type "bt" at that prompt and press enter. The resulting stack trace shows more information about what led up to the crash.

Debugging memory related crashes in C isn't simple. Just because it crashes in libpq doesn't mean the fault is in libpq. It might be, or it might be that some other library is corrupting memory that causes libpq to crash later, corrupting the stack, returning a bad pointer from a function call, library headers not matching linked library sizes so returned struct sizes/offsets are wrong, and lots lots more. It's not simple.

--
Craig Ringer

On 10/08/2011 3:43 PM, Venkat Balaji wrote:

        libpq.so.5 => /opt/Postgres9.0.1/lib/libpq.so.5

OK, so it looks like it's runtime linking to the correct libpq, assuming that the copy in /opt/Postgres9.0.1/ is the one for ... well .. 9.0.1 . That was my first suspicion about what might be wrong, that it was being runtime linked to a different libpq from a different build.

Program received signal SIGSEGV, Segmentation fault.
0x00002aaaaaac3b8c in resetPQExpBuffer () from
/opt/Postgres9.0.1/lib/libpq.so.5
(gdb) bt\
#0  0x00002aaaaaac3b8c in resetPQExpBuffer () from
/opt/Postgres9.0.1/lib/libpq.so.5
#1  0x00002aaaaaabd610 in pqGets_internal () from
/opt/Postgres9.0.1/lib/libpq.so.5
#2  0x0000000000658d70 in ?? ()
#3  0x0000000000658d70 in ?? ()
#4  0x0000000000659188 in ?? ()
#5  0x0000000000000000 in ?? ()

While I'm no expert in the interpretation of wonky stack traces, I strongly suspect you have a corrupt stack here. Those addresses don't look right, and frame 5 is flat-out impossible AFAIK.

It's possible that the lack of debug symbols for other libraries and/or the fact that you didn't rebuild with --enable-debug is the cause, but I doubt it.

The trouble with stack corruption is finding the culprit. It's likely to be something specific to your system, like a custom-built library you installed in /usr/local that's being used for headers (includes) but not for the library, or where the library in /usr/local is being used for compile-time linking then a copy in /usr/lib is being used for runtime linking. Stuff like that is often the cause... but it can also be a genuine bug in libpq, psql, or one of the libraries that's only being triggered under some circumstance particular to your system or build.

The first thing to do is see if you can make this bug happen on another system that's configured as close to your current one as possible. If you can't, figure out what's different about them. If you can still reproduce it on a clean RHEL system, send the configure command, installed package list and any other information required to reproduce the bug to this mailing list.

Most likely you'll find that you can't reproduce it on a clean RHEL system and will find something like a dodgy old copy of a library somewhere, an installer that's overwritten a library in /usr/lib64 with its own incompatible version, or something fun like that.

Unfortunately this kind of thing can be very hard to debug remotely. It *might* help if you upload your config.log from your compile somewhere and send a link to this mailing list (do not attach the file!), but it might not land up telling me anything. Similarly, rebuilding Pg with --enable-debug, running "make install" and re-testing might get a better backtrace - or might be similarly useless.

--
Craig Ringer

Thanks Craig for your detailed explanation !

I had rebuilt the PG 9.0.1 with --enable-debug option.

As of now, i had downloaded a fresh package and re-installed. Its perfectly working fine for me.

I will further try to debug this issue. Will let you know what i find.

Thanks

Nag

On Wed, Aug 10, 2011 at 7:21 PM, Craig Ringer <ringerc@ringerc.id.au> wrote:

On 10/08/2011 3:43 PM, Venkat Balaji wrote:

        libpq.so.5 => /opt/Postgres9.0.1/lib/libpq.so.5

OK, so it looks like it's runtime linking to the correct libpq, assuming that the copy in /opt/Postgres9.0.1/ is the one for ... well .. 9.0.1 . That was my first suspicion about what might be wrong, that it was being runtime linked to a different libpq from a different build.

Program received signal SIGSEGV, Segmentation fault.
0x00002aaaaaac3b8c in resetPQExpBuffer () from
/opt/Postgres9.0.1/lib/libpq.so.5
(gdb) bt\
#0  0x00002aaaaaac3b8c in resetPQExpBuffer () from
/opt/Postgres9.0.1/lib/libpq.so.5
#1  0x00002aaaaaabd610 in pqGets_internal () from
/opt/Postgres9.0.1/lib/libpq.so.5
#2  0x0000000000658d70 in ?? ()
#3  0x0000000000658d70 in ?? ()
#4  0x0000000000659188 in ?? ()
#5  0x0000000000000000 in ?? ()

While I'm no expert in the interpretation of wonky stack traces, I strongly suspect you have a corrupt stack here. Those addresses don't look right, and frame 5 is flat-out impossible AFAIK.

It's possible that the lack of debug symbols for other libraries and/or the fact that you didn't rebuild with --enable-debug is the cause, but I doubt it.

The trouble with stack corruption is finding the culprit. It's likely to be something specific to your system, like a custom-built library you installed in /usr/local that's being used for headers (includes) but not for the library, or where the library in /usr/local is being used for compile-time linking then a copy in /usr/lib is being used for runtime linking. Stuff like that is often the cause... but it can also be a genuine bug in libpq, psql, or one of the libraries that's only being triggered under some circumstance particular to your system or build.

The first thing to do is see if you can make this bug happen on another system that's configured as close to your current one as possible. If you can't, figure out what's different about them. If you can still reproduce it on a clean RHEL system, send the configure command, installed package list and any other information required to reproduce the bug to this mailing list.

Most likely you'll find that you can't reproduce it on a clean RHEL system and will find something like a dodgy old copy of a library somewhere, an installer that's overwritten a library in /usr/lib64 with its own incompatible version, or something fun like that.

Unfortunately this kind of thing can be very hard to debug remotely. It *might* help if you upload your config.log from your compile somewhere and send a link to this mailing list (do not attach the file!), but it might not land up telling me anything. Similarly, rebuilding Pg with --enable-debug, running "make install" and re-testing might get a better backtrace - or might be similarly useless.

--
Craig Ringer