Обсуждение: Socket Timeouts and fatal errors (please help)

Поиск
Список
Период
Сортировка

Socket Timeouts and fatal errors (please help)

От
"Doug Lochart"
Дата:
First let me divulge the pertinent info:

database server:
-----------------------------
Linux 2.6.8.1
postgres 8.1.0

webserver:
-------------------
Linux 2.6.8.1
Apache 2.0.53, mod_jk, tomcat 5.0.28
jdbc-driver 8.1-404.jdbc3
libpq.so.4.1

We have Java client objects using jdbc, some JNI C code using libpq
and some standalone c applications using libpq.
Both Java and C apps are using connection pools.  we have our max
connections set to 100 in postgres.  Our max pool size is 20 in the
java realm and 10 in the C side.

Our production server is not used heavily at the moment.  What we are
experiencing is the following:

(java side)
Sporadic (once or twice a month) Socket timeout exceptions like this:
The call in question is simply doing some data cleanup when a user
session expires.  Don't the the word 'Lock' worry you ... its all
data.  This call failed on a query.

org.postgresql.util.PSQLException: An I/O error occured while sending
to the backend.
        at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:214)
        at org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:430)
        at org.postgresql.jdbc2.AbstractJdbc2Statement.executeWithFlags(AbstractJdbc2Statement.java:332)
        at org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:324)
        at com.capecomputing.mensana.dataaccess.MensanaConnection.getConnection(MensanaConnection.java:116)
        at com.capecomputing.mensana.dataaccess.MensanaConnection.<init>(MensanaConnection.java:83)
        at
com.capecomputing.mensana.dataaccess.DatabaseUpdateAction.getEncryptedParameters(DatabaseUpdateAction.java:65)
        at
com.capecomputing.mensana.dataaccess.DatabaseUpdateAction.createEncryptedParamterList(DatabaseUpdateAction.java:39)
        at
com.capecomputing.mensana.dataaccess.CleanupLockedPrepaidClientInvoicesAction.<init>(CleanupLockedPrepaidClientInvoicesAction.java:33)
        at
com.capecomputing.mensana.dataaccess.MensanaDataAccess.cleanupLockedPrepaidInvoices(MensanaDataAccess.java:1685)
        at
com.capecomputing.mensana.webportal.application.SessionEventListener.cleanupLockedPrepaidInvoices(SessionEventListener.java:62)
        ...
Caused by: java.net.SocketException: Connection timed out
        at java.net.SocketInputStream.socketRead0(Native Method)
        at java.net.SocketInputStream.read(SocketInputStream.java:129)
        at java.io.BufferedInputStream.fill(BufferedInputStream.java:183)
        at java.io.BufferedInputStream.read(BufferedInputStream.java:201)
        at org.postgresql.core.PGStream.ReceiveChar(PGStream.java:254)
        at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1163)
        at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:188)
        ... 49 more


On the JNI C side the error was like this:  We retrieved a connection
from the pool.  Checked its status it was good.  Performed a query an
checked the result status and all that was returned was a fatal error.
 The JNI C side tried another connection from the pool and made
another query, this one caused a tomcat crash (the query dumped this
hs_err file)

An unexpected exception has been detected in native code outside the VM.
Unexpected Signal : 11 occurred at PC=0x400EA4D3
Function=strlen+0x33
Library=/lib/libc.so.6
...

The last message in the log was "about to call PQexec()"

Also in the standalone C application we experienced the same fatal
error message.

One of the days when we noticed the java socket timeouts we had a
power hit.  We chalked it up to that but the last one was not.  This
never happens in our non prod environments ... probably because we are
always bouncing the apps at least once a week for tweaks and tests.

Does anyone have any ideas?

I will post a sperate issue/question that may or may not be related to
the issues I stated here.  It will be titled pgadmin3 and tomcat
hangs.

thanks in advance

Doug







--
What profits a man if he gains the whole world yet loses his soul?

Re: Socket Timeouts and fatal errors (please help)

От
Koen Martens
Дата:
Doug Lochart wrote:
> Our production server is not used heavily at the moment.  What we are
> experiencing is the following:
>
> (java side)
> Sporadic (once or twice a month) Socket timeout exceptions like this:
> The call in question is simply doing some data cleanup when a user
> session expires.  Don't the the word 'Lock' worry you ... its all
> data.  This call failed on a query.
>
> org.postgresql.util.PSQLException: An I/O error occured while sending
> to the backend.
>         at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:214)
>         at org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:430)
>         at org.postgresql.jdbc2.AbstractJdbc2Statement.executeWithFlags(AbstractJdbc2Statement.java:332)
>         at org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:324)
>         at com.capecomputing.mensana.dataaccess.MensanaConnection.getConnection(MensanaConnection.java:116)
>         at com.capecomputing.mensana.dataaccess.MensanaConnection.<init>(MensanaConnection.java:83)
>         at
com.capecomputing.mensana.dataaccess.DatabaseUpdateAction.getEncryptedParameters(DatabaseUpdateAction.java:65)
>         at
com.capecomputing.mensana.dataaccess.DatabaseUpdateAction.createEncryptedParamterList(DatabaseUpdateAction.java:39)
>         at
com.capecomputing.mensana.dataaccess.CleanupLockedPrepaidClientInvoicesAction.<init>(CleanupLockedPrepaidClientInvoicesAction.java:33)
>         at
com.capecomputing.mensana.dataaccess.MensanaDataAccess.cleanupLockedPrepaidInvoices(MensanaDataAccess.java:1685)
>         at
com.capecomputing.mensana.webportal.application.SessionEventListener.cleanupLockedPrepaidInvoices(SessionEventListener.java:62)
>         ...
> Caused by: java.net.SocketException: Connection timed out
>         at java.net.SocketInputStream.socketRead0(Native Method)
>         at java.net.SocketInputStream.read(SocketInputStream.java:129)
>         at java.io.BufferedInputStream.fill(BufferedInputStream.java:183)
>         at java.io.BufferedInputStream.read(BufferedInputStream.java:201)
>         at org.postgresql.core.PGStream.ReceiveChar(PGStream.java:254)
>         at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1163)
>         at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:188)
>         ... 49 more

Don't know if it helps, but i had similar problems a while back.
Turned out it was caused by a misconfigured firewall reload
(flushing all the firewall rules followed by re-inserting them)
which caused outstanding network connections to be interrupted.
Doing firewall reload the proper way (load new rules in inactive
table, swap active & inactive) fixed it for me.

> An unexpected exception has been detected in native code outside the VM.
> Unexpected Signal : 11 occurred at PC=0x400EA4D3
> Function=strlen+0x33
> Library=/lib/libc.so.6
> ...

And this I have seen on occasions where the compiled libc was out of
sync with the kernel version used, but might be anything..

Best,

Koen

--
K.F.J. Martens, Sonologic, http://www.sonologic.nl/
Networking, hosting, embedded systems, unix, artificial intelligence.
Public PGP key: http://www.metro.cx/pubkey-gmc.asc
Wondering about the funny attachment your mail program
can't read? Visit http://www.openpgp.org/