Обсуждение: Socket Timeouts and fatal errors (please help)
First let me divulge the pertinent info: database server: ----------------------------- Linux 2.6.8.1 postgres 8.1.0 webserver: ------------------- Linux 2.6.8.1 Apache 2.0.53, mod_jk, tomcat 5.0.28 jdbc-driver 8.1-404.jdbc3 libpq.so.4.1 We have Java client objects using jdbc, some JNI C code using libpq and some standalone c applications using libpq. Both Java and C apps are using connection pools. we have our max connections set to 100 in postgres. Our max pool size is 20 in the java realm and 10 in the C side. Our production server is not used heavily at the moment. What we are experiencing is the following: (java side) Sporadic (once or twice a month) Socket timeout exceptions like this: The call in question is simply doing some data cleanup when a user session expires. Don't the the word 'Lock' worry you ... its all data. This call failed on a query. org.postgresql.util.PSQLException: An I/O error occured while sending to the backend. at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:214) at org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:430) at org.postgresql.jdbc2.AbstractJdbc2Statement.executeWithFlags(AbstractJdbc2Statement.java:332) at org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:324) at com.capecomputing.mensana.dataaccess.MensanaConnection.getConnection(MensanaConnection.java:116) at com.capecomputing.mensana.dataaccess.MensanaConnection.<init>(MensanaConnection.java:83) at com.capecomputing.mensana.dataaccess.DatabaseUpdateAction.getEncryptedParameters(DatabaseUpdateAction.java:65) at com.capecomputing.mensana.dataaccess.DatabaseUpdateAction.createEncryptedParamterList(DatabaseUpdateAction.java:39) at com.capecomputing.mensana.dataaccess.CleanupLockedPrepaidClientInvoicesAction.<init>(CleanupLockedPrepaidClientInvoicesAction.java:33) at com.capecomputing.mensana.dataaccess.MensanaDataAccess.cleanupLockedPrepaidInvoices(MensanaDataAccess.java:1685) at com.capecomputing.mensana.webportal.application.SessionEventListener.cleanupLockedPrepaidInvoices(SessionEventListener.java:62) ... Caused by: java.net.SocketException: Connection timed out at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.read(SocketInputStream.java:129) at java.io.BufferedInputStream.fill(BufferedInputStream.java:183) at java.io.BufferedInputStream.read(BufferedInputStream.java:201) at org.postgresql.core.PGStream.ReceiveChar(PGStream.java:254) at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1163) at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:188) ... 49 more On the JNI C side the error was like this: We retrieved a connection from the pool. Checked its status it was good. Performed a query an checked the result status and all that was returned was a fatal error. The JNI C side tried another connection from the pool and made another query, this one caused a tomcat crash (the query dumped this hs_err file) An unexpected exception has been detected in native code outside the VM. Unexpected Signal : 11 occurred at PC=0x400EA4D3 Function=strlen+0x33 Library=/lib/libc.so.6 ... The last message in the log was "about to call PQexec()" Also in the standalone C application we experienced the same fatal error message. One of the days when we noticed the java socket timeouts we had a power hit. We chalked it up to that but the last one was not. This never happens in our non prod environments ... probably because we are always bouncing the apps at least once a week for tweaks and tests. Does anyone have any ideas? I will post a sperate issue/question that may or may not be related to the issues I stated here. It will be titled pgadmin3 and tomcat hangs. thanks in advance Doug -- What profits a man if he gains the whole world yet loses his soul?
Doug Lochart wrote: > Our production server is not used heavily at the moment. What we are > experiencing is the following: > > (java side) > Sporadic (once or twice a month) Socket timeout exceptions like this: > The call in question is simply doing some data cleanup when a user > session expires. Don't the the word 'Lock' worry you ... its all > data. This call failed on a query. > > org.postgresql.util.PSQLException: An I/O error occured while sending > to the backend. > at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:214) > at org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:430) > at org.postgresql.jdbc2.AbstractJdbc2Statement.executeWithFlags(AbstractJdbc2Statement.java:332) > at org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:324) > at com.capecomputing.mensana.dataaccess.MensanaConnection.getConnection(MensanaConnection.java:116) > at com.capecomputing.mensana.dataaccess.MensanaConnection.<init>(MensanaConnection.java:83) > at com.capecomputing.mensana.dataaccess.DatabaseUpdateAction.getEncryptedParameters(DatabaseUpdateAction.java:65) > at com.capecomputing.mensana.dataaccess.DatabaseUpdateAction.createEncryptedParamterList(DatabaseUpdateAction.java:39) > at com.capecomputing.mensana.dataaccess.CleanupLockedPrepaidClientInvoicesAction.<init>(CleanupLockedPrepaidClientInvoicesAction.java:33) > at com.capecomputing.mensana.dataaccess.MensanaDataAccess.cleanupLockedPrepaidInvoices(MensanaDataAccess.java:1685) > at com.capecomputing.mensana.webportal.application.SessionEventListener.cleanupLockedPrepaidInvoices(SessionEventListener.java:62) > ... > Caused by: java.net.SocketException: Connection timed out > at java.net.SocketInputStream.socketRead0(Native Method) > at java.net.SocketInputStream.read(SocketInputStream.java:129) > at java.io.BufferedInputStream.fill(BufferedInputStream.java:183) > at java.io.BufferedInputStream.read(BufferedInputStream.java:201) > at org.postgresql.core.PGStream.ReceiveChar(PGStream.java:254) > at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1163) > at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:188) > ... 49 more Don't know if it helps, but i had similar problems a while back. Turned out it was caused by a misconfigured firewall reload (flushing all the firewall rules followed by re-inserting them) which caused outstanding network connections to be interrupted. Doing firewall reload the proper way (load new rules in inactive table, swap active & inactive) fixed it for me. > An unexpected exception has been detected in native code outside the VM. > Unexpected Signal : 11 occurred at PC=0x400EA4D3 > Function=strlen+0x33 > Library=/lib/libc.so.6 > ... And this I have seen on occasions where the compiled libc was out of sync with the kernel version used, but might be anything.. Best, Koen -- K.F.J. Martens, Sonologic, http://www.sonologic.nl/ Networking, hosting, embedded systems, unix, artificial intelligence. Public PGP key: http://www.metro.cx/pubkey-gmc.asc Wondering about the funny attachment your mail program can't read? Visit http://www.openpgp.org/