Обсуждение: Remove usage of finalizers ?

Hey,

the other day we ran into a situation where our app ran into a OOME situation on heavy
load. It turned out that we had around 290k objects on the Finalizer queue, that
were Statements.

There has been a discussion in the past about finalizers (
see e.g. around  http://www.postgresql.org/message-id/BCD42993-CB7B-453F-95B4-09E84A956AB0@me.com ).

I understand that Connection objects are supposed implement a finalizer (is that actually true?),
but here the penalty is not that high, as Connections usually are pooled and are thus long living.

Other JDBC objects like Statements are extremely short lived and the creation rate can be
on a busy application much higher than the finalization rate (which is what we were seeing).

So I wonder if the driver could be rewritten in a way that either
- uses no finalizers for the short lived objects
or
- exist in 2 flavors: a debug version that does excessive logging in the
 finalizer if the objects were not yet closed (and stays as is wrt the extra work)
 and a production version where
 the finalize() methods are removed, so that the objects do not end up
 in the finalizer queue and can't pile up under high load.

Developers could then use the devel version and the other driver for production.

Relying on the finalize() method to close/free objects on the PG server is a bad
idea anyway, as there is no guarantee when finalizers run or if they run at all.

  Thanks
       Heiko


--
Heiko Rupp   hwr@pilhuhn.de
Blog: http://pilhuhn.blogspot.com   @pilhuhn

Heiko W. Rupp wrote
> the other day we ran into a situation where our app ran into a OOME
> situation on heavy
> load. It turned out that we had around 290k objects on the Finalizer
> queue, that
> were Statements.

It would help to have an idea of the kind of load that would cause 290k
statement objects to sit waiting in the finalizer queue when each one's
finalizer routine is a no-op (assumes you previously closed these statements
properly).  Any idea of the pace of accumulation involved?


> There has been a discussion in the past about finalizers (
> see e.g. around  http://www.postgresql.org/message-id/

> BCD42993-CB7B-453F-95B4-09E84A956AB0@

>  ).

That thread references an alternative driver and does so with respect to a
different use-case.  The programmer also came to the conclusion that using a
finalizer was a good idea though as applied to Statements no comment was
rendered.


> Other JDBC objects like Statements are extremely short lived and the
> creation rate can be
> on a busy application much higher than the finalization rate (which is
> what we were seeing).

I am suspicious about this claim as made though my limited experience is
part of the reason.  I do not doubt the queue can get backlogged but a
significant number of large companies make use of this driver and the
encounter rate of this problem does not seem that large.  The first thing to
check is to try and see if it is truly the rate of object
creation/destruction that is the driver or is it that the time needed to
destroy a single object is simply too long (i.e., you are not closing your
statements properly).


> So I wonder if the driver could be rewritten in a way that either
> - uses no finalizers for the short lived objects
> or
> - exist in 2 flavors: a debug version that does excessive logging in the
>  finalizer if the objects were not yet closed (and stays as is wrt the
> extra work)
>  and a production version where
>  the finalize() methods are removed, so that the objects do not end up
>  in the finalizer queue and can't pile up under high load.

This seems to be a straight-forward change to make and the project is
open-source.  Before it becomes part of a production release someone would
have to try it and report back on their findings anyway - it is too large a
change to just go ahead and make blindly.  Ideally bench-marking would be
involved as well but even just putting it into production and seeing if/when
the system crashes (if ever) would be a worthwhile activity.  Is this
something you'd be interested in doing?

Is there some kind of pathologically bad application out there where it is
possible to test how quickly a poorly implemented application would fail
under various algorithms but using the same hardware?

Having separate drivers is undesirable but it is possible to run-time decide
on an implementation.  Simply no-op'ing the finalizer method, to my
knowledge and as implied here, does not solve the problem if indeed it is
the act of being added to the queue that is to blame.  I do not know enough
of the JVM to know if this is indeed the case.  It is possible that a
constant no-op would cause the JVM (at least some of them) to optimize away
the finalizer call during run-time but that is probably better not relied
upon.

David J.





--
View this message in context: http://postgresql.1045698.n5.nabble.com/Remove-usage-of-finalizers-tp5775186p5775195.html
Sent from the PostgreSQL - jdbc mailing list archive at Nabble.com.

Long finalized queue means that some of your objects finalizes long. This means that either you don't close some of your statements explicitly or you have different type of objects (even if it's small number) that take long time to finalize. Check stack trace of finalized thread.
As of driver, removing finalize altogether is not good as statements won't be closing implicitly. Possible options are:
1) Move processing out of finalizer thread with reference queues.
2) Warn when something is implicitly closed to indicate usage problem.

Am 21.10.2013 um 08:29 schrieb Vitalii Tymchyshyn:

> Long finalized queue means that some of your objects finalizes long. This means that either you don't close some of
yourstatements explicitly or you have different type  

Actually this is a false assumption. The finalizer daemon thread has a low priority and can sometimes
not keep up with the most trivial cases. See e.g. http://www.fasterj.com/articles/finalizer1.shtml

And yes, closing statements and other objects (e.g. LOBs) explicitly makes a lot of sense.


> of objects (even if it's small number) that take long time to finalize. Check stack trace of finalized thread.
> As of driver, removing finalize altogether is not good as statements won't be closing implicitly. Possible options
are:
> 1) Move processing out of finalizer thread with reference queues.
> 2) Warn when something is implicitly closed to indicate usage problem.

That is what I meant: have 2 drivers:

- one for devlopment that does the warning in the finalizer and the closing, but very loud warning
- one for production where the finalize() methods are gone.

Again: there is no guarantee that a finalize() method is ever called from the JVM. So relying on
it may create bogus results

>
> 20 жовт. 2013 23:33, користувач "Heiko W. Rupp" <hwr@pilhuhn.de> написав:
> Hey,
>
> the other day we ran into a situation where our app ran into a OOME situation on heavy
> load. It turned out that we had around 290k objects on the Finalizer queue, that
> were Statements.
>
> There has been a discussion in the past about finalizers (
> see e.g. around  http://www.postgresql.org/message-id/BCD42993-CB7B-453F-95B4-09E84A956AB0@me.com ).
>
> I understand that Connection objects are supposed implement a finalizer (is that actually true?),
> but here the penalty is not that high, as Connections usually are pooled and are thus long living.
>
> Other JDBC objects like Statements are extremely short lived and the creation rate can be
> on a busy application much higher than the finalization rate (which is what we were seeing).
>
> So I wonder if the driver could be rewritten in a way that either
> - uses no finalizers for the short lived objects
> or
> - exist in 2 flavors: a debug version that does excessive logging in the
>  finalizer if the objects were not yet closed (and stays as is wrt the extra work)
>  and a production version where
>  the finalize() methods are removed, so that the objects do not end up
>  in the finalizer queue and can't pile up under high load.
>
> Developers could then use the devel version and the other driver for production.
>
> Relying on the finalize() method to close/free objects on the PG server is a bad
> idea anyway, as there is no guarantee when finalizers run or if they run at all.
>
>   Thanks
>        Heiko
>
>
> --
> Heiko Rupp   hwr@pilhuhn.de
> Blog: http://pilhuhn.blogspot.com   @pilhuhn
>
>
>
> --
> Sent via pgsql-jdbc mailing list (pgsql-jdbc@postgresql.org)
> To make changes to your subscription:
> http://www.postgresql.org/mailpref/pgsql-jdbc

--
Heiko Rupp   hwr@pilhuhn.de
Blog: http://pilhuhn.blogspot.com   @pilhuhn

A case in point demonstrating why finalizers are inherently useless/bad: unpredictability. I agree with the approach that they should be used only conditionally (such a in debug mode).

Péter

Are you sure that you have priority problem? I'd say it's possible but not too probable. Have you checked your finalizer stack traces?
Note that removing finalizers without introducing some other mechanism, like reference queues, would introduce server side leaks on poorly written yet correct applications!

A case in point demonstrating why finalizers are inherently useless/bad: unpredictability. I agree with the approach that they should be used only conditionally (such a in debug mode).

Péter

On Oct 21, 2013 10:07 AM, "Heiko W. Rupp" <hwr@pilhuhn.de> wrote:

Am 21.10.2013 um 08:29 schrieb Vitalii Tymchyshyn:

> Long finalized queue means that some of your objects finalizes long. This means that either you don't close some of your statements explicitly or you have different type

Actually this is a false assumption. The finalizer daemon thread has a low priority and can sometimes
not keep up with the most trivial cases. See e.g. http://www.fasterj.com/articles/finalizer1.shtml

And yes, closing statements and other objects (e.g. LOBs) explicitly makes a lot of sense.


> of objects (even if it's small number) that take long time to finalize. Check stack trace of finalized thread.
> As of driver, removing finalize altogether is not good as statements won't be closing implicitly. Possible options are:
> 1) Move processing out of finalizer thread with reference queues.
> 2) Warn when something is implicitly closed to indicate usage problem.

That is what I meant: have 2 drivers:

- one for devlopment that does the warning in the finalizer and the closing, but very loud warning
- one for production where the finalize() methods are gone.

Again: there is no guarantee that a finalize() method is ever called from the JVM. So relying on
it may create bogus results

>
> 20 жовт. 2013 23:33, користувач "Heiko W. Rupp" <hwr@pilhuhn.de> написав:
> Hey,
>
> the other day we ran into a situation where our app ran into a OOME situation on heavy
> load. It turned out that we had around 290k objects on the Finalizer queue, that
> were Statements.
>
> There has been a discussion in the past about finalizers (
> see e.g. around  http://www.postgresql.org/message-id/BCD42993-CB7B-453F-95B4-09E84A956AB0@me.com ).
>
> I understand that Connection objects are supposed implement a finalizer (is that actually true?),
> but here the penalty is not that high, as Connections usually are pooled and are thus long living.
>
> Other JDBC objects like Statements are extremely short lived and the creation rate can be
> on a busy application much higher than the finalization rate (which is what we were seeing).
>
> So I wonder if the driver could be rewritten in a way that either
> - uses no finalizers for the short lived objects
> or
> - exist in 2 flavors: a debug version that does excessive logging in the
>  finalizer if the objects were not yet closed (and stays as is wrt the extra work)
>  and a production version where
>  the finalize() methods are removed, so that the objects do not end up
>  in the finalizer queue and can't pile up under high load.
>
> Developers could then use the devel version and the other driver for production.
>
> Relying on the finalize() method to close/free objects on the PG server is a bad
> idea anyway, as there is no guarantee when finalizers run or if they run at all.
>
>   Thanks
>        Heiko
>
>
> --
> Heiko Rupp   hwr@pilhuhn.de
> Blog: http://pilhuhn.blogspot.com   @pilhuhn
>
>
>
> --
> Sent via pgsql-jdbc mailing list (pgsql-jdbc@postgresql.org)
> To make changes to your subscription:
> http://www.postgresql.org/mailpref/pgsql-jdbc

--
Heiko Rupp   hwr@pilhuhn.de
Blog: http://pilhuhn.blogspot.com   @pilhuhn



--
Sent via pgsql-jdbc mailing list (pgsql-jdbc@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-jdbc

Am 21.10.2013 um 11:34 schrieb Vitalii Tymchyshyn:

> Are you sure that you have priority problem? I'd say it's possible but not too probable. Have you checked your
finalizerstack traces? 

Did you read the article? This clearly shows that under load those problems can arise even with very trivial
finalizers.
The problem is normally not visible under light load, as here the creation of new Statements etc. is not that fast and
the 
app itself has idle phases so that the finalizer daemon gets time to run.
Under higher load you have more work on the application threads and are creating much more objects.
This also means that the garbage collector needs more time to run and thus the finalizer thread gets less.
As objects are not finalized, the garbage collector has even more work to do, which reduces the available time
for the finalizer even more.

> Note that removing finalizers without introducing some other mechanism, like reference queues, would introduce server
sideleaks on poorly written yet correct applications! 

Server side leaks are introduced the moment that you rely on a finalizer being called (close to the time the object is
nolonger needed). 
Per JLS, the finalizers are a) not guaranteed to ever run,  b) run in any specific order c) at any given time.

And the other mechanisms exist -- close() calls. Software that relies on the finalizer for this cleanup may show
unpredictable 
behavior.

Another reason to get rid of finalize() methods is that Object creation is several hundred times slower with the
finalize()
methods present, as not only the Statement etc. is created, but also the Finalizer and the later needs to be put on the
finalizer
queue. Josh Bloch talks about 5.6ns vs 2400ns for a simple object in "Effective Java, 2nd edition".
That is a penalty everyone has to pay - even if they close Statements etc correctly.

  Heiko


>
> 21 жовт. 2013 11:06, користувач "Heiko W. Rupp" <hwr@pilhuhn.de> написав:
>
> Am 21.10.2013 um 08:29 schrieb Vitalii Tymchyshyn:
>
> > Long finalized queue means that some of your objects finalizes long. This means that either you don't close some of
yourstatements explicitly or you have different type 
>
> Actually this is a false assumption. The finalizer daemon thread has a low priority and can sometimes
> not keep up with the most trivial cases. See e.g. http://www.fasterj.com/articles/finalizer1.shtml
>
> And yes, closing statements and other objects (e.g. LOBs) explicitly makes a lot of sense.
>
>
> > of objects (even if it's small number) that take long time to finalize. Check stack trace of finalized thread.
> > As of driver, removing finalize altogether is not good as statements won't be closing implicitly. Possible options
are:
> > 1) Move processing out of finalizer thread with reference queues.
> > 2) Warn when something is implicitly closed to indicate usage problem.
>
> That is what I meant: have 2 drivers:
>
> - one for devlopment that does the warning in the finalizer and the closing, but very loud warning
> - one for production where the finalize() methods are gone.
>
> Again: there is no guarantee that a finalize() method is ever called from the JVM. So relying on
> it may create bogus results
>
> >
> > 20 жовт. 2013 23:33, користувач "Heiko W. Rupp" <hwr@pilhuhn.de> написав:
> > Hey,
> >
> > the other day we ran into a situation where our app ran into a OOME situation on heavy
> > load. It turned out that we had around 290k objects on the Finalizer queue, that
> > were Statements.
> >
> > There has been a discussion in the past about finalizers (
> > see e.g. around  http://www.postgresql.org/message-id/BCD42993-CB7B-453F-95B4-09E84A956AB0@me.com ).
> >
> > I understand that Connection objects are supposed implement a finalizer (is that actually true?),
> > but here the penalty is not that high, as Connections usually are pooled and are thus long living.
> >
> > Other JDBC objects like Statements are extremely short lived and the creation rate can be
> > on a busy application much higher than the finalization rate (which is what we were seeing).
> >
> > So I wonder if the driver could be rewritten in a way that either
> > - uses no finalizers for the short lived objects
> > or
> > - exist in 2 flavors: a debug version that does excessive logging in the
> >  finalizer if the objects were not yet closed (and stays as is wrt the extra work)
> >  and a production version where
> >  the finalize() methods are removed, so that the objects do not end up
> >  in the finalizer queue and can't pile up under high load.
> >
> > Developers could then use the devel version and the other driver for production.
> >
> > Relying on the finalize() method to close/free objects on the PG server is a bad
> > idea anyway, as there is no guarantee when finalizers run or if they run at all.
> >
> >   Thanks
> >        Heiko
> >
> >
> > --
> > Heiko Rupp   hwr@pilhuhn.de
> > Blog: http://pilhuhn.blogspot.com   @pilhuhn
> >
> >
> >
> > --
> > Sent via pgsql-jdbc mailing list (pgsql-jdbc@postgresql.org)
> > To make changes to your subscription:
> > http://www.postgresql.org/mailpref/pgsql-jdbc
>
> --
> Heiko Rupp   hwr@pilhuhn.de
> Blog: http://pilhuhn.blogspot.com   @pilhuhn
>

--
Heiko Rupp   hwr@pilhuhn.de
Blog: http://pilhuhn.blogspot.com   @pilhuhn

On 21.10.2013 13:09, Heiko W. Rupp wrote:
> Am 21.10.2013 um 11:34 schrieb Vitalii Tymchyshyn:
>> Note that removing finalizers without introducing some other mechanism, like reference queues, would introduce
serverside leaks on poorly written yet correct applications! 
>
> Server side leaks are introduced the moment that you rely on a finalizer being called (close to the time the object
isno longer needed). 
> Per JLS, the finalizers are a) not guaranteed to ever run,  b) run in any specific order c) at any given time.
>
> And the other mechanisms exist -- close() calls. Software that relies on the finalizer for this cleanup may show
unpredictable
> behavior.

If we can't get rid of the finalizer completely, I wonder if we could
arrange things so that we could avoid the finalizer overhead when there
are server-side resources associated with the statement. I would guess
that in most cases that you don't close a Statement properly, it's a
very short-lived Statement used to run a single query. Such a statement
will typically not have any server-side resources associated with it.

Actually, we already seem to use PhantomReferences to clean up orphaned
SimpleQuery and Portal objects. AFAICS those are the things that
actually represent server-side resources. Why do we even have a
finalizer in AbstractJdbc2Statement? Perhaps we used to rely on the
finalizer to clean up things, but later added the more fine-grained
PhantomReferences to the underlying objects, making the finalizer redundant?

> Another reason to get rid of finalize() methods is that Object creation is several hundred times slower with the
finalize()
> methods present, as not only the Statement etc. is created, but also the Finalizer and the later needs to be put on
thefinalizer 
> queue. Josh Bloch talks about 5.6ns vs 2400ns for a simple object in "Effective Java, 2nd edition".
> That is a penalty everyone has to pay - even if they close Statements etc correctly.

That's still negligible compared to sending a statement to the remote
host and getting the result back, though.

- Heikki

On 21 October 2013 23:03, Dave Cramer <pg@fastcrypt.com> wrote:
> Please send patches for possible solutions. I think I would be OK with
> removing them and letting people find their Statement leaks

One option if we want to keep the existing behaviour available in some
way would be to remove the finalize method from AbstractJdbc2Statement
and create subclasses of the various concrete statement classes which
could be used when a debug flag is switched on. e.g.
DebugJdbc3Statement, DebugJdbc3PreparedStatement,
DebugJdbc3CallableStatement etc which have the finalizer present.

There would be quite a few of them though, and we'd need to have a
switch everywhere that one of those is instantiated, or introduce a
factory. It's a bit gross.

IMO tracing non-closed statement leaks is probably better done in a
connection pool or jdbc debugging library anyway, so maybe we
shouldn't bother and just remove it.

One question is this, though: how many users out there aren't closing
their statements currently and are relying, accidentally or
deliberately, on the current behaviour?

Tom

As for me, introducing server-side leak would be plain wrong. If it will be done, please announce in the list, I will stop recommending using postgresql in java projects.

That's rather harsh. How do you see this as introducing a server side leak ? Statements are supposed to be closed by applications as are connections.

Dave Cramer

dave.cramer(at)credativ(dot)ca
http://www.credativ.ca

On Tue, Oct 22, 2013 at 8:36 PM, Vitalii Tymchyshyn <vit@tym.im> wrote:

As for me, introducing server-side leak would be plain wrong. If it will be done, please announce in the list, I will stop recommending using postgresql in java projects.

23 жовт. 2013 03:12, користувач "Tom Dunstan" <pgsql@tomd.cc> написав:

On 21 October 2013 23:03, Dave Cramer <pg@fastcrypt.com> wrote:
> Please send patches for possible solutions. I think I would be OK with
> removing them and letting people find their Statement leaks

One option if we want to keep the existing behaviour available in some
way would be to remove the finalize method from AbstractJdbc2Statement
and create subclasses of the various concrete statement classes which
could be used when a debug flag is switched on. e.g.
DebugJdbc3Statement, DebugJdbc3PreparedStatement,
DebugJdbc3CallableStatement etc which have the finalizer present.

There would be quite a few of them though, and we'd need to have a
switch everywhere that one of those is instantiated, or introduce a
factory. It's a bit gross.

IMO tracing non-closed statement leaks is probably better done in a
connection pool or jdbc debugging library anyway, so maybe we
shouldn't bother and just remove it.

One question is this, though: how many users out there aren't closing
their statements currently and are relying, accidentally or
deliberately, on the current behaviour?

Tom

On 23 October 2013 11:06, Vitalii Tymchyshyn <vit@tym.im> wrote:
> As for me, introducing server-side leak would be plain wrong. If it will be
> done, please announce in the list, I will stop recommending using postgresql
> in java projects.

It's not even clear that leaked (aka non-closed) java statement
objects will even have any impact - see Heikki's email which points
out that they don't actually correspond to real pgsql server
resources.

The current finalizer on statement objects just calls the close()
method. All java code which uses JDBC should be calling close() on
created statement objects explicitly in a finally block, or using a
framework which does that automatically. To not call close() on a
statement object is a bug in the calling code. The existing finalizer
is trying to save users from the possible bad effects of such a bug in
their code, but it's not clear what those effects are, and the
finalizer clearly has a non-zero performance cost in some situations
(that's what kicked off the thread). So users whose code is doing the
right thing may be getting penalized for the sake of saving other
users with buggy code.

I suggested a way to remove the cost of the finalizer while still
allowing users to switch the old behaviour back on using a parameter,
but as I said it's a bit ugly code-wise.

We need to determine whether not calling close() on thrown-away
statement objects actually causes any real resource leaks or not.
Without that info we don't know what the trade-offs are.

Cheers

Tom

According to jdbc javadoc Statement.close simply releases resources without waiting for them to be closed automatically.
And good behaving libs in java look for their external resources allocated. The application may have problems. Last leak I've seen was something like  this.getClass().getResourceAsStream(...) != null. But this was traceable because one could find unclosed streams in heap dump.
What is proposed here is server side leak that will be invisible while analyzing application. I can't afford using such a lib in my projects.

On 23 October 2013 12:29, Vitalii Tymchyshyn <vit@tym.im> wrote:
> According to jdbc javadoc Statement.close simply releases resources without
> waiting for them to be closed automatically.

Interestingly, as recently as JDK 1.5 the javadoc for Statement said:
 > Note: A Statement object is automatically closed when it is garbage
collected.
( http://docs.oracle.com/javase/1.5.0/docs/api/java/sql/Statement.html#close() )

That language seems to have been removed as of JDK 1.6, presumably
because relying on finalizers for cleanup of any real resource hasn't
been considered good practice since well before even then. And that
was in 2006.

I'm a little bit frightened now that there may be code out there
relying on this behaviour, bad practice though it may be. Once again
I'd like to know if this case makes any difference or not, given our
other use of PhantomReferences etc.

Tom

Vitalii,

Am 23.10.2013 um 02:36 schrieb Vitalii Tymchyshyn:

> As for me, introducing server-side leak would be plain wrong. If it will be done, please announce in the list, I will
stoprecommending using postgresql in java projects. 

*relying* on Finalizers to clean up server side state may introduce those leaks in the first place, as
finalizers are not guaranteed to be called at all.
In my case we had 290k Statements in the queue to be finalized, when the JVM started to throw
OutOfMemoryErrors. At that point no one will those finalize() methods be called here.

Also there is no guarantee when the finalizer runs other than after completed construction
and before the memory is finally released by the garbage collector.
Objects to be finalized could thus hold references to pg-server side resources for hours,
which is not what you want for short lived simple statements.

And as Dave said, Statements are supposed to be explicitly close()d by the application.

Am 23.10.2013 um 02:12 schrieb Tom Dunstan:
> and create subclasses of the various concrete statement classes which
> could be used when a debug flag is switched on. e.g.
> DebugJdbc3Statement, DebugJdbc3PreparedStatement,

While this could be beneficial from the "one jar to rule them all"
standpoint, I think this is too much hassle.

> IMO tracing non-closed statement leaks is probably better done in a
> connection pool or jdbc debugging library anyway, so maybe we
> shouldn't bother and just remove it.

Yep.

I think we could just have two jars

psql-debug.jar  -- provide very loud logging if Statements and Connections
are not explicitly closed, which can be used for development and debugging.
psql-prod.jar -- without finalize() calls for integration, performance testing and
production

> One question is this, though: how many users out there aren't closing
> their statements currently and are relying, accidentally or
> deliberately, on the current behaviour?

Deliberately relying on the current behavior is just bogus and I guess
no one will know if they accidentally rely on it :-)

Besides that: tools like Findbugs or IntelliJ IDEA warn you when Statements
are not closed as they consider that a leak.

--
Heiko Rupp   hwr@pilhuhn.de
Blog: http://pilhuhn.blogspot.com   @pilhuhn

I am not big fan of finalizes. But you propose to remove mechanism that works most times, without introducing anything else.
You are saying that as in my incorrect application (and I am 95% sure you've got incorrect application if you are getting finalizing objects piled up) it does not work, let's remove it altogether and make driver also wrong.
Look, java pretty much guaranties garbage collecting, being it heap objects or out of heap objects like files or sockets. And you propose to introduce out-of-heap leak in the driver. I am very strongly against it.
P.S. If we will start talking about Errors, like OOM, any application should be considered broken after almost any Error being thrown, no matter if it's thrown in the finalized thread or not.

On 10/23/2013 12:06 AM, Vitalii Tymchyshyn wrote:
> I am not big fan of finalizes. But you propose to remove mechanism
> that works most times, without introducing anything else.
> You are saying that as in my incorrect application (and I am 95% sure
> you've got incorrect application if you are getting finalizing objects
> piled up) it does not work, let's remove it altogether and make driver
> also wrong.

as an outsider to this stuff, with only a superficial understanding of
these finalizers, it seems to me they should log a warning if anything
leaks to them.  you don't get rid of them entirely, they provide backup
sweeping up of neglected debris..


--
john r pierce                                      37N 122W
somewhere on the middle of the left coast

On Oct 23, 2013, at 12:22 AM, John R Pierce <pierce@hogranch.com> wrote:

> On 10/23/2013 12:06 AM, Vitalii Tymchyshyn wrote:
>> I am not big fan of finalizes. But you propose to remove mechanism that works most times, without introducing
anythingelse. 
>> You are saying that as in my incorrect application (and I am 95% sure you've got incorrect application if you are
gettingfinalizing objects piled up) it does not work, let's remove it altogether and make driver also wrong. 
>
> as an outsider to this stuff, with only a superficial understanding of these finalizers, it seems to me they should
loga warning if anything leaks to them.  you don't get rid of them entirely, they provide backup sweeping up of
neglecteddebris.. 
>

The specific problem with this is that the mere presence of a non-empty finalizer increases the cost of object
allocationand deallocation by orders of magnitude (as is claimed by many Java performance resources).  All correct
applicationspretty much have to deallocate these objects explicitly, as the GC can leave the resources lying around for
unboundedtime.  So having these finalizers penalizes the correct applications (which I have not observed in practice,
butOP seemingly has) by some amount. 


Maybe the best thing would be for the OP to distill the problem down into a self-contained code example so we can
verifythat there aren't any other errors that might be contributing to this problem?  It seems that if the problem was
reallythat big, someone else would have run across it. 

Best,
Steven

Seems to me that PhantomReferences can solve this problem as they will provide the ability to the cleanup without
causingthe performance issues with the finalizers. I am not familiar with the postgres jdbc driver code base so I have
noidea how much work is involved in switching from finalizers to phantom references.  

Can anyone with knowledge of the jdbc code base comment on the practicality of moving from finalizers to phantom
reference.

On 2013-10-23, at 1:47 PM, Steven Schlansker <stevenschlansker@gmail.com> wrote:

>
> On Oct 23, 2013, at 12:22 AM, John R Pierce <pierce@hogranch.com> wrote:
>
>> On 10/23/2013 12:06 AM, Vitalii Tymchyshyn wrote:
>>> I am not big fan of finalizes. But you propose to remove mechanism that works most times, without introducing
anythingelse. 
>>> You are saying that as in my incorrect application (and I am 95% sure you've got incorrect application if you are
gettingfinalizing objects piled up) it does not work, let's remove it altogether and make driver also wrong. 
>>
>> as an outsider to this stuff, with only a superficial understanding of these finalizers, it seems to me they should
loga warning if anything leaks to them.  you don't get rid of them entirely, they provide backup sweeping up of
neglecteddebris.. 
>>
>
> The specific problem with this is that the mere presence of a non-empty finalizer increases the cost of object
allocationand deallocation by orders of magnitude (as is claimed by many Java performance resources).  All correct
applicationspretty much have to deallocate these objects explicitly, as the GC can leave the resources lying around for
unboundedtime.  So having these finalizers penalizes the correct applications (which I have not observed in practice,
butOP seemingly has) by some amount. 
>
>
> Maybe the best thing would be for the OP to distill the problem down into a self-contained code example so we can
verifythat there aren't any other errors that might be contributing to this problem?  It seems that if the problem was
reallythat big, someone else would have run across it. 
>
> Best,
> Steven
>
>
>
> --
> Sent via pgsql-jdbc mailing list (pgsql-jdbc@postgresql.org)
> To make changes to your subscription:
> http://www.postgresql.org/mailpref/pgsql-jdbc

On Oct 23, 2013, at 10:58 AM, Adib Saikali <adib.saikali@gmail.com> wrote:

> Seems to me that PhantomReferences can solve this problem as they will provide the ability to the cleanup without
causingthe performance issues with the finalizers. I am not familiar with the postgres jdbc driver code base so I have
noidea how much work is involved in switching from finalizers to phantom references.  
>
> Can anyone with knowledge of the jdbc code base comment on the practicality of moving from finalizers to phantom
reference.
>

Do we in fact know that phantom (or weak?) references are cheaper than Finalizers?

It may be that switching to References makes turning a "debug mode" on or off much easier (i.e. no need to subclass)
butit would be nice to have some evidence as to whether it will be faster or not, and under what workloads. 

> On 2013-10-23, at 1:47 PM, Steven Schlansker <stevenschlansker@gmail.com> wrote:
>
>>
>> On Oct 23, 2013, at 12:22 AM, John R Pierce <pierce@hogranch.com> wrote:
>>
>>> On 10/23/2013 12:06 AM, Vitalii Tymchyshyn wrote:
>>>> I am not big fan of finalizes. But you propose to remove mechanism that works most times, without introducing
anythingelse. 
>>>> You are saying that as in my incorrect application (and I am 95% sure you've got incorrect application if you are
gettingfinalizing objects piled up) it does not work, let's remove it altogether and make driver also wrong. 
>>>
>>> as an outsider to this stuff, with only a superficial understanding of these finalizers, it seems to me they should
loga warning if anything leaks to them.  you don't get rid of them entirely, they provide backup sweeping up of
neglecteddebris.. 
>>>
>>
>> The specific problem with this is that the mere presence of a non-empty finalizer increases the cost of object
allocationand deallocation by orders of magnitude (as is claimed by many Java performance resources).  All correct
applicationspretty much have to deallocate these objects explicitly, as the GC can leave the resources lying around for
unboundedtime.  So having these finalizers penalizes the correct applications (which I have not observed in practice,
butOP seemingly has) by some amount. 
>>
>>
>> Maybe the best thing would be for the OP to distill the problem down into a self-contained code example so we can
verifythat there aren't any other errors that might be contributing to this problem?  It seems that if the problem was
reallythat big, someone else would have run across it. 
>>
>> Best,
>> Steven
>>
>>
>>
>> --
>> Sent via pgsql-jdbc mailing list (pgsql-jdbc@postgresql.org)
>> To make changes to your subscription:
>> http://www.postgresql.org/mailpref/pgsql-jdbc
>
>
>
> --
> Sent via pgsql-jdbc mailing list (pgsql-jdbc@postgresql.org)
> To make changes to your subscription:
> http://www.postgresql.org/mailpref/pgsql-jdbc

On 23.10.2013 21:04, Steven Schlansker wrote:
>
> On Oct 23, 2013, at 10:58 AM, Adib Saikali<adib.saikali@gmail.com>  wrote:
>
>> Seems to me that PhantomReferences can solve this problem as they will provide the ability to the cleanup without
causingthe performance issues with the finalizers. I am not familiar with the postgres jdbc driver code base so I have
noidea how much work is involved in switching from finalizers to phantom references. 
>>
>> Can anyone with knowledge of the jdbc code base comment on the practicality of moving from finalizers to phantom
reference.
>
> Do we in fact know that phantom (or weak?) references are cheaper than Finalizers?

Well, we *already* use phantom references to clean up server-side
resources in the SimpleQuery and Portal classes. I think that's actually
already enough to avoid server-side resource leaks if an application
neglects to close Statements.

Rather than discussing what might be the best way to use finalizers or
phantom references to replace the current finalizer in the Statement
class, I wish someone would just try to remove it and see what happens.

- Heikki

Well, references:
1) Can be created only when needed (when there is something to dispose)
2) Can be cleared if close was called (most cases)
3) Processing can be skipped if connection is closed as a whole
4) Cleanup can be done during regular connection calls thus ensuring no concurrent problems are added and clean-up requests don't pile up because of CPU overload

References are much more superior to finalizers