Обсуждение: autovacuum ignore tables
If I were to specify in the pg_autovacuum catalog that certain high volume partitioned tables(that get dropped daily) be ignored, then when autovacuum finishes, will it update the transaction id wraparound counter (this way, I can get autovacuum to finish quickly )
OR
Will I still need to periodically do vacuumdb –a to take care of the wraparound problem.
Sriram Dandapani wrote: > > If I were to specify in the pg_autovacuum catalog that certain high > volume partitioned tables(that get dropped daily) be ignored, then > when autovacuum finishes, will it update the transaction id wraparound > counter (this way, I can get autovacuum to finish quickly ) > > OR > > Will I still need to periodically do vacuumdb –a to take care of the > wraparound problem. > You don't need to do a manual vacuumdb -a since autovacuum will do this once it decides you are getting too close to the wraparound point. I believe this has been improved in the upcoming 8.2 release where autovacuum no longer need to vacuum the whole database at once, rather XID wraparound is now tracked on a per table basis.
The only issue I have with autovacuum is the fact that I have to briefly stop/restart postgres every couple of days, which kills autovacuum and it has no memory of previous work done. I work with several databases with partitioned tables having high daily volume. Dropping partitioned tables locks out jdbc inserts and the drop command itself goes into a WAIT state. Hence, I have to stop postgres,update pg_hba.conf to prevent access,restart postgres,drop tables and update pg_hba to allow access(Crazy, but I have no choice because Postgres deadlocks on drop child tables while inserts happen on the parent) -----Original Message----- From: Matthew T. O'Connor [mailto:matthew@zeut.net] Sent: Thursday, September 28, 2006 9:22 PM To: Sriram Dandapani Cc: pgsql-admin@postgresql.org Subject: Re: [ADMIN] autovacuum ignore tables Sriram Dandapani wrote: > > If I were to specify in the pg_autovacuum catalog that certain high > volume partitioned tables(that get dropped daily) be ignored, then > when autovacuum finishes, will it update the transaction id wraparound > counter (this way, I can get autovacuum to finish quickly ) > > OR > > Will I still need to periodically do vacuumdb -a to take care of the > wraparound problem. > You don't need to do a manual vacuumdb -a since autovacuum will do this once it decides you are getting too close to the wraparound point. I believe this has been improved in the upcoming 8.2 release where autovacuum no longer need to vacuum the whole database at once, rather XID wraparound is now tracked on a per table basis.
Sriram Dandapani wrote: > The only issue I have with autovacuum is the fact that I have to briefly > stop/restart postgres every couple of days, which kills autovacuum and > it has no memory of previous work done. I work with several databases > with partitioned tables having high daily volume. Dropping partitioned > tables locks out jdbc inserts and the drop command itself goes into a > WAIT state. Hence, I have to stop postgres,update pg_hba.conf to prevent > access,restart postgres,drop tables and update pg_hba to allow > access(Crazy, but I have no choice because Postgres deadlocks on drop > child tables while inserts happen on the parent) Sounds like a fairly serious problem, but I'll let someone else talk to that. As for the autovac issue, as of 8.1 and autovac being integrated into core, this isn't true. It does remember where it was, at least as long as you don't have the GUC var reset_stats_on_restart (something like that) set. Matt
"Sriram Dandapani" <sdandapani@counterpane.com> writes: > The only issue I have with autovacuum is the fact that I have to briefly > stop/restart postgres every couple of days, which kills autovacuum and > it has no memory of previous work done. As already stated, this isn't true as of 8.1. > I work with several databases > with partitioned tables having high daily volume. Dropping partitioned > tables locks out jdbc inserts and the drop command itself goes into a > WAIT state. Hence, I have to stop postgres,update pg_hba.conf to prevent > access,restart postgres,drop tables and update pg_hba to allow > access(Crazy, but I have no choice because Postgres deadlocks on drop > child tables while inserts happen on the parent) It sounds to me like you have a problem with failing to commit transactions promptly. The DROP will wait for existing transactions to release their locks on the doomed table, but its lock request will block any incoming transactions that try to acquire new locks on the table. So basically you should see a hiccup of length equal to the longest normal transaction using that table. In a well-designed concurrent system that should not be a problem. It might be worth doing the DROP as begin; lock table parent_table; drop table child_table; commit; so that SELECTs on the parent do simply block and don't risk getting errors from trying to access a just-dropped child table. regards, tom lane
I tried your code with the lock table followed by drop. The lock table goes into a wait state AND all INSERTS (even new ones coming in) go into a wait state. Ps -ef | grep waiting shows the function call (that does the lock and drop) in wait state as well as all INSERTS. All jdbc inserts are committed cleanly. When the commit is done..the autocommit for the connection is set to true (to flush out any open transactions) -----Original Message----- From: Tom Lane [mailto:tgl@sss.pgh.pa.us] Sent: Friday, September 29, 2006 7:45 AM To: Sriram Dandapani Cc: Matthew T. O'Connor; pgsql-admin@postgresql.org Subject: Re: [ADMIN] autovacuum ignore tables "Sriram Dandapani" <sdandapani@counterpane.com> writes: > The only issue I have with autovacuum is the fact that I have to briefly > stop/restart postgres every couple of days, which kills autovacuum and > it has no memory of previous work done. As already stated, this isn't true as of 8.1. > I work with several databases > with partitioned tables having high daily volume. Dropping partitioned > tables locks out jdbc inserts and the drop command itself goes into a > WAIT state. Hence, I have to stop postgres,update pg_hba.conf to prevent > access,restart postgres,drop tables and update pg_hba to allow > access(Crazy, but I have no choice because Postgres deadlocks on drop > child tables while inserts happen on the parent) It sounds to me like you have a problem with failing to commit transactions promptly. The DROP will wait for existing transactions to release their locks on the doomed table, but its lock request will block any incoming transactions that try to acquire new locks on the table. So basically you should see a hiccup of length equal to the longest normal transaction using that table. In a well-designed concurrent system that should not be a problem. It might be worth doing the DROP as begin; lock table parent_table; drop table child_table; commit; so that SELECTs on the parent do simply block and don't risk getting errors from trying to access a just-dropped child table. regards, tom lane
"Sriram Dandapani" <sdandapani@counterpane.com> writes: > I tried your code with the lock table followed by drop. > The lock table goes into a wait state AND all INSERTS (even new ones > coming in) go into a wait state. Well, it's waiting for something. Try looking in pg_locks to see which transaction is blocking it. regards, tom lane
I see RowExclusiveLocks for all the jdbc inserts Real question is: Why do the INSERTS go into wait state as soon as the lock table statement is issued on the parent? -----Original Message----- From: Tom Lane [mailto:tgl@sss.pgh.pa.us] Sent: Friday, September 29, 2006 11:10 AM To: Sriram Dandapani Cc: Matthew T. O'Connor; pgsql-admin@postgresql.org Subject: Re: [ADMIN] autovacuum ignore tables "Sriram Dandapani" <sdandapani@counterpane.com> writes: > I tried your code with the lock table followed by drop. > The lock table goes into a wait state AND all INSERTS (even new ones > coming in) go into a wait state. Well, it's waiting for something. Try looking in pg_locks to see which transaction is blocking it. regards, tom lane
"Sriram Dandapani" <sdandapani@counterpane.com> writes: > Real question is: Why do the INSERTS go into wait state as soon as the > lock table statement is issued on the parent? If you were just inserting directly into other child tables, a lock on either the parent or the target child table shouldn't affect them. I wonder if you are using conditional rules to redirect the inserts, and the rules include a reference to the target table? If that's the case, you really need to drop the relevant rule before you remove the child table, anyway. regards, tom lane
The jdbc inserts go into the main parent table and check constraints redirect them to child tables. If I were to drop the rule, that would immediately affect the inserts and they would go the the parent table. And I cannot lock the parent due to the problem I mentioned earlier. -----Original Message----- From: Tom Lane [mailto:tgl@sss.pgh.pa.us] Sent: Friday, September 29, 2006 2:38 PM To: Sriram Dandapani Cc: Matthew T. O'Connor; pgsql-admin@postgresql.org Subject: Re: [ADMIN] autovacuum ignore tables "Sriram Dandapani" <sdandapani@counterpane.com> writes: > Real question is: Why do the INSERTS go into wait state as soon as the > lock table statement is issued on the parent? If you were just inserting directly into other child tables, a lock on either the parent or the target child table shouldn't affect them. I wonder if you are using conditional rules to redirect the inserts, and the rules include a reference to the target table? If that's the case, you really need to drop the relevant rule before you remove the child table, anyway. regards, tom lane
"Sriram Dandapani" <sdandapani@counterpane.com> writes: > The jdbc inserts go into the main parent table and check constraints > redirect them to child tables. A check constraint can't redirect anything. Are you saying that you use a trigger to try to insert the row into *each* child table, relying on the constraints to make all except one insert fail? Ugh. You'd do a lot better to duplicate the partitioning-rule knowledge in the trigger, and do only one insert that should succeed --- no wasted cycles, and no need for a subtransaction. regards, tom lane
I meant to say...rules are used on the parent table to redirect to child. Just like the way the postgres 8.1 documentation has it. -----Original Message----- From: Tom Lane [mailto:tgl@sss.pgh.pa.us] Sent: Friday, September 29, 2006 2:48 PM To: Sriram Dandapani Cc: Matthew T. O'Connor; pgsql-admin@postgresql.org Subject: Re: [ADMIN] autovacuum ignore tables "Sriram Dandapani" <sdandapani@counterpane.com> writes: > The jdbc inserts go into the main parent table and check constraints > redirect them to child tables. A check constraint can't redirect anything. Are you saying that you use a trigger to try to insert the row into *each* child table, relying on the constraints to make all except one insert fail? Ugh. You'd do a lot better to duplicate the partitioning-rule knowledge in the trigger, and do only one insert that should succeed --- no wasted cycles, and no need for a subtransaction. regards, tom lane