Обсуждение: Stuck cvs lock on anoncvs repository
Don't think I have login privs on that machine --- would someone
who does blow away this lock?
regards, tom lane
------- Forwarded Message
Date: Fri, 12 Jun 2009 17:45:14 +0900
From: Fujii Masao <masao.fujii@gmail.com>
To: PostgreSQL-development <pgsql-hackers@postgresql.org>
Subject: [HACKERS] cannot update to the latest CVS sources
Hi,
$ cvs update
....
cvs update: Updating src/backend/commands
cvs update: [08:32:37] waiting for anoncvs's lock in
/projects/cvsroot/pgsql/src/backend/commands
cvs update: [08:33:07] waiting for anoncvs's lock in
/projects/cvsroot/pgsql/src/backend/commands
cvs update: [08:33:37] waiting for anoncvs's lock in
/projects/cvsroot/pgsql/src/backend/commands
cvs update: [08:34:07] waiting for anoncvs's lock in
/projects/cvsroot/pgsql/src/backend/commands
cvs update: [08:34:37] waiting for anoncvs's lock in
/projects/cvsroot/pgsql/src/backend/commands
cvs update: [08:35:07] waiting for anoncvs's lock in
/projects/cvsroot/pgsql/src/backend/commands
The cvs's lock remains held? Why?
Regards,
--
Fujii Masao
NIPPON TELEGRAPH AND TELEPHONE CORPORATION
NTT Open Source Software Center
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
------- End of Forwarded Message
On Fri, Jun 12, 2009 at 3:12 PM, Tom Lane<tgl@sss.pgh.pa.us> wrote: > Don't think I have login privs on that machine --- would someone > who does blow away this lock? Done. -- Dave Page EnterpriseDB UK: http://www.enterprisedb.com
Hi, On Fri, Jun 12, 2009 at 11:15 PM, Dave Page<dpage@pgadmin.org> wrote: > On Fri, Jun 12, 2009 at 3:12 PM, Tom Lane<tgl@sss.pgh.pa.us> wrote: >> Don't think I have login privs on that machine --- would someone >> who does blow away this lock? > > Done. I encountered the same problem again. Can you release the lock? cvs update: [01:31:47] waiting for anoncvs's lock in /projects/cvsroot/pgsql/src/backend/catalog cvs update: [01:32:17] waiting for anoncvs's lock in /projects/cvsroot/pgsql/src/backend/catalog cvs update: [01:32:47] waiting for anoncvs's lock in /projects/cvsroot/pgsql/src/backend/catalog cvs update: [01:33:17] waiting for anoncvs's lock in /projects/cvsroot/pgsql/src/backend/catalog cvs update: [01:33:47] waiting for anoncvs's lock in /projects/cvsroot/pgsql/src/backend/catalog Regards, -- Fujii Masao NIPPON TELEGRAPH AND TELEPHONE CORPORATION NTT Open Source Software Center
Fujii Masao <masao.fujii@gmail.com> writes:
> I encountered the same problem again. Can you release the lock?
Huh. We should look into why that's happening. The previous time was
right after tagging rc1, and this after tagging final ... seems like
there must be some connection. Does the lockfile show anything about
who took the lock?
regards, tom lane
Tom Lane wrote: > Fujii Masao <masao.fujii@gmail.com> writes: >> I encountered the same problem again. Can you release the lock? > > Huh. We should look into why that's happening. The previous time was > right after tagging rc1, and this after tagging final ... seems like > there must be some connection. Does the lockfile show anything about > who took the lock? should be working again now. I'm not sure why this happened but it seems to coincide with the daily "forced" push of the repo just after midnight. I suspect that we are somehow rsyncing too much state to anoncvs (especially the #cvs-lock directory) and we are simply missing a --exclude '#cvs.*' flag for rsync. Stefan
Stefan Kaltenbrunner wrote: > Tom Lane wrote: >> Fujii Masao <masao.fujii@gmail.com> writes: >>> I encountered the same problem again. Can you release the lock? >> >> Huh. We should look into why that's happening. The previous time was >> right after tagging rc1, and this after tagging final ... seems like >> there must be some connection. Does the lockfile show anything about >> who took the lock? > > should be working again now. I'm not sure why this happened but it seems > to coincide with the daily "forced" push of the repo just after midnight. > I suspect that we are somehow rsyncing too much state to anoncvs > (especially the #cvs-lock directory) and we are simply missing a > --exclude '#cvs.*' flag for rsync. FWIW I modified the push script on cvs-master to exclude any lock (directories) that might intermediatly show up during the run. I'm not sure why this only started to happen now - but I suspect it correlates with the fact that tagging seems to take a while and the particular time marc did the tagging so that collided with the forced push. Stefan
do we not have --delete on the rsync command? why doesn't the problem go away on the next sync if we do? -- greg http://mit.edu/~gsstark/resume.pdf
Greg Stark wrote: > do we not have --delete on the rsync command? why doesn't the problem > go away on the next sync if we do? we do use --delete - but nobody commited after we tagged and before the problem got reported so the regular on-commit code didn't kick in and forced syncs only happen once a day. Stefan
Stefan Kaltenbrunner escribió: > Greg Stark wrote: >> do we not have --delete on the rsync command? why doesn't the problem >> go away on the next sync if we do? > > we do use --delete - but nobody commited after we tagged and before the > problem got reported so the regular on-commit code didn't kick in and > forced syncs only happen once a day. Are we worried about bandwidth wastage that we don't run it more often? -- Alvaro Herrera http://www.CommandPrompt.com/ The PostgreSQL Company - Command Prompt, Inc.
On Sat, 27 Jun 2009, Alvaro Herrera wrote: > Stefan Kaltenbrunner escribió: >> Greg Stark wrote: >>> do we not have --delete on the rsync command? why doesn't the problem >>> go away on the next sync if we do? >> >> we do use --delete - but nobody commited after we tagged and before the >> problem got reported so the regular on-commit code didn't kick in and >> forced syncs only happen once a day. > > Are we worried about bandwidth wastage that we don't run it more often? More often then when ppl commit? What would be the point to that? AT least, the way I understood it setup, when someone does a commit, it initiates a push to the anoncvs repository ... ---- Marc G. Fournier Hub.Org Networking Services (http://www.hub.org) Email . scrappy@hub.org MSN . scrappy@hub.org Yahoo . yscrappy Skype: hub.org ICQ . 7615664
On Sat, Jun 27, 2009 at 8:09 PM, Marc G. Fournier<scrappy@hub.org> wrote: > On Sat, 27 Jun 2009, Alvaro Herrera wrote: > >> Stefan Kaltenbrunner escribió: >>> >>> Greg Stark wrote: >>>> >>>> do we not have --delete on the rsync command? why doesn't the problem >>>> go away on the next sync if we do? >>> >>> we do use --delete - but nobody commited after we tagged and before the >>> problem got reported so the regular on-commit code didn't kick in and >>> forced syncs only happen once a day. >> >> Are we worried about bandwidth wastage that we don't run it more often? > > More often then when ppl commit? What would be the point to that? Well, the whole point of using rsync is that it only copies what has changed, so there is very little overhead involved in running it when nothing has changed. If our current system is adding an additional layer of "did anything change?" checking, it's unnecessarily duplicating functionality which is already implemented really, really well by rsync. If updating the exclude list is sufficient to fix the problem, maybe it's not worth worrying about. But on the other hand maybe we should just rsync it every 15 minutes and forget about it. ...Robert
Robert Haas <robertmhaas@gmail.com> writes:
> On Sat, Jun 27, 2009 at 8:09 PM, Marc G. Fournier<scrappy@hub.org> wrote:
>> More often then when ppl commit? �What would be the point to that?
> Well, the whole point of using rsync is that it only copies what has
> changed, so there is very little overhead involved in running it when
> nothing has changed.
This is kind of missing the point, which is that we are talking about
blowing away lockfiles. Having that happen frequently and automatically
doesn't sound like a particularly good idea.
One thing I'm wondering is why cvs is taking out locks at all, since
it's certainly not committing to the repository. I guess even a "cvs
update" locks things momentarily, but can we prevent that?
regards, tom lane
Marc G. Fournier escribió: > On Sat, 27 Jun 2009, Alvaro Herrera wrote: > >> Stefan Kaltenbrunner escribió: >>> Greg Stark wrote: >>>> do we not have --delete on the rsync command? why doesn't the problem >>>> go away on the next sync if we do? >>> >>> we do use --delete - but nobody commited after we tagged and before the >>> problem got reported so the regular on-commit code didn't kick in and >>> forced syncs only happen once a day. >> >> Are we worried about bandwidth wastage that we don't run it more often? > > More often then when ppl commit? What would be the point to that? Avoiding the precise problem that this thread is about, of course. -- Alvaro Herrera http://www.CommandPrompt.com/ PostgreSQL Replication, Consulting, Custom Development, 24x7 support
Robert Haas wrote: > On Sat, Jun 27, 2009 at 8:09 PM, Marc G. Fournier<scrappy@hub.org> wrote: >> On Sat, 27 Jun 2009, Alvaro Herrera wrote: >> >>> Stefan Kaltenbrunner escribió: >>>> Greg Stark wrote: >>>>> do we not have --delete on the rsync command? why doesn't the problem >>>>> go away on the next sync if we do? >>>> we do use --delete - but nobody commited after we tagged and before the >>>> problem got reported so the regular on-commit code didn't kick in and >>>> forced syncs only happen once a day. >>> Are we worried about bandwidth wastage that we don't run it more often? >> More often then when ppl commit? What would be the point to that? > > Well, the whole point of using rsync is that it only copies what has > changed, so there is very little overhead involved in running it when > nothing has changed. If our current system is adding an additional > layer of "did anything change?" checking, it's unnecessarily > duplicating functionality which is already implemented really, really > well by rsync. The point is that there is a lot less overhead, not necessarily very little. And most of the difference is in network overhead - not disk (or cpu). It still has to scan the full repository on both sides and compute the hashes for all files. Then transfer those across the net. Just to find out if there is anything to do. Whereas our current solution requires checking the existence of a single file, and then it will know if there is anything to do. Running rsync every 5 minutes *does* have a serious performance impact. The reason our current system was invented at all is that with rsync, we simply couldn't sync often enough. We had it at bi-hourly I think, and that work but created a noticeable load on the anoncvs machine. > If updating the exclude list is sufficient to fix the problem, maybe > it's not worth worrying about. But on the other hand maybe we should > just rsync it every 15 minutes and forget about it. Remember that we can't push the timing too far on it, because the cvs server will send out links to cvsweb right away, and those point to the anoncvs machine. 15 minutes *might* work, but not much more than that. Fact is, that *removing* the forced sync would likely fix this issue, since it will then perform a sync when the tag operation is complete - not in the middle of it. But I still think it's a good thing to have a fallback sync. We could make it run a bit more often (say 4 times a day instead of 1 or something) which would shorten the time if this happens again. -- Magnus HaganderSelf: http://www.hagander.net/Work: http://www.redpill-linpro.com/
Tom Lane wrote: > Robert Haas <robertmhaas@gmail.com> writes: >> On Sat, Jun 27, 2009 at 8:09 PM, Marc G. Fournier<scrappy@hub.org> wrote: >>> More often then when ppl commit? What would be the point to that? > >> Well, the whole point of using rsync is that it only copies what has >> changed, so there is very little overhead involved in running it when >> nothing has changed. > > This is kind of missing the point, which is that we are talking about > blowing away lockfiles. Having that happen frequently and automatically > doesn't sound like a particularly good idea. > > One thing I'm wondering is why cvs is taking out locks at all, since > it's certainly not committing to the repository. I guess even a "cvs > update" locks things momentarily, but can we prevent that? the problem is that we transferred the locks from cvs.postgresql.org to anoncvs.postgresql.org which confused cvs running on anoncvs. The system as set up is by default triggered by somebody commiting - however there is also a cronjob that does one forced sync every day. The forced sync ran concurrently with marcs tagging of 8.4.0 and caused some lockdirectories to get synced over to anoncvs. I think that simply excluding those lock directories is enough to fix this for our purpose. Stefan
On Sun, Jun 28, 2009 at 9:19 AM, Magnus Hagander<magnus@hagander.net> wrote: > Remember that we can't push the timing too far on it, because the cvs > server will send out links to cvsweb right away, and those point to the > anoncvs machine. 15 minutes *might* work, but not much more than that. Yeuch. I often look at paches as soon as I see the email on -committers. Not having to wait for the changes to propagate was a definite improvement that I wouldn't want to see reverted. Besides, as far as I can see - Stefan's fix should resolve the issue entirely. -- Dave Page EnterpriseDB UK: http://www.enterprisedb.com
On Jun 28, 2009, at 4:28 AM, Dave Page <dpage@pgadmin.org> wrote: > On Sun, Jun 28, 2009 at 9:19 AM, Magnus > Hagander<magnus@hagander.net> wrote: > >> Remember that we can't push the timing too far on it, because the cvs >> server will send out links to cvsweb right away, and those point to >> the >> anoncvs machine. 15 minutes *might* work, but not much more than >> that. > > Yeuch. I often look at paches as soon as I see the email on > -committers. Not having to wait for the changes to propagate was a > definite improvement that I wouldn't want to see reverted. > > Besides, as far as I can see - Stefan's fix should resolve the issue > entirely. Triggered is definitely the way to go if it can be made robust, which it sounds like it can. Otherwise, reducing the recovery time when there *is* a problem is second-best. ...Robert
Stefan Kaltenbrunner <stefan@kaltenbrunner.cc> writes:
> Tom Lane wrote:
>> One thing I'm wondering is why cvs is taking out locks at all, since
>> it's certainly not committing to the repository. I guess even a "cvs
>> update" locks things momentarily, but can we prevent that?
> the problem is that we transferred the locks from cvs.postgresql.org to
> anoncvs.postgresql.org which confused cvs running on anoncvs.
Ah, now I understand. Yes, sounds like a simple --exclude will fix it.
regards, tom lane
On Sun, Jun 28, 2009 at 4:17 PM, Tom Lane<tgl@sss.pgh.pa.us> wrote: > Stefan Kaltenbrunner <stefan@kaltenbrunner.cc> writes: >> the problem is that we transferred the locks from cvs.postgresql.org to >> anoncvs.postgresql.org which confused cvs running on anoncvs. > > Ah, now I understand. Yes, sounds like a simple --exclude will fix it. From our experience, it's always better to put the locks outside of the repository. You should be able to set: LockDir=/var/lock/cvs/your-repo in the CVSROOT/config file. (with /var/lock/cvs/your-repo a directory where every user accessing the repository can write - readonly users included). -- Guillaume