mainSavannah Administration - Support: sr #109208, Spurious lock on gnubg cvs...

 
 

sr #109208: Spurious lock on gnubg cvs repository

Submitter:  Philippe Michel <plm>
Submitted:  Tue 20 Dec 2016 07:33:33 PM UTC
   
 
Category:  Source code repositories - developer access Priority:  5 - Normal
Severity:  3 - Normal Status:  In Progress
Privacy:  Public Assigned to:  agn
Operating System:  *BSD Open/Closed:  Closed
* Mandatory Fields

Add a New Comment Rich Markup
   

Jump to the original submission

Wed 21 Dec 2016 06:05:36 AM UTC, comment #6: 

Assaf, My NFS comment was a red herring. That didn't apply here. Sorry. I was mistaken. I believe the nobody files in the /var/lock/cvs are created by the pserver running as user nobody.

Those top level lock directories need to exist for cvs or cvs will complain with a fatal error.  Lower level directories are made by cvs on the fly. I believe that pserver also requires it even though pserver shouldn't need to lock anything. (An easy test is to take a test project and remove the top level directory and then check out using the pserver.)

With a default, default cvs installation those lock files go into the repository.  All ssh access is by the user which has write access by group access.  But with the pserver running as a non-priviledged user it can't do this.  With many pserver installations all cvs users and the pserver would be in a "pubcvs" group.  But that isn't good for isolating different savannah projects from each other.  Therefore the pserver running as any single group is problematic.  Therefore the lock directory is moved elsewhere out of the repository.

The file system holding those locks is on a tmpfs ram file system and evaporates every reboot.  Therefore at boot time they need to be created. If you look on vcs in /etc/init.d/cvs_lockdirs you will see the script someone previously wrote to do this. I migrated that to vcs0 in /etc/rc.local (for others reading, rc.local because the Trusquel 7 system uses upstart but the next one will be systemd and I didn't want to do three different init scripts). In rc.local I wrote the lock direction creation somewhat differently using find and -print0 because I didn't like using ls | xargs which has data dependent failure possibilities.

You can remove lower level directories.  They will be recreated as they are needed by cvs.  The top level directory needs to exist however. If created by an ssh user they will be owned by the ssh user. Probably we should ensure that the directories below are of the project group. I don't think that was being done before.

I have been looking on vcs expecting to find a cronjob that will create those top level directories when new cvs projects are created.  So far I have not found one. I have only found the creation of them at boot time. This looks like a bug that should be addressed. Probably not so many new cvs projects are being created these days and so not being tickled.

Bob Proulx <rwp>
Site Administrator
Wed 21 Dec 2016 04:40:24 AM UTC, comment #5: 

The root cause of the problem was a stale lock. I removed the stale lock file /run/lock/cvs/sources/gnubg/gnubg/#cvs.rfl.vcs0.15281 from the vcs0 system.  It was not associated with any running process and was three days old.

This was almost certainly due to some associated action with all of recent cvs migration activity.  Unless it appears again I am going to close the issue and think it is resolved.  However we are probably missing a routine cvs monitoring, reporting or cleanup action for stale lock files.

Bob Proulx <rwp>
Site Administrator
Wed 21 Dec 2016 04:24:43 AM UTC, comment #4: 

Hi Bob,

Not sure if it's related, but there's a cvs/lock directory on vcs0 in /var/lock/cvs/ .

It has files owned by 'nobody':

  root@vcs0:/tmp# ls -ld /var/lock/cvs/sources/gnubg
  drwxrwxrwx 3 root root 60 Dec 15 23:32 /var/lock/cvs/sources/gnubg

  root@vcs0:/tmp# ls -lA /var/lock/cvs/sources/gnubg
  total 0
  drwxrwxrwx 23 nobody nogroup 480 Dec 20 22:28 gnubg

  root@vcs0:/tmp# find /var/lock/cvs/sources/gnubg | wc -l
  50

  root@vcs0:/tmp# find /var/lock/cvs/sources/gnubg | head
  /var/lock/cvs/sources/gnubg
  /var/lock/cvs/sources/gnubg/gnubg
  /var/lock/cvs/sources/gnubg/gnubg/#cvs.rfl.vcs0.15281
  /var/lock/cvs/sources/gnubg/gnubg/xpm
  /var/lock/cvs/sources/gnubg/gnubg/win32
  /var/lock/cvs/sources/gnubg/gnubg/textures
  /var/lock/cvs/sources/gnubg/gnubg/sounds
  /var/lock/cvs/sources/gnubg/gnubg/scripts
  /var/lock/cvs/sources/gnubg/gnubg/po
  /var/lock/cvs/sources/gnubg/gnubg/po/gnubg-langs


Perhaps they were created by the new cvs binary running under "nobody" ?
I wonder if they are safe to remove, and we can test whether they are re-created...
I would think that cvs running from SSH runs as the actual user and thus not owned by 'nobody',
and 'cvs' running as 'pserver' shouldn't need to lock anything...


-assaf



Assaf Gordon <agn>
Site Administrator
Wed 21 Dec 2016 03:55:32 AM UTC, comment #3: 

I think I know some of the problem. For one the new machine is NFS mounting the old machine's file system. This is where the "nobody" comes into play.  I have intentionally not turned off the NFS root_squash and therefore anything root does is mapped to the nobody user.

Previously configured for CVS the locks are not kept in the repository.  This would not work for permissions from the pserver which runs as a non-priviledge nobody user.  Instead locks are kept off in a separate directory tree.  Previously this message would have reported it as root's lock.  Meanwhile I don't think this is significant to the root cause of the problem.

Having said all of that I don't know why gnubg wouldn't be able to commit.  I see no lock files there.  I am looking into the problem too.

Bob Proulx <rwp>
Site Administrator
Tue 20 Dec 2016 09:03:29 PM UTC, comment #2: 

Exactly. My CVS/Root is plm@cvs.savannah.gnu.org:/sources/gnubg
 and I got these  messages when trying to commit a change.


Philippe Michel <plm>
Tue 20 Dec 2016 08:25:02 PM UTC, comment #1: 

Hello,

Thank you for the report.

Just to double-check:
You see these messages when you connect with SSH to
    -email is unavailable- ?

-assaf

Assaf Gordon <agn>
Site Administrator
Tue 20 Dec 2016 07:33:33 PM UTC, original submission:  

For a few days (since the server migration probably) write operations on the gnubg repository hang with the following message :

cvs commit: [19:29:29] waiting for nobody's lock in /sources/gnubg/gnubg
cvs commit: [19:29:59] waiting for nobody's lock in /sources/gnubg/gnubg
cvs commit: [19:30:29] waiting for nobody's lock in /sources/gnubg/gnubg

Philippe Michel <plm>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by rwp (Posted a comment)
  • -email is unavailable- added by agn (Posted a comment)
  • -email is unavailable- added by plm (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only logged-in users can vote.

     

    Follow 3 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2016-12-21 rwp Open/ClosedOpen Closed
    2016-12-20 agn StatusNone In Progress
        Assigned toNone agn

    Back to the top

    Powered by Savane 3.13-0329.
    Corresponding source code