mainSavannah Administration - Support: sr #108973, Problem with SVN server

 
 

sr #108973: Problem with SVN server

Submitter:  Joris van der Hoeven <vdhoeven>
Submitted:  Fri 19 Feb 2016 03:00:06 PM UTC
   
 
Category:  Source code repositories - developer access Priority:  5 - Normal
Severity:  3 - Normal Status:  Done
Privacy:  Public Assigned to:  rwp
Operating System:  None Open/Closed:  Closed
* Mandatory Fields

Add a New Comment Rich Markup
   

Jump to the original submission

Mon 29 Feb 2016 09:51:42 PM UTC, comment #7: 

To address the performance problems Ruben configured ext4 barriers off.  This makes ext4 perform like ext3 which defaults to barriers off. This is also the default in 2.6.37 and later.

After the config change this morning the disk latency and utilization metrics look much improved. For the past several days the disk utilization has been 100%. After this change the utilization is no longer pegged at 100% and latency is hugely reduced. This change makes the system usable.

Previously the number of service processes running would range between 40-60 active processes.  They just couldn't get enough I/O to finish.  After this change the number of service processes are between 10 and 20.  Much improved.

Bob Proulx <rwp>
Site Administrator
Sun 28 Feb 2016 07:42:48 PM UTC, comment #6: 

The vcs virtual machine and the underlying host is in a perpetual state of overload.  It is currently running a load of 20 for  the past few days.  It had a spike to 75 a few hours ago.  RAM use is reasonable.  cpu utilization is mostly negligent.  This is typical when the VM can't get I/O bandwidth from the underlying host server and so all tasks get blocked in the ready-to-run-but-waiting-for-I/O queue.

The VM system has quite a few I/O processes stacked up in the process queues.

  vcs:~# ps -ef | grep -c -e git -e cvs -e svn -e bzr -e hg
  45

Since it is a virtual machine the actual system performance stats are relatively fake so can't really be trusted.  One would need to look at the dom0 host for real stats.  It would be interesting to see the I/O rates.  But I assume that 45 download processes is more than it can comfortably handle.

This VM is shared with other VMs.  My general belief is that the total I/O for the entire stack of VMs overloads the underlying host system. When that happens even simple disk I/O in the VM is very slow.  That causes everything to queue for long times.

I will also note that the I/O bandwidth seen by the VMs is usually above 90 MB/sec.  Currently on vcs hdparm reports 7.67 MB/sec after being shared with all of the other I/O processes.  I have to assume that is what each of the above processes are seeing for disk I/O bandwidth.  Which is why they are crawling along so slowly.

Bob Proulx <rwp>
Site Administrator
Sat 27 Feb 2016 03:29:54 PM UTC, comment #5: 

Still extremely slow to run "svn commit". It's been like this for the last few weeks.

Gavin D. Smith <gavin>
Sat 27 Feb 2016 12:37:05 AM UTC, comment #4: 

Today it seems slow again, though not as bad as last week. For example, here are four timed attempts to sync a local working copy to the nongnu/lmi svn repo (which is small and has had no changes for a week or so):


$date --utc +'%Y-%m-%dT%H:%M:%SZ'; time svn update
2016-02-26T22:50:07Z
Updating '.':
At revision 6508.
svn update  0.08s user 0.22s system 44% cpu 0.656 total

$date --utc +'%Y-%m-%dT%H:%M:%SZ'; time svn update
2016-02-26T22:50:12Z
Updating '.':
svn: E170013: Unable to connect to a repository at URL 'http://svn.savannah.nongnu.org/svn/lmi/lmi/trunk'
svn: E175012: Connection timed out
svn update  0.03s user 0.12s system 0% cpu 10:00.23 total

$date --utc +'%Y-%m-%dT%H:%M:%SZ'; time svn update
2016-02-27T00:23:52Z
Updating '.':
At revision 6508.
svn update  0.06s user 0.20s system 2% cpu 11.203 total

$date --utc +'%Y-%m-%dT%H:%M:%SZ'; time svn update
2016-02-27T00:25:52Z
Updating '.':
At revision 6508.
svn update  0.06s user 0.19s system 52% cpu 0.469 total


Greg Chicares <chicares>
Sat 20 Feb 2016 06:03:50 PM UTC, comment #3: 

It took an hour of tinkering but eventually the problem was resolved.  I wish I could say there was one precise solution but it was impossible to know exactly what collection of services in total was overloading the system.

It is now the next day and things seem to be holding.  Therefore I am closing this ticket.

Bob Proulx <rwp>
Site Administrator
Fri 19 Feb 2016 11:24:26 PM UTC, comment #2: 

The system appears to be suffering from some type of DDoS attack.  The FSF admins are aware of this and are working on the problem.  See the out of band status information page for up to the moment status information.

https://pumprock.net/fsfstatus

All of the version control services (git, svn, cvs, and so forth) all live on the vcs server and all will be suffering from the same problems until it is resolved.

I will assign this ticket to me for tracking but I am simply monitoring the situation.  The FSF admins are the ones actually working on the problem.

Bob Proulx <rwp>
Site Administrator
Fri 19 Feb 2016 05:15:11 PM UTC, comment #1: 

This might be a server-wide outage, Git and CVS (which seems to be on
the same server at 208.118.235.72) are affected too; see also sr #108972.

I noticed the significant response latency on CGit access a day
before the outage too (had to wait nearly half-minute for each commit
page to show up).

Nutchanon Wetchasit <nachanon>
Fri 19 Feb 2016 03:00:06 PM UTC, original submission:  

This morning, I have been unable to do 'svn up' on my computer at work (nearby Paris). Pinging 'svn.savannah.gnu.org' works as expected, so it seems that the problem concerns the SVN service. I also noticed a significant slow-down of svn during the last one or two weeks, both on my computer at home and at my work.

Joris van der Hoeven <vdhoeven>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by gavin (Posted a comment)
  • -email is unavailable- added by chicares (Posted a comment)
  • -email is unavailable- added by chicares
  • -email is unavailable- added by rwp (Posted a comment)
  • -email is unavailable- added by nachanon (Posted a comment)
  • -email is unavailable- added by vdhoeven (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only logged-in users can vote.

     

    Follow 5 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2016-02-27 chicares Carbon-Copy- Added chicares
    2016-02-20 rwp StatusIn Progress Done
        Open/ClosedOpen Closed
    2016-02-19 rwp StatusNone In Progress
        Assigned toNone rwp

    Back to the top

    Powered by Savane 3.13-f8d8.
    Corresponding source code