mainSavannah Administration - Support: sr #107341, made some disorder in my download...

 
 

sr #107341: made some disorder in my download directory ...

Submitter:  Torsten Mueller <muellerto>
Submitted:  Wed 14 Apr 2010 07:08:56 AM UTC
   
 
Category:  Download area - developer access Priority:  5 - Normal
Severity:  3 - Normal Status:  Done
Privacy:  Public Assigned to:  karl
Operating System:  None Open/Closed:  Closed
* Mandatory Fields

Add a New Comment Rich Markup
   

Jump to the original submission

Fri 30 Apr 2010 05:11:49 PM UTC, comment #12: 

I systematically checked all our listed mirrors and removed the half dozen or so that were inaccessible and mailed the admins.

Surprisingly, every mirror which could be accessed at all had the correct pdrx directory contents, so everyone was deleting via rsync.

The zerg maintainer apparently fixed his problem, since http://mirrors.zerg.biz/nongnu/pdrx/ comes up clean now.  I reenabled that one.

If anyone notices continuing problems with mirrors, please say.

karl

Karl Berry <karl>
Site Administrator
Fri 23 Apr 2010 12:06:19 AM UTC, comment #11: 

i've disabled zerg -- temporarily, i hope -- and notified its admin.

dunno if we want to use this ticket to track the general issue.  i still have not found the time to systematically check the other mirrors.  if anyone else could pursue that, it would be great.

k

Karl Berry <karl>
Site Administrator
Wed 21 Apr 2010 09:43:56 PM UTC, comment #10: 

i've pinged the zerg admin once again (just before seeing this mail, as it happens :).  he's responded in the past.  if he doesn't get back in another day or so, i think we should just disable zerg from the redirects while it's straightened.

also, i think we need to systematically try every mirror so we can contact any others that aren't deleting.  it's been on my list ever since Torsten made his comment about "several mirrors".

Karl Berry <karl>
Site Administrator
Wed 21 Apr 2010 09:38:15 PM UTC, comment #9: 

Apparently mirrors.zerg.biz still has files in the top-level directory:


      bin-win32/              12-Apr-2010 19:01    -
      bin/                    14-Apr-2010 09:44    -
      pdrx-0.3.3.tar.bz2      31-Mar-2010 14:26   86K
      pdrx-0.3.4.tar.bz2      02-Apr-2010 12:54   88K
      pdrx-0.3.5-win32-bin..> 12-Apr-2010 09:27  1.7M
      pdrx-0.3.5.tar.bz2      12-Apr-2010 09:30  124K
      src/


On Savannah there's only:

[DIR]        bin/        14-Apr-2010 06:44          -
[DIR]        src/        12-Apr-2010 15:58          -


Apparently there was a problem with the cron (it didn't start - I suspect a weird naming convention issue in /etc/cron.d), it should be fix now. Anyway, this shouldn't have interfered, because I had already run the cron manually a couple times.

If within 24h there's still a problem on zerg, while other mirrors are fine (e.g. http://mirrors.aixtools.net/sv/pdrx/), then I suppose there's a problem on their side.

Sylvain Beucler <Beuc>
Mon 19 Apr 2010 11:51:42 PM UTC, comment #8: 


> chmod -r a+rwX


Maybe not -w ;)

I cron'd this:
cd /srv/download && (find -type d -print0 | xargs -r -0 chmod 2775) && (find -type f -print0 | xargs -r -0 chmod 664)

It only needs a few seconds to complete.

I chose not to mess with symlinks to be sure this won't start chmod'ing files outside of /srv/download - but symlink permissions cannot be changed under GNU/Linux afaict.


find . ! -perm /o=r
exibited a few files that had mistaken permissions, and a few top-level directories for private projects that had no files in them.  Hence I think it's safe to continue in this direction.

Sylvain Beucler <Beuc>
Mon 19 Apr 2010 11:28:59 PM UTC, comment #7: 

it is true that we used to recommend --ignore-errors.  But I noticed yesterday that the rsync documentation says this about it:

> Tells  --delete to go ahead and delete files even when there are I/O errors.


That does not seem like a good idea.  I/O errors are very likely real.  I don't see any options to ignore permission errors.  And I agree with you that it's better to fix them than to ignore them.

Summing up, something like chmod -r a+rwX from cron sounds like an excellent idea to me.



Karl Berry <karl>
Site Administrator
Mon 19 Apr 2010 10:58:37 PM UTC, comment #6: 


> I thought maintainers intentionally were given the power to
> change permissions in their release area. Not that I have a
> problem myself with making them all readable.


A few months ago I ran some 'find' to check what kind of permissions were in use.  It appears that all files were publicly available except for mistakes.  I also don't see why people would use a mirrorable download area to store private files (and people don't).

At that time I had fixed all permissions that could cause a rsync replication to choke.  Maybe we need to do that again, I'm not sure yet.

I think that at a point we asked mirror maintainers to use --ignore-errors, I'm not sure what the status of this now? Anyway, I think it is better if we don't use that option and fix permissions instead.

Sylvain Beucler <Beuc>
Mon 19 Apr 2010 10:37:29 PM UTC, comment #5: 

Yeah, I realized too late I was completely off-base with the -noredirect.  I fixed http://www.gnu.org/server/mirror.html.

I can contact all the mirror maintainers (or at least the ones with stale files), but first I want to be sure about what's going on.  I've been talking with the zerg.biz maintainer and he says he is including --delete but nonetheless the deletions are not happening.  I'm asking for a full transcript, etc...

If you can fix the permission errors, that would be all to the good.  but how can you fix them?  I thought maintainers intentionally were given the power to change permissions in their release area.  Not that I have a problem myself with making them all readable.

k


Karl Berry <karl>
Site Administrator
Mon 19 Apr 2010 10:35:17 AM UTC, comment #4: 

For clarity: rsync://dl.sv.gnu.org/releases-noredirect is invalid, so the documentation needs to be fixed back.

The big job is contacting the early mirror maintainers so they update their scripts.

Also,
"There will probably be some permission errors; just ignore them. They are due to choices or mistakes by individual savannah project maintainers.)"
=> I'm considering fixing these permissions errors by cron.  I added something similar at Gna! and it works fine so far.

Sylvain Beucler <Beuc>
Mon 19 Apr 2010 06:23:09 AM UTC, comment #3: 

Not at all!
All mirrors use rsync://, which doesn't redirect, and the 'no-redirect' is our internal URL for our own mirror.

If mirrors maintainers need to do something, it's update their rsync options so that removed files are also removed on their side (as documented).

Sylvain Beucler <Beuc>
Sun 18 Apr 2010 11:00:36 PM UTC, comment #2: 

Hi -- I think I see the problem now, namely that when we changed dl.savannah.nongnu.org to do an automatic redirect, we forgot that that was what all the mirrors were mirroring from.  So now mirrors may easily end up mirroring from another mirror, rather than us.

I will write the mirror maintainers about updating their target to rsync://dl.sv.gnu.org/releases-noredirect.  I updated our instructions for potential mirrors already.

Thanks for writing,
karl

Karl Berry <karl>
Site Administrator
Fri 16 Apr 2010 07:31:17 AM UTC, comment #1: 

I've seen now it's a problem on several mirrors. Some are up to date even in the directory structure. Others are not.

Torsten Mueller <muellerto>
Wed 14 Apr 2010 07:08:56 AM UTC, original submission:  

Two days ago I used sftp to manage my download place. I created some subdirectories to get more clearness and moved my files by downloading and uploading again onto the right place. Then I deleted the files on their old places. If I connect to my upload server (download.savannah.nongnu.org, /releases/pdrx) everything looks like it should be.

The mirroring process did recognize the new subdirectories and the files in them but not the removing of the files on their old places. If I look with the user's eyes onto a download mirror (mirrors.zerg.biz, /nongnu/pdrx) I see a mix of old an new files.

Could someone please repair this (these??) mirror(s)?

I'm sorry for this. I hope I will not produce such disorder in future. I learned a lot with this.

Torsten Mueller <muellerto>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by Beuc (Posted a comment)
  • -email is unavailable- added by karl (Posted a comment)
  • -email is unavailable- added by muellerto (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only logged-in users can vote.

     

    Follow 5 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2010-04-30 karl Open/ClosedOpen Closed
    2010-04-19 Beuc Open/ClosedClosed Open
    2010-04-18 karl StatusNone Done
        Open/ClosedOpen Closed
        Assigned toNone karl

    Back to the top

    Powered by Savane 3.13-02a9.
    Corresponding source code