bugGNU Wget - Bugs: bug #20522, Issues with --retr-symlinks and...


bug #20522: Issues with --retr-symlinks and timestamps

Submitter:  Micah Cowan <micahcowan>
Submitted:  Wed 18 Jul 2007 07:46:37 PM UTC
Category:  Program Logic Severity:  3 - Normal
Priority:  6 Status:  Confirmed
Privacy:  Public Assigned to:  None
Originator Name:  Open/Closed:  Open
Release:  1.8.2 Operating System:  GNU/Linux
Reproducibility:  None Fixed Release:  None
Planned Release:  1.14 Regression:  None
Work Required:  0 - Hours Patch Included:  None
* Mandatory Fields

Add a New Comment Rich Markup

Wed 18 Jul 2007 07:46:37 PM UTC, original submission:  

Forwarded from http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=162283


Wget always downloads retrieved symlinks, even when the retrieved
file has already been successfully downloaded.

I invoked wget like this,

    wget \
              --cut-dirs=2 \
              --glob=on \
              --input-file=NetBSD-1.6-urlist \
              --append-output=NetBSD-1.6-log \
              --retr-symlinks \
              --timestamping \
              --force-directories \
              --no-host-directories \
              --non-verbose \

the files in question have this ftp directory listing,

    lrwxrwxrwx   1 ftpuser  ftpusers 27 Sep 12 03:35 man.tgz -> ../../../shared/ALL/man.tgz
    lrwxrwxrwx   1 ftpuser  ftpusers 28 Sep 12 03:35 misc.tgz -> ../../../shared/ALL/misc.tgz

I ran wget and these two files, {man,misc}.tgz, were downloaded
successfully, ie, the symlinks were followed, and the
linked-to-files were downloaded, in addition, during this run many
other regular, ie, not symlinked files, were also downloaded.

The first indication of a problem is that the downloaded symlinked
files were given the mtime of the symlink, rather than the mtime
of the linked-to-file,

    -rw-r--r--    1 jeff     jeff 6640184 Sep 12 03:35 man.tgz
    -rw-r--r--    1 jeff     jeff 2333002 Sep 12 03:35 misc.tgz

Then, I ran wget again, and here is where the problem occurred.
What happened is that the previously downloaded regular files were
not downloaded again, as expected, however, the two symlinked
files were downloaded again, in their entirety.

This is a tremendous waste of resources, and it negates the whole
purpose for using timestamping in the first place.

Jeff Sheinberg

Micah Cowan <micahcowan>


(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:

No files currently attached


Depends on the following items: None found

Items that depend on this one: None found


Carbon-Copy List
  • -email is unavailable- added by micahcowan (Submitted the item)

    Follow 5 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2008-08-21 micahcowan StatusNone Confirmed
        Planned Release1.12 1.14
    2007-08-01 micahcowan Priority5 - Normal 6
    2007-07-21 micahcowan Work RequiredNone 0 - Hours
    2007-07-20 micahcowan Planned ReleaseNone 1.12

    Back to the top

    Powered by Savane 3.12.
    Corresponding source code