bugGNU Wget - Bugs: bug #20522, Issues with --retr-symlinks and...

 
 

bug #20522: Issues with --retr-symlinks and timestamps

Submitter:  Micah Cowan <micahcowan>
Submitted:  Wed 18 Jul 2007 07:46:37 PM UTC
   
 
Category:  Program Logic Severity:  3 - Normal
Priority:  6 Status:  Confirmed
Privacy:  Public Assigned to:  None
Originator Name:  Open/Closed:  Open
Release:  1.8.2 Operating System:  GNU/Linux
Reproducibility:  None Fixed Release:  None
Planned Release:  1.14 Regression:  None
Work Required:  0 - Hours Patch Included:  None
* Mandatory Fields

Add a New Comment Rich Markup
   

Sat 03 May 2025 03:51:22 PM UTC, comment #1: 

I had a look at this and it seems it is due to a limitation of FTP, and perhaps quite impractical to implement.

Files are discovered by listing contents of a directory and parsing the output. Therefore, we can only get the path of a symlink's target by parsing `ls' output.

Some FTP servers server provide a MDTM command which returns the timestamp of a file given its path: https://www.solarwinds.com/serv-u/tutorials/mdtm-mkd-mode-noop-sscn-xcrc-ftp-command

This is however not always available.

To provide support for resolving the timestamp of the target of a symlink, I see 2 ways off the top of my head:

  • maintain a data structure to
    • populate given the paths of all files to download
    • traverse to link symlinks to their file nodes
    • traverse again to download


  • or, for each symlink encountered
    • cd into the directory of the linked to file
    • list contents again
    • parse the contents again and compare file by file to find the target file


While the latter might be okay for one file, - although I would argue, not a very clean solution - it has the potential to explode in computational complexity once we do a recursive download with many symlinks.

I don't know what is being done about this in V2, but perhaps the best way to circumvent this is to use --retr-symlinks=no. This will download symlinks normal files.

As long as the source is trusted, the security concerns of having dangling symlinks that resolve to outside the local mirror might not be a practical concern.

mattiadb271 <mattiadb271>
Wed 18 Jul 2007 07:46:37 PM UTC, original submission:  

Forwarded from http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=162283

Hi,

Wget always downloads retrieved symlinks, even when the retrieved
file has already been successfully downloaded.

I invoked wget like this,


    wget \
              --cut-dirs=2 \
              --glob=on \
              --input-file=NetBSD-1.6-urlist \
              --append-output=NetBSD-1.6-log \
              --retr-symlinks \
              --timestamping \
              --force-directories \
              --no-host-directories \
              --non-verbose \
            &


the files in question have this ftp directory listing,


    lrwxrwxrwx   1 ftpuser  ftpusers 27 Sep 12 03:35 man.tgz -> ../../../shared/ALL/man.tgz
    lrwxrwxrwx   1 ftpuser  ftpusers 28 Sep 12 03:35 misc.tgz -> ../../../shared/ALL/misc.tgz


I ran wget and these two files, {man,misc}.tgz, were downloaded
successfully, ie, the symlinks were followed, and the
linked-to-files were downloaded, in addition, during this run many
other regular, ie, not symlinked files, were also downloaded.

The first indication of a problem is that the downloaded symlinked
files were given the mtime of the symlink, rather than the mtime
of the linked-to-file,


    -rw-r--r--    1 jeff     jeff 6640184 Sep 12 03:35 man.tgz
    -rw-r--r--    1 jeff     jeff 2333002 Sep 12 03:35 misc.tgz


Then, I ran wget again, and here is where the problem occurred.
What happened is that the previously downloaded regular files were
not downloaded again, as expected, however, the two symlinked
files were downloaded again, in their entirety.

This is a tremendous waste of resources, and it negates the whole
purpose for using timestamping in the first place.

Thanks,
--
Jeff Sheinberg

Micah Cowan <micahcowan>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by mattiadb271 (Posted a comment)
  • -email is unavailable- added by micahcowan (Submitted the item)
  •  

    Follow 5 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2008-08-21 micahcowan StatusNone Confirmed
        Planned Release1.12 1.14
    2007-08-01 micahcowan Priority5 - Normal 6
    2007-07-21 micahcowan Work RequiredNone 0 - Hours
    2007-07-20 micahcowan Planned ReleaseNone 1.12

    Back to the top

    Powered by Savane 3.15.
    Corresponding source code