Mirroring at Savannah

NOTE: This is an updated version of the previous Mirmon page.

Savannah offers two public-facing mirroring services:

  1. ftpmirror.gnu.org - redirects to world-wide mirrors of ftp.gnu.org. These mirrors carry official releases of GNU pacakges. ftpmirror.gnu.org and gnu mirror are used interchangeably in this page.

  2. download.savannah.{nongnu,gnu}.org - redirects to world-wide mirrors of savannah's DownloadArea. Typically used for non-gnu software, but gnu packages can store files there as well. download and nongnu mirror are used interchangeably in this page.

Client-side overview

A user requesting a download from ftpmirror.gnu.org will receive an HTTP/302 response redirecting to a mirror closest to the user's originating IP:

$ curl -o /dev/null -D /dev/stdout --silent http://ftpmirror.gnu.org/sed/sed-4.4.tar.xz
HTTP/1.1 302 Found
Location: http://gnu.mirrors.hoobly.com/gnu/sed/sed-4.4.tar.xz
[...]

A user requested a download from download.savannah.gnu.org will be redirected to an intermediate redirection URL, and then to the mirror:

$ curl -o /dev/null -D /dev/stdout --silent http://download.savannah.gnu.org/releases/test-project/README
HTTP/1.1 302 Found
Location: http://download.savannah.gnu.org/releases-redirect/test-project/README
[...]

$ curl -o /dev/null -D /dev/stdout --silent http://download.savannah.gnu.org/releases-redirect/test-project/README
HTTP/1.1 302 Found
Location: http://nongnu.askapache.com//test-project/README
[...]

The technical reason for this extra redirection is that download.savannah.gnu.org serves directory listing directly, and only redirects actual files. See nginx details below.

Server-side redirection overview

Redirection is implemented using a perl cgi script, called from nginx/fcgiwrap running on download0.sv.gnu.org.

Both gnu and nongnu redirections are served from download0.savannah.gnu.org VM. ftpmirror.gnu.org is CNAME'd to ftpmirror.sv.gnu.org, which points to the same IP as download0 (remember savannah admin's only control *.sv.gnu.org DNS entries).

The nginx configuration files are: download0:/etc/nginx/sites-available/ftpmirror, download0:/etc/nginx/sites-available/ftpmirror-common.inc, download0:/etc/nginx/sites-available/download, download0:/etc/nginx/sites-available/download-common.inc. The *-common.inc files contain the cgi configurations. The other files include them twice (once for http, once for https. TODO: consolidate them, like Bob did for vcs0).

The CGI configuration for gnu/ftpmirror is:

location / {
    fastcgi_param PATH_INFO            $uri;
    fastcgi_param MIRROR_FILE          /opt/savannah/mirror-lists/gnu-ftpmirrors.txt;
    fastcgi_param SCRIPT_FILENAME      /opt/savannah/cgi-bin/mirrors/mirror-redirect.pl;
    include fastcgi_params;
    fastcgi_pass unix:/var/run/fcgiwrap.socket;
}

The CGI configuration for nongun/download is:

# redirect to the CGI if it's a file URL
rewrite ^/releases/(.*[^/])$ /releases-redirect/$1 last;

# directory listing if it's a directory
location /releases {
    alias /srv/download;
    fancyindex on;              # Enable fancy indexes.
}

# actual redirection mechanism
location /releases-redirect {
    fastcgi_split_path_info            ^(/releases-redirect)(/?.+)$;
    fastcgi_param PATH_INFO            $fastcgi_path_info;
    fastcgi_param MIRROR_FILE          /opt/savannah/mirrors/active-mirror-lists/download-mirrors.txt;
    fastcgi_param SCRIPT_FILENAME      /opt/savannah/cgi-bin/mirrors/mirror-redirect.pl;
    include fastcgi_params;
    fastcgi_pass unix:/var/run/fcgiwrap.socket;
}

The mirror list text files (gnu-ftpmirror and download-mirrors.txt) contain an auto-generated list of active mirrors:

# head -n5 /opt/savannah/mirrors/active-mirror-lists/gnu-ftpmirror
http://mirror.tochlab.net/pub/gnu/ ru
http://mirror.cedia.org.ec/gnu/ ec
http://www.gutscheinrausch.de/mirror/gnu/ de
http://www.artfiles.org/gnu.org/ de
http://gnu.mirrors.linux.ro/ ro

See 'mirmon' section below to learn how these files are generated.

The perl script mirror-redirect.pl uses Geo::IP and Geo::Mirror modules to read the mirror file, detect the closest region based on the user's IP, and return an HTTP redirect to that mirror.

Server-side Mirmon overview

NOTE: Most of this section was copied from the Mirmon page and updated as needed.

  1. The authoratative lists of gnu/nongnu mirrors are managed manually. For gnu/ftpmirrors, FSF admins or GNU webmasters update the lists (http://www.gnu.org/prep/ftp.html, generated from http://www.gnu.org/prep/FTP). For nongnu/download mirrors, savannah admins update this file (and html version).

  2. mirmon is used to checks each mirror and determine how up to date it is.

    We maintain 3 separate mirmon configurations:

    • gnu - list of HTTP-only mirrors for ftpmirror.gnu.org. The gnu list is used later by the redirection cgi script.
    • allgnu - list of HTTP,FTP,RSYNC mirrors for ftpmirror.gnu.org. The allgnu list is used only by human admins to check which mirrors are up-to-date.
    • nongnu - list of HTTP-only mirrors for download.savannah.gnu.org. The nongnu list is used later by the redirection cgi script.
  3. The mirmon input lists (of mirrors to check) are here:

    The lists are stored here:

    $ ls -lhog /opt/savannah/mirrors/mirmon-lists/
    -rw-r--r-- 1 7.7K Feb 20 23:30 allgnu-mirror-list.txt
    -rw-r--r-- 1 4.0K Feb 20 23:30 gnu-mirror-list.txt
    lrwxrwxrwx 1   28 Feb 20 23:28 nongnu-mirror-list.txt -> /srv/download/00_MIRRORS.txt
    

    The script /opt/savannah/mirrors/scripts/get-gnu-mirror-list.pl fetches the gnu/ftpmirror list and saves it locally in the two versions (gnu and allgnu). The nongnu list is symlinked to the manually-managed file.

    These files are the input to mirmon.

  4. The three mirmon configuration files are:

    $ ls -1 /etc/mirmon/
    mirmon-allgnu.conf
    mirmon-gnu.conf
    mirmon-nongnu.conf
    

    When mirmon is executed (once for each configuration), it reads the corresponding list file, and generates an HTML report page and a 'state' textual file:

    /opt/savannah/www/mirmon/gnu/index.html
    /opt/savannah/www/mirmon/nongnu/index.html
    /opt/savannah/www/mirmon/allgnu/index.html
    
    # ls -lhog /var/lib/mirmon/state-*
    -rw-r--r-- 1  16K Feb 21 19:42 /var/lib/mirmon/state-allgnu
    -rw-r--r-- 1 7.3K Feb 21 20:22 /var/lib/mirmon/state-gnu
    -rw-r--r-- 1 3.5K Feb 21 19:32 /var/lib/mirmon/state-nongnu
    

    A 'state' file contains the status for each mirror:

    $ head -n5 /var/lib/mirmon/state-gnu
    http://ca.mirror.babylon.network/gnu/ 1487646961 ok 1487651999 s 1487651999-s 1487651999
    http://download.nust.na/pub/ftp.gnu.org/gnu/ 1487643361 ok 1487651999 s 1487651999-s 1487651999
    http://fr.mirror.babylon.network/gnu/ 1487643361 ok 1487651999 s 1487651999-s 1487651999
    http://ftp-gnu-org.ip-connect.vn.ua/ 1487625361 ok 1487651999 s 1487651999-s 1487651999
    http://ftp.acc.umu.se/mirror/gnu.org/gnu/ 1487556961 ok 1487651999 s 1487651999-s 1487651999
    
  5. Two auxiliary scripts /opt/savannah/mirrors/scripts/update-active-mirrors-{gnu,nongnu}.sh read the gnu/nongnu state files, and generate a simple text file containing only the active and up-to-date mirrors:

    $ ls -1 /opt/savannah/mirrors/active-mirror-lists/
    download-mirrors.txt
    gnu-ftpmirrors.txt
    

    These are the two files used in the nginx/cgi scripts (see above section).

  6. The 3 mirmon invocations are executed as cronjobs in /etc/cron.d/sv-mirmon:

    # Update the authoratative list of GNU ftp mirrors (mirroring ftp.gnu.org).
    3 1 * * * root /opt/savannah/mirrors/scripts/get-gnu-mirror-list.pl --http-only /opt/savannah/mirrors/mirmon-lists/gnu-mirror-list.txt
    6 1 * * * root /opt/savannah/mirrors/scripts/get-gnu-mirror-list.pl --all       /opt/savannah/mirrors/mirmon-lists/allgnu-mirror-list.txt
    
    # Check the status of each mirror (and regenarate the HTML report page).
    # For gnu/nongnu - also update the list of active mirrors (for the mirror CGIs).
    22 * * * * root /usr/bin/mirmon -c /etc/mirmon/mirmon-gnu.conf -q -get update >/var/log/mirmon/gnu 2>&1 && /opt/savannah/mirrors/scripts/update-active-mirrors-gnu.sh
    
    32 * * * * root /usr/bin/mirmon -c /etc/mirmon/mirmon-nongnu.conf -q -get update >/var/log/mirmon/nongnu 2>&1 && /opt/savannah/mirrors/scripts/update-active-mirrors-nongnu.sh
    
    42 * * * * root /usr/bin/mirmon -c /etc/mirmon/mirmon-allgnu.conf -q -get update >/var/log/mirmon/allgnu 2>&1
    

Diagnostics and Troubleshooting

  1. The three mirmon HTML reports are viewable here: http://download0.savannah.gnu.org/mirmon/.

  2. An additional CGI script exercise the Geo::IP module and the generated files: http://download0.savannah.gnu.org/mirror-check/ and http://ftpmirror0.savannah.gnu.org/mirror-check/.

    The script is /opt/savannah/cgi-bin/mirrors/mirror-check.pl.

    The NGINX configurations are ftpmirror-common.inc and download-common.inc:

    location /mirror-check/ {
        fastcgi_param GNU_MIRROR_FILE      /opt/savannah/mirrors/active-mirror-lists/gnu-ftpmirrors.txt;
        fastcgi_param NONGNU_MIRROR_FILE   /opt/savannah/mirrors/active-mirror-lists/download-mirrors.txt;
        fastcgi_param SCRIPT_FILENAME      /opt/savannah/cgi-bin/mirrors/mirror-check.pl;
        include fastcgi_params;
        fastcgi_pass unix:/var/run/fcgiwrap.socket;
    }
    
  3. Mirmon cronjob logs are saved in /var/log/mirmon. Current implementation saves only the most recent run.

  4. Mirmon state files should be generated at 22/32/42 minutes of each hour (defined in the cron file):

    # ls -lhog /var/lib/mirmon/
    total 60K
    -rw-r--r-- 1 7.3K Feb 21 20:22 state-gnu
    -rw-r--r-- 1 3.5K Feb 21 20:32 state-nongnu
    -rw-r--r-- 1  16K Feb 21 20:42 state-allgnu
    

    The active mirror list files should have similar timestamp:

    # ls -lhog /opt/savannah/mirrors/active-mirror-lists/
    total 8.0K
    -rw-r--r-- 1  869 Feb 21 20:32 download-mirrors.txt
    -rw-r--r-- 1 2.6K Feb 21 20:22 gnu-ftpmirrors.txt