Savannah Internals

This page describes inner workings of the GNU Savannah platform (web frontend + backend processes).

Some terminology

GNU Savannah (or just Savannah)
The name of the entire platform, encompassing the various hosts (see SavannahArchitecture), services (see SavannahServices), people (see SavannahTeam) and other odds and ends.
Savane (French for 'savannah')
The code name for the source code which runs many parts of the Savannah platform (most notably: the web frontend at savannah.gnu.org).

While usually these are distinguished, they are sometimes used interchangeably, and should be appropriately understood by the context.

groups
In savane code, software development projects are an informal subset of "groups" (examples of groups that are not projects in that sense are GNU and FSF audio and video and the Spanish GNU Translation Team. The database table is named `groups`, and several PHP files are named `groupXXX.php`. `group_id=N` in URLS refer to group internal ID.
Savannah Administrators
Volunteers with special access permissions on the Savannah frontend website (see 'becoming super-user' below) and administrator ssh access to the Savannah hosts. See SavannahTeam and List of current administrators.
Group Administrators
Savannah users who have administrative access to their group. As an example, these people are group administrators for GNU AWK.

Savannah source code

The source code for the PHP frontend and most of the backend scripts is available at Savane Git repository. N.B. That's the administration group, and the git repository named savane within it. This repository is the only official one.

Warning about repositories

There are several other repositories/clones/forks which look like the savane repositories (e.g., savane-cleanup). All code running on Savannah hosts comes from the administration group, using several version control systems there: git, cvs, subversion; each of which holds different stuff.

Savannah hosts

See SavannahArchitecture and SavannahServices.

savannah-internals-2015.png

savannah-internals-2015.dia

Workflow - overview

  1. A user submits a new group. Savannah administrators approve the group (see ApprovingSubmission).
  2. Group administrators of existing group can change group settings (e.g., add a mailing list, switch from cvs to git, etc.).
  3. The PHP code on frontend updates the MySQL database, and optionally runs one-time creation scripts.
  4. On vcs and download, cron jobs regularly query the database for updates, and make the necessary changes (e.g., create a new git repository for a new group).
  5. The cvsweb repositories hold the files which are the source for a group web pages. When these are updated, a special hook calls a CGI script on www.gnu.org to request a website update.
  6. A cron job on www.gnu.org regularly checks for update requests, and cvs updates the requested repositories.
  7. Obsolete.
  8. Publishing a new tarball release for GNU packages is performed by an automated process, explained in the GNU Maintainer's guide. It is not managed by Savannah administrators, and only mentioned here for completeness.
  9. GNU Bug Tracker (http://debbugs.gnu.org) is based on Debian Bug Tracking System. It is not managed by Savannah administrators. Packages hosted on Savannah, either GNU or non-GNU, can request to use the bug tracker. More information: http://debbugs.gnu.org/Using.html.
  10. For non-GNU groups, group members can upload new files to be available for download as 'releases' (e.g., http://download.savannah.gnu.org/releases/freetype/). While GNU groups can also store files this way, it is very strongly recommended that official GNU packages use ftp.gnu.org via the automated ftp uploads (see item 8, above). For details, see download section in SavannahServices. See also UserAuthentication.
  11. Group members can push code to the repositories on vcs using ssh. See 'vcs' section in SavannahServices for configuration details. See CvsGettingStarted, Svn, UsingHg, UsingBzr, UsingGit for usage information, and UserAuthentication for ssh authentication details.

User registration (frontend)

Anyone with a valid email account can register on GNU Savannah.

Warning: the email that Savannah sends will often be considered spam by big email providers, or outright rejected.

For further details, see IdleAccounts.

Group submission (frontend)

Any Savannah user can submit a new group request. This procedure is for non-GNU groups only (for GNU packages, the first and additional step is evaluating software for GNU.

  • Public URL: https://savannah.gnu.org/register/
  • Source code: /frontend/php/register/index.php (and see all files in /frontend/php/register/ ).
  • After form submission, The php confirmation.php file in register/:
    • updates the database record in the groups table, with STATUS='P'.
    • sends a notification email to savannah-register-public@gnu.org.
    • Creates a new task in the task tracker (FIXME: how?)
    • FIXME: the source code mentions something about cookbook and recipes database tables?
  • The group is in pending state until approved by Savannah administrators.

Group approval, creation (frontend)

Savannah admins approve and create groups after careful and manual review. Group creation is the same for GNU groups.

  • Public URL: https://savannah.nongnu.org/siteadmin/groupedit.php?group_id=N (only available to Savannah admins in superuser mode).
  • The URL is display as 'group administration' link when viewing a pending group ticket.
  • Source code: /frontend/php/siteadmin/groupedit.php.
  • In this page the following changes can be made:
    • system name (i.e. 'short name') - the name will be used in URLs, as a Unix group name for access control, and as directory names in various locations. Must be unique.
    • status: pending,active,deleted,maintenance,incomplete.
    • license
    • visibility (public/private)

    These changes are written to the groups table.

  • Two special commands are available in this page as well:
    savannah-internals-group-edit-icons.png
    • The green "V" icon: calls groupedit.php again with cgi parameters updatefast=1 and status=A, thus marking the group as active in the groups table, and in Savannah.
    • The configure (screwdriver and wrench) icon: calls /frontend/php/siteadmin/triggercreation.php, which calls the necessary one-time group creation scripts.
  • The triggercreation.php script does the following:
    • Sets group defaults (e.g., having a homepage, having 'downloads', using cvs repository, using task, support, bug trackers, using news, etc.). This are fields such as use_homepage=1 in the groups table.
      The items set by default are hard-coded in triggercreation.php. (this is one reason why by default, all new groups on Savannah start with a cvs repository). These settings can later be changed by the group administrators.
    • Sends an email to the group administrator (based on the Savannah user who registered the group).
  • FIXME: triggercreation.php mentions "site-specific triggers executed"
    • which ones are these and where are they executed?

Group administration - general information (frontend)

Group admins can modify various items for their groups using the Savannah web interface.

Group administration - feature selection (frontend)

Group admins can enable and disable features for their groups using the Savannah web interface.

  • features in this context are the facilities used by the group, such as git, cvs, homepage, downloads, news, bugs/support/patches/tasks trackers, mailing lists, etc.
    Each feature has a corresponding field in the groups table (e.g., use_git, use_homepage, use_downloads) which is set to 1 or 0.

  • Some features are associated with a URL. The URLs for these features are initialized to the Savannah defaults at group creation, but can be later changed by the group administrators. Examples:

    1. If use_homepage=1, the url_homepage field defaults to http://www.gnu.org/software/GROUP (for GNU groups) or http://www.nongnu.org/GROUP (for non-GNU groups). Group administrators can change this, although GNU packages should not do so.
    2. If use_git=1, the url_git field defaults to git/?group=GROUP (which resolves to https://savannah.gnu.org/git/?group=GROUP, a page containing instructions about accessing git on Savannah for this group). Group administrators can change this to any other URL, although they should not do so, especially not GNU packages. (The purpose of Savannah is to host source code.)

  • Public URL: https://savannah.gnu.org/project/admin/editgroupfeatures.php?group=GROUP (only available to group admins).

  • Source code: /frontend/php/project/admin/editgroupfeatures.php.

  • When updating features (selecting or deselecting features), the changes are saved to the database, in the groups table. No other changes are performed.

  • These database changes will be picked up by the cron jobs running on other VMs (see below). This is the reason why some features take time to propagate (e.g., when a group administrator turns on git, the git repository is not created immediately). It would be better to do things synchronously, or at least very quickly.

Group administration - mailing lists (frontend)

Group admins can add, configure and delete mailing lists for their groups using the Savannah web interface.

  • Mailing lists require special handling: the mailing list server (lists.gnu.org, using GNU Mailman) is administered by the FSF, not Savannah.
  • Special configuration on lists.gnu.org allows remote execution of shell scripts from frontend to automate list management. More info: ListServer.
  • Public URL: https://savannah.gnu.org/mail/admin/?group=GROUP (only available to group admins).
  • The PHP code updates the mail_group_list table and runs the scripts to update the mailing lists on lists.gnu.org.

MySQL Database

See SavannahDatabaseSchema and SavannahDatabaseExamples.

Cron job on vcs

Some cron jobs may still be running on the old vcs VM still in use and accessible as vcs. The intent is to have everything moved off of there eventually.

Stored in vcs0:/etc/cron.d/sv:

# remake user list for outgoing mail.
*/10 * * * *    root    nice -n 11 sv_aliases --cron && sed -e '/# Savannah include start/,/# Savannah include end/s/\([^:]\+\): .*/\1: INVALID.NOREPLY@gnu.org/' /etc/email-addresses -e 's,This is /etc/email-addresses,DO NOT EDIT - GENERATED FROM email-addresses.,' > /etc/email-addresses_SENDER

# remake list of git repositories in each group.
*/15 * * * *    root    sv_cgit.pl

# remake list of groups for the vcs.
35,05 * * * *   root    nice -n 11 sv_groups --cron --only-cvs --only-homepage --only-svn --only-git --only-hg --only-bzr

  • /backend/mail/sv_aliases - This script prepares an alias-type file for for exim4 mail server, used when sending commit updates to mailing lists.

    • The script queries the MySQL database, extracting user names and emails for all Savannah users. Additionally, emails for 'squads' (per-group mailing groups) are queried. The results are saved in vcs0:/etc/email-addresses.

    • vcs0:/etc/email-addresses is symlinked to the standard vcs0:/etc/exim4/email-addresses file. Example:

        agn: assafgordon@gmail.com
  karl: karl@gnu.org

    • The sed command in the cronjob file replaces each user's email with INVALID.NOREPLY@gnu.org, and saves the results in vcs0:/etc/email-addresses_SENDER. Example:

        agn: INVALID.NOREPLY@gnu.org
  karl: INVALID.NOREPLY@gnu.org

    • FIXME: where is /etc/email-addresses_SENDER mentioned/used? in an exim4 config file?

  • sv_cgit.pl - this script updates the vcs0:/srv/git/cgitrepos file which is used by the cgit CGI application.

    • cgit configuration file is vcs0:/etc/cgitrc.

  • sv_groups - This script creates new repositories for groups in the Savannah database

    • reminder: when a new group is approved on Savannah, or when a group administrator selects a new repository feature the frontend PHP code only updates the database. this script picks up the changes and executes them.
    • The script queries the groups table, and creates the needed repositories.
      Example: For group XXX, if groups.use_git=1, it look for vcs0:/srv/git/XXX.git. If it doesn't exist, create it with git init and additional Savannah-specific configurations (e.g. permissions, hooks, xattrs).
    • Repository initialization and configuration is delegated to the relevant perl module. See Git.pm, Cvs.pm, Bzr.pm, Svn.pm, Hg.pm, Download.pm.
    • Two types of CVS repositories can be created: cvs (Using CvsMakeArea in Cvs.pm) - source code repository; webcvs (Using WebCvMakeAreaSavannahGNU in Cvs.pm) - webcvs repository (files will be available on www.gnu.org/www.nongnu.org). The same routines also update existing CVS hooks.
    • The script takes parameteres determining which items to create (e.g. --only-git).
    • On vcs host, the git/hg/bzr/cvs/cvsweb repositories are created.
    • On download host, the download are created.

Cron jobs on download

Some cron jobs may still be running on the old download VM still in use but now accessible as olddownload. The intent is to have everything moved off of there eventually.

stored in download0:/etc/cron.d/sv:

*/30 * * * *    root    sv_groups --cron --only-download
  • sv_groups - This script creates new repositories for groups listed in the Savannah database. See description for sv_groups in the above section.

Cron jobs on frontend

Basically, none. Wiki refresh is run once an hour, but this isn't really needed.

Cron jobs on mgt1

Stored in mgt1:/etc/cron.d/savannah:

# Assign uidNumber and gidNumber in the database (and rebuild cache
# for libnss-mysql-bg). 77211 was the latest group created by the old
# sv_groups cron - maybe one day we'll move everything to the 70000 range.
*/5 * * * *   root   sv_assign_uid_gid --min-gidNumber=77212

# Frontend-related:
#
# This script does not need root access, access to the database is enough.
*/25 * * * *    root    sv_cleaner
48 12 2 * *     root    sv_cleaner --big-cleanup

4 4,18 * * *    root    sv_reminder

  • sv_assign_uid_gid - creates unique user ids and group ids for users and groups in the Savannah system.

    • reminder: across all Savannah hosts, Unix user/group ids are managed with nsswitch/libnss-mysql (see UserAuthentication for details).
    • When a new user (or group) is created on Savannah, the PHP code on frontend does not assign uid/gid.
    • This script picks up new users (SQL: user.uidNumber is NULL) and new groups (SQL: groups.gidNumber is NULL) and assigns them ids.

  • sv_cleaner - Removes parts of the database that belong to removed entities.

  • sv_reminder - sends email reminders to users.

www.gnu.org update mechanism

Mailing list configuration

Additional functionality

super-user and impersonate on frontend

frontend configuration files

SavaneConfiguration

git hooks on vcs

cvsweb hooks on vcs