This file tries to shed some light on Guile's current module system.

Introduction
************

This document contains all information about the module system I have been able to deduce from the source code or from mailing list conversation.  I have written down everything while trying to figure out how the Guile module system actually works, and some of the text is taken from commentary in the file boot-9.scm.

The information contained herein is surely not complete, and I will be happy to receive additions, corrections and suggestions for improving it.

Also note that the information contained in this document reflects the current state as of the time writing, and the facts stated are not guaranteed to be stable.  A complete redesign of the module system is planned for a long time now, and actually might happen some day.

There are basically three views on the module system:

  * Data-centric: The module as a data type, with all procedures for creating and manipulating modules.  This is documented in the Module Data Type.

  * Declarational: How to use declarations like use-modules or define-module for dealing with modules and the scoping mechanisms they provide.  This issue is documented in the Guile Reference Manual, and will not be covered in detail here.

  * Internal: How do the module system and the Guile evaluator interact? This is documented in Modules and Evaluation.

Module Data Type
****************

This chapter will describe the module system from the point of the data type module.  Thus we will first see what this data type looks like and what operations are defined on it.

Modules in Guile are instances of a data type module.  A module is a record with the following fields.

obarray
     This is a hash table which contains all bindings made in the module.

uses-list
     List of the modules imported by the module.  A lot of search procedures search through all modules in this list when a specified binding cannot be found in the module's obarray.

lazy-binding-proc
     A procedure for determining a binding for a module lazily.  This is invoked if a binding for a name is requested from the module, but does not exist.  The procedure can create a new binding, by loading additional code or fetching it from another module/data structure.

eval-closure
     A procedure for determining a binding in a module.  The evaluator uses this procedure for accessing top-level variables.

transformer
     The syntax transformer used for all evaluations in the module.

name
     The name of the module.  This is a list of symbols, such as (guile) or (ice-9 popen).

kind
     A symbol describing the type of the module.  Normal modules are of type module, interfaces are of type interface and directories (modules which implement the hierarchical namespace) are of kind directory).  Autoload modules (placeholder for not-yet-loaded modules) are of type autoload.  FIXME: Are there other types?

observers
     A list of procedures to call when one of the module's bindings is removed or modified or a new binding is created.

weak-observers
     Similar to OBSERVERS, but this is a hash table from which the observer procedures will magically disappear when there are no other references to the procedure left except from the module they observe.

observer-id
     This is an integer, which specifies the identifer the next weak observer will get.  It is incremented each time a weak observer is added.

The Data Type
=============

A new module is created using make-module.  Several procedures are defined for accessing the members of a module.

 -- procedure: make-module size uses-list lazy-binding-proc
     Create a new module and initialize its fields with the parameters.

     size
          Size of the module's obarray.

     uses-list
          List of the modules imported by the module.

     lazy-binding-proc
          The procedure for determining a binding for a module lazily.

The various fields of a module can be accessed and modified using the following procedures:

 -- procedure: module-obarray module
 -- procedure: set-module-obarray! module obj
 -- procedure: module-uses module
 -- procedure: set-module-uses! module obj
 -- procedure: module-binder module
 -- procedure: set-module-binder! module obj
 -- procedure: module-eval-closure module
 -- procedure: set-module-eval-closure! module obj
 -- procedure: module-transformer module
 -- procedure: set-module-transformer! module obj
 -- procedure: module-name module
 -- procedure: set-module-name! module obj
 -- procedure: module-kind module
 -- procedure: set-module-kind! module obj
 -- procedure: module-observers module
 -- procedure: set-module-observers! module obj
 -- procedure: module-weak-observers module
 -- procedure: module-observer-id module
 -- procedure: set-module-observer-id! module obj
     Read the corresponding member of MODULE, or write the value OBJ into the slot.

 -- procedure: module? obj
     This is the type predicate for modules, which will return #t if its argument is a module, and #f otherwise.

 -- procedure: make-scm-module
     This will create a module which represents Guile's builtin bindings. Initially, it is empty, but when variable lookups are made in it, the requested bindings will get copied into the module from the internal obarray.  Bindings will get copied even if only accessed.

 -- procedure: make-root-module
     Create a new module which works on the internal obarrary.  The difference to the modules returned by make-scm-module is that bindings are only copied to the obarray if they are explicitly defined.

Module Loading
==============

The normal way of loading modules in Scheme programs is to use the special form use-modules, which loads the source of one or more specified modules and imports their public bindings into the current module.  This procedure is documented in the Guile Reference Manual, so I will not go into details here.  This section rather contains the underlying mechanisms, on which use-modules is built.

Normally, you will not need to create new modules explicitly, adding definitions to it manually.  Far more often, you will want to load some Scheme code from a file, and install its definitions into a module, only exporting the public bits.  This is what the procedure resolve-module is for.

 -- procedure: resolve-module name [maybe-autoload]
     If the module NAME already exists, return it.  Otherwise, try to load the Scheme code for NAME into a newly created module, adding the exports to the public interface.

     The parameter MAYBE-AUTOLOAD controls the behaviour when the module code is going to be loaded.

     MAYBE-AUTOLOAD == #f
          Do not defer loading.

     MAYBE-AUTOLOAD == #t
          Defer loading of the source code until a binding from the module is requested.

     MAYBE-AUTOLOAD not given
          Like MAYBE-AUTOLOAD == #t.

Modules and Variables
=====================

Modules can be regarded as mappings from symbols (variable names) to bindings (variable locations).  The procedures documented in this section can be used to test whether such a mapping exists for a given variable name, how to add mappings and how to retrieve a variables value.

We sometimes want to look for properties of a symbol just within the obarray of one module.  If the property holds, then it is said to hold ``locally'' as in, ``The symbol display is locally rebound in the module safe-guile.''

Other times, we want to test for a symbol property in the obarray of M and, if it is not found there, try each of the modules in the uses list of M.  This is the normal way of testing for some property, so we state these properties without qualification as in: ``The symbol 'fnord is interned in module M because it is interned locally in module M2 which is a member of the uses list of M.''

In the descriptions below, the term ``well-defined'' means: Any other value than the special object SCM_UNDEFINED, which is used internally for marking undefined locations.

 -- procedure: module-locally-bound? module sym
     Test if SYM is bound in MODULE directly, i.e. it does not suffice that SYM is bound in one of the used modules of MODULE.  Bound means that the symbol is interned and bound to some well-defined value.

 -- procedure: module-bound? module sym
     Return true if SYM is bound in MODULE or one of the modules in MODULE's uses list.  The search looks in all transitively used modules.  Bound means that the symbol is interned and bound to some well-defined value.

 -- procedure: module-symbol-locally-interned? module sym
     Test if SYM is interned in MODULE directly, i.e. it does not suffice that SYM is bound in one of the used modules of MODULE.  Unlike module-locally-bound, the symbol is not required to be bound to a well-defined value.

 -- procedure: module-symbol-interned? module sym
     Return true if SYM is interned in MODULE or one of the modules in MODULE's uses list.  The search looks in all transitively used modules.  Unlike module-bound, the symbol is not required to be bound to a well-defined value.

 -- procedure: module-local-variable module sym
     Return a variable object for SYM in the module MODULE, or #f if no such symbol is defined in MODULE.  If the symbols is not found at first, but the module has a lazy binder, then try the binder.

 -- procedure: module-variable module sym
     Return a variable object for SYM in the module MODULE or one of its used modules, or #f if no such symbol is defined in MODULE or its uses.

 -- procedure: module-symbol-local-binding module symbol [opt-value]
     Return the value of the binding called SYMBOL in MODULE, or OPT-VAL if no such binding exists.  If no OPT-VALUE is given and no binding exists, an error is thrown.

 -- procedure: module-symbol-binding module symbol [opt-value]
     Return the value of the binding called SYMBOL in MODULE, or OPT-VAL if no such binding exists.  If no OPT-VALUE is given and no binding exists, an error is thrown.  Unlike module-symbol-local-binding, this will search all used modules as well as MODULE.

 -- procedure: module-make-local-var! module symbol
     Create a binding for a variable called SYMBOL in MODULE and return the variable object representing the new location in the module. If SYMBOL is already defined in MODULE, nothing happens.

 -- procedure: module-add! module symbol var
     Add the variable VAR to MODULE under the name SYMBOL. VAR must be of type variable, for example a value returned by make-variable or make-undefined-variable.

 -- procedure: module-remove! module symbol
     Remove the binding for SYMBOL in MODULE.  The return value is not specified.

 -- procedure: module-clear! module
     Remove all bindings from MODULE.

Iterating over Module Bindings
==============================

 -- procedure: module-for-each proc module
     Apply PROC to every binding in MODULE.  PROC is called with two parameters, the name and variable for each binding.  PROC is only applied to the local bindings of MODULE, not to possible bindings in its uses list.

 -- procedure: module-map proc module
     Apply PROC to every binding in MODULE and return a list of the results of all applications of PROC.  PROC is called with two parameters, the name and variable for each binding.  PROC is only applied to the local bindings of MODULE, not to possible bindings in its uses list.

The Lazy Binder
===============

The lazy binding procedures which are connected to modules are invoked every time a binding is searched in a module, but is not present.  A binder is called with three arguments.

When a lazy binder returns a variable object, the search is successful and the return value will be used.  If the return value is #f, the search is continued in the modules from the uses list.  FIXME: Is this always the case or only in the standard eval closure?

module
     The module for which the binding is requested.

symbol
     The name of the searched symbol.

define?
     #t if the binding should be defined, #f otherwise.

     If define is true, no recursion occurs, as the binding either exists in the first level module, or is created in the first level module.

Module Observers
================

A module can have a number of observers attached.  These are procedures which are called whenever something withing the module changes.  This can be the creation, deletion or modification of a binding.

When a change occurs, the procedure module-modified (documented below in this section) will be called which in turn will apply all observer procedures to the modified module.

 -- procedure: module-observe module proc
     Add the observer PROC to MODULE and return a pair of MODULE and PROC.  The returned value can be used with module-unobserve.

 -- procedure: module-observe-weak module proc
     Add PROC as a weak observer to MODULE and erturn a pair of MODULE and a unique integer, the observer ID.  The returned value can be used with module-unobserve.

 -- procedure: module-unobserve token
     Remove an observer from a module.  The module and the observer to be removed are taken from TOKEN, which must be returned by module-observe or module-unobserve.

 -- procedure: module-modified m
     Signal a modification of module M to all associated observers.

The Current Module
==================

For all evaluations, Guile maintains a so-called current module. (A current module does not exist until Guile has been completely booted, that means until boot-9.scm has been loaded. But this should be no issue unless you are doing weird things with the module system, which might be a bad idea, but YMMV.)  This is used for all top-level definitions and variable lookups.  When the current module changes, new definitions will go to the new module.  The procedures in this section manipulate the notion of the current module.

The current module is also used when C code calls scm_make_gsubr for creating new primitives or scm_sysintern for interning symbols.  From C, the current module can be set by calling scm_set_current_module, which will return the old module.  This returned module can later be used to switch back to the old module after creating a new one and installing bindings there.

 -- procedure: set-current-module module
     Make MODULE the current module, into which all following definitions will go.  Return the old module in effect before the call to set-current-module.

 -- procedure: current-module
     Return the module which is currently registered as the current module.

High-Level Module Access
========================

The procedure in the previous chapter are not for general use.  The current chapter will document all procedures which are meant to be used by users who need to work with modules.

The parameter MODULE in the following descriptions must be a module, NAME must be a symbol (which most probably will need to be quoted).

 -- procedure: module-ref module name [default]
     Return the value of a variable called NAME in MODULE or any of its used modules.  If there is no such variable (or the variable is not boudn to a well-defined value), then if the optional third argument DEFAULT is present, it is returned; otherwise an error is signaled.

     The difference between this procedure and module-symbol-binding (see Modules and Variables) is that the latter may return an undefined value, whereas the former will throw an exception in that case.

 -- procedure: module-set! module name value
     Sets the variable called NAME in MODULE (or in a module that MODULE uses) to VALUE; if there is no such variable, an error is signaled.

 -- procedure: module-define! module name value
     Sets the variable called NAME in MODULE to VALUE; if there is no such variable, it is added first.

 -- procedure: module-defined? module name
     Return #t if NAME is defined in MODULE (or in a module that MODULE uses).

 -- procedure: module-use! module interface
     Add INTERFACE to the list of interfaces used by MODULE.  For information what an interface is, see Modules and Interfaces.

 -- procedure: module-export! module names
     Add all variables from NAMES (a list of symbols) to the public interface of MODULE (see Modules and Interfaces).

Recursive Namespaces
====================

A hierarchical namespace emerges if we consider some module to be root, and variables bound to modules as nested namespaces.

The modules which implement the internal nodes are of kind directory. FIXME: Is this correct?

The routines in this chapter manage variable names in hierarchical namespace.  Each variable name is a list of elements, looked up in successively nested modules.

     (nested-ref some-root-module '(foo bar baz))
     =>
     <value of a variable named baz in the module bound to bar in
     the module bound to foo in some-root-module>

When an invalid module path is given, nested-ref and local-ref silently return #f, and the other procedures simply fail without an error.

Nested references can start in any module, so you can even pass your own modules as the ROOT parameter to the nested-* procedures below.

 -- procedure: nested-ref root names
     Look up the variable identified by the symbol list NAMES, starting in the module ROOT.

 -- procedure: nested-set! root names val
     Set the variable identified by the symbol list NAMES to VAL, starting the variable lookup in module ROOT.  The return value is not specified.

 -- procedure: nested-define! root names val
     Set the variable identified by the symbol list NAMES to VAL, starting the variable lookup in module ROOT.  If the variable does not exist, create it before setting its value.  The return value is not specified.

 -- procedure: nested-remove! root names
     Remove the variable identified by the symbol list NAMES, starting the variable lookup in module ROOT.  The return value is not specified.

 -- procedure: local-ref names
 -- procedure: local-set! names val
 -- procedure: local-define! names val
 -- procedure: local-remove! names
     Like the nested-ref, nested-set!, nested-define! and nested-remove! procedures above, but start the variable lookup in the module returned by current-module.

Module Hierarchy
================

Currently, the following entries are defined in the hierarchical namespace.

(app)
     This is the root of all named objects which are not in the top level.

(app modules)
     This is the directory of all modules.

(app modules guile)
     This is the standard root module.

User modules which are loaded into Guile as well as the modules shipped with the Guile distribution are installed under (app modules) as well.

NJFIXME - so why do module install directories not include "app/modules" in their path? - or is that a silly question?

MGFIXME - Honestly, I don't have a clue.  I think that the (app modules) internal module name prefix was once introduced to make other module name spaces possible.  Maybe (app core ...) or something.

Modules and Interfaces
======================

Interfaces are modules of kind interface.  They always belong to another module and contain the bindings which are exported from that module.  Interfaces are the means by which the different scopes of a module (private vs. public bindings) are implemented.

Every module can define a special variable called %module-public-interface, which is bound to the module's interface.

Whenever a variable is exported (with the export form or the :export keyword in the define-module form), this variable is added to the defining module's interface.  Because importing a module means adding other modules' interfaces to the uses list, the exported variables become visible in the importing module.

 -- procedure: module-public-interface m
     Return the public interface of module M, or #f if M does not have a public interface.

 -- procedure: set-module-public-interface! m i
     Set the public interface of the module M to I.

Modules and Environments
========================

An environment belongs to a specific module, which can be determined by calling environment-module.

 -- procedure: environment-module env

Modules Miscellanea
===================

This chapter contains all miscellaneous information and procedure documentation which I have not been able to include elsewhere.  If someone knows how to include them into other chapters, suggestions are welcome.

 -- procedure: set-system-module! m s
     Set the system-module property of the module M to S. S should be a boolean telling whether M is a system module or not.  System modules are treated specially in some cases, for example procedures defined in system modules are excluded from backtraces. FIXME: Is this last sentence true?

Modules and Evaluation
**********************

Up to here, we have seen how modules are implemented as a data type, which can be manipulated by C and Scheme code to implement module system work like providing private and public name spaces, loading of modules and creating new modules.

This chapter will describe the connection between the module system and the Guile evaluator.  Top-level variables (that is, variables not lexically bound) need to be resolved in the current module, and if not defined there, in the used modules, and so on, until the root module has been asked for the bindings.

First, we have to recall how Guile normally figures out the location for a given variable when evaluating a form.  The evaluator starts by scanning the lexical environment it maintains.  It first looks in each slot in the top-most environment frame, continuing in the next frame and so on, until it reaches the end of the lexical environment chain.

The CAR of the last pair of the environment chain is either #f, or it is a procedure.  When it is #f, the normal system obarray is searched for the variable, otherwise the procedure is called for returning the requested variable.  This procedure is the current module's eval closure, and is responsible for searching a variable's binding, installing it if necessary.

When a variable is finally found, the reference to the variable in the currently executed Scheme code is replaced by a special value (a so-called gloc), so that this environment search is not necessary the next time the variable is looked up.