This works for me now, with a fresh clone of Octave. Default ./bootstrap pulls upstream gnulib and updates to a8e125d24942. The environment overrides GNULIB_REVISION and GNULIB_URL also work as documented for me.

This was done on the stable and default branches

• https://hg.savannah.gnu.org/hgweb/octave/rev/ece72b94486f
• https://hg.savannah.gnu.org/hgweb/octave/rev/9d4711b6cd43

Mike (comment #38), in the past this did not really happen very often, and the fix is more than obvious.  In the same spirit some developers (including me) face from time to time problems with stale build trees, if there where breaking changes that require deleting all old stuff and starting from bootstrap again.

My overall impression is Octave wins much more by doing this change, than it looses.

It fails any time the revision is not found in the local history. It does not do a pull or fetch of new revisions at all. In my estimation this will fail for everyone every time GNULIB_REVISION is updated.

I think this "failure" you expect is that one "fatal: reference is not a tree":

GNULIB_REVISION=24d438e5e579478a4c4d4fece2f10a68a1836268 ./bootstrap

./bootstrap: Bootstrapping from checked-out octave sources...
fatal: reference is not a tree: 24d438e5e579478a4c4d4fece2f10a68a1836268
<dead>

In this case bootstrap was not able to checkout the correct git id and died.  The reason was in my scenario (comment #34) that git could not find this id in the gnulib repo (as I cut the internet connection and was in an old state).

As soon as the git ID is fetched, it is really hard to provoke a failure scenario.  bootstrap does really a good job, if you do not make it suffer too much with border cases.

Regarding comment #32 (jwe): No objections, please apply =)

Regarding comment #33 (Mike), comment #27 (Mike):

I tried to reproduce your example

bootstrap.conf
: ${GNULIB_REVISION=a8e125d24942bc581fda1628a61a60b979ab7087}


https://git.savannah.gnu.org/cgit/gnulib.git (orign/master)
24d438e5e579478a4c4d4fece2f10a68a1836268 (01/21/2020 05:59:10 PM -0800)

LOCAL (orign/master)
020f5190955808cafb670d2be6afb7d5faf69621 (01/16/2020 09:41:34 PM +0530)

LOCAL (master)
1fabf601fa87579a9eaed42c92c6bb9a97922c1d (12/13/2019 12:19:56 AM +0100)


DISABLE INTERNET

./bootstrap

./bootstrap: Bootstrapping from checked-out octave sources...
Note: checking out 'a8e125d24942bc581fda1628a61a60b979ab7087'.

... successful



GNULIB_REVISION=020f5190955808cafb670d2be6afb7d5faf69621 ./bootstrap

./bootstrap: Bootstrapping from checked-out octave sources...
Previous HEAD position was a8e125d24 * lib/mknodat.c: Remove incorrect comments.
HEAD is now at 020f51909 vcs-to-changelog: Fix parsing of fndecl without args.

... successful



GNULIB_REVISION=24d438e5e579478a4c4d4fece2f10a68a1836268 ./bootstrap

./bootstrap: Bootstrapping from checked-out octave sources...
fatal: reference is not a tree: 24d438e5e579478a4c4d4fece2f10a68a1836268
<dead>


GNULIB_REVISION= ./bootstrap

./bootstrap: Bootstrapping from checked-out octave sources...
./bootstrap: getting gnulib files...
Cloning into 'gnulib'...
fatal: Unable to look up git.sv.gnu.org (port 9418) (Name or service not known)
<dead>


ENABLE INTERNET

GNULIB_REVISION=24d438e5e579478a4c4d4fece2f10a68a1836268 ./bootstrap

./bootstrap: Bootstrapping from checked-out octave sources...
./bootstrap: getting gnulib files...
Cloning into 'gnulib'...

<old "gnulib" folder removed, startover with a fresh clone>


GNULIB_REVISION= ./bootstrap

<just take what is there>

You are right.  It is maybe 99% of what we want.  But most important, we can easily control via one hash inside bootstrap.conf for a plain standard Octave developer a very easy to maintain and understand way to get the gnulib right.

I things go wrong, bootstrap is smart enough to issue an "tree"-error about git id mismatches.  Thus lets do the change now than later and gain new experience with these border cases.

I'm ready to push the attached change to stable and merge with default.  Any objection?


(file #48280)

Yeah, I just tested Kai's original patch and it does force the gnulib directory to 'ff29c880629e73def7cd6c4a954e919b174d31b6', even with --gnulib-srcdir. Even if I run './bootstrap --gnulib-srcdir=../../gnulib --no-git', it still forces my gnulib repo to the pinned revision.

Personally, I would be fine with passing GNULIB_REVISION as an environment variable to workaround this to get what I want, for example 'GNULIB_REVISION=master ./bootstrap ...'. Would this be ok? Change the patch to bootstrap.conf to do this instead? This works for me.

: ${GNULIB_REVISION=ff29c880629e73def7cd6c4a954e919b174d31b6}

Actually I now have some doubts about whether bootstrap correctly fetches new revisions from the gnulib directory after the initial clone. I'll test that next.

Mike: It looks like if you use

./bootstrap --gnulib-srcdir=path/to/gnulib

then the version used will be whatever is current in the separate gnulib repo.

Yeah, I think a warning message would be fine, either 'configure' or 'make', either now or later, when there's a mismatch, so users know they should re-run bootstrap.

If Kai thinks it's safe to integrate this into the stable branch for the 5.2 release, seems ok to me. We will need people with various versions of Texinfo to confirm that the latest gnulib fixes the doc problems that Rik reported on the maintainers list, and doesn't introduce any more problems.

I've been building both default and stable with a more recent gnulib branch for a few months and I haven't had run into any problems.

OK.  I guess my question now is, should there be a warning that the version of gnulib is not the expected one?  I don't think we should prevent that mismatch, but would it be useful to warn about it?

If we don't have gnulib as a subrepo, how are things supposed to work?  After updating, do you now always have to use bootstrap to make sure that you are at the expected gnulib revision?  Will there be any warning from configure or make if you are not?

Isn't adding this check into configure a bit of overkill?

When we track gnulib as a hg subrepository, it's true that using "hg update" will automatically switch to the correct revision. However, the user who is bisecting or tracking development still always needs to use bootstrap in order to copy those changes from "gnulib" into "libgnu". Otherwise, they definitely may be building with out of date copies of source files in libgnu.

Adding this check into configure is a step above and beyond what we had been doing before, but it does check for something that would normally have been enforced by hg. However, it's still no guarantee that the user has run bootstrap to copy the sources from gnulib into libgnu. I'm not sure it's useful.

In the attached patch, I opted for checking if the bootstrap script is in the source tree. That file isn't in the distribution tarball.

IIRC, Mike uses a git repository for Octave. So checking for the ".hg" directory might cause problems.

(file #48107)

Well, it wasn't completely my idea but what jwe asked for in comment #11.
I haven't tested with a distribution tarball. But I think it would fail without the switch. Good catch.

Is there a way that configure can tell that it is called from a distribution tarball?

The attached patch adds a test to the configure script to check whether the current gnulib revision is the expected one. If this is not the case, configure fails.
For this, the git revision is stored in a new file "gnulib_rev" that is used by bootstrap and by configure. That file is part of the hg repository. So every hg revision will continue to be linked to a specific gnulib version.
The file could also be hidden if preferred.

The test can be skipped by adding --without-gnulib-rev-check to the configure switches.

The patch applies on top of Kai's patch from comment #10.
It only works if the gnulib repository is at the default location in the source tree. For other configurations, --without-gnulib-rev-check must be used.


(file #48103)

Indeed, bootstrap is a very "intelligent" script, that can ensure using the correct "GNULIB_REVISION" all by its own.

Regarding older releases, soon 5.2.0 and older:  There the files ".hgsub" and ".hgsubstate" are still present in the repository and will have to use Octave's gnulib hg subrepo https://hg.octave.org/gnulib/ to work in the future.  Thus we cannot delete Octave's gnulib hg subrepo, if we want to be able to know the gnulib version of that releases.

Regarding releases 6.1 and onwards: ".hgsub" and ".hgsubstate" are removed.  The "GNULIB_REVISION" is maintained in "bootstrap.conf" inside Octave's main hg repository.  Thus it is clear for every commit, which "GNULIB_REVISION" was in use.  The hard work is done by bootstrap itself.

Scenario #1: In a fresh Octave hg repo.  Just calling

./bootstrap

will check for "./gnulib/" to be a clone of git://git.sv.gnu.org/gnulib (https://git.savannah.gnu.org/cgit/gnulib.git) and clone it if not existing.  If existing, bootstrap tries to checkout "GNULIB_REVISION".  This works, even if the "GNULIB_REVISION" is in the future, i.e. not fetched inside "./gnulib/".

Scenario #2: In a fresh Octave hg repo.  Custom gnulib git repo

./bootstrap --gnulib-srcdir=path/to/gnulib

and the procedure is as in #1.

As said before in comment #10.  Updating gnulib from 6.1 will be pretty easy compared to now.  Never had such a close look at bootstrap before.

During the preparation for the 6.1 release, I stumbled over this issue again.  gnulib does not make it easy to understand the bootstrap script.  But basically all that is said in the bootstrap file.  Octave does currently not make use of the

GNULIB_REVISION=03eccb2d010e0755fd55458e2f012fcfb0735c2d

https://hg.octave.org/gnulib/rev/c34f677e6117

https://git.savannah.gnu.org/cgit/gnulib.git/commit/?id=03eccb2d010e0755fd55458e2f012fcfb0735c2d

feature.  I tried around with it:

+ Works offline (if the revision is fetched inside the git repo).
+ Works with self-hosted gnulib-git repo (git version managed by bootstrap, "--gnulib-srcdir" and "GNULIB_SRCDIR" are dominant)
- `git` becomes a new build requirement (honestly, today not a drawback at all)

With this approach, updating gnulib is now as easy as changing the hash in bootstrap.conf

GNULIB_REVISION=ff29c880629e73def7cd6c4a954e919b174d31b6

and run before committing to the Octave hg repo:

./bootstrap --bootstrap-sync

No more hg subrepo in Octave!! =D

The attached short changeset should be used with a clean clone of Octave and automatically cloned the "gnulib" repo removed:

diff -r 1d04f049a859 -r 1a63b0fbafd3 .hgsub
--- a/.hgsub        Tue Dec 10 17:55:29 2019 -0800
+++ /dev/null        Thu Jan 01 00:00:00 1970 +0000
@@ -1,1 +0,0 @@
-gnulib = http://hg.octave.org/gnulib
diff -r 1d04f049a859 -r 1a63b0fbafd3 .hgsubstate
--- a/.hgsubstate        Tue Dec 10 17:55:29 2019 -0800
+++ /dev/null        Thu Jan 01 00:00:00 1970 +0000
@@ -1,1 +0,0 @@
-c34f677e6117d1c293338328a6c9b9ce18e9ef53 gnulib
diff -r 1d04f049a859 -r 1a63b0fbafd3 bootstrap.conf
--- a/bootstrap.conf        Tue Dec 10 17:55:29 2019 -0800
+++ b/bootstrap.conf        Wed Dec 11 14:39:32 2019 +0900
@@ -176,10 +176,15 @@ gnulib_tool_option_extras="--libtool"
 gnulib_name="libgnu"
 source_base="libgnu"

-# Don't use git to update gnulib sources. We keep gnulib under a
-# Mercurial subrepository instead.
-use_git=false
-GNULIB_SRCDIR=gnulib
+# Go to https://git.savannah.gnu.org/cgit/gnulib.git to find out the latest
+# revision ID.  Paste it below and run
+#
+#   ./bootstrap --bootstrap-sync
+#
+# To update the ./bootstrap file with that from ./gnulib/build-aux/bootstrap
+#
+GNULIB_REVISION=ff29c880629e73def7cd6c4a954e919b174d31b6
+

 # Don't check for translations since we don't have any in Octave yet.
 # This avoids the need for sha1sum or compatible utility in bootstrap.

(file #48044)

jwe - I know Jordi's talked about the pros and cons of hg subrepos before. Do you have a specific new layout you want to use, or are you musing generally about the problems with the current layout?

Personally I would probably prefer that we just not have our own gnulib mirror at all anymore, and use bootstrap the way it was intended, to always pull the latest gnulib from git on savannah into a subdirectory, when the user runs ./bootstrap.

I know another option was to have a new top-level project with no files, that tracks octave and gnulib as subrepos. I'm not really a fan of that idea. I think it would be confusing to need to pull from a new project name, and have a layout that differs from every other GNU / Gnulib / autotools style project out there.

We might also reconsider our decision to include gnulib as an hg subrepo contained in a subdirectory of the Octave source tree.

Heads up to other maintainers and developers, yes I am hosting git mirrors of the Octave source tree in an experiment to see if doing so brings in contributions or raises visibility. If you don't care about git, you can ignore their existence and bugs like this one :)

The way I build is to use an external gnulib git clone. This could be either in the octave source tree, or in another directory. Here is my exact process.

cd ~/src
git clone https://git.savannah.gnu.org/git/gnulib.git
cd ~/src/octave/default
./bootstrap --with-gnulib=../../gnulib

Gnulib is already a hg subrepository, I'm not sure there is a nice way to make it also a git subrepository without things getting very confusing (both vcs would need to track subrepository state). I don't advocate trying to do this in the Octave official repository.

I may end up making another top-level repository that contains octave and gnulib as subrepos, and configures one to talk to the other. But we can discuss that elsewhere.

This bug is only relevant to the gitlab/github mirrors that mtmiller has setup.

# git clone https://gitlab.com/mtmiller/octave.git
# cd octave
# ./bootstrap
./bootstrap: Error: --no-git requires --gnulib-srcdir

Is this a problem we can solve with git submodules?  Or some tweaking to `bootstrap`.  It would be nice if it worked OOTB.