bugGNU Common Lisp - Bugs: bug #67062, build of gcl-2.7.1 fails on...

 
 

bug #67062: build of gcl-2.7.1 fails on Maneage commit 9ca097e8 - 'Killed'

Submitter:  Boud Roukema <boud>
Submitted:  Thu 01 May 2025 01:20:29 PM UTC
   
 
Category:  None Severity:  3 - Normal
Item Group:  None Status:  None
Privacy:  Public Assigned to:  None
Open/Closed:  Open
* Mandatory Fields

Add a New Comment Rich Markup
   

Jump to the original submission

Tue 03 Jun 2025 09:14:54 PM UTC, comment #13: 


comment #12:

> Greetings!  OK I will await confirmation of Colin's build, then will update the webpage/readme with parallel build instructions and a note that LTO is not supported at present. 


From my end we can consider it "good"; my next blocker for further testing would be the inclusion of ASDF, which has recently been discussed on the mailing list.

Colin Woodbury <fosskers>
Tue 03 Jun 2025 02:56:42 PM UTC, comment #12: 

Greetings!  OK I will await confirmation of Colin's build, then will update the webpage/readme with parallel build instructions and a note that LTO is not supported at present.  FYI, I've looked at it briefly, and it does appear that it would be simple to support LTO in the next release.

Regarding info::

gcl27
GCL (GNU Common Lisp)  2.7.1 Thu Apr 10 09:38:27 PM EDT 2025  CLtL1    git: Version_2_7_2pre6
Source License: LGPL(gcl,gmp), GPL(unexec,bfd,xgcl)
Binary License:  GPL due to GPL'ed components: (XGCL UNEXEC)
Modifications of this banner must retain notice of a compatible license
Dedicated to the memory of W. Schelter

Use (help) to get some basic information on how to use GCL.
Temporary directory for compiler files set to /tmp/

>(describe "Environment")


"Environment" - simple string
 dimension:  11

>(si::info "Environment")


 0: (gcl27-si.info)Environment Variables.
 1: Environment.
 2: (gcl27.info)Introduction to Environments.
 3: The Global Environment.
 4: Dynamic Environments.
 5: Lexical Environments.
 6: The Null Lexical Environment.
 7: Environment Objects.
 8: Environment.
 9: The External Environment.
 10: Environment Inquiry.
 11: Environment Dictionary.
Enter n, all, none, or multiple choices eg 1 3 : 0

Info from file /usr/share/info/gcl27.info:
9.3 Environment Variables
=========================

Several environment variables affect GCL:

 -- Environment Variable: GCL_MEM_MULTIPLE
     A positive float indicating the fraction of available memory GCL
     should use.  Defaults to 1.0.

 -- Environment Variable: GCL_MEM_BOUND
     A positive integer bounding GCL's heap to 1<<(n+1) bytes.  Trumps
     GCL_MEM_MULTIPLE. Defaults to sizeof(long)-1.

 -- Environment Variable: GCL_GC_ALLOC_MIN
     A positive float indicating the minimum fraction of heap to be
     allocated between garbage collection (GC) cycles.  Defaults to
     0.05.

 -- Environment Variable: GCL_GC_PAGE_MIN
     A positive float indicating the minimum fraction of heap to be
     allocated before garbage collection (GC) commences.  Defaults to
     0.5.

 -- Environment Variable: GCL_GC_PAGE_MAX
     A positive float indicating the maximum fraction of heap to be
     allocated after which garbage collection (GC) is mandatory.
     Defaults to 0.75.

 -- Environment Variable: GCL_MULTIPROCESS_MEMORY_POOL
     A string when set indicating a directory in which to place the file
     gcl_pool used for coordinating memory management among multiple GCL
     processes.  This should be a local directory for performance
     reasons.  Default is unset.

 -- Environment Variable: GCL_WAIT_ON_ABORT
     A non-negative integer indicating how many seconds to sleep before
     aborting on fatal error.  Defaults to 0.

NIL

>



Take care,

Camm Maguire <camm>
Group administrator
Mon 02 Jun 2025 12:27:43 PM UTC, comment #11: 

Strictly speaking, this is a separate bug, but it's easy to fix: while I'm happy to believe that the three ways of handling multiprocessing are in the release notes for one or more particular versions, many people are likely to only read the README file in the git repository.

I think that brief info on multiprocessing should be added to the README file: single cpu machines are quite rare today. Not everybody will think of looking at 'gcl/debian/rules' for hints, even if that's often quite wise :).

I searched the code of gcl-2.7.2pre5-91ab92223:

$ grep -rn MULTIPROCESS gcl
gcl/o/main.c:376:  multiprocess_memory_pool=getenv("GCL_MULTIPROCESS_MEMORY_POOL");
gcl/debian/rules:23:GCL_MULTIPROCESS_MEMORY_POOL?=$(shell pwd)
gcl/debian/rules:24:export GCL_MULTIPROCESS_MEMORY_POOL
gcl/info/system.texi:463:@defvr {Environment Variable}  GCL_MULTIPROCESS_MEMORY_POOL


I tried

info gcl

but that doesn't seem to have the string "MULTIPROC" anywhere - despite it being in 'gcl/info/system.texi'.



Boud Roukema <boud>
Sun 01 Jun 2025 09:44:46 PM UTC, comment #10: 


comment #8:

> I cannot reproduce with just ' -fplt -fcf-protection=none'.  Are you using LTO optimizations? 


It turns out that I had LTO enabled. Explicitly disabling it made the build succeed. Please wait a bit while I confirm the behaviour of the executable, etc., before we close this.

Colin Woodbury <fosskers>
Sun 01 Jun 2025 02:12:52 PM UTC, comment #9: 

Greetings!

One other helpful item would be:

objdump -t gcl0/gcl_s.o

Take care,

Camm Maguire <camm>
Group administrator
Sun 01 Jun 2025 01:48:22 PM UTC, comment #8: 

Greetings!

OK, just to clarify my understanding and to make sure this thread is discussing the same thing, Boud reports success with 2e2818a318, and then a few days later Colin reports the unrelocated symbol failure.  Are we starting from the same base and if so what has changed?

I cannot reproduce with just ' -fplt -fcf-protection=none'.  Are you using LTO optimizations?  If so I would suggest turning this off.  Trying now to reproduce with that assumption with no success.  I will need to see the full configure and build output if this does not clear it up.

Take care,

Camm Maguire <camm>
Group administrator
Sun 01 Jun 2025 10:33:50 AM UTC, comment #7: 


comment #6:

> Please post the previous few lines of the log, where the 'unrelocated non-local' symbol is identified.


It's a bit hard to tell but I'll paste here what I see.

```
;; End of Pass 2.
OPTIMIZE levels: Safety=3, Space=0, Speed=3
;; Finished compiling gcl0/gcl_s.o.
[ "gcl_s" = "gcl_c" ] || [ "gcl_s" = "gcl_listlib" ] || \
        ln -f gcl0/gcl_s.o $(echo gcl0/gcl_s.o |sed 's,\.o,\.done,g') #FIXME directory link
unixport/gcl0 -eval "(mapc 'load (directory \"gcl0/*.done\"))" -compile lsp/gcl_typep.lsp -o gcl0/gcl_typep.o
Unrelocated non-local symbol: __gnu_lto_slim

Error:ERROR:  The assertion !emsg("Unrelocated non-local symbol: %s\n",st1+sym->st_name) on line 248 of o/sfaslelf.c in function relocate_symbols failed: Success
Signalled by IF.

ERROR:  The assertion !emsg("Unrelocated non-local symbol: %s\n",st1+sym->st_name) on line 248 of o/sfaslelf.c in function relocate_symbols failed: Success
```

For the record, here is how I am compiling:

```
  CFLAGS="${CFLAGS} -fplt -fcf-protection=none -std=gnu17" ./configure --prefix=/usr
  export GCL_MEM_MULTIPLE="1/2"
  make -j 1
  make DESTDIR=build install
```

Colin Woodbury <fosskers>
Sat 31 May 2025 01:44:26 PM UTC, comment #6: 

Greetings, and thanks for your report!

Please post the previous few lines of the log, where the 'unrelocated non-local' symbol is identified.

Typically this results from changing CFLAGS and not restarting with 'make clean'

Take care,

Camm Maguire <camm>
Group administrator
Thu 08 May 2025 06:12:49 AM UTC, comment #5: 

Although I did manage to make some progress with the posted enviroment variables, my build still fails in the second stage (using gcl0, I believe) with:

ERROR:  The assertion !emsg("Unrelocated non-local symbol: %s\n",st1+sym->st_name) on line 248 of o/sfaslelf.c in function relocate_symbols failed: Success

Any ideas?

Colin Woodbury <fosskers>
Mon 05 May 2025 09:05:59 PM UTC, comment #4: 

I did a new Maneage_dev test branch commit, so [5] will probably be clearer than [4] (to avoid you having to search through the tree).

[5] https://codeberg.org/boud/maneage_dev/commit/2e2818a318b6976e29e97524e5bf4a0360c24636

Boud Roukema <boud>
Mon 05 May 2025 08:47:25 PM UTC, comment #3: 

hi Camm,

comment #2:

> Greetings!
>
> I recommend you take a quick glance at the homepage, and the release notes:
>
> https://www.gnu.org/software/gcl/
> https://www.gnu.org/software/gcl/RELEASE-2.7.1.html
>
> First please note the small errata in the released tarball.


There's also a small erratum in the count of errata: it should be plural on the web page, not singular. ;)

> Second, GCL will build in parallel, but by default each instance will think it has access to your available memory, and oom is near certain.  There are three ways to address this, as mentioned in the release notes:
>
> 1) export GCL_MEM_MULTIPLE=<fraction> -- each instance has this fraction of memory
> 2) export GCL_MEM_BOUND=<integer> -- each instance has (1UL<<<integer>+1) bytes of memory
> 3) export GCL_MULTIPROCESS_MEMORY_POOL=<directory> -- write a lockfile in this directory for all gcl instances to coordinate their memory usage.


For the Maneage context, 1) would require a bit of extra work to calculate the number of threads and 2) would need calculating both the number of threads and the amount of acceptably usable memory; so 3) seems the easiest choice.

With 3) GCL_MULTIPROCESS_MEMORY_POOL=<directory> given as a parameter to 'make', I got a full compile with Maneage commit 2e2818a318 [4].

So I think that this bug can be either closed or maybe downgraded to a 'wishlist' item: it should only take a moderate amount of work to see if the number of available threads is greater than 1, and if also all three of these GCL*MEM* environment variables are empty, and in that case print a big warning (or maybe a fatal build error?) informing the user about these three options and pointing them to the release notes.

At least as far as Maneage is concerned, I'm happy to consider this bug closed.

For the record, in the successful run with 3), i.e. Maneage commit 2e2818a318, I did not apply the three errata. Since Maneage ignores the debian/ build rules, only the third erratum is relevant to Maneage. (My only reason for not applying it was that initially I didn't see where the errata were; now I do see them.)

Cheers
Boud

[4] https://codeberg.org/boud/maneage_dev/src/branch/debug_gcl_try3/reproduce/software/make/high-level.mk#L570

Boud Roukema <boud>
Mon 05 May 2025 04:27:29 PM UTC, comment #2: 

Greetings!

I recommend you take a quick glance at the homepage, and the release notes:

https://www.gnu.org/software/gcl/
https://www.gnu.org/software/gcl/RELEASE-2.7.1.html

First please note the small errata in the released tarball.

Second, GCL will build in parallel, but by default each instance will think it has access to your available memory, and oom is near certain.  There are three ways to address this, as mentioned in the release notes:

1) export GCL_MEM_MULTIPLE=<fraction> -- each instance has this fraction of memory
2) export GCL_MEM_BOUND=<integer> -- each instance has (1UL<<<integer>+1) bytes of memory
3) export GCL_MULTIPROCESS_MEMORY_POOL=<directory> -- write a lockfile in this directory for all gcl instances to coordinate their memory usage.


Please let me know if there are any problems with this.

Take care,

Camm Maguire <camm>
Group administrator
Thu 01 May 2025 07:44:06 PM UTC, comment #1: 

hi maintainers,

WORKAROUND:

Doing the build with

./project configure --existing-conf --jobs=1

which has the effect of giving 'make' a '-j1' parameter for single-thread building provides a workaround for this bug. A cleaner workaround (in the Maneage context) is only to force single-thread building for 'gcl'; this is done in commit 7d6ad66c [3].

Forcing a single-thread build leads to a full build of gcl with no errors reported. Running the binary within the Maneage environment gives

$ gcl --version
GCL 2.7.1 git tag Version_2_7_0

(I'm not going to complain about the misleading git tag - though in principle that's probably a bug.)

Interactive use of 'top' during the build suggested that the maximum RES RAM used was about 1Gb or so instead of 80 Gb. This is a lot more reasonable.

SUGGESTIONS:

It seems to me that either:
(i) the gcl-2.7.1 internal build rules should force a single-thread build; or
(ii) the huge RAM (maybe ramdisk) usage and eventual killing by, presumably, the OOM killer, in a multi-threaded build should be traced and debugged.

If a specific hack of one (or more) of the gcl source files is proposed to try to trace the multi-threading bug, I'd be happy to test that.

Cheers
Boud

[3] https://codeberg.org/boud/maneage_dev/commit/7d6ad66ce8834d280383b6e98e65497dc6416b5e

Boud Roukema <boud>
Thu 01 May 2025 01:20:29 PM UTC, original submission:  

DESCRIPTION:

The build of gcl-2.7.1 within Maneage, in the same context as bug #67034 [1] but with a lower-case symbolic link created to work around what appears to be the bug reported there, fails with no diagnostic information that I can see except 'Killed' for several .o files.

CONTEXT:

Identical to that of bug #67034 plus the symlink workaround.

LOG:

The last 1000 lines of the log are at [2]. An extract of the lines where 'Killed' appears is here:

  3646        OPTIMIZE levels: Safety=3, Space=0, Speed=3
  3647        ;; Finished compiling gcl0/gcl_cmpvs.o.
  3648        [ "gcl_cmpvs" = "gcl_c" ] || [ "gcl_cmpvs" = "gcl_listlib" ] || \
  3649                ln -f gcl0/gcl_cmpvs.o $(echo gcl0/gcl_cmpvs.o |sed 's,\.o,\.done,g') #FIXME directory link
  3650        ;; End of Pass 1.
  3651        unixport/gcl0 -eval "(mapc 'load (directory \"gcl0/*.done\"))" -compile lsp/gcl_top.lsp -o gcl0/gcl_top.o
  3652
  3653        ;; Compiling lsp/gcl_top.lsp.
  3654        ;; End of Pass 2.
  3655        ;; End of Pass 1.
  3656        make[2]: *** [Makefile:4645: gcl0/gcl_typeof.o] Killed
  3657        unixport/gcl0 -eval "(mapc 'load (directory \"gcl0/*.done\"))" -compile lsp/gcl_trace.lsp -o gcl0/gcl_trace.o
  3658        make[2]: *** [Makefile:4645: gcl0/gcl_cmptop.o] Killed
  3659        unixport/gcl0 -eval "(mapc 'load (directory \"gcl0/*.done\"))" -compile lsp/gcl_sloop.lsp -o gcl0/gcl_sloop.o
  3660        make[2]: *** [Makefile:4645: gcl0/gcl_bit.o] Killed
  3661        unixport/gcl0 -eval "(mapc 'load (directory \"gcl0/*.done\"))" -compile lsp/gcl_debug.lsp -o gcl0/gcl_debug.o
  3662        make[2]: *** [Makefile:4645: gcl0/gcl_cmpinline.o] Killed
  3663        unixport/gcl0 -eval "(mapc 'load (directory \"gcl0/*.done\"))" -compile lsp/gcl_info.lsp -o gcl0/gcl_info.o
  3664        ;; End of Pass 2.
  3665        ;; End of Pass 2.
  3666        ;; End of Pass 2.
  3667        ;; End of Pass 1.
  3668        make[2]: *** [Makefile:4645: gcl0/gcl_describe.o] Killed


Lines around 4645 of the top Makefile:

  4644  gcl0/%.o: | unixport/gcl0
  4645          $| -eval "(mapc 'load (directory \"$(@D)/*.done\"))" -compile $< -o $@
  4646          [ "$*" = "gcl_c" ] || [ "$*" = "gcl_listlib" ] || \
  4647                  ln -f $@ $$(echo $@ |sed 's,\.o,\.done,g') #FIXME directory link


An independent run after a full rm of the gcl build directory gives a somewhat different ordering of the errors, since 'make' is run by Maneage in parallel mode (by default). The first five kills on this independent run are:

  3647  OPTIMIZE levels: Safety=3, Space=0, Speed=3
  3648  ;; Finished compiling gcl0/gcl_cmpvs.o.
  3649  [ "gcl_cmpvs" = "gcl_c" ] || [ "gcl_cmpvs" = "gcl_listlib" ] || \
  3650          ln -f gcl0/gcl_cmpvs.o $(echo gcl0/gcl_cmpvs.o |sed 's,\.o,\.done,g') #FIXME directory link
  3651  unixport/gcl0 -eval "(mapc 'load (directory \"gcl0/*.done\"))" -compile lsp/gcl_top.lsp -o gcl0/gcl_top.o
  3652
  3653  ;; Compiling lsp/gcl_top.lsp.
  3654  ;; End of Pass 2.
  3655  make[2]: *** [Makefile:4645: gcl0/gcl_s.o] Killed
  3656  unixport/gcl0 -eval "(mapc 'load (directory \"gcl0/*.done\"))" -compile lsp/gcl_trace.lsp -o gcl0/gcl_trace.o
  3657  ;; End of Pass 1.
  3658  make[2]: *** [Makefile:4645: gcl0/gcl_sym.o] Killed
  3659  unixport/gcl0 -eval "(mapc 'load (directory \"gcl0/*.done\"))" -compile lsp/gcl_sloop.lsp -o gcl0/gcl_sloop.o
  3660  make[2]: *** [Makefile:4645: gcl0/gcl_cmpwt.o] Killed
  3661  unixport/gcl0 -eval "(mapc 'load (directory \"gcl0/*.done\"))" -compile lsp/gcl_debug.lsp -o gcl0/gcl_debug.o
  3662  ;; End of Pass 2.
  3663  make[2]: *** [Makefile:4645: gcl0/gcl_deftype.o] Killed
  3664  unixport/gcl0 -eval "(mapc 'load (directory \"gcl0/*.done\"))" -compile lsp/gcl_info.lsp -o gcl0/gcl_info.o
  3665  ;; End of Pass 1.
  3666  ;; End of Pass 2.
  3667  make[2]: *** [Makefile:4645: gcl0/gcl_describe.o] Killed


HYPOTHESIS:

I've noticed that during the build, 'top' showed that the RES (physical memory) got up to about 80 Gb of RAM (and over 100 Gb of VIR virtual ram). This particular machine (multi-core) has about 190 Gb of RAM. This high apparent RAM usage is probably mostly an area of /dev/shm created as a ramdisk by Maneage, based on the motivation that there's no need to thrash a disk with a temporary build area; if RAM is available, then it's faster and can be safely given back to the kernel when no longer needed.

An example of high memory usage:

$ ps -A uf|grep gcl0 |grep -v grep
boud       69618 98.8 46.2 112532076 91545264 pts/1 R+ 14:54   8:35      |                           \_ unixport/gcl0 -eval (mapc 'load (directory "gcl0/*.done")) -compile lsp/gcl_listlib.lsp -o gcl0/gcl_listlib.o
boud       69622 98.8 46.2 112523896 91540116 pts/1 R+ 14:54   8:35      |                           \_ unixport/gcl0 -eval (mapc 'load (directory "gcl0/*.done")) -compile lsp/gcl_seqlib.lsp -o gcl0/gcl_seqlib.o


My guess about "Killed" is that this is from the OOM killer deciding that insufficient RAM is available. The kernel is 6.1.0-33-amd64 (Debian/bookworm).

Could there be a closed loop requiring an infinite amount of disk space or ram (I guess this would be a type of race condition)?

[1] https://savannah.gnu.org/bugs/index.php?67034

[2] https://paste.debian.net/hidden/6bfdc292

Boud Roukema <boud>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by fosskers (Posted a comment)
  • -email is unavailable- added by camm (Posted a comment)
  • -email is unavailable- added by boud (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only logged-in users can vote.

     

    No changes have been made to this item

    Back to the top

    Powered by Savane 3.15-e6e5.
    Corresponding source code