bugGNU gettext - Bugs: bug #63371, build fails on MinGW due to...

 
 

You are not allowed to post comments on this tracker with your current authentication level.

bug #63371: build fails on MinGW due to missing gnulib close module

Submitter:  Vaclav Slavik <vslavik>
Submitted:  Tue 15 Nov 2022 06:15:05 PM UTC
   
 
Category:  Build Severity:  3 - Normal
Item Group:  None Status:  Works For Me
Privacy:  Public Assigned to:  haible
Open/Closed:  Closed

Jump to the original submission

Thu 29 Jun 2023 12:39:44 PM UTC, comment #12: 

Vaclav Slavik wrote:

> I'm seeing build failures in up-to-date MSYS2 environment


The original reason why I call MSYS2 unreliable is this hack:

export MSYS2_ARG_CONV_EXCL="-DLOCALEDIR=;-DLIBDIR=;-DLOCALE_ALIAS_PATH="


Seen in
https://github.com/msys2/MINGW-packages/blob/master/mingw-w64-gettext/PKGBUILD

A system in which, in a program invocation, the program by default receives different arguments than the ones that the caller passed is unreliable.

Anyone who sees this and https://github.com/msys2/MSYS2-packages/issues/1130 should
1. vomit,
2. turn away from MSYS2, and
3. use Cygwin as dev environment instead.

Bruno Haible <haible>
Group administrator
Sun 25 Jun 2023 03:05:21 PM UTC, comment #11: 


> I have further bisected this to one of the following gnulib commits.


Unfortunately, this bisect result does not help: The gnulib change from 2021-04-11 was huge, and was subsequently stabilized over the following three months.

The only reasonable way forward is to look at the results of autoconfiguration, that is:
  $ grep close config.cache
  $ grep CLOSE config.status
for the config.cache file that is generated when you configure gettext with option '-C', and for the various config.status files that are generated as well.

> when attempting to cross compile from x86_64-unknown-linux-gnu to x86_64-w64-mingw32


Are you doing this with Wine enabled, or without? If it's without Wine, it's pointless to invest any effort into such a build. If it's with Wine, it may be worth your time, since afterwards "make check" can give you an indication about the quality of the resulting binaries.

Bruno Haible <haible>
Group administrator
Sun 25 Jun 2023 02:17:50 PM UTC, comment #10: 

I have further bisected this to one of the following gnulib commits.  (The commits are not contiguous, but I've verified it's not any of the commits in between by dropping them in a rebase.)  I can't narrow it down any further than this, because it I get different (earlier) build errors if I try to build any intermediate commit in this range.  Vaclav, I hope this is helpful to you, as I've reached the limits of my abilities given my limited understanding of gnulib.

commit 80a344e3237b878734295cbe696b7f1e4a8541f0
Author: Bruno Haible <bruno@clisp.org>
Date:   Sun Apr 11 21:25:54 2021 +0200

    Support several gnulib-tool invocations under the same configure.ac.

    Reported by Reuben Thomas <rrt@sc3d.org> in
    <https://lists.gnu.org/archive/html/bug-gnulib/2021-04/msg00104.html>.

    This is done by defining the Gnulib module indicator variables per
    gnulib-tool invocation. So that a generated .h file is no longer
    influenced by the set of modules used in other gnulib-tool invocations.

commit 8b857ed1ad1cbbf265d43016ee7ee93c7b7c6cb4
Author: Bruno Haible <bruno@clisp.org>
Date:   Sun Apr 11 15:54:18 2021 +0200

    Rename GNULIB_OVERRIDES_CHAR16_T, GNULIB_OVERRIDES_CHAR32_T.

commit db16856ae761bc213942c17f6fc2b7e9655014b4
Author: Bruno Haible <bruno@clisp.org>
Date:   Sun Apr 11 15:50:35 2021 +0200

    Rename GNULIB_OVERRIDES_WINT_T.

commit 2bdca6a667a32d5b92a6ffc3311524748f08e506
Author: Bruno Haible <bruno@clisp.org>
Date:   Sun Apr 11 02:38:27 2021 +0200

    Simplify GNULIB_* variable initializations.

Alyssa Ross <qyliss>
Thu 22 Jun 2023 07:54:49 AM UTC, comment #9: 


> Wine is not involved in cross-compilation. Only a compiler and Mingw headers and import libraries are.


Oh, you are talking about an environment where
- all Autoconf tests that involve AC_RUN_IFELSE are guesses with manually predetermined results,
- it is not possible to run "make check"?

As a builder, you have not the slightest clue about the reliability of the resulting binaries. I don't even want to think about such environments.

> this is precisely the motivation for avoiding Cygwin... When only Mingw headers and libraries are involved, you're guaranteed that the output is compatible with real (Cygwin-less) Windows.


True. But since the GNU Build system is based on a POSIX 'make' program and a POSIX 'sh' program, and native Windows doesn't have these, they must come from somewhere. Either from MSYS2, which is a minified Cygwin and thus has the same problem that you want to avoid; or from a Linux-based cross-compilation environment with Wine; or from WSL. Maybe someone knows whether WSL is usable for such cross-compilation?

> As Cygwin and Mingw both use Windows PE format, it's all too easy to accidentally pull in a part of Cygwin (been there, done that too many times) or inadvertently run a runtime check against Cygwin instead of Mingw target.


While this is theoretically possible, I think/hope I'm avoiding most of these pitfalls because:
- I also run 32-bit mingw compilations on 64-bit cygwin; the object formats of 32-bit mingw and 64-bit cygwin are not the same.
- I routinely check the "dumpbin /imports" of binaries.
- Most of the Autoconf tests use AC_LINK_IFELSE and AC_RUN_IFELSE, which use the correct compiler.

Bruno Haible <haible>
Group administrator
Thu 22 Jun 2023 06:43:00 AM UTC, comment #8: 

comment #7:

> The problem with x86_64-w64-mingw32 in a build environment of type x86_64-unknown-linux-gnu is that it adds another component, namely Wine, to the mix. Which adds a source of complexity/bugs


Wine is not involved in cross-compilation. Only a compiler and Mingw headers and import libraries are. It actually avoids an extra component the supported build requires, namely Cygwin.

Your preferred environment is conceptually similar: you're cross-compiling from Cygwin to Mingw runtime, only without the added protection of binary formats being incompatible and not executable on the host. As Cygwin and Mingw both use Windows PE format, it's all too easy to accidentally pull in a part of Cygwin (been there, done that too many times) or inadvertently run a runtime check against Cygwin instead of Mingw target. Indeed, I suspect something of the kind may be happening here with the close module, which is why it fails in native and cross-compile-from-linux environments, but not in cross-compile-from-cygwin one.

By the way: this seems to have affected mlocati's binaries (the ones you link to) too: they used cross-compilation setup and per https://github.com/mlocati/gettext-iconv-windows/issues/14 gave up on trying to build 0.21.1 in an "official" Cygwin environment.

> And what is it worth if there is the possibility that the resulting binaries work fine in this environment, but not in real Windows?


For what it's worth, I couldn't agree more; this is precisely the motivation for avoiding Cygwin... When only Mingw headers and libraries are involved, you're guaranteed that the output is compatible with real (Cygwin-less) Windows.

> There is no continuum of operating systems between POSIX and Windows. It's two isolated groups of OSes:


I don't think I of all people need this explained :) My point was that the issue is in the check and is likely related to (cross-)compilation and not Windows per se.

> If you find it worth fixing, then feel free to spend the effort on it. I don't want to spend effort on it,


Of course, understood.

Vaclav Slavik <vslavik>
Wed 21 Jun 2023 08:29:10 PM UTC, comment #7: 


> the supported way is non-standard for MinGW and rather complicated to setup, especially in any automated fashion


The problem with x86_64-w64-mingw32 in a build environment of type x86_64-unknown-linux-gnu is that it adds another component, namely Wine, to the mix. Which adds a source of complexity/bugs. And what is it worth if there is the possibility that the resulting binaries work fine in this environment, but not in real Windows?

> this may indicate problems on other platforms too. The point of using Autoconf-based build system surely is to detect and handle runtime conditions without having specific hard-coded support for specific platforms (e.g., require 2 POSIX-y runtimes to build for Windows), right? Similarly, the point of gnulib is to provide missing bits in a more general fashion, no?


The Autoconf principle with configure tests works fine as long as the behaviour differences are small (e.g. a parameter supporting a certain value or not). Whereas for native Windows support, in many places, specific code is needed. And then it makes no difference whether we use an Autoconf test or just a specific 'case $host_os in mingw*) ...'. There is no continuum of operating systems between POSIX and Windows. It's two isolated groups of OSes: POSIX and Unix-like on one hand, and native Windows. It would be useless to write "general" code that works on a continuum of operating systems that does not exist. Because such code would not be testable, and code which isn't testable is usually not working.

> That seems worth fixing regardless of anything else.


If you find it worth fixing, then feel free to spend the effort on it. I don't want to spend effort on it, since there don't need to be several ways to arrive at bative Windows binaries. I chose the way that seems most reliable to me.

Bruno Haible <haible>
Group administrator
Wed 21 Jun 2023 05:54:47 PM UTC, comment #6: 

Ah, I hoped the issue would go away on its own in 0.22, but no such luck then.

comment #5:

> This is not a supported build environment for building for x86_64-w64-mingw32; see the file INSTALL.windows for the supported build environments.


We understand. But that aside (FWIW, the supported way is non-standard for MinGW and rather complicated to setup, especially in any automated fashion), this may indicate problems on other platforms too. The point of using Autoconf-based build system surely is to detect and handle runtime conditions without having specific hard-coded support for specific platforms (e.g., require 2 POSIX-y runtimes to build for Windows), right? Similarly, the point of gnulib is to provide missing bits in a more general fashion, no?

The fact that this gnulib module broke in some rather vanilla (albeit unsupported/unwanted by you) environments - two of them now - indicates that there's some subtle bug in the detection or compilation logic of the module that wasn't there before and isn't there for other modules. That seems worth fixing regardless of anything else.

I of course understand why it's not a priority for you. I wouldn't have filed this issue without a tested patch if gnulib wasn't as impenetrable to outsiders as it is, but I'll give it another go - I hope you don't mind accepting verified working patches even for unsupported environments.

> I don't see any hints either. @GL_GNULIB_CLOSE@ is 1, that is, the module 'close' is in use when write-catalog.c gets compiled.


Doesn't this:

> catalog.o:write-catalog.:(.text+0x6e4): undefined reference to `close_used_without_requesting_gnulib_module_close'


contradict that, though?

Vaclav Slavik <vslavik>
Wed 21 Jun 2023 05:13:07 PM UTC, comment #5: 


> I have the same problem when attempting to cross compile from x86_64-unknown-linux-gnu to x86_64-w64-mingw32.


This is not a supported build environment for building for x86_64-w64-mingw32; see the file INSTALL.windows for the supported build environments.

Bruno Haible <haible>
Group administrator
Wed 21 Jun 2023 02:34:03 PM UTC, comment #4: 

I have the same problem when attempting to cross compile from x86_64-unknown-linux-gnu to x86_64-w64-mingw32.  I have bisected the regression, and found that it was introduced by one of these commits:

commit acb7f1cc9b4df499b23576c488f66540330ca836
Author: Bruno Haible <bruno@clisp.org>
Date:   Sun Jun 20 14:19:39 2021 +0200

    Update after gnulib changed.

    Previously, different gnulib-tool invocations within the scope of the same
    configure.ac interfered with each other (through the GNULIB_* indicator macros,
    used in the generated .h files), and libgrep relied on it. Now, each gnulib-tool
    invocation needs its own copy of the generated .h files, in particular locale.h.
    Needed for the localeconv() override on native Windows.

    * autogen.sh: When invoking gnulib-tool for libgrep, don't avoid 'locale' and
    the other header file modules.
    * gettext-tools/configure.ac: Set GL_GRGL_GNULIB_SETLOCALE_NULL to 1.

commit f8c28f094dd7f85f1dc0631435f6b76c19b50242
Author: Bruno Haible <bruno@clisp.org>
Date:   Sun Jun 20 03:47:14 2021 +0200

    Update after gnulib changed.

    * gettext-runtime/libasprintf/autosprintf.cc: Include <config.h>. This is
    needed because of these new gnulib module dependencies:
      vasnprintf -> free-posix -> stdlib -> unistd,
    and the generated unistd.h gets included indirectly by <string> on OpenBSD 6.7
    and gives a #error "Please include config.h first."

commit dbcae010af498da55cb1fdac2c22d91206da34c0
Author: Bruno Haible <bruno@clisp.org>
Date:   Sun Jun 20 03:44:25 2021 +0200

    Update to the newest gnulib.

    * gnulib-local/lib/localename.c.diff: Remove file.
    * gnulib-local/lib/localename-table.h.diff: Remove file.
    * gnulib-local/lib/tsearch.c.diff: Remove file.
    * gnulib-local/modules/tsearch.diff: Remove file.
    * gnulib-local/Makefile.am (EXTRA_DIST): Remove them.
    * libtextstyle/configure.ac: Update for changed naming convention of module
    indicator variables.

Alyssa Ross <qyliss>
Mon 13 Feb 2023 10:54:10 PM UTC, comment #3: 


> Attaching cache and status; I wasn't able to find meaningful (to me) hints in there.


I don't see any hints either. @GL_GNULIB_CLOSE@ is 1, that is, the module 'close' is in use when write-catalog.c gets compiled.

Bruno Haible <haible>
Group administrator
Wed 16 Nov 2022 05:16:57 PM UTC, comment #2: 


> The patch you provided is probably a no-op, because GNULIB_MODULES_TOOLS_FOR_SRC contains 'fopen', and 'fopen' depends on 'close'.


OK... In my defense, it is explicitly included in GNULIB_MODULES_LIBGETTEXTPO (described as "This is a subset of the GNULIB_MODULES_TOOLS_FOR_SRC" - though it was not). To be honest, I didn't even realize this could be an issue with my (unsupported) build environment; if I did, I'd try to dig deeper myself first.

I don't see any relevant variables in set's output besides perhaps those:

MSYSTEM=MINGW32
MSYSTEM_CARCH=i686
MSYSTEM_CHOST=i686-w64-mingw32
MSYSTEM_PREFIX=/mingw32

That is, I'm building 32bit executables. Configure command is this:

./configure -C --prefix=c:/usr/local --disable-static --disable-dependency-tracking --enable-silent-rules --with-libiconv-prefix=/d/a/gettext-tools-windows/gettext-tools-windows/build/stage/usr/local --disable-rpath --enable-nls --disable-csharp --disable-java --enable-threads=windows --enable-relocatable CFLAGS="-O2" CXXFLAGS="-O2" LDFLAGS="-Wl,--dynamicbase -Wl,--nxcompat -Wl,--no-seh"

Attaching cache and status; I wasn't able to find meaningful (to me) hints in there.

(file #53964, file #53965)

Vaclav Slavik <vslavik>
Wed 16 Nov 2022 02:37:22 AM UTC, comment #1: 

The patch you provided is probably a no-op, because GNULIB_MODULES_TOOLS_FOR_SRC contains 'fopen', and 'fopen' depends on 'close'.

Since I had tested the gettext-0.21.1 tarball on mingw-w64 before releasing it (in a Cygwin environment, as described in the INSTALL.windows file), it's likely that there are subtle differences between your environment and mine.

Can you please
1) Redo the configuration, this time with option -C, so that a config.cache is created,
2) Show the commands you use to configure and build the tarball, with all relevant environment variables (CC, CXX, CPPFLAGS, etc.),
3) Attach the generated gettext-tools/config.cache and gettext-tools/config.status files.
Thanks.

Bruno Haible <haible>
Group administrator
Tue 15 Nov 2022 06:15:05 PM UTC, original submission:  

I'm seeing build failures in up-to-date MSYS2 environment when trying to build 0.21.1.



D:/a/_temp/msys64/mingw32/bin/../lib/gcc/i686-w64-mingw32/12.2.0/../../../../i686-w64-mingw32/bin/ld.exe: .libs/libgettextsrc_la-write-catalog.o:write-catalog.:(.text+0x6e4): undefined reference to `close_used_without_requesting_gnulib_module_close'
5766
collect2.exe: error: ld returned 1 exit status
5767
make[4]: *** [Makefile:3782: libgettextsrc.la] Error 1
5768
make[4]: Leaving directory '/d/a/gettext-tools-windows/gettext-tools-windows/build/compile/gettext-0.21.1/gettext-tools/src'
5769
make[3]: *** [Makefile:3629: all] Error 2


(Eventually, more files that use close() would be affected (msgexec.c, msggrep.c, urlget.c, write-catalog.c), this is just the first failure.)

I think this patch, which adds close to GNULIB_MODULES_TOOLS_FOR_SRC, is sufficient to fix:


diff --git a/autogen.sh b/autogen.sh
index b896c3f1e..02a5fe130 100755
--- a/autogen.sh
+++ b/autogen.sh
@@ -134,6 +134,7 @@ if ! $skip_gnulib; then
     c-strcasestr
     c-strstr
     clean-temp
+    close
     closedir
     closeout
     configmake


But I have difficulty successfully running "make dist" here to be able to test it, and all the relevant files are generated and not included in git.

Vaclav Slavik <vslavik>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attached Files
file #53964:  config.status added by vslavik (178KiB - application/octet-stream)
file #53965:  config.cache added by vslavik (43KiB - application/octet-stream)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by qyliss (Posted a comment)
  • -email is unavailable- added by haible (Posted a comment)
  • -email is unavailable- added by vslavik (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

     

    Follow 6 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2023-02-13 haible StatusNeed Info Works For Me
        Open/ClosedOpen Closed
    2022-11-16 vslavik Attached File- Added config.status, #53964
        Attached File- Added config.cache, #53965
    2022-11-16 haible StatusNone Need Info
        Assigned toNone haible

    Back to the top

    Powered by Savane 3.13-3230.
    Corresponding source code