bugGNU Octave - Bugs: bug #61024, [mxe-octave] build-markupsafe...

 
 

bug #61024: [mxe-octave] build-markupsafe fails to build if build-python3 is built without libffi

Submitter:  John W. Eaton <jwe>
Submitted:  Mon 09 Aug 2021 06:14:40 PM UTC
   
 
Category:  Configuration and Build System Severity:  3 - Normal
Priority:  5 - Normal Item Group:  Build Failure
Status:  In Progress Assigned to:  None
Originator Name:  jwe Open/Closed:  * Open
Release:  * dev Operating System:  * Microsoft Windows
Fixed Release:  None Planned Release:  None
* Mandatory Fields

Add a New Comment Rich Markup
   

Jump to the original submission

Tue 10 Aug 2021 08:42:29 PM UTC, comment #15: 

related, Ive added build-std to some updates Im working on for for msys2.

Will probally be pushing that tommorrow

John Donoghue <lostbard>
Group Member
Tue 10 Aug 2021 05:12:47 PM UTC, comment #14: 

Yeah. That is probably the best solution.

The "Requirements" section of the index.html already lists some packages. It is probably not up-to-date. I didn't know this list existed and never added anything on to it (like e.g. `zstd`).

Markus Mützel <mmuetzel>
Group administrator
Tue 10 Aug 2021 03:59:10 PM UTC, comment #13: 

To get past this problem, I can easily install libffi on my systems.  But at some point it might be best if we were building our own?  Or if we came up with some other way to ensure a proper and consistent set of build tools for everyone using mxe-octave?

John W. Eaton <jwe>
Group administrator
Tue 10 Aug 2021 03:57:40 PM UTC, comment #12: 

My previous change allows the _ctypes module to be built but I see the following messages in the build-python3 log file:


*** WARNING: renaming "_ctypes" since importing it failed: libffi.so.8: cannot open shared object file: No such file or directory

[...]

Following modules built successfully but were removed because they could not be imported:
_ctypes


If I pass "--without-system-ffi" to the configure command in build-python3.mk I see the same problem as above.

I don't know to fix that.  Maybe some LD_LIBRARY_PATH setting, but I'm not seeing how to make it work.

John W. Eaton <jwe>
Group administrator
Tue 10 Aug 2021 03:18:49 PM UTC, comment #11: 

Oops, wait, I checked that build-python3 was built without errors but build-markupsafe is still failing.  I'll look at it again now.  Ugh.

John W. Eaton <jwe>
Group administrator
Tue 10 Aug 2021 03:13:18 PM UTC, comment #10: 

After looking at my build (not one of the buildbots) I noticed this problem in the build-python3 log file:


gcc -pthread -fPIC -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -std=c99 -Wextra -Wno-unused-result -Wno-unused-parameter -Wno-missing-field-initializers -Werror=implicit-function-declaration -I/scratch/build/mxe-octave-w64/default/tmp-build-python3/Python-3.8.3/Include/internal -I/scratch/build/mxe-octave-w64/default/usr/x86_64-w64-mingw32/include -I. -I/scratch/build/mxe-octave-w64/default/usr/include -IObjects -IPython -I/usr/include/x86_64-linux-gnu -I/usr/local/include -I/scratch/build/mxe-octave-w64/default/tmp-build-python3/Python-3.8.3/Include -I/scratch/build/mxe-octave-w64/default/tmp-build-python3/Python-3.8.3/.build -c /scratch/build/mxe-octave-w64/default/tmp-build-python3/Python-3.8.3/Modules/_ctypes/_ctypes.c -o build/temp.linux-x86_64-3.8/scratch/build/mxe-octave-w64/default/tmp-build-python3/Python-3.8.3/Modules/_ctypes/_ctypes.o
In file included from /scratch/build/mxe-octave-w64/default/usr/x86_64-w64-mingw32/include/corecrt.h:10,
                 from /scratch/build/mxe-octave-w64/default/usr/x86_64-w64-mingw32/include/crtdefs.h:10,
                 from /scratch/build/mxe-octave-w64/default/usr/x86_64-w64-mingw32/include/limits.h:6,
                 from /scratch/build/mxe-octave-w64/default/tmp-build-python3/Python-3.8.3/Include/Python.h:11,
                 from /scratch/build/mxe-octave-w64/default/tmp-build-python3/Python-3.8.3/Modules/_ctypes/_ctypes.c:104:
/scratch/build/mxe-octave-w64/default/usr/x86_64-w64-mingw32/include/_mingw.h:272:2: error: #error Only Win32 target is supported!
  272 | #error Only Win32 target is supported!
      |  ^~~~~


I was not expecting to see "-I/scratch/build/mxe-octave-w64/default/usr/x86_64-w64-mingw32/include" in the compiler flags when building a native build tool.  If I understand correctly, this is happening because the python3 configure script is using the pkg-config program that we build to tell us what -I option we should use to find libffi headers.  But by default our pkg-config program looks for .pc files in directories that are appropriate for the cross built packages, not the native ones.  We try to fix that up in the build-*.mk files where needed by using "PKG_CONFIG_PATH='$(BUILD_PKG_CONFIG_PATH)'" on the configure command line, but the value of BUILD_PKG_CONFIG_PATH is set in the mxe-octave configure script before we build the build-pkg-config program.

I checked in the following change but I'm not sure it is the best solution.

https://hg.octave.org/mxe-octave/rev/4e1ae8cbc9ec

Let's see if this helps with the buildbots.  If not, I'll take another look.

John W. Eaton <jwe>
Group administrator
Tue 10 Aug 2021 09:43:07 AM UTC, comment #9: 

It still doesn't build on your buildbots:
http://buildbot.octave.org:8010/#/builders/29/builds/1030/steps/7/logs/stdio

Could you please upload the build log of build-python from that run? Maybe that contains hints as to what is going wrong...

Markus Mützel <mmuetzel>
Group administrator
Tue 10 Aug 2021 08:42:38 AM UTC, comment #8: 

And another one that adds build-libtool to the dependencies of build-libffi:
https://hg.octave.org/mxe-octave/rev/5cb9ea0261ba

Unfortunately, building those build tools seems to depend on which packages are installed on the build system. So it looks like it can fail on some build systems even if it works for me locally.
I hope we'll reach a point where this is working reliably (and less dependent on the build system) soon.

Markus Mützel <mmuetzel>
Group administrator
Tue 10 Aug 2021 08:04:10 AM UTC, comment #7: 

I pushed a patch adding a build-libffi package here:
https://hg.octave.org/mxe-octave/rev/f61f106184dc

Marking as ready for test.

Markus Mützel <mmuetzel>
Group administrator
Tue 10 Aug 2021 06:37:23 AM UTC, comment #6: 

That should have read: "The buildbots are failing to compile build-markupsafe now."

The error looks very similar to the one that you saw on your new system.

Markus Mützel <mmuetzel>
Group administrator
Tue 10 Aug 2021 06:01:50 AM UTC, comment #5: 

The buildbots are failing to compile build-python now. Maybe, we would have needed a build-libffi package instead?

Re-opening.

Markus Mützel <mmuetzel>
Group administrator
Mon 09 Aug 2021 08:29:00 PM UTC, comment #4: 

Works for me so closing as fixed.  Thanks!

John W. Eaton <jwe>
Group administrator
Mon 09 Aug 2021 07:44:15 PM UTC, comment #3: 
Markus Mützel <mmuetzel>
Group administrator
Mon 09 Aug 2021 07:35:41 PM UTC, comment #2: 

The change seems fine to me.  Go ahead and push it and I'll uninstall libffi-dev on my system and try another build.

John W. Eaton <jwe>
Group administrator
Mon 09 Aug 2021 07:13:22 PM UTC, comment #1: 

I might have had libffi-dev installed on my system. So I never noticed this.

Would the attached patch be ok?

(file #51754)

Markus Mützel <mmuetzel>
Group administrator
Mon 09 Aug 2021 06:14:40 PM UTC, original submission:  

I hit the following problem when doing an mxe-octave build on a new system:


[build]    build-markupsafe

Failed to build package build-markupsafe!
------------------------------------------------------------
make[2]: Entering directory '/scratch/build/mxe-octave-w64/default'
rm -rf   '/scratch/build/mxe-octave-w64/default/tmp-build-markupsafe' '/scratch/build/mxe-octave-w64/default/tmp-build-markupsafe-install'
mkdir -p '/scratch/build/mxe-octave-w64/default/tmp-build-markupsafe'
( cd '/scratch/build/mxe-octave-w64/default/tmp-build-markupsafe' &&  tar xzf '/share/mxe-octave/pkg-src/MarkupSafe-1.0.tar.gz' ) ||  false
test ! -d '/home/jwe/src/mxe-octave/src/build-markupsafe' || cp -a '/home/jwe/src/mxe-octave/src/build-markupsafe' '/scratch/build/mxe-octave-w64/default/tmp-build-markupsafe'
cd '/scratch/build/mxe-octave-w64/default/tmp-build-markupsafe/MarkupSafe-1.0'
cd '/scratch/build/mxe-octave-w64/default/tmp-build-markupsafe/MarkupSafe-1.0' && python3 setup.py install --prefix='/scratch/build/mxe-octave-w64/default/usr'
Traceback (most recent call last):
  File "setup.py", linlibffi-dee 6, in <module>
    from setuptools import setup, Extension, Feature
  File "/scratch/build/mxe-octave-w64/default/usr/lib/python3.8/site-packages/setuptools/__init__.py", line 20, in <module>
    from setuptools.dist import Distribution, Feature
  File "/scratch/build/mxe-octave-w64/default/usr/lib/python3.8/site-packages/setuptools/dist.py", line 35, in <module>
    from setuptools import windows_support
  File "/scratch/build/mxe-octave-w64/default/usr/lib/python3.8/site-packages/setuptools/windows_support.py", line 2, in <module>
    import ctypes
  File "/scratch/build/mxe-octave-w64/default/usr/lib/python3.8/ctypes/__init__.py", line 7, in <module>
    from _ctypes import Union, Structure, Array
ModuleNotFoundError: No module named '_ctypes'
make[2]: *** [/scratch/build/mxe-octave-w64/default/Makefile:977: build-only-build-markupsafe] Error 1


After installing libffi-dev on the build system and rebuilding the build-python3 package in mxe-octave, the problem is gone and the build-markupsafe package builds successfully.  But maybe we should have rules to build the libffi package as part of the mxe-octave build rules so we can control the version that is used and to avoid this problem in the future?

John W. Eaton <jwe>
Group administrator

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

Attached Files
file #51754:  bug61024-mxe-libffi.patch added by mmuetzel (3KiB - application/octet-stream)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by lostbard (Posted a comment)
  • -email is unavailable- added by mmuetzel (Updated the item)
  • -email is unavailable- added by jwe (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only group members can vote.

     

    Follow 7 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2021-08-10 mmuetzel StatusReady For Test In Progress
    2021-08-10 mmuetzel StatusIn Progress Ready For Test
    2021-08-10 mmuetzel StatusFixed In Progress
        Open/ClosedClosed Open
    2021-08-09 jwe StatusNone Fixed
        Open/ClosedOpen Closed
    2021-08-09 mmuetzel Attached File- Added bug61024-mxe-libffi.patch, #51754

    Back to the top

    Powered by Savane 3.13-4448.
    Corresponding source code