bugGNU Octave - Bugs: bug #61024, [mxe-octave] build-markupsafe...

 
 

bug #61024: [mxe-octave] build-markupsafe fails to build if build-python3 is built without libffi

Submitted by:  John W. Eaton <jwe>
Submitted on:  Mon 09 Aug 2021 06:14:40 PM UTC  
 
Category:  Configuration and Build System Severity:  3 - Normal
Priority:  5 - Normal Item Group:  Build Failure
Status:  In Progress Assigned to:  None
Originator Name:  jwe Open/Closed:  Open
Release:  dev Operating System:  Microsoft Windows

Add a New Comment (Rich Markup)
   

You are not logged in

Please log in, so followups can be emailed to you.

 

( Jump to the original submission)

Tue 10 Aug 2021 08:42:29 PM UTC, comment #15: 

related, Ive added build-std to some updates Im working on for for msys2.

Will probally be pushing that tommorrow

John Donoghue <lostbard>
Project Member
Tue 10 Aug 2021 05:12:47 PM UTC, comment #14: 

Yeah. That is probably the best solution.

The "Requirements" section of the index.html already lists some packages. It is probably not up-to-date. I didn't know this list existed and never added anything on to it (like e.g. `zstd`).

Markus Mützel <mmuetzel>
Project Member
Tue 10 Aug 2021 03:59:10 PM UTC, comment #13: 

To get past this problem, I can easily install libffi on my systems.  But at some point it might be best if we were building our own?  Or if we came up with some other way to ensure a proper and consistent set of build tools for everyone using mxe-octave?

John W. Eaton <jwe>
Project Administrator
Tue 10 Aug 2021 03:57:40 PM UTC, comment #12: 

My previous change allows the _ctypes module to be built but I see the following messages in the build-python3 log file:

*** WARNING: renaming "_ctypes" since importing it failed: libffi.so.8: cannot open shared object file: No such file or directory

[...]

Following modules built successfully but were removed because they could not be imported:
_ctypes

If I pass "--without-system-ffi" to the configure command in build-python3.mk I see the same problem as above.

I don't know to fix that.  Maybe some LD_LIBRARY_PATH setting, but I'm not seeing how to make it work.

John W. Eaton <jwe>
Project Administrator
Tue 10 Aug 2021 03:18:49 PM UTC, comment #11: 

Oops, wait, I checked that build-python3 was built without errors but build-markupsafe is still failing.  I'll look at it again now.  Ugh.

John W. Eaton <jwe>
Project Administrator
Tue 10 Aug 2021 03:13:18 PM UTC, comment #10: 

After looking at my build (not one of the buildbots) I noticed this problem in the build-python3 log file:

gcc -pthread -fPIC -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -std=c99 -Wextra -Wno-unused-result -Wno-unused-parameter -Wno-missing-field-initializers -Werror=implicit-function-declaration -I/scratch/build/mxe-octave-w64/default/tmp-build-python3/Python-3.8.3/Include/internal -I/scratch/build/mxe-octave-w64/default/usr/x86_64-w64-mingw32/include -I. -I/scratch/build/mxe-octave-w64/default/usr/include -IObjects -IPython -I/usr/include/x86_64-linux-gnu -I/usr/local/include -I/scratch/build/mxe-octave-w64/default/tmp-build-python3/Python-3.8.3/Include -I/scratch/build/mxe-octave-w64/default/tmp-build-python3/Python-3.8.3/.build -c /scratch/build/mxe-octave-w64/default/tmp-build-python3/Python-3.8.3/Modules/_ctypes/_ctypes.c -o build/temp.linux-x86_64-3.8/scratch/build/mxe-octave-w64/default/tmp-build-python3/Python-3.8.3/Modules/_ctypes/_ctypes.o
In file included from /scratch/build/mxe-octave-w64/default/usr/x86_64-w64-mingw32/include/corecrt.h:10,
                 from /scratch/build/mxe-octave-w64/default/usr/x86_64-w64-mingw32/include/crtdefs.h:10,
                 from /scratch/build/mxe-octave-w64/default/usr/x86_64-w64-mingw32/include/limits.h:6,
                 from /scratch/build/mxe-octave-w64/default/tmp-build-python3/Python-3.8.3/Include/Python.h:11,
                 from /scratch/build/mxe-octave-w64/default/tmp-build-python3/Python-3.8.3/Modules/_ctypes/_ctypes.c:104:
/scratch/build/mxe-octave-w64/default/usr/x86_64-w64-mingw32/include/_mingw.h:272:2: error: #error Only Win32 target is supported!
  272 | #error Only Win32 target is supported!
      |  ^~~~~

I was not expecting to see "-I/scratch/build/mxe-octave-w64/default/usr/x86_64-w64-mingw32/include" in the compiler flags when building a native build tool.  If I understand correctly, this is happening because the python3 configure script is using the pkg-config program that we build to tell us what -I option we should use to find libffi headers.  But by default our pkg-config program looks for .pc files in directories that are appropriate for the cross built packages, not the native ones.  We try to fix that up in the build-*.mk files where needed by using "PKG_CONFIG_PATH='$(BUILD_PKG_CONFIG_PATH)'" on the configure command line, but the value of BUILD_PKG_CONFIG_PATH is set in the mxe-octave configure script before we build the build-pkg-config program.

I checked in the following change but I'm not sure it is the best solution.

https://hg.octave.org/mxe-octave/rev/4e1ae8cbc9ec

Let's see if this helps with the buildbots.  If not, I'll take another look.

John W. Eaton <jwe>
Project Administrator
Tue 10 Aug 2021 09:43:07 AM UTC, comment #9: 

It still doesn't build on your buildbots:
http://buildbot.octave.org:8010/#/builders/29/builds/1030/steps/7/logs/stdio

Could you please upload the build log of build-python from that run? Maybe that contains hints as to what is going wrong...

Markus Mützel <mmuetzel>
Project Member
Tue 10 Aug 2021 08:42:38 AM UTC, comment #8: 

And another one that adds build-libtool to the dependencies of build-libffi:
https://hg.octave.org/mxe-octave/rev/5cb9ea0261ba

Unfortunately, building those build tools seems to depend on which packages are installed on the build system. So it looks like it can fail on some build systems even if it works for me locally.
I hope we'll reach a point where this is working reliably (and less dependent on the build system) soon.

Markus Mützel <mmuetzel>
Project Member
Tue 10 Aug 2021 08:04:10 AM UTC, comment #7: 

I pushed a patch adding a build-libffi package here:
https://hg.octave.org/mxe-octave/rev/f61f106184dc

Marking as ready for test.

Markus Mützel <mmuetzel>
Project Member
Tue 10 Aug 2021 06:37:23 AM UTC, comment #6: 

That should have read: "The buildbots are failing to compile build-markupsafe now."

The error looks very similar to the one that you saw on your new system.

Markus Mützel <mmuetzel>
Project Member
Tue 10 Aug 2021 06:01:50 AM UTC, comment #5: 

The buildbots are failing to compile build-python now. Maybe, we would have needed a build-libffi package instead?

Re-opening.

Markus Mützel <mmuetzel>
Project Member
Mon 09 Aug 2021 08:29:00 PM UTC, comment #4: 

Works for me so closing as fixed.  Thanks!

John W. Eaton <jwe>
Project Administrator
Mon 09 Aug 2021 07:44:15 PM UTC, comment #3: 
Markus Mützel <mmuetzel>
Project Member
Mon 09 Aug 2021 07:35:41 PM UTC, comment #2: 

The change seems fine to me.  Go ahead and push it and I'll uninstall libffi-dev on my system and try another build.

John W. Eaton <jwe>
Project Administrator
Mon 09 Aug 2021 07:13:22 PM UTC, comment #1: 

I might have had libffi-dev installed on my system. So I never noticed this.

Would the attached patch be ok?

(file #51754)

Markus Mützel <mmuetzel>
Project Member
Mon 09 Aug 2021 06:14:40 PM UTC, original submission:  

I hit the following problem when doing an mxe-octave build on a new system:

[build]    build-markupsafe

Failed to build package build-markupsafe!
------------------------------------------------------------
make[2]: Entering directory '/scratch/build/mxe-octave-w64/default'
rm -rf   '/scratch/build/mxe-octave-w64/default/tmp-build-markupsafe' '/scratch/build/mxe-octave-w64/default/tmp-build-markupsafe-install'
mkdir -p '/scratch/build/mxe-octave-w64/default/tmp-build-markupsafe'
( cd '/scratch/build/mxe-octave-w64/default/tmp-build-markupsafe' &&  tar xzf '/share/mxe-octave/pkg-src/MarkupSafe-1.0.tar.gz' ) ||  false
test ! -d '/home/jwe/src/mxe-octave/src/build-markupsafe' || cp -a '/home/jwe/src/mxe-octave/src/build-markupsafe' '/scratch/build/mxe-octave-w64/default/tmp-build-markupsafe'
cd '/scratch/build/mxe-octave-w64/default/tmp-build-markupsafe/MarkupSafe-1.0'
cd '/scratch/build/mxe-octave-w64/default/tmp-build-markupsafe/MarkupSafe-1.0' && python3 setup.py install --prefix='/scratch/build/mxe-octave-w64/default/usr'
Traceback (most recent call last):
  File "setup.py", linlibffi-dee 6, in <module>
    from setuptools import setup, Extension, Feature
  File "/scratch/build/mxe-octave-w64/default/usr/lib/python3.8/site-packages/setuptools/__init__.py", line 20, in <module>
    from setuptools.dist import Distribution, Feature
  File "/scratch/build/mxe-octave-w64/default/usr/lib/python3.8/site-packages/setuptools/dist.py", line 35, in <module>
    from setuptools import windows_support
  File "/scratch/build/mxe-octave-w64/default/usr/lib/python3.8/site-packages/setuptools/windows_support.py", line 2, in <module>
    import ctypes
  File "/scratch/build/mxe-octave-w64/default/usr/lib/python3.8/ctypes/__init__.py", line 7, in <module>
    from _ctypes import Union, Structure, Array
ModuleNotFoundError: No module named '_ctypes'
make[2]: *** [/scratch/build/mxe-octave-w64/default/Makefile:977: build-only-build-markupsafe] Error 1

After installing libffi-dev on the build system and rebuilding the build-python3 package in mxe-octave, the problem is gone and the build-markupsafe package builds successfully.  But maybe we should have rules to build the libffi package as part of the mxe-octave build rules so we can control the version that is used and to avoid this problem in the future?

John W. Eaton <jwe>
Project Administrator

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

Attached Files
file #51754:  bug61024-mxe-libffi.patch added by mmuetzel (3KiB - application/octet-stream)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by lostbard (Posted a comment)
  • -email is unavailable- added by mmuetzel (Updated the item)
  • -email is unavailable- added by jwe (Submitted the item)
  •  

    Do you think this task is very important?
    If so, you can add your encouragement to it.
    This task has 0 encouragements so far.

    Only project members can vote.

     

     

     

    Follow 7 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2021-08-10 mmuetzel StatusReady For Test => In Progress
    2021-08-10 mmuetzel StatusIn Progress => Ready For Test
    2021-08-10 mmuetzel StatusFixed => In Progress
        Open/ClosedClosed => Open
    2021-08-09 jwe StatusNone => Fixed
        Open/ClosedOpen => Closed
    2021-08-09 mmuetzel Attached File- => Added bug61024-mxe-libffi.patch, #51754

    Back to the top


    Powered by Savane 3.6