bugGNU Octave - Bugs: bug #58689, [MXE Octave] Update mesa to use...

 
 

bug #58689: [MXE Octave] Update mesa to use meson as (native) build tool

Submitter:  Markus Mützel <mmuetzel>
Submitted:  Thu 02 Jul 2020 04:05:38 PM UTC
   
 
Category:  Configuration and Build System Severity:  3 - Normal
Priority:  5 - Normal Item Group:  Other
Status:  Fixed Assigned to:  None
Originator Name:  Open/Closed:  * Closed
Release:  * other Operating System:  * GNU/Linux
Fixed Release:  None Planned Release:  None
* Mandatory Fields

Add a New Comment Rich Markup
   

Jump to the original submission

Sat 01 Aug 2020 09:48:37 AM UTC, comment #43: 

Seems to work for me with the updated version.

Closing as fixed.

Markus Mützel <mmuetzel>
Group administrator
Fri 31 Jul 2020 06:49:56 AM UTC, comment #42: 

The stripping errors seem to have disappeared with:
https://hg.octave.org/mxe-octave/rev/beb3380b1665

Thanks.

Markus Mützel <mmuetzel>
Group administrator
Thu 30 Jul 2020 11:09:31 AM UTC, comment #41: 

looks like msys2 python installs some 64 bit dist tools as well as 32 bit, so probally need to just remove the 64 bit ones

John Donoghue <lostbard>
Group Member
Thu 30 Jul 2020 06:30:34 AM UTC, comment #40: 

The messages from comment #39 already appeared in build #814:
http://buildbot.octave.org:8010/#/builders/14/builds/814/steps/8/logs/stdio

That was before the changes from comment #38.

Might be related to this change instead:
Use msys2 python3
https://hg.octave.org/mxe-octave/rev/dace3d372190

Markus Mützel <mmuetzel>
Group administrator
Wed 29 Jul 2020 08:29:03 PM UTC, comment #39: 

The 32bit Windows buildbot is having problems stripping some files:

stripping files...
/scratch/buildbot/workers/jwe-debian-x86_64-0/w32-stable-on-debian/src/usr/bin/i686-w64-mingw32-strip: /scratch/buildbot/workers/jwe-debian-x86_64-0/w32-stable-on-debian/src/dist/octave-2020-07-28-12-31-w32/usr/lib/python3.8/distutils/command/wininst-9.0-amd64.exe: file format not recognized
/scratch/buildbot/workers/jwe-debian-x86_64-0/w32-stable-on-debian/src/usr/bin/i686-w64-mingw32-strip: /scratch/buildbot/workers/jwe-debian-x86_64-0/w32-stable-on-debian/src/dist/octave-2020-07-28-12-31-w32/usr/lib/python3.8/distutils/command/wininst-10.0-amd64.exe: file format not recognized
/scratch/buildbot/workers/jwe-debian-x86_64-0/w32-stable-on-debian/src/usr/bin/i686-w64-mingw32-strip: /scratch/buildbot/workers/jwe-debian-x86_64-0/w32-stable-on-debian/src/dist/octave-2020-07-28-12-31-w32/usr/lib/python3.8/distutils/command/wininst-14.0-amd64.exe: file format not recognized


I didn't notice that on earlier tests.

Markus Mützel <mmuetzel>
Group administrator
Wed 29 Jul 2020 10:08:40 AM UTC, comment #38: 

I pushed the following changes to use meson as the build tool for mesa:
https://hg.octave.org/mxe-octave/rev/455bb9c9a7eb
https://hg.octave.org/mxe-octave/rev/4edf49596bf0
https://hg.octave.org/mxe-octave/rev/be0a7ce498e6

They are not perfect at the moment (e.g. they probably don't use ccache correctly). But that can be fixed with some follow up changes imho.
It should be easier to update mesa now.

Marking as ready for test.

Markus Mützel <mmuetzel>
Group administrator
Mon 27 Jul 2020 06:07:13 PM UTC, comment #37: 

It looks like (at least for the time being) we cannot fully replace python2 with python3 for all packages.
That means that it is not necessary to push everything here in one go. I hope that the risk of breaking something is smaller when doing incremental steps (or at least this way it might be easier to find the error in case something breaks).
As the initial steps I pushed John Donogue's change that allows the python scripts in MXE Octave to run with python3 here:
https://hg.octave.org/mxe-octave/rev/898c27394c57

Additionally, I pushed this change to add a build rule for python3:
https://hg.octave.org/mxe-octave/rev/fb50ae69121c

With these changes, some of the other patches probably no longer apply cleanly. I'll update them when I come around to it.

Markus Mützel <mmuetzel>
Group administrator
Mon 20 Jul 2020 01:23:24 PM UTC, comment #36: 

Running the BIST's is a great idea. I just finished "__run_test_suite__" on 32bit + 64bit and unfortunately (or not unfortunately) here is everything fine. So i can't reproduce the closing problem. But lets keep track on that in report #58790.

Furthermore i see 9 fails in the 32 bit version whereas none in the 64bit. The results are:


Summary:

  PASS                            16211
  FAIL                                0
  XFAIL (reported bug)               30
  SKIP (missing feature)             52
  SKIP (run-time condition)          50


Summary:

  PASS                            16202
  FAIL                                9
  XFAIL (reported bug)               30
  SKIP (missing feature)             52
  SKIP (run-time condition)          50


I picked one example:


test('xlim')


This test creates a plot with an x-axis in the range 0..1.2 but the test checks for a range between 0..1.4. However the BIST plot cmnd is:


h = plot3 ([0,1.1], [0,1], [0, 1]);


So IMHO "1.2" and "1.4" are both valid. However that doesn't answer the question why the figure axes-limits are set differently between 32/64bit. However the fails in the 32bit maybe for this report OT too and may require some additional attention.

Coming back to this "bug" report my conclusion is that your mesa update patches are good.

Hg200 <hg200>
Mon 20 Jul 2020 12:51:00 PM UTC, comment #35: 

Octave 32bit closes unexpectedly for me also without the patches here. I filed bug #58790 for that issue.

Markus Mützel <mmuetzel>
Group administrator
Mon 20 Jul 2020 10:44:56 AM UTC, comment #34: 

The mingw triplets are a little bit misleading when it comes to 32bit or 64bit targets. Basically, "i686-w64-mingw32" corresponds to 32-bit Windows targets, and "x86_64-w64-mingw32" corresponds to 64-bit Windows targets. [1]

Octave started correctly for me, too. I also didn't see it closing immediately during some random testing.
Only when running the complete test suite with "__run_test_suite__", Octave was closing late into it (during the tests for .m files in the sparse directory).

I haven't had time to look into it in more detail. But I think I saw some failing tests related to sparse matrices earlier in the test suite, too. So that error might be caused by something completely unrelated to the changes here. But I haven't tested that yet.

[1]: https://sourceforge.net/p/mingw-w64/wiki2/TypeTriplets/

Markus Mützel <mmuetzel>
Group administrator
Mon 20 Jul 2020 10:30:44 AM UTC, comment #33: 

Okay. With your config switches and your patches, on my Windows 7 Octave-gui runs nicely. I started stream* demos. However the reason why i am asking for is that Octave comes up with the message "i686-w64-mingw32". but presume the execeutable seems to be 32 indeed:


octave-2020-07-20-09-13-w32-installer.exe
file octave-7.0.0.exe
octave-7.0.0.exe: PE32 executable (console) Intel 80386, for MS Windows


whereas yesterday i had:


octave-2020-07-19-14-27-w64-64-installer.exe
file octave-7.0.0.exe
octave-7.0.0.exe: PE32+ executable (console) x86-64, for MS Windows


By the way - on Windows i don't have installed the java environment. But that does not sound to me like it is the root cause in your case ...

Hg200 <hg200>
Sun 19 Jul 2020 10:40:13 PM UTC, comment #32: 

I am using the following switches to configure for a 32bit Windows build (quite close to the buildbots):

./configure --with-pkg-dir=../mxe-octave-pkg --with-ccache --enable-octave=default --enable-binary-packages --enable-qt5 --enable-devel-tools --disable-windows-64 --disable-64 --disable-fortran-int64 --disable-system-opengl


Markus Mützel <mmuetzel>
Group administrator
Sun 19 Jul 2020 04:11:55 PM UTC, comment #31: 

That's nice. Win64 build also finished here. I installed Octave on an old Windows 7 computer and everything looks fine.

How can i build for Win32? Is it enough to just omit all *64 switches on the configure prompt?

Hg200 <hg200>
Sun 19 Jul 2020 10:29:56 AM UTC, comment #30: 

Cross-compiling the 32bit Windows build finished without errors.
However, Octave closes unexpectedly while running the test suite. "gdb" can't catch an error because Octave "just closes" (with error code 3).
I can't reproduce this behavior with a 64bit Windows build which is otherwise the same.

I haven't tested 32bit Windows builds in some time. So, I can't tell if this is caused by these patches or if it already happened without them.

I'll try again with a clean 32bit build without the patches from this bug report.

Markus Mützel <mmuetzel>
Group administrator
Sat 18 Jul 2020 05:52:00 PM UTC, comment #29: 

The attached patch "2_python3_v2.patch" adds a new build rule for python3 instead of updating python2 to python3. It also takes care of explicitly calling scons with python2 in nsis.mk. So the patch "7_nsis_python2.patch" should no longer be necessary.

I noticed that I forgot to update dist-files.mk in the patches that add build rules for ninja and meson. I'm attaching updated patches for those that also update their dependency to python3.

I started a clean build for win32-32 with these changes again and waited until mesa built. I'll report in case it fails later on.

(file #49505, file #49506, file #49507)

Markus Mützel <mmuetzel>
Group administrator
Sat 18 Jul 2020 04:13:17 PM UTC, comment #28: 

Thank you for testing.
Nice catch. I completely forgot to test the nsis-installer target (although it was mentioned earlier in comment #13).
It looks like the scons rules for nsis are incompatible with python3. See also this upstream bug report:
https://sourceforge.net/p/nsis/bugs/1242/
I tried to cherry-pick the patch mentioned there:
https://sourceforge.net/p/nsis/code/7150/
But that doesn't make any difference for me.

Only if I call scons with python2 (that is installed on my system and is no longer part of MXE Octave), compilation completes and an installer is created.

I am attaching a patch. But I don't know if we want to have python2 (>=2.7 according to the nsis docs) as an external dependency of MXE Octave.
Maybe instead, we want to build both python2 and python3 as part of MXE Octave?
I'll try and figure out how to do that.

Markus Mützel <mmuetzel>
Group administrator
Sat 18 Jul 2020 02:38:46 PM UTC, comment #27: 

I applied "1_use_python3_in_scripts, 2_python3, 3_ninja, 4_meson_v2, 5_mesa_20_v2" and started various new (virgin) builds on Fedora for a Win64 target. Python, ninja, meson, mesa, llvm are downloaded and built in the intended versions. but nsis fails for "make nsis-installer 7z-dist".

The error message is weired and I can't figure out the exact reason but if i compare with a reference build (without the patches 1..5) it looks a bit like a "block" is missing and then nothing can be installed because nothing was build and then it stops. Maybe comment #13 kicks in?

Markus, if you are able to build with your patches, please let me know. I will check again.

Hg200 <hg200>
Thu 16 Jul 2020 06:17:12 PM UTC, comment #26: 

The attached patch "4_meson_v2.patch" contains a change that removes any single quotes from the MXE_CC (and others) variables when writing to the toolchain file. I believe that is good enough for now.

"5_mesa_20_v2.patch" no longer installs the opengl32.dll.a static library.

I'll try building again with these patches from a clean tree.

(file #49493, file #49494)

Markus Mützel <mmuetzel>
Group administrator
Mon 13 Jul 2020 08:02:53 AM UTC, comment #25: 

The meson build rule for mesa additionally installs a static library (opengl32.dll.a). The mesa one shadows another one that was previously installed by mingw (libopengl32.a). It looks like the packages need the static library from mingw (for some reason).
After removing that static library, the patch "6_qmake_opengl.patch" is no longer needed.
I'll prepare an updated patch when I find some time to test it.

Additionally, there is an issue with the meson toolchain file: The commands in the file have to be enclosed in a single pair of single quotes. The value in MXE_CC (and others) is sometimes already enclosed in single quotes. Other times it isn't (dependent on the configuration).
It also looks like the strings can't contain any single quotes (or they must be encoded).
Is there a way to make sure the strings with the commands written to the meson toolchain file are enclosed in only one pair of single quote characters? Maybe by removing pairs of single quotes at start and end before adding a single pair when writing to the file?
At the same time (maybe less priority), can we escape single quotes in these strings?

Markus Mützel <mmuetzel>
Group administrator
Sat 11 Jul 2020 01:51:36 PM UTC, comment #24: 

And here the last two patches.

(file #49467, file #49468)

Markus Mützel <mmuetzel>
Group administrator
Sat 11 Jul 2020 01:50:56 PM UTC, comment #23: 

I pushed two patches that update glu and llvm here:
https://hg.octave.org/mxe-octave/rev/005f1b76cbab
https://hg.octave.org/mxe-octave/rev/0e70b5b31362

They don't need any of the other patches here. And they shouldn't be too disruptive (apart from maybe unknown consequences of using LLVM 7 with the JIT compiler in Octave).

The other patches need to be applied at once. So they don't go necessarily in a particular order. Nevertheless, I prepended numbers in the order I had them applied for my testing.

They are mostly clean now. But I still have two doubts:
0. MESON_TOOLCHAIN_FILE is set the same value in build-meson.mk and mesa.mk. Should it instead be set in the root makefile (similar to what is done for the cmake toolchain files)?
0. The projects using qmake can't find the opengl32 library unless explicitly passed to. A similar issue happens already when compiling qtbase (see patch "6_qmake_opengl.patch"). Is there a "toolchain file" or something similar for qmake that could be used to direct the packages to the correct location of the library.
I'm also not sure why this is an issue now. Previously, we installed the library manually and it could be found with no issues. Now the meson build rule does the installing. But the dll ends up in the same location.



(file #49463, file #49464, file #49465, file #49466)

Markus Mützel <mmuetzel>
Group administrator
Sat 11 Jul 2020 09:12:22 AM UTC, comment #22: 

Thanks again for your help. The "mingw-" prefix for patch files is exactly what I was looking for.

I tried to scan for the packages that add mesa as a dependency.
That dependency seems to be conditional on USE_SYSTEM_OPENGL being set to "no" everywhere (except for in glu which itself is only build when USE_SYSTEM_OPENGL is "no"). AFAICT mesa is never built if USE_SYSTEM_OPENGL is "yes".
So it is probably ok to remove the logic around USE_SYSTEM_OPENGL from the mesa.mk file.

In the meantime, the first successful builds completed with mesa 20.0.8 for w64-mingw and native Linux.
I'll clean up the patches and post them here soon.

Markus Mützel <mmuetzel>
Group administrator
Fri 10 Jul 2020 06:04:02 PM UTC, comment #21: 

Im not sure on USE_SYSTEM_OPENGL, it might have been from when we were providing offscreen mesa as well as ability to build the opengl library.

John Donoghue <lostbard>
Group Member
Fri 10 Jul 2020 06:00:42 PM UTC, comment #20: 

You can create a patch named mingw-pkgname-X-XXXXX.patch and it will be applied only when building for mingw/cross

There are a few other patches in there that are mingw only for some other packages.


John Donoghue <lostbard>
Group Member
Thu 09 Jul 2020 08:15:40 PM UTC, comment #19: 

I wanted to write that I could not figure out how to cross build llvm 10.0.

Markus Mützel <mmuetzel>
Group administrator
Thu 09 Jul 2020 07:36:00 PM UTC, comment #18: 

I tried to update LLVM to the latest version (10.0.0) but I could figure out how to get it to cross compile.
So I opted for a smaller update from 6.0.1 to 7.0.0.
Good thing is that the host triplet in that version is correctly set to the cross target for the native version of llvm-config. However, the configure flags are still incorrectly set.
I added a horrible patch that hard codes the settings we have in our make rule into llvm-config.
With that hack, I was able to compile (and finally also link) mesa built with meson.
Is there a way to have a patch only apply for mingw targets?

I'm attaching the two patches.

I still haven't tried a native build. But I hope we'll be able to use the same build rule.

I don't know how to tread USE_SYSTEM_OPENGL. Why do we need mesa if not for the software OpenGL driver?

(file #49454, file #49455)

Markus Mützel <mmuetzel>
Group administrator
Wed 08 Jul 2020 04:30:28 PM UTC, comment #17: 

There ar probally more reported opengl bugs/issues than JIT ones I bet - at least in windows.


John Donoghue <lostbard>
Group Member
Wed 08 Jul 2020 03:00:34 PM UTC, comment #16: 

I see the same as you. In particular:

./usr/x86_64-w64-mingw32/bin/x86_64-w64-mingw32-llvm-config  --system-libs
-lz -lrt -ldl -ltinfo -lpthread -lm -lxml2


Similarly for the shared libraries (which mesa should probably link to anyway):

./usr/x86_64-w64-mingw32/bin/x86_64-w64-mingw32-llvm-config  --system-libs --link-shared
llvm-config: error: libLLVM-6.0.so is missing


That is not what a Windows dynamic library would be named. (It should be looking for something named "LLVM.dll" or similarly.)

Both looks like a bug in llvm to me. IIUC, the native llvm-config that is built when cross-compiling llvm should help other applications that want to be cross-compiled. (But I may have misunderstood.)

Maybe that has been fixed in a newer version of llvm.

If I recall correctly, we keep llvm at the current version (6.1) for the JIT compiler in Octave. But the JIT compiler has limited benefit as it is right now anyway, and no one seems to use it currently (more so in cross-builds).
So, do you think it would be ok to try to update llvm (and maybe breaking the option to compile Octave with JIT compiler in MXE)?

Markus Mützel <mmuetzel>
Group administrator
Tue 07 Jul 2020 03:00:42 PM UTC, comment #15: 

It looks like it is getting the additional libraries it is failing with from llvm-config, which makes sence as its using the native llvm config there rather than the cross one.

./usr/x86_64-w64-mingw32/bin/x86_64-w64-mingw32-llvm-config  --libs --system-libs

-lLLVMLTO -lLLVMPasses -lLLVMObjCARCOpts -lLLVMSymbolize -lLLVMDebugInfoPDB -lLLVMDebugInfoDWARF -lLLVMFuzzMutate -lLLVMTableGen -lLLVMDlltoolDriver -lLLVMLineEditor -lLLVMOrcJIT -lLLVMCoverage -lLLVMMIRParser -lLLVMObjectYAML -lLLVMLibDriver -lLLVMOption -lLLVMWindowsManifest -lLLVMX86Disassembler -lLLVMX86AsmParser -lLLVMX86CodeGen -lLLVMGlobalISel -lLLVMSelectionDAG -lLLVMAsmPrinter -lLLVMDebugInfoCodeView -lLLVMDebugInfoMSF -lLLVMX86Desc -lLLVMMCDisassembler -lLLVMX86Info -lLLVMX86AsmPrinter -lLLVMX86Utils -lLLVMMCJIT -lLLVMInterpreter -lLLVMExecutionEngine -lLLVMRuntimeDyld -lLLVMCodeGen -lLLVMTarget -lLLVMCoroutines -lLLVMipo -lLLVMInstrumentation -lLLVMVectorize -lLLVMScalarOpts -lLLVMLinker -lLLVMIRReader -lLLVMAsmParser -lLLVMInstCombine -lLLVMTransformUtils -lLLVMBitWriter -lLLVMAnalysis -lLLVMProfileData -lLLVMObject -lLLVMMCParser -lLLVMMC -lLLVMBitReader -lLLVMCore -lLLVMBinaryFormat -lLLVMSupport -lLLVMDemangle
-lz -lrt -ldl -ltinfo -lpthread -lm -lxml2


Im guessing the cross built llv-config is an exe file so wouldn't work in the detect, so would need to come up with a modified way of getting the libraries etc of llvm

John Donoghue <lostbard>
Group Member
Tue 07 Jul 2020 02:49:17 PM UTC, comment #14: 

build-setuptools woked after i did a clean build and I see the same fail in mesa as you are

John Donoghue <lostbard>
Group Member
Mon 06 Jul 2020 12:56:18 PM UTC, comment #13: 

I misunderstood. It isn't scons itself that requires python 2.7. But it is the scons buildrule for mesa that requires that specific version:
https://docs.mesa3d.org/install.html#third-party-extra-tools

At least it looks like there is a chance that scons will still work for building nsis after a python3 update.

Markus Mützel <mmuetzel>
Group administrator
Mon 06 Jul 2020 10:13:01 AM UTC, comment #12: 

Comment #10 - its quite possible it is (on your or my system) pulling in something that not from the mxe tree. I'll look some more when I get the chance.

A quick look at latest version of scons says it needs python 3.5+ so it may be ok with an update

John Donoghue <lostbard>
Group Member
Mon 06 Jul 2020 07:25:00 AM UTC, comment #11: 

Would a better approach be to compile both python2 and python3?
Would it be possible to select to use one of those versions per build rule?

Markus Mützel <mmuetzel>
Group administrator
Mon 06 Jul 2020 07:21:47 AM UTC, comment #10: 

Interesting. I tried again with:

make clean
./bootstrap
./configure --enable-devel-tools --enable-windows-64 --enable-octave=default --enable-binary-packages --with-ccache --with-pkg-dir=../mxe-octave-pkg
make all 7z-dist JOBS=3


It builds for me.
I don't know how python modules work. Could it be that it finds modules that are installed for my system's python (not the one in MXE Octave)?

Markus Mützel <mmuetzel>
Group administrator
Sun 05 Jul 2020 08:55:41 PM UTC, comment #9: 

I dont even get that that far with the current patches, failing a build-setuptools:



Traceback (most recent call last):
  File "setup.py", line 9, in <module>
    import setuptools
  File "/home/jade/mxe-octave/build-64/tmp-build-setuptools/setuptools-49.1.0/setuptools/__init__.py", line 23, in <module>
    from setuptools.dist import Distribution
  File "/home/jade/mxe-octave/build-64/tmp-build-setuptools/setuptools-49.1.0/setuptools/dist.py", line 34, in <module>
    from setuptools import windows_support
  File "/home/jade/mxe-octave/build-64/tmp-build-setuptools/setuptools-49.1.0/setuptools/windows_support.py", line 2, in <module>
    import ctypes
  File "/home/jade/mxe-octave/build-64/usr/lib/python3.8/ctypes/__init__.py", line 7, in <module>
    from _ctypes import Union, Structure, Array
ModuleNotFoundError: No module named '_ctypes'


John Donoghue <lostbard>
Group Member
Sun 05 Jul 2020 08:28:48 PM UTC, comment #8: 

Scons is already required for nsis.

John Donoghue <lostbard>
Group Member
Sun 05 Jul 2020 07:24:31 PM UTC, comment #7: 

Could only attach four file at a time. So here the patch that changes the cross build rule for mesa (mesa_meson.patch).
I haven't touched the native build rule yet.

(file #49436)

Markus Mützel <mmuetzel>
Group administrator
Sun 05 Jul 2020 07:22:23 PM UTC, comment #6: 

I tried to update build-python to python3 (python3.patch).
With that, ninja and meson compiled fine (build-ninja_v2.patch and build-meson_v2.patch).

Unfortunately, scons needs python 2.7 and is incompatible with python3. Thus, the native build rule for mesa broke.
I figured that this wouldn't be too bad after all because we wanted to change to meson anyway. (Primarily as the native build tool. But using the same build tool for cross and native builds would probably not hurt imho.)

It turns out meson needs llvm-config to be able to detect llvm (llvm_config.patch).

But I am still stuck at trying to cross-compile mesa with meson.
The build seems to work ok. But at the last step, it fails with the following linker error:

[921/921] Linking target src/gallium/targets/libgl-gdi/opengl32.dll
FAILED: src/gallium/targets/libgl-gdi/opengl32.dll
/home/osboxes/Documents/Repositories/Octave/mxe-octave/usr/bin/x86_64-w64-mingw32-g++  -o src/gallium/targets/libgl-gdi/opengl32.dll 'src/gallium/targets/libgl-gdi/c96d1e6@@opengl32@sha/libgl_gdi.c.obj' -Wl,--allow-shlib-undefined -shared ../src/gallium/targets/libgl-gdi/../../state_trackers/wgl/opengl32.def -Wl,--start-group -Wl,--out-implib=src/gallium/targets/libgl-gdi/opengl32.dll.a -Wl,--whole-archive src/gallium/state_trackers/wgl/libwgl.a -Wl,--no-whole-archive -Wl,--nxcompat -Wl,--dynamicbase src/gallium/auxiliary/libgallium.a src/compiler/nir/libnir.a src/compiler/libcompiler.a src/util/libmesa_util.a src/util/format/libmesa_format.a src/compiler/glsl/libglsl.a src/compiler/glsl/glcpp/libglcpp.a src/mesa/libmesa_gallium.a src/mesa/libmesa_common.a src/mesa/libmesa_sse41.a src/gallium/winsys/sw/gdi/libwsgdi.a src/mapi/glapi/libglapi_static.a src/gallium/drivers/llvmpipe/libllvmpipe.a src/gallium/drivers/softpipe/libsoftpipe.a -lws2_32 /home/osboxes/Documents/Repositories/Octave/mxe-octave/usr/x86_64-w64-mingw32/lib/libz.dll.a -pthread -lm -lrt -ldl -lpthread -lLLVMCoroutines -lLLVMipo -lLLVMInstrumentation -lLLVMVectorize -lLLVMLinker -lLLVMIRReader -lLLVMAsmParser -lLLVMX86Disassembler -lLLVMX86AsmParser -lLLVMX86CodeGen -lLLVMGlobalISel -lLLVMSelectionDAG -lLLVMAsmPrinter -lLLVMDebugInfoCodeView -lLLVMDebugInfoMSF -lLLVMCodeGen -lLLVMScalarOpts -lLLVMInstCombine -lLLVMTransformUtils -lLLVMBitWriter -lLLVMX86Desc -lLLVMMCDisassembler -lLLVMX86Info -lLLVMX86AsmPrinter -lLLVMX86Utils -lLLVMMCJIT -lLLVMExecutionEngine -lLLVMTarget -lLLVMAnalysis -lLLVMProfileData -lLLVMRuntimeDyld -lLLVMObject -lLLVMMCParser -lLLVMBitReader -lLLVMMC -lLLVMCore -lLLVMBinaryFormat -lLLVMSupport -lLLVMDemangle -lz -ltinfo -lxml2 -lLLVMCoroutines -lLLVMipo -lLLVMInstrumentation -lLLVMVectorize -lLLVMLinker -lLLVMIRReader -lLLVMAsmParser -lLLVMX86Disassembler -lLLVMX86AsmParser -lLLVMX86CodeGen -lLLVMGlobalISel -lLLVMSelectionDAG -lLLVMAsmPrinter -lLLVMDebugInfoCodeView -lLLVMDebugInfoMSF -lLLVMCodeGen -lLLVMScalarOpts -lLLVMInstCombine -lLLVMTransformUtils -lLLVMBitWriter -lLLVMX86Desc -lLLVMMCDisassembler -lLLVMX86Info -lLLVMX86AsmPrinter -lLLVMX86Utils -lLLVMMCJIT -lLLVMExecutionEngine -lLLVMTarget -lLLVMAnalysis -lLLVMProfileData -lLLVMRuntimeDyld -lLLVMObject -lLLVMMCParser -lLLVMBitReader -lLLVMMC -lLLVMCore -lLLVMBinaryFormat -lLLVMSupport -lLLVMDemangle -lz -ltinfo -lxml2 -lLLVMCoroutines -lLLVMipo -lLLVMInstrumentation -lLLVMVectorize -lLLVMLinker -lLLVMIRReader -lLLVMAsmParser -lLLVMX86Disassembler -lLLVMX86AsmParser -lLLVMX86CodeGen -lLLVMGlobalISel -lLLVMSelectionDAG -lLLVMAsmPrinter -lLLVMDebugInfoCodeView -lLLVMDebugInfoMSF -lLLVMCodeGen -lLLVMScalarOpts -lLLVMInstCombine -lLLVMTransformUtils -lLLVMBitWriter -lLLVMX86Desc -lLLVMMCDisassembler -lLLVMX86Info -lLLVMX86AsmPrinter -lLLVMX86Utils -lLLVMMCJIT -lLLVMExecutionEngine -lLLVMTarget -lLLVMAnalysis -lLLVMProfileData -lLLVMRuntimeDyld -lLLVMObject -lLLVMMCParser -lLLVMBitReader -lLLVMMC -lLLVMCore -lLLVMBinaryFormat -lLLVMSupport -lLLVMDemangle -lz -ltinfo -lxml2 -lkernel32 -luser32 -lgdi32 -lwinspool -lshell32 -lole32 -loleaut32 -luuid -lcomdlg32 -ladvapi32 -Wl,--end-group
/home/osboxes/Documents/Repositories/Octave/mxe-octave/usr/bin/x86_64-w64-mingw32-ld: cannot find -lrt
/home/osboxes/Documents/Repositories/Octave/mxe-octave/usr/bin/x86_64-w64-mingw32-ld: cannot find -ldl
/home/osboxes/Documents/Repositories/Octave/mxe-octave/usr/bin/x86_64-w64-mingw32-ld: cannot find -ltinfo
/home/osboxes/Documents/Repositories/Octave/mxe-octave/usr/bin/x86_64-w64-mingw32-ld: cannot find -lxml2
/home/osboxes/Documents/Repositories/Octave/mxe-octave/usr/bin/x86_64-w64-mingw32-ld: cannot find -ltinfo
/home/osboxes/Documents/Repositories/Octave/mxe-octave/usr/bin/x86_64-w64-mingw32-ld: cannot find -lxml2
/home/osboxes/Documents/Repositories/Octave/mxe-octave/usr/bin/x86_64-w64-mingw32-ld: cannot find -ltinfo
/home/osboxes/Documents/Repositories/Octave/mxe-octave/usr/bin/x86_64-w64-mingw32-ld: cannot find -lxml2
collect2: error: ld returned 1 exit status
ninja: build stopped: subcommand failed.
make[2]: *** [/home/osboxes/Documents/Repositories/Octave/mxe-octave/Makefile:941: build-only-mesa] Error 1
make[2]: Leaving directory '/home/osboxes/Documents/Repositories/Octave/mxe-octave'


These libraries shouldn't be linked in for a win32 library. (At least dl, rt, and tinfo wouldn't work pretty certainly. Not so sure about xml2.)
I don't know where these flags come from. Any help would be very much appreciated.

(file #49432, file #49433, file #49434, file #49435)

Markus Mützel <mmuetzel>
Group administrator
Sun 05 Jul 2020 10:12:14 AM UTC, comment #5: 

Thank you again for the update.
I'm not sure if this won't cause some issues though.
Other packages might chose to encode their files in UTF-8 (which might arguably be a better choice).
Reading such files in Latin-1 might result in mojibake for non-ASCII characters. That might not be a problem per se. But writing these strings back to a file (without specifying the encoding) encodes that mojibake in UTF-8 and results in a wrong file.
It's maybe good enough to read the files in Latin-1 and treat the read strings more like "byte streams". When writing those "byte streams" back to files, the encoding could be set to Latin-1 as well. That probably ensures that what we write back to the files is what we read from the source (in a byte-wise manner).

The attached updated patch does that.

A "real solution" might be to read and write byte streams from and to the files. But that would probably require a few other changes. And I'm not sure it is worth the effort...

I'll try updating build-python.mk to python3 next and see if that breaks anything.

(file #49431)

Markus Mützel <mmuetzel>
Group administrator
Sat 04 Jul 2020 04:17:53 PM UTC, comment #4: 

Updated patch

(file #49429)

John Donoghue <lostbard>
Group Member
Sat 04 Jul 2020 09:00:11 AM UTC, comment #3: 

Thank you very much for your help.

I applied your patch on top of hg id e60d29a560ab and tried to test by building for Windows including binary packages.
The compilation failed at of-tsa with the following error in the build log:

Creating... PKG_ADD
cleaning up
Traceback (most recent call last):
  File "/home/osboxes/Documents/Repositories/Octave/mxe-octave/tools/pkg-install.py", line 618, in <module>
    pkg(sys.argv[1:])
  File "/home/osboxes/Documents/Repositories/Octave/mxe-octave/tools/pkg-install.py", line 609, in pkg
    install_pkg(a, env)
  File "/home/osboxes/Documents/Repositories/Octave/mxe-octave/tools/pkg-install.py", line 375, in install_pkg
    create_pkgadddel (env, packdir, "PKG_ADD");
  File "/home/osboxes/Documents/Repositories/Octave/mxe-octave/tools/pkg-install.py", line 133, in create_pkgadddel
    for a in extract_pkg(f, '^[#%][#%]* *' + nm + ': *(.*)$'):
  File "/home/osboxes/Documents/Repositories/Octave/mxe-octave/tools/pkg-install.py", line 56, in extract_pkg
    lines = f.read().splitlines()
  File "/usr/lib/python3.8/codecs.py", line 322, in decode
    (result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb2 in position 1054: invalid start byte
make[1]: *** [/home/osboxes/Documents/Repositories/Octave/mxe-octave/Makefile:941: build-only-of-tsa] Error 1
make[1]: Leaving directory '/home/osboxes/Documents/Repositories/Octave/mxe-octave'



This is probably because the file sinvest2.m is encoded in Latin1 and contains the character "²" (superscript 2).

Should that be considered a bug in of-tsa (wrong encoding, should use UTF-8)? Or should we change the script pkg-install.py to take whatever encoding it is fed?

Markus Mützel <mmuetzel>
Group administrator
Fri 03 Jul 2020 03:48:33 PM UTC, comment #2: 

Patch for update of scripts to py3 (assuming a python3 is available)




(file #49419)

John Donoghue <lostbard>
Group Member
Fri 03 Jul 2020 11:27:00 AM UTC, comment #1: 

Several of the scripts we use in the tools directory would need be updated, but dont think would have to change much.


John Donoghue <lostbard>
Group Member
Thu 02 Jul 2020 04:05:38 PM UTC, original submission:  

In MXE Octave, we use autotools to build mesa on Linux natively.
Mesa has dropped support for autotools in later versions. So we are stuck at version 19.0.8.

I tried to add build rules for meson with the goal to switch to meson for building mesa.

Meson depends on ninja.

I was able to make some changes to build ninja (see attached patch build-ninja.patch).

But building meson (see attached patch build-meson.patch) fails early on with:

ERROR: Tried to install Meson with an unsupported Python version:
2.7.15 (default, Jun 11 2020, 17:37:14)
[GCC 9.3.0]
Meson requires Python 3.5.2 or greater


Is it possible to update python to python3? Or would that break a lot?

Even if building meson should succeed, I have not yet tried to build mesa with meson.

Needless to say that all of this is very preliminary.

Any help would be appreciated.

Markus Mützel <mmuetzel>
Group administrator

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

Attached Files
file #49505:  2_python3_v2.patch added by mmuetzel (6KiB - application/octet-stream)
file #49506:  3_ninja_v2.patch added by mmuetzel (2KiB - application/octet-stream)
file #49507:  4_meson_v3.patch added by mmuetzel (3KiB - application/octet-stream)
file #49493:  4_meson_v2.patch added by mmuetzel (3KiB - application/octet-stream)
file #49494:  5_mesa_20_v2.patch added by mmuetzel (8KiB - application/octet-stream)
file #49467:  5_mesa_20.patch added by mmuetzel (8KiB - application/octet-stream)
file #49468:  6_qmake_opengl.patch added by mmuetzel (2KiB - application/octet-stream)
file #49463:  1_use_python3_in_scripts.patch added by mmuetzel (13KiB - application/octet-stream)
file #49464:  2_python3.patch added by mmuetzel (3KiB - application/octet-stream)
file #49465:  3_ninja.patch added by mmuetzel (2KiB - application/octet-stream)
file #49466:  4_meson.patch added by mmuetzel (3KiB - application/octet-stream)
file #49454:  llvm.7.0.0.patch added by mmuetzel (5KiB - application/octet-stream)
file #49455:  mesa_meson_v2.patch added by mmuetzel (6KiB - application/octet-stream)
file #49436:  mesa_meson.patch added by mmuetzel (5KiB - application/octet-stream)
file #49432:  python3.patch added by mmuetzel (3KiB - application/octet-stream)
file #49433:  build-ninja_v2.patch added by mmuetzel (2KiB - application/octet-stream)
file #49434:  build-meson_v2.patch added by mmuetzel (3KiB - application/octet-stream)
file #49435:  llvm_config.patch added by mmuetzel (1KiB - application/octet-stream)
file #49431:  py3scripts_v3.patch added by mmuetzel (13KiB - application/octet-stream)
file #49429:  py3scripts_v2.patch added by lostbard (12KiB - application/octet-stream)
file #49419:  py3scripts.patch added by lostbard (11KiB - application/octet-stream)
file #49410:  build-ninja.patch added by mmuetzel (2KiB - application/octet-stream)
file #49411:  build-meson.patch added by mmuetzel (2KiB - application/octet-stream)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by hg200 (Posted a comment)
  • -email is unavailable- added by lostbard (Posted a comment)
  • -email is unavailable- added by mmuetzel (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only group members can vote.

     

    Follow 25 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2020-08-01 mmuetzel StatusReady For Test Fixed
        Open/ClosedOpen Closed
    2020-07-29 mmuetzel StatusIn Progress Ready For Test
    2020-07-27 mmuetzel StatusPatch Submitted In Progress
    2020-07-18 mmuetzel Attached File- Added 2_python3_v2.patch, #49505
        Attached File- Added 3_ninja_v2.patch, #49506
        Attached File- Added 4_meson_v3.patch, #49507
    2020-07-16 mmuetzel Attached File- Added 4_meson_v2.patch, #49493
        Attached File- Added 5_mesa_20_v2.patch, #49494
        StatusNeed Info Patch Submitted
    2020-07-11 mmuetzel Attached File- Added 5_mesa_20.patch, #49467
        Attached File- Added 6_qmake_opengl.patch, #49468
    2020-07-11 mmuetzel Attached File- Added 1_use_python3_in_scripts.patch, #49463
        Attached File- Added 2_python3.patch, #49464
        Attached File- Added 3_ninja.patch, #49465
        Attached File- Added 4_meson.patch, #49466
    2020-07-09 mmuetzel Attached File- Added llvm.7.0.0.patch, #49454
        Attached File- Added mesa_meson_v2.patch, #49455
        StatusIn Progress Need Info
    2020-07-05 mmuetzel Attached File- Added mesa_meson.patch, #49436
    2020-07-05 mmuetzel Attached File- Added python3.patch, #49432
        Attached File- Added build-ninja_v2.patch, #49433
        Attached File- Added build-meson_v2.patch, #49434
        Attached File- Added llvm_config.patch, #49435
    2020-07-05 mmuetzel Attached File- Added py3scripts_v3.patch, #49431

    Back to the top

    Powered by Savane 3.13-f8d8.
    Corresponding source code