mainGNU Libtool - Support: sr #110901, libtool hangs indefinitely on...

 
 

sr #110901: libtool hangs indefinitely on windows when used in msys due to cmd.exe call bug

Submitter:  Mitch <mitchc>
Submitted:  Wed 19 Jul 2023 07:23:17 AM UTC
   
 
Category:  None Priority:  5 - Normal
Severity:  4 - Important Status:  In Progress
Privacy:  Public Assigned to:  None
Open/Closed:  Open Operating System:  Microsoft Windows
* Mandatory Fields

Add a New Comment Rich Markup
   

Jump to the original submission

Sat 31 May 2025 12:58:24 PM UTC, comment #14: 

comment #13:

> > This sounds fine to me. Could you write a patch for this, or specify the ENV variable and parts of path conversion that you would like bypassed?
>
>
> I updated
> https://github.com/mitchcapper/libtool/compare/add_bypass_path_conversion_opt

Thank you for the updated patch, but this will not have the desired behaviour. Additional options are currently being passed to func_convert_core_msys_to_w32 for cygpath, so if LIBTOOL_MSYS_NO_PATH_CONVERSION=1, func_convert_core_msys_to_w32_result would equal "-w" instead of the path.

> and have attached the patch too.  In short setting LIBTOOL_MSYS_NO_PATH_CONVERSION=1 completely bypasses all conversion.  Obviously if there is a better name by all means.

I like the idea of adding a path conversion bypass configuration option to libtool more than relying on an ENV variable. It could be beneficial for other windows path conversions if it is designed like aix-soname. While it is annoying to have to modify a source package before building, I think making a small edit to configure.ac for LT_INIT is okay.

> One additional option for even general improvement could be to simply check for the presence of a slash before running the conversion.

I would like to have a check cached for "cmd //c" or "cmd /c", but I am not sure about how best to test this. It may only lead to libtool hanging much sooner...

Ileana Dumitrescu <ildumi>
Group administrator
Fri 30 May 2025 06:15:39 PM UTC, comment #13: 


> This sounds fine to me. Could you write a patch for this, or specify the ENV variable and parts of path conversion that you would like bypassed?



I updated
https://github.com/mitchcapper/libtool/compare/add_bypass_path_conversion_opt

and have attached the patch too.  In short setting LIBTOOL_MSYS_NO_PATH_CONVERSION=1 completely bypasses all conversion.  Obviously if there is a better name by all means.

One additional option for even general improvement could be to simply check for the presence of a slash before running the conversion.

Mitch <mitchc>
Fri 30 May 2025 05:47:37 PM UTC, comment #12: 

comment #10:

> Looks like MSys is still in (very inactive) use.
> The project page has two years old activity.

Could you link the project page here?

> I wouldn't drop support for now, as it is easy to check for 'cygpath' availability and use it, if it is present.

I will work on a patch for this or something similar.

> Not directly related: libtool could be inspected for other path conversions as cygpath is much faster than conversion via cmd.

I agree that this would be good to look at improving. I am not sure if I could complete this before the next release, but I will look into it.


comment #11:

> I would still highly recommend adding a bypass ENV var as its fairly straightforward to just use posix style compat windows paths. Giving the user (or software) the ability to turn it off would seem beneficial.  

This sounds fine to me. Could you write a patch for this, or specify the ENV variable and parts of path conversion that you would like bypassed?

> Libtool is quite slow on windows in many cases. A large part of this on windows due to all the process spawning.   While I don't mind manually modifying the libtool script to strip this function out of automated builds there is a bit of a dance to make sure it doesn't get restored over.

Testing on windows for libtool is quite slow, but hopefully I can improve that soon.

Ileana Dumitrescu <ildumi>
Group administrator
Fri 30 May 2025 01:39:52 AM UTC, comment #11: 

I would still highly recommend adding a bypass ENV var as its fairly straightforward to just use posix style compat windows paths. Giving the user (or software) the ability to turn it off would seem beneficial.  

Libtool is quite slow on windows in many cases. A large part of this on windows due to all the process spawning.   While I don't mind manually modifying the libtool script to strip this function out of automated builds there is a bit of a dance to make sure it doesn't get restored over.

Mitch <mitchc>
Thu 29 May 2025 07:25:14 PM UTC, comment #10: 

Looks like MSys is still in (very inactive) use.
The project page has two years old activity.
Some large projects, like libcurl, are still testing for the original MSys/MinGW.
I wouldn't drop support for now, as it is easy to check for 'cygpath' availability and use it, if it is present.

Not directly related: libtool could be inspected for other path conversions as cygpath is much faster than conversion via cmd.

Evgeny Grin <karlson2k>
Thu 29 May 2025 06:40:14 PM UTC, comment #9: 

comment #8:

> комментарий №7:
> > I have applied a patch for this issue on the development branch [1], which should migrate to master after more testing. If this does not seem to fix it, I will reopen.
> >
> > [1]https://cgit.git.savannah.gnu.org/cgit/libtool.git/commit/?h=development&id=1377b68b4aed0d6ee3c390131eb763d26e5b642d
>
> This made libtool incompatible with original MSys version, which does not have cygpath.
> MSys2 has cygpath and with MSys2 it should work fine.


Thank you for the correction. Is the original MSYS used still? From what I have read, it has been a dead project for many years, so it should be removed from newer versions of libtool. If this migrates to master, I will update the NEWS to say support is being dropped for MSYS but that MSYS2 is still supported.

Ileana Dumitrescu <ildumi>
Group administrator
Thu 29 May 2025 05:37:34 PM UTC, comment #8: 

комментарий №7:

> I have applied a patch for this issue on the development branch [1], which should migrate to master after more testing. If this does not seem to fix it, I will reopen.
>
> [1]https://cgit.git.savannah.gnu.org/cgit/libtool.git/commit/?h=development&id=1377b68b4aed0d6ee3c390131eb763d26e5b642d


This made libtool incompatible with original MSys version, which does not have cygpath.
MSys2 has cygpath and with MSys2 it should work fine.

Evgeny Grin <karlson2k>
Thu 29 May 2025 05:10:09 PM UTC, comment #7: 

I have applied a patch for this issue on the development branch [1], which should migrate to master after more testing. If this does not seem to fix it, I will reopen.

[1]https://cgit.git.savannah.gnu.org/cgit/libtool.git/commit/?h=development&id=1377b68b4aed0d6ee3c390131eb763d26e5b642d

Ileana Dumitrescu <ildumi>
Group administrator
Mon 02 Dec 2024 09:34:54 PM UTC, comment #6: 

Sorry, I forgot to add one way to slightly increase performance as this does seem msys specific would be to call cygpath as we do elsewhere.  That would reduce it from 8 to 2 processes spawned every time I believe.

Mitch <mitchc>
Mon 02 Dec 2024 09:32:38 PM UTC, comment #5: 

My original quick-fix suggestion was to simply unset MSYS_ARG_CONV_EXCL in libtool so the hack inplace would work. 

Later for my own GNU builds I started to just add a flag that controlled of the cmd.exe call should be as it is "//c"  or without the MSYS hack, and only a single slash.

Honestly though these are both poor as this function really leads to horrible performance in windows.


Instead I would recommend something like:
https://github.com/mitchcapper/libtool/compare/add_bypass_path_conversion_opt

Yes it adds a new ENV var so nothing benefits from the fix initially.  In fact due to the nature of the bug no project will likely implement the flag in their build process (unless they used the forward slash notation with relative which would ensure msys escaping would not happen.

Instead this would be a user level flag that they would set similar to how they set MSYS2_ARG_CONV_EXCL to control escaping.

While it would potentially make more sense for it to be a libtool command line arg the problem there is the user would likely need to edit the build /configure scripts manually to do so which would be a big pain.

Now this straight up bypasses the conversion rather than using the singular slash as I originally did with a flag. The reason being for properly configured filenames no translation is needed and the performance hit with even the correctly working cmd.exe call is pretty huge.  Windows has a "high" cost for process spawning and that one command that is called on every path spawns many processes.  Part of this is sh/bash.exe on windows spawns an additional copy of itself every time it goes to launch a process.  In short a single file conversion call causes:
A 'forked' sh.exe to then run'env.exe bash /usr/bin/cmd /c echo .libs/'.  That spawns env.exe, which spawns bash.exe.   Thankfully echo is a builtin of cmd.exe so no spawn there.   Then we use sed to trim the result so that is another copy of sh.exe that then calls sed.exe.

All in all to convert ".libs/" to the proper path you are talking about 8 processes started and terminated in series.

So bypassing any spawns is a huge performance increase.

If we don't want to do this sort of bypass option we should still give users some way to avoid the double slash hack without them resorting to editing libtool or the build commands itself.  Heck most users probably just give up instead when it freezes not knowing the issue.

One change we could do would be to be smarter about if we need to use the hack at all.  Relative paths, for example, I don't think ever need to be converted.  One could replace any backslashes with forward slashes as just about anything windows should understand forward slashed paths.

Another option would be to potentially detect if cmd.exe needs a single slash or double slash to be past in the configure script or just at the very start of a libtool spawn.  That won't help performance but would prevent hangs.

Mitch <mitchc>
Tue 20 Aug 2024 02:16:41 PM UTC, comment #4: 

This is a similar report to both of the following:
  https://debbugs.gnu.org/cgi/bugreport.cgi?bug=49246
  https://debbugs.gnu.org/cgi/bugreport.cgi?bug=10949

Maybe an update to libtool's documentation should be made, since this has been recurring? A minor revision of the quote below should be helpful in doc/notes.texi:

comment #2:

> When starting any program under MSYS (and MSYS2), the MSYS[2] checks whether the lunched program is linked with msys-*.dll.
> If program is linked with this DLL then the program expects POSIX-like environment so no path translation is performed.
> If program is not linked with this DLL then the program is native Windows binary, which expects native Windows paths. In such case the MSYS[2] scans all arguments for POSIX-like paths and converts all of them to Windows native paths. The problem is that all Windows drives are mounted as single letter in the root so the path like "/d/path/file" should be translated into "D:/path/file", while "/c" is translated into "C:/"
> To prevent this translation double slash could be used. Double slash is replaced with single slash.


An update to documentation and/or libtool could also be made for the variable MSYS2_ARG_CONV_EXCL, but I am not sure...

Ileana Dumitrescu <ildumi>
Group administrator
Fri 28 Jul 2023 10:59:38 PM UTC, comment #3: 

OK! Sorry for not investigating further, once I found the mailing list thread that seemed like it terminated with the proper fix (and verifying it fixed it for me) I stopped debugging the issue.

It is now clear.  I think this relates to the MSYS2 variable MSYS2_ARG_CONV_EXCL  which controls what is or is not escaped.  By default msys tries escaping any and all command line args with slashes thus leading do the cmd //c fix in use.

Of course once you start trying to do a large variety of program calls MSYS2 escaping causes more errors than fixes.  MSYS2 resolves this by using MSYS2_ARG_CONV_EXCL to specify what patterns should not be escaped, but if you are like me (and I am guessing the person who originally posted to the ML) you might just do MSYS2_ARG_CONV_EXCL="*"  to turn of escaping all together.  Now there is the problem that libtool has those built in hacks to get around the msys2 limitations but msys2 won't be escaping it leading to the literal //c being passed to cmd.exe causing this hang.

The best solution here may be for libtool to unset MSYS2_ARG_CONV_EXCL prior  to executing a cmd shell thus guaranteeing the brute escaping it expects.

Personally I think getting to the other position (forcing all escaping off) is a better long term solution as you don't have to worry about any potential funky modification when making native calls.   MSYS2 and native Windows do share a path format that works for both in nearly all situations and that is:
c:/test/my/file.txt   virtually all msys2 and windows executables accept that without issue and there are no backslashes to worry about fixing.  It has the side effect that msys also won't try to do any path conversion on the way out.   There are a few limitations to this:
*) You need to make sure your paths are in that format, if you don't know if something is a path or not that may not be doable
*) If you are trying to pass a msys2 filesystem file to a native command the prefixing has to be done manually (ie something that is in the msys2 root like /etc/myconfig.conf needs translating to c:/msys64/etc/myconfig.conf ).


That may be a pipe dream not worth considering, and the first solution of setting the ENV var before the call may be the best way to go.

I will say I have been compiling a massive amount of foss software (https://github.com/mitchcapper/WIN64LinuxBuild) as close to native windows as possible from inside msys2 and despite having the escape fixes turned off this is the first time I have run into this problem.  Prior to disabling the automatic conversion though there were a good bit of issues.

Mitch <mitchc>
Tue 25 Jul 2023 04:21:14 PM UTC, comment #2: 

When starting any program under MSYS (and MSYS2), the MSYS[2] checks whether the lunched program is linked with msys-*.dll.
If program is linked with this DLL then the program expects POSIX-like environment so no path translation is performed.
If program is not linked with this DLL then the program is native Windows binary, which expects native Windows paths. In such case the MSYS[2] scans all arguments for POSIX-like paths and converts all of them to Windows native paths. The problem is that all Windows drives are mounted as single letter in the root so the path like "/d/path/file" should be translated into "D:/path/file", while "/c" is translated into "C:/"
To prevent this translation double slash could be used. Double slash is replaced with single slash.

Examples:

------------------------
MSYS ~
$ cmd /c echo "/c"
Microsoft Windows [Version 10.0.19045.3208]
(c) Microsoft Corporation. All rights reserved.

C:\msys64\home\user>exit

MSYS ~
$ cmd //c echo "/c"
C:/

MSYS ~
$ cmd //c echo "//c"
/c

MSYS ~
$ cmd //c echo "///c"
//c

MSYS ~
$ echo.exe "/c"
/c

MSYS ~
$ echo.exe "//c"
//c
------------------------
"echo.exe" is Msys program. cmd is not Msys program.

Evgeny Grin <karlson2k>
Tue 25 Jul 2023 04:05:07 PM UTC, comment #1: 
Evgeny Grin <karlson2k>
Wed 19 Jul 2023 07:23:17 AM UTC, original submission:  

Seems the bug was reported with fix a few years ago but sadly did not get applied:

https://lists.nongnu.org/archive/html/libtool/2021-06/msg00001.html

Mitch <mitchc>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by ildumi (Posted a comment)
  • -email is unavailable- added by karlson2k (Posted a comment)
  • -email is unavailable- added by mitchc (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only logged-in users can vote.

     

    Follow 5 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2025-05-30 mitchc Attached File- Added libtool-37ce24a-libtool@Add@msys@bypass@path@conversion@env@var@LIBTOOL_MSYS_NO_PATH_CONVERSION.patch, #57261
    2025-05-30 ildumi StatusDone In Progress
        Open/ClosedClosed Open
    2025-05-29 ildumi StatusNone Done
        Open/ClosedOpen Closed

    Back to the top

    Powered by Savane 3.15-e283.
    Corresponding source code