bugGNU Octave - Bugs: bug #49839, [mxe-octave] perl 5.8 utf8 module...

 
 

bug #49839: [mxe-octave] perl 5.8 utf8 module case-folding bug caused by installing symbolic-2.4.0 package

Submitter:  Jihang <neur1n>
Submitted:  Wed 14 Dec 2016 12:16:04 AM UTC
   
 
Category:  Configuration and Build System Severity:  3 - Normal
Priority:  3 - Low Item Group:  Incorrect Result
Status:  Fixed Assigned to:  None
Originator Name:  Open/Closed:  * Closed
Release:  * dev Operating System:  * Microsoft Windows
Fixed Release:  None Planned Release:  None
* Mandatory Fields

Add a New Comment Rich Markup
   

Jump to the original submission

Thu 23 Aug 2018 05:27:32 PM UTC, comment #25: 

Its been set as the default for windows now, so I would imagine will be in 5.0 unless something major comes up

John Donoghue <lostbard>
Group Member
Thu 23 Aug 2018 05:08:39 PM UTC, comment #24: 

Excellent.  Is the msys2 "done" or just proposed?  Is it for 5.0?

Colin Macdonald <cbm>
Thu 23 Aug 2018 05:03:14 PM UTC, comment #23: 

And using dev octave with the currently mssy2:

>> s = _makeinfo_( "@documentencoding UTF-8\nX @var{x}")

s = X X

>>



John Donoghue <lostbard>
Group Member
Thu 23 Aug 2018 04:58:49 PM UTC, comment #22: 

Current msys2-perl being now installed is  5.24.4

John Donoghue <lostbard>
Group Member
Thu 23 Aug 2018 04:55:15 PM UTC, comment #21: 

Re: the minimal test case, maybe my patch at https://savannah.gnu.org/patch/?9464 would help make that automatically testable?  The patch allows one get to get codes back from `__makeinfo__`.  I'd appreciate a review.

Did I see talk of moving to msys2 on the devlist?  From below, I gather that would include an up-to-date perl.

Colin Macdonald <cbm>
Sun 05 Aug 2018 03:25:52 PM UTC, comment #20: 

texinfo was updated to 6.5 in the meantime and I now see the following line in the log file:

checking whether we can build Perl extension (XS) modules... yes


However, I still get the perl error with a recent MXE cross-build on Windows:

>> s = __makeinfo__( "@documentencoding UTF-8\nX @var{x}")
Use of uninitialized value in lc at /usr/lib/perl5/5.8/utf8_heavy.pl line 123.
s = X X


Markus Mützel <mmuetzel>
Group administrator
Tue 16 May 2017 04:22:52 PM UTC, comment #19: 

Ok, I hadn't updated mxe-octave in a while, I see that now.

From the texinfo log file:

> checking whether we can build Perl extension (XS) modules... no


Yeah that's a bad thing that is allowing this bug to be hit, and also making makeinfo slower than it needs to be for Windows users every time they run a help command.

Mike Miller <mtmiller>
Group Member
Mon 15 May 2017 01:14:05 PM UTC, comment #18: 

Also, just for info, mxe-octave uses texinfo 6.3, however I guess doesn't not beuild any of the extensions in C

John Donoghue <lostbard>
Group Member
Fri 12 May 2017 09:41:48 PM UTC, comment #17: 

Slight optimization, this is now the shortest possible example I can come up with: "@documentencoding UTF-8\nX @var{x}". The @documentencoding is required, and it seems to require use of the @var macro, but it has to have some normal text as well.

This is a useful case to demonstrate whether makeinfo and/or perl are affected by this bug, but since there is no way to capture the error stream output, I don't see an obvious way to test whether we are affected, aside from adding this test, and hoping people will notice it in the command window and report it when they run the full test suite.

Mike Miller <mtmiller>
Group Member
Fri 12 May 2017 09:19:12 PM UTC, comment #16: 

That does it.



>> s = __makeinfo__("@documentencoding UTF-8\n@defvar @var{x}\n@end defvar\n")
Use of uninitialized value in lc at /usr/lib/perl5/5.8/utf8_heavy.pl line 123.
s =  -- Variable: X
>>


John Donoghue <lostbard>
Group Member
Fri 12 May 2017 08:53:41 PM UTC, comment #15: 

The following string should be enough to trigger this bug.

John, can you see if this one causes the perl error message?


s = __makeinfo__("@documentencoding UTF-8\n@defvar @var{x}\n@end defvar\n")


That should return the right text string " -- Variable: X", but it should also print the error message on stderr.

Now even if that's so, I don't know how to capture this error, because the text is sent to stderr, which system() ignores, and the exit status of makeinfo is still success.

Mike Miller <mtmiller>
Group Member
Fri 12 May 2017 08:28:34 PM UTC, comment #14: 

Ok, I am able to reproduce this on Debian with some wrangling.

I set up a CentOS 5 container, which comes with perl 5.8.8 by default, which is affected by this utf8 module bug.

I built texinfo 6.0 manually from source, turned the syms.m doc string into a texi file, ran makeinfo, and was able to reproduce the same error message as everyone is seeing on Windows.

If I updated to texinfo 6.1 or 6.3, this error has been resolved, makeinfo runs without any diagnostics, and the doc string displays correctly.

I believe this is because of the following change mentioned in the texinfo 6.1 NEWS:

> Some Perl modules have been rewritten in C to increase speed.
> If Perl extensions can be created, they are used by default;
> otherwise the pure Perl implementations are still used.
> Disable at build time with "configure --disable-perl-xs".  The
> environment variable TEXINFO_XS controls how they are used by
> texi2any.


Using texinfo 6.3, as I said I get no warning message. If I add TEXINFO_XS=omit to my environment and run makeinfo the same way, I get the original error message again. So texinfo 6.1 and newer avoid this perl incompatibility by building their own compiled perl extensions for some functions, which are now preferred over the pure perl implementations that came with texinfo 6.0.

I would guess that updating mxe-octave to build texinfo 6.3 would resolve this problem, if that's easier than updating or patching perl 5.8.8 that comes from the msys distribution.

Next I will try to reduce the syms texinfo doc string to a test case that can trigger this error to appear so we can add that to our test suite.

Mike Miller <mtmiller>
Group Member
Fri 12 May 2017 06:30:03 PM UTC, comment #13: 

Thanks for clarifying that the symbolic package does install successfully. I was misunderstanding this as a package installation failure.

It's still important to fix, as these messages are very distracting and misleading, and we know that the version of perl from msys that makeinfo is using is quite old and buggy.

And I would still like to figure out how to trigger this particular message from perl with a simple test case using makeinfo that we can put into a unit test. If running `help syms` triggers it, then I would probably try copying the doc string to a new function file and paring it down until I can figure out what the minimal text is to make it happen. Unfortunately I don't know how to reproduce this bug on any systems I have access to.

Mike Miller <mtmiller>
Group Member
Thu 11 May 2017 12:47:53 PM UTC, comment #12: 

Yes I get the Use of uninitialized value in lc message, line 123 when trying install 2.5.0 of the symbolic package.

The package appears to install ok despite the warning.

Running load symbolic; help syms I get another print out of the uninitialized value in lc and then the help text for the syms function.

For the test calls:


> s = __makeinfo__ (["@documentencoding UTF-8\n", 0xcf, 0x80])
s = π
> s = __makeinfo__ (char ([0xe2, 0x8e, 0xa2]))
s = ⎢
> s = __makeinfo__ (char ([0xe2, 0x80, 0x9c]))
s = “
> s = __makeinfo__ (char ([0xe2, 0x88, 0x9a]))
s = √
> s = __makeinfo__ (char ([0xe6, 0x88, 0x91]))
s = 我






John Donoghue <lostbard>
Group Member
Wed 10 May 2017 06:07:21 PM UTC, comment #11: 

Or maybe it needs the @documentencoding tag to trigger this, how about


s = __makeinfo__ (["@documentencoding UTF-8\n", 0xcf, 0x80])


and the same with the other wide character samples?

Mike Miller <mtmiller>
Group Member
Wed 10 May 2017 05:56:11 PM UTC, comment #10: 

And you do get the "Use of uninitialized value in lc at ..." error message when you try to install symbolic?

If so, then I would still like to find a specific wide character that can expose this error when running makeinfo so we can add it as a test case.

How about the same test with one of

  • [0xe2, 0x8e, 0xa2]
  • [0xe2, 0x80, 0x9c]
  • [0xe2, 0x88, 0x9a]
  • [0xe6, 0x88, 0x91]
Mike Miller <mtmiller>
Group Member
Wed 10 May 2017 12:18:25 PM UTC, comment #9: 

On mine, octave 4.2.1:


>> s = __makeinfo__ (char ([0xcf, 0x80]))
s = π
>> isequal (s, [0xcf, 0x80])
ans = 1
>> system('perl -v')

This is perl, v5.8.8 built for msys-64int



John Donoghue <lostbard>
Group Member
Mon 08 May 2017 11:14:49 PM UTC, comment #8: 

John, or anyone else with Octave 4.2 on Windows affected by this bug, with Perl unpatched, how does the following behave, is the same error thrown?


>> s = __makeinfo__ (char ([0xcf, 0x80]))
s = π
>> isequal (s, [0xcf, 0x80])
ans = 1


If this triggers the same error, I think it might make a good unit test to ensure that Octave's Texinfo support is compatible with UTF-8 doc strings.

Mike Miller <mtmiller>
Group Member
Wed 04 Jan 2017 06:36:00 PM UTC, comment #7: 

I had started looking at how hard it would be to use msys2 rather than msys, which has more up-to-date packages.

John Donoghue <lostbard>
Group Member
Wed 14 Dec 2016 08:13:41 PM UTC, comment #6: 

We currently use a binary package from MSYS to provide Perl.  I don't see a newer version yet.  I also don't know how hard it would be to just cross compile Perl from source.

John W. Eaton <jwe>
Group administrator
Wed 14 Dec 2016 08:07:56 PM UTC, comment #5: 

Thanks for all the tips, the patch works now. I really appreciate it. And sorry about that I forgot to mention that I've tried to install symbolic-win-py-bundle-2.4.0, but came up with the same problem. I don't know if it just doesn't work in my machine.

Jihang <neur1n>
Wed 14 Dec 2016 07:44:32 PM UTC, comment #4: 

The attached patch (adapted from https://github.com/cbm755/octsympy/issues/610 and http://www.nntp.perl.org/group/perl.modules/2010/01/msg69392.html) was suggested to fix this error, if we want to stick with Perl 5.8 in mxe-octave.

(file #39225)

Mike Miller <mtmiller>
Group Member
Wed 14 Dec 2016 07:36:26 PM UTC, comment #3: 

This has been reported earlier: https://github.com/cbm755/octsympy/issues/610

This is probably occurring when any package is installed and it runs the `makeinfo` command to compile the doc strings in the package. The `makeinfo` command is a perl script, and uses the `utf8` perl module.

And it may only affect certain locales, or may only affect certain Unicode characters that are used in the symbolic package doc strings.

This could be fixed by either patching the perl library that is built and provided with the Octave Windows bundle, or by updating to a newer version of perl.

Mike Miller <mtmiller>
Group Member
Wed 14 Dec 2016 07:01:07 PM UTC, comment #2: 

An easier way for you would have been to install the symbolic-win-py-bundle-2.4.0.zip from
https://github.com/cbm755/octsympy/releases

That includes about everything you need on Windows.

Philip Nienhuis <philipnienhuis>
Group Member
Wed 14 Dec 2016 12:36:36 AM UTC, comment #1: 

Hmm... Looks like the Makeup tags didn't worked as I expected. How do I edit this post? I couldn't find a guide around. Thanks.

Jihang <neur1n>
Wed 14 Dec 2016 12:16:04 AM UTC, original submission:  

Let's start with the very beginning:

1: Tried to install symbolic-2.4.0 package by

pkg install -forge symbolic

but failed as an error said it couldn't read URL. It seems that I need cURL.

2. Downloaded symbolic-2.4.0.tar.gz and tried to install by

pkg install symbolic-2.4.0.tar.gz


3. Then kept popping up this message

Use of uninitialized value in lc at /usr/lib/perl5/5.8/utf8_heavy.pl line 123.

Codes related to this line are (line 123 in bold)

my $canonical = lc $type;
$canonical =~ s/(?<=[a-z\d])(?:\s+|[-_])(?=[a-z\d])//g;
print "canonical = $canonical\n" if DEBUG;

require "unicore/Canonical.pl";
*if (my $base = ($utf8::Canonical{$canonical} || $utf8::Canonical{ lc $utf8::PropertyAlias{$canonical} })) {*
    $file = "unicore/lib/gc_sc/$base.pl";
    last GETFILE;
}


4. Commented out these lines then the package could be installed successfully. Then I uncommented them afterwards and tried the following in Octave

pkg load symbolic
help syms

Then the help info popped up, but after another line of

Use of uninitialized value in lc at /usr/lib/perl5/5.8/utf8_heavy.pl line 123.


6. My search result of "perl lc", which seems like a case-sensitive matching bug of Perl. And I found this as well, which says at the beginning of the page: _A perl Unicode case-folding bug, present in at least perl-5.8.[678],
fixed in perl-5.8.9, where the utf8_heavy.pl has been substantially
rewritten_.

6. So I think that is the problem. Then I tried to get Perl 5.8.9 installed for Octave, and what I used is strawberry-perl-5.8.9.5.msi.

7. I installed this release of Perl and firstly just simply modify the utf8_heavy.pl, PVA.pl and Canonical.pl files in Octave according to those in Perl 5.8.9. It failed.

8. Tried to substituted all the files of Perl in Octave with Perl 5.8.9 and failed. Gave out compilation and version errors.

Appreciate your reading!

Jihang <neur1n>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

Attached Files
file #39225:  perl.diff added by mtmiller (606B - text/x-diff)

 

Depends on the following items: None found

Items that depend on this one

Digest:
   bug dependencies.

 

Carbon-Copy List
  • -email is unavailable- added by rik5 (Updated the item)
  • -email is unavailable- added by cbm (Posted a comment)
  • -email is unavailable- added by mmuetzel (Posted a comment)
  • -email is unavailable- added by lostbard (Posted a comment)
  • -email is unavailable- added by jwe (Posted a comment)
  • -email is unavailable- added by neur1n (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only group members can vote.

     

    Follow 13 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2018-12-13 rik5 StatusReady For Test Fixed
        Open/ClosedOpen Closed
    2018-10-02 mtmiller Carbon-CopyRemoved 80942 -
    2018-10-02 mtmiller StatusConfirmed Ready For Test
    2018-01-30 mtmiller Dependencies- bugs #53024 is dependent
    2017-09-06 mtmiller CategoryInterpreter Configuration and Build System
        Priority5 - Normal 3 - Low
        Release4.2.0 dev
        Summarymxe-octave: perl 5.8 utf8 module case-folding bug caused by installing symbolic-2.4.0 package [mxe-octave] perl 5.8 utf8 module case-folding bug caused by installing symbolic-2.4.0 package
    2016-12-14 mtmiller Attached File- Added perl.diff, #39225
    2016-12-14 mtmiller Item GroupNone Incorrect Result
        StatusNone Confirmed
        SummaryPerl 5.8.8 Unicode case-folding bug caused by installing symbolic-2.4.0 package mxe-octave: perl 5.8 utf8 module case-folding bug caused by installing symbolic-2.4.0 package

    Back to the top

    Powered by Savane 3.13-d3ae.
    Corresponding source code