bugGNU Octave - Bugs: bug #49509, [mxe] failing tests on WinXP,...

 
 

bug #49509: [mxe] failing tests on WinXP, including "test help"

Submitter:  Hartmut <hardy>
Submitted:  Tue 01 Nov 2016 08:47:03 PM UTC
   
 
Category:  Interpreter Severity:  3 - Normal
Priority:  5 - Normal Item Group:  Regression
Status:  Fixed Assigned to:  None
Originator Name:  Open/Closed:  * Closed
Release:  * 4.2.0-rc3 Operating System:  * Microsoft Windows
Fixed Release:  None Planned Release:  None
* Mandatory Fields

Add a New Comment Rich Markup
   

Jump to the original submission

Mon 07 Nov 2016 04:53:24 PM UTC, comment #16: 

Typing or pasting Umlaut characters in the command window still doesn't work in Windows 10 (and never has as far as I can tell). There might be a problem with how we use stdin? But I can't figure out what it is. All non-ASCII characters seem to be eaten up by WriteConsoleInput and never make it to the console...

However, all Unicode characters (of the used fonts) at least display correctly when the codepage of the console is set to 65001 (UTF-8) in Windows 10 (and maybe Windows 8/8.1?).

Markus Mützel <mmuetzel>
Group administrator
Mon 07 Nov 2016 04:47:51 PM UTC, comment #15: 

Okay, I pushed the change to stable here (http://hg.savannah.gnu.org/hgweb/octave/rev/2d74b4653e5d).  Closing report.

Rik <rik5>
Group administrator
Mon 07 Nov 2016 04:36:16 PM UTC, comment #14: 

If I have followed all the bug reports regarding "international" characters correctly, then typing umlauts in the GUI console never worked so far under Windows. It only started working recently (with cset 8c749f33ce0a), and only with the version "Win10" and no earlier windows versions.

@Rik: So in my opinion: Yes please, go ahead an apply this patch (file #38889) to stable.

Hartmut <hardy>
Mon 07 Nov 2016 04:18:12 PM UTC, comment #13: 

I applied the patch and built a Windows version with MXE.  My only test platform is a Windows XP virtual machine.  The tests that were failing earlier now work, but I am not able to enter an umlaut character with cut and paste.  Is that expected?

Since this fixes a much more serious issue than umlauts, can I apply the patch to stable and figure out the umlaut problem later?

Rik <rik5>
Group administrator
Fri 04 Nov 2016 10:38:43 PM UTC, comment #12: 

I have finished _run_test_suite_ on my mxe-octave build of Octave-4.2.0-rc3 including the "Win7plus-patch" on a WinXP machine, now. Everything looks FINE. I only get three failing tests, all happen without this patch as well (see bug #49501 and bug #49040).

So I can just encourage to use this patch (file #38889).

Hartmut <hardy>
Fri 04 Nov 2016 08:11:40 PM UTC, comment #11: 

"test help" passes now on WinXP, with Octave 4.2.0-rc3 and the "Win7plus-patch" (file #38889) from comment #9. I'll let you know the results from a full run of the test suite soon.

Hartmut <hardy>
Fri 04 Nov 2016 06:51:03 PM UTC, comment #10: 

(compiling...)

Hartmut <hardy>
Fri 04 Nov 2016 06:36:53 PM UTC, comment #9: 

Hartmut, can you please try the attached patch and test with Windows XP. It compiles for me and works as expected in Windows 10.

(file #38889)

Markus Mützel <mmuetzel>
Group administrator
Fri 04 Nov 2016 06:29:46 PM UTC, comment #8: 

I have now tested a fresh mxe-octave built of Octave-4.2.0-rc3 plus the "IsValidCodepage-patch" (file #38884). Running this on a WinXP machine and doing "test help" gives me still the same failure. So this patch does definitively NOT help (as was expected in comment #7).

So I see two ways of fixing this:

  • (A) completly remove the "65001-patch" (cset 8c749f33ce0a) from stable Octave. The only drawback of this would be that we destroy the utf8 capabilities of the terminal under Win10. (But this never worked before in any other release, and it still does not work in any other Windows version):
  • (B) add some code switch option to only use this "65001-patch" for Windows versions of 7 or higher. This would keep the benefit of a working utf8-support in Win10 terminals, and still not spoil the help function in WinXP. (But I don't know of a feasible way to formulate such a code switch option that depends on the Windows version. This version will only be known at runtime of Octave, not at compile time.)


@Rik: After fixing this, I would propose to KEEP the second text encoding patch ("cstrings-patch", cset f88de8f27259) in stable octave, and proceed to a new release candidate rc4 that might hopefully then turn into the 4.2 release.


Hartmut <hardy>
Fri 04 Nov 2016 02:36:41 PM UTC, comment #7: 

I think in this case my patch from comment #3 wouldn't help.
Instead the codepage should probably be set depending on the Windows version.
Since the 65001-patch did help for the command window in Windows 10 (but not in Windows 7) and it breaks things in Windows XP (but not in Windows 7), it might be an idea to change the codepage only on Windows 7 or later.
What do you think?

Markus Mützel <mmuetzel>
Group administrator
Fri 04 Nov 2016 02:19:19 PM UTC, comment #6: 

When I do the commands of Markus comment #5 in unpatched 4.2.0-rc3 on WinXP then I get:


>> system ("chcp 65001")
Aktive Codepage: 65001.
ans = 0
>> system ("chcp")
Aktive Codepage: 65001.
ans = 0
>>


My mxe-compilation of 4.2.0-rc3 without the "65001-patch" is done now. Result: "test help" works fine! Passes 5/5. So this is the proof that this "65001-patch" is definitly the source of the trouble in this bug report.

I will now start an mxe-octave compilation of 4.2.0-rc3 with the new "IsValidCodepage-patch" from comment #3. Let's see...

Hartmut <hardy>
Fri 04 Nov 2016 12:57:05 PM UTC, comment #5: 

Maybe try the following:

system ("chcp 65001")
system ("chcp")


That might display some error or give indications what is going on.

Markus Mützel <mmuetzel>
Group administrator
Fri 04 Nov 2016 12:47:49 PM UTC, comment #4: 

I have tested on WinXP with 4.2.0-rc3:
test help -> fails
system("chcp 850")
test help -> passes 5/5

So we are on the right track!

Compiling mxe-octave will take a while, maybe I'll have results this evening.

Is there a (faster) way to test your "IsValidCodePage (65001)" command from within a running rc3 installation? I.e. without recompiling mxe-octave?

Hartmut <hardy>
Fri 04 Nov 2016 12:20:43 PM UTC, comment #3: 

Hartmut,
can you test whether setting the codepage manually to 850 (Latin-1) lets "test help" pass again in Windows XP?

system ("chcp 850")


I do not have a WinXP installation to check. I also have no idea how that patch could have affected those tests.

It might be possible that codepage 65001 is not loaded by default in WinXP (or not available at all?). It is difficult to find definitive documentation about this on Microsoft's webpages.
But if this were the case, setting it might lead to unexpected results.
Can you try if the attached patch fixes the issues?

(file #38884)

Markus Mützel <mmuetzel>
Group administrator
Fri 04 Nov 2016 09:27:59 AM UTC, comment #2: 

I have done some more testing. The patch from comment #1 does NOT help at all. Building rc3 with and without it doesn't make a difference regarding the failure of "test help" under WinXP.

But I have an (uncorrelated) built of default-octave with and without the "65001-patch". Without this patch "test help" works fine under XP, and only applying this single patch makes "test help" fail under XP. So I have the suspicion that this "65001-patch" is causing those troubles under WinXP.

I will now do a 4.2.0-rc3 built, with just removing this "65001-patch" (see http://hg.savannah.gnu.org/hgweb/octave/rev/8c749f33ce0a) and see if solves this issue.

Hartmut <hardy>
Tue 01 Nov 2016 08:52:36 PM UTC, comment #1: 

The attached PATCH solves this issue with 4.2.0-rc3. (It changes all instances of QString::toLatin1 into QString::toLocal8Bit, and the same for QString::fromLatin1)

This patch is not ment to be applied directly. It might have unwanted side effects, or even lesser lines of patching might be enough. But I hope that someone with more insite can gain some more understanding from this.

Btw: do we still support WinXP? (It did work flawlessly until 4.2.0-rc2.)

(file #38861)

Hartmut <hardy>
Tue 01 Nov 2016 08:47:03 PM UTC, original submission:  

I have observed this with the current pre-release 4.2.0-rc3  under Windows XP (without a Java JRE installed, I don't know if this is relevant).

The following tests of the test suite fail:

test __unimplemented__
test get_first_help_sentence
test help
test test


The function calls in those test just return nothing instead of the expected strings. This is especially bad becaus this also means that "help" does not work at all.

On Windows 7 (but with JRE) this does NOT happen.

On the same WinXP machine, this does NOT happen with Octave 4.2.0-rc2. So we spoilt something between rc2 and rc3.

Here are the details from the fntests.log file:


>>>>> processing C:\Octave\Octave-4.2.0-rc3\share\octave\4.2.0-rc3\m\help\__unimplemented__.m
***** test
 str = __unimplemented__ ("no_name_function");
 assert (isempty (str));
 str = __unimplemented__ ("quad2d");
 assert (str(1:51), "quad2d is not implemented.  Consider using dblquad.");
 str = __unimplemented__ ("MException");
 assert (str(1:58), "the 'MException' function is not yet implemented in Octave");
!!!!! test failed
str(51): out of bound 0

>>>>> processing C:\Octave\Octave-4.2.0-rc3\share\octave\4.2.0-rc3\m\help\get_first_help_sentence.m
***** assert (get_first_help_sentence ('get_first_help_sentence'), ...
        "Return the first sentence of a function's help text.")
!!!!! test failed
ASSERT errors for:  assert (get_first_help_sentence ('get_first_help_sentence'),"Return the first sentence of a function's help text.")

  Location  |  Observed  |  Expected  |  Reason
     []                   Return the first sentence of a function's help text.   Strings don't match

>>>>> processing C:\Octave\Octave-4.2.0-rc3\share\octave\4.2.0-rc3\m\help\help.m
***** assert (! isempty (strfind (help ("ls"), "List directory contents")))
!!!!! test failed
assert (!isempty (strfind (help ("ls"), "List directory contents"))) failed

>>>>> processing C:\Octave\Octave-4.2.0-rc3\share\octave\4.2.0-rc3\m\testfun\test.m
***** fail ("test (1,2,3,4)", "usage.*test") # too many args, generates usage()
!!!!! test failed
expected error <usage.*test>
but got <Invalid call to test.  Correct usage is:
 >


I have also found (by chance) a patch that resolves this issue for me (already discussed before in bug #49118 comment #25). I will attach this as soon as I have a bug number.


Hartmut <hardy>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

Attached Files

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by rik5 (Posted a comment)
  • -email is unavailable- added by hardy
  • -email is unavailable- added by mmuetzel (Updated the item)
  • -email is unavailable- added by hardy (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only group members can vote.

     

    Follow 7 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2016-11-07 rik5 StatusIn Progress Fixed
        Open/ClosedOpen Closed
    2016-11-07 rik5 StatusNone In Progress
    2016-11-04 mmuetzel Attached File- Added bug49509_65001_Windows7plus.patch, #38889
    2016-11-04 hardy Carbon-Copy- Added rik5
    2016-11-04 mmuetzel Attached File- Added bug49509_IsValidCodePage_65001.patch, #38884
    2016-11-01 hardy Attached File- Added QtTextEncodingToLocal8Bit.patch, #38861

    Back to the top

    Powered by Savane 3.13-f8d8.
    Corresponding source code