bugGNU Octave - Bugs: bug #64331, unicode2native demands codepage...

 
 

bug #64331: unicode2native demands codepage input on Windows

Submitter:  Qianqian Fang <fangq>
Submitted:  Wed 21 Jun 2023 04:14:01 PM UTC
   
 
Category:  Octave Function Severity:  3 - Normal
Priority:  5 - Normal Item Group:  Regression
Status:  Fixed Assigned to:  None
Originator Name:  Qianqian Fang Open/Closed:  * Closed
Release:  * 8.2.0 Operating System:  * Any
Fixed Release:  8.3.0 Planned Release:  8.3.0
* Mandatory Fields

Add a New Comment Rich Markup
   

Jump to the original submission

Mon 26 Jun 2023 11:25:16 AM UTC, comment #10: 

Thank you for testing. That seems to be compatible to Matlab now.
With Matlab R2023a:

>> unicode2native('abc', [])
Error using unicode2native
Encoding must be a text scalar.

>> unicode2native('abc', '')

ans =

  1×3 uint8 row vector

   97   98   99


Closing as fixed (again).

Markus Mützel <mmuetzel>
Group administrator
Fri 23 Jun 2023 08:55:25 PM UTC, comment #9: 


>> unicode2native('test'," ")
ans =

  116  101  115  116

>> unicode2native('test',"")
ans =

  116  101  115  116

>> unicode2native('test','')
ans =

  116  101  115  116

>> unicode2native('test',[])
error: unicode2native: CODEPAGE must be a string
error: called from
    unicode2native at line 63 column 5

>> unicode2native('test')
ans =

  116  101  115  116

>> version -hgid
ans = 59681cc864e7


Dmitri.
--

Dmitri A. Sergatskov <dasergatskov>
Fri 23 Jun 2023 06:56:11 PM UTC, comment #8: 

The wording in our documentation is "If @var{codepage} is omitted or empty". So, we should also support an empty second argument. That is the same for native2unicode and for unicode2native.
That also matches the online documentation of TheMathworks for Matlab.

I changed both functions on the stable branch here:
https://hg.savannah.gnu.org/hgweb/octave/rev/e8497a0fb249

Re-opening and marking as ready for test again.

Markus Mützel <mmuetzel>
Group administrator
Fri 23 Jun 2023 05:15:45 PM UTC, comment #7: 

@Dmitri: Thanks for testing.

Closing as fixed.

Markus Mützel <mmuetzel>
Group administrator
Fri 23 Jun 2023 05:14:20 PM UTC, comment #6: 

Btw, using `unicode2native` without specifying the output encoding might give "unexpected" results even if the input is ASCII-only. If the system locale is, e.g., IBM-037 (EBCDIC), you'd get the following (even without the second argument):

>> unicode2native('abc', 'ibm-037')
ans =

  129  130  131


Whereas on an ASCII-compatible locale, you'd expect:

>> unicode2native('abc', 'latin1')
ans =

  97  98  99


Markus Mützel <mmuetzel>
Group administrator
Fri 23 Jun 2023 05:09:01 PM UTC, comment #5: 

This fixed it for me for both 8.2.1 (fab3e312a0b4) and 9.0.0 (3815dfaf27e3):


octave:1> unicode2native('test')
ans =

  116  101  115  116

octave:2> version -hgid
ans = fab3e312a0b4


Dmitri.
--

Dmitri A. Sergatskov <dasergatskov>
Fri 23 Jun 2023 02:21:51 PM UTC, comment #4: 

It's likely a regression from here:
https://hg.savannah.gnu.org/hgweb/octave/rev/e9771f0e180c

In particular, this part:

     error ("unicode2native: UTF8_STR must be a character vector");
   endif

-  if (! ischar (codepage))
+  if (! (ischar (codepage) && isrow (codepage)))
     error ("unicode2native: CODEPAGE must be a string");
   endif


The default value of `codepage` is the empty string `""`. `isrow ("")` results in false. But it should be allowed.


Should be fixed with this change:
https://hg.savannah.gnu.org/hgweb/octave/rev/fab3e312a0b4

Marking as ready for test.

Markus Mützel <mmuetzel>
Group administrator
Wed 21 Jun 2023 04:44:36 PM UTC, comment #3: 

In #60480 changeset I see:

if (nargin == 2 && ! (ischar (codepage) && isrow (codepage)))


and in the current source (9.0.0 hgid :

if (! ischar (utf8_str) || ! isvector (utf8_str))


Not sure why and when this change was made.

Dmitri.
--

Dmitri A. Sergatskov <dasergatskov>
Wed 21 Jun 2023 04:25:56 PM UTC, comment #2: 

thanks for confirming - the most recent octave on Linux I have is 6.4 on Ubuntu 22.04, which does not have this error. so, it is likely the issue was introduced in 7.x series.

by the way, please ignore my comment on #60480, as it has been already marked as fixed&closed.

Qianqian Fang <fangq>
Wed 21 Jun 2023 04:18:38 PM UTC, comment #1: 

Confirm on linux as well (9.0.0 dev):

octave:1> unicode2native('test')
error: unicode2native: CODEPAGE must be a string
error: called from
    unicode2native at line 63 column 5
octave:2> version -hgid
ans = addd99059d4c


Dmitri.
--

Dmitri A. Sergatskov <dasergatskov>
Wed 21 Jun 2023 04:14:01 PM UTC, original submission:  

I have 3 versions of octave installed on a windows machine for testing. For octave 5.2, the following command returned no error

octave:1> unicode2native('test')
ans =

  116  101  115  116


however, the same command failed on 7.3 and 8.2, complaining the optional parameter codepage is missing.



octave:1> unicode2native('test')
error: unicode2native: CODEPAGE must be a string
error: called from
    unicode2native at line 63 column 5



from the latest documentation, codepage input is optional - same as matlab's counterpart

https://docs.octave.org/v8.2.0/String-encoding.html

I consider this as a regression, and would be appreciated if this can be fixed - as it now breaks my code working for both matlab and octave.


by the way, it appears that a previously reported bug #60480 (https://savannah.gnu.org/bugs/?60480)  has been fixed at least based on my test (see screenshot)

Qianqian Fang <fangq>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

Attached Files
file #54852:  octave_unicode2native.png added by fangq (56KiB - image/png)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by mmuetzel (Posted a comment)
  • -email is unavailable- added by dasergatskov (Posted a comment)
  • -email is unavailable- added by fangq (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only group members can vote.

     

    Follow 11 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2023-06-26 mmuetzel StatusReady For Test Fixed
        Open/ClosedOpen Closed
    2023-06-23 mmuetzel StatusFixed Ready For Test
        Open/ClosedClosed Open
    2023-06-23 mmuetzel StatusReady For Test Fixed
        Open/ClosedOpen Closed
    2023-06-23 mmuetzel StatusNone Ready For Test
        Operating SystemMicrosoft Windows Any
        Fixed ReleaseNone 8.3.0
        Planned ReleaseNone 8.3.0
    2023-06-21 fangq Attached File- Added octave_unicode2native.png, #54852

    Back to the top

    Powered by Savane 3.13-bb6a.
    Corresponding source code