bugGNU Octave - Bugs: bug #57235, Document that startup file...

 
 

bug #57235: Document that startup file .octaverc is always read in the system encoding

Submitter:  None
Submitted:  Thu 14 Nov 2019 01:50:58 PM UTC
   
 
Category:  Interpreter Severity:  3 - Normal
Priority:  5 - Normal Item Group:  Documentation
Status:  Fixed Assigned to:  None
Originator Name:  Ata Sevinç Originator Email:  -email is unavailable-
Open/Closed:  * Closed Release:  * dev
Operating System:  * Any Fixed Release:  None
Planned Release:  None
* Mandatory Fields

Add a New Comment Rich Markup
   

Jump to the original submission

Mon 27 Jul 2020 03:07:31 PM UTC, comment #16: 

Closing as fixed.

Markus Mützel <mmuetzel>
Group administrator
Mon 27 Jul 2020 01:15:25 PM UTC, comment #15: 

OK to close this report as fixed?

John W. Eaton <jwe>
Group administrator
Sat 27 Jun 2020 10:04:34 AM UTC, comment #14: 

I pushed a change to the documentation here:
http://hg.savannah.gnu.org/hgweb/octave/rev/6bd9d77c7105

I hope it is ok to reference the undocumented function "__mfile_encoding__" in the documentation (which makes it kind of semi-documented).

Marking as ready for test.

Markus Mützel <mmuetzel>
Group administrator
Fri 26 Jun 2020 06:52:18 AM UTC, comment #13: 

If I recall correctly, when this bug was created, Octave ignored the system's locale charset on non-Windows platforms. That has been fixed in the meantime.

One drawback might be that is is not possible (at least not without breaking a lot of other programs) to set a UTF-8 locale on Windows. But that is not Octave's fault and I'm not sure if it is worth trying to implement a workaround for that shortcoming.

It's probably good enough to document somewhere that the startup files are interpreted (initially) in the locale charset. That is until a user manually changes the .m file encoding and triggers re-parsing the .m files (see comment #4) or the GUI sets the .m file encoding from the saved GUI options.

Markus Mützel <mmuetzel>
Group administrator
Thu 25 Jun 2020 09:32:04 PM UTC, comment #12: 

Sorry for letting this issue drop.  I promised to do something but now I'm not sure what the problem is.

It looks like the default for input_system::m_file_encoding is "system" and if that is used, then we look at the environment when deciding what encoding to use.  If I set LANG=C in the environment and start Octave, then the first time that file_reader::get_input is called, it uses the value returned from octave_locale_charset_wrapper ("ANSI_X3.4-1968") and if I have LANG=en_US.UTF-8 the encoding used is "UTF-8".  Is that wrong?

I see that in file_reader::get_input, we are comparing the value of the encoding variable with "utf-8" and it appears to be a case-sensitive comparison.  Is that the correct thing to do?

If we need to be able to set the encoding before reading config files, then using an environment variable or a command-line option seems like the right thing to do.  Since it seems like we are paying attention to the LANG value in the environment, I don't understand what the problem is.

John W. Eaton <jwe>
Group administrator
Mon 18 Nov 2019 03:57:09 PM UTC, comment #11: 

OK, I think we can arrange for the encoding to be set properly but I need a little time to think about how it should work.

There is also a larger issue of how settings should really be handled.  I will try to post a message about that to the maintainers list and/or open a separate bug report about that.

  


John W. Eaton <jwe>
Group administrator
Mon 18 Nov 2019 03:47:09 PM UTC, comment #10: 

The encoding is set globally. The property "m_mfile_encoding" has to be set in the "input_system" object (using input_system::mfile_encoding).

At the moment this is done in main_window::notice_settings. But that only works when using the GUI and it is too late for the startup files.

Markus Mützel <mmuetzel>
Group administrator
Mon 18 Nov 2019 03:28:46 PM UTC, comment #9: 

I haven't really paid much attention to encoding issues.

For other code in .m files, is this something that is handled for each file individually?  If so, what needs to happen to detect and set the encoding properly for the octaverc startup files?

John W. Eaton <jwe>
Group administrator
Sun 17 Nov 2019 06:00:57 AM UTC, comment #8: 

Unfortunately, there is no work-around for the second issue mentioned in comment #4. You'll have to wait for Octave 6.1.
If nothing else changes until then, you'll still need one of the work-around for the first issue mentioned in comment #4.

Markus Mützel <mmuetzel>
Group administrator
Sat 16 Nov 2019 05:09:53 PM UTC, comment #7: 

None of the workarounds is successful, but thanks for your efforts.

Anonymous
Sat 16 Nov 2019 04:36:49 PM UTC, comment #6: 

Re-titling to better reflect the remaining issue.

I don't completely understand the Octave start-up process. But here is my analysis/guess of what is the issue:

  • The startup files are executed early on when initializing the interpreter.
  • The .m file encoding is restored from the value in the Qt settings file by the GUI process.
  • It looks like the latter is done after the startup files are executed.


I don't know how this should be handled.
Is there a settings file or something similar that holds properties for the interpreter (and not so much for the GUI)?

CC'ing jwe because I think he best understands the startup process.

Markus Mützel <mmuetzel>
Group administrator
Sat 16 Nov 2019 04:26:52 PM UTC, comment #5: 

I pushed a change that fixes the second issue for me here:
http://hg.savannah.gnu.org/hgweb/octave/rev/12a53552db92

This will be part of the next major release Octave 6.1.

Markus Mützel <mmuetzel>
Group administrator
Sat 16 Nov 2019 04:12:12 PM UTC, comment #4: 

Thanks, for your explanation.

jwe is currently working on a replacement for the command window which will hopefully fix the issue you describe in comment #3.

Wrt your comment #2: That is a corner case we haven't covered yet.
There are actually two issues that are causing the current behaviour:
1. Octave doesn't know the encoding that is used to read .m files so early on in the start up process. I don't know how to fix this in Octave though.
As a work-around you could use two startup files called in a row. The first startup file (.octaverc) is used to set the .m file encoding:

__mfile_encoding__ ("utf-8");  # or whichever encoding you prefer
clear ("functions");   # necessary (?) to re-read the .m files in the correct encoding
second_octaverc;


Put the non-ASCII code in a second startup file (second_octaverc.m):

addpath ('C:\Türkçe');


Another work-around that you could use: Save the .octaverc file in your system encoding (likely "Windows-1254"?).


2. However, the above work-arounds currently fail because of another issue. There is still some code that isn't Unicode-aware on Windows. I am currently working on a fix for that and waiting for the cross-compilation to finish.


Markus Mützel <mmuetzel>
Group administrator
Sat 16 Nov 2019 01:10:54 AM UTC, comment #3: 

In addition, at the console of Octave 5.1.0, after copying 'Mützel' if you paste it to the console, you see

>> Mtzel

Anonymous
Sat 16 Nov 2019 01:03:07 AM UTC, comment #2: 

1) Make a folder "C:\Türkçe".
2) Write a command line into "octaverc" file as

addpath('C:\Türkçe')

3) Run Octave 5.1.0
It gives the following warnings without adding that folder into path:

warning: addpath: C:\Türkçe: No such file or directory
warning: called from
    C:\Octave\OCTAVE~1.0\mingw64\share\octave\site\m\startup/octaverc at line 8 column 1

Anonymous
Fri 15 Nov 2019 08:14:16 AM UTC, comment #1: 

Thank you for your report.
Could you please give step-by-step instructions how to reproduce the error?

Markus Mützel <mmuetzel>
Group administrator
Thu 14 Nov 2019 01:50:58 PM UTC, original submission:  

Hello,
Neither GUI nor CLI versions allows using special characters in other languages, for example Turkish letters "ç,ı,ş". If the program file includes these letters, the interpreter just omits them without leaving spaces instead. These letters can be in the path name, so it might be necessary to use them. We need character set UTF-8 or Windows-1254 or iso-8859-9 to be accepted.
Thanks and regards.

Anonymous

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by jwe (Posted a comment)
  • -email is unavailable- added by mmuetzel
  • -email is unavailable- added by mmuetzel (Posted a comment)
  • -email is unavailable- added by None (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only group members can vote.

     

    Follow 13 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2020-07-27 mmuetzel StatusReady For Test Fixed
        Open/ClosedOpen Closed
    2020-06-27 mmuetzel StatusConfirmed Ready For Test
    2020-06-26 mmuetzel Item GroupNone Documentation
        Operating SystemMicrosoft Windows Any
        SummaryStartup file .octaverc is always read in the system encoding Document that startup file .octaverc is always read in the system encoding
    2019-11-16 mmuetzel StatusReady For Test Confirmed
        SummaryCharacter encoding problems Startup file .octaverc is always read in the system encoding
        Carbon-Copy- Added jwe
    2019-11-16 mmuetzel StatusIn Progress Ready For Test
    2019-11-16 mmuetzel StatusNeed Info In Progress
        Release5.1.0 dev
    2019-11-15 mmuetzel StatusNone Need Info

    Back to the top

    Powered by Savane 3.13-f8d8.
    Corresponding source code