bugGNU Octave - Bugs: bug #49339, False error reports while...

 
 

bug #49339: False error reports while producing latex with non-ascii chars under gnuplot

Submitted by:  None
Submitted on:  Thu 13 Oct 2016 08:14:55 PM UTC  
 
Category: Plotting with gnuplotSeverity: 3 - Normal
Priority: 5 - NormalItem Group: Inaccurate Result
Status: Need InfoAssigned to: None
Originator Name: R.S.CarmenesOriginator Email: -unavailable-
Open/Closed: OpenRelease: 4.2.0
Operating System: GNU/Linux

Add a New Comment(Rich Markup)
   

You are not logged in

Please log in, so followups can be emailed to you.

 

(Jump to the original submission Jump to the original submission)

Sat 17 Jun 2017 03:06:45 AM UTC, comment #7:

OK, just keep in mind we should probably make the gnuplot toolkit behave the same as the core behaves. Hopefully it is as easy as placing

"set encoding locale"

right after gnuplot is launched (and prior to the setting of the terminal type).

Dan Sebald <sebald>
Fri 16 Jun 2017 10:11:59 PM UTC, comment #6:

This really has nothing to do with gnuplot as I understand it, this has to do with Octave not initializing the process locale state from the environment when it starts up in non-interactive non-graphical mode.

I think the untested snippet I wrote in comment #4 is the right way to solve this, I just haven't bothered to look into the best place to place this code.

Mike Miller <mtmiller>
Project Administrator
Thu 15 Jun 2017 11:35:49 PM UTC, comment #5:

Any progress on this issue? Perhaps "progress" isn't the right word, as it seems that this one isn't difficult to address, but more a question of what should be done.

One question is whether this is a gnuplot toolkit bug or a ft_text_renderer:: bug or not a bug and just a simple warning that can be ignored.

From the ft_text_renderer:: perspective, this warning does seem like the thing it should do. If I understand correctly, the figure property is set to interpret the text as C and there is some non-C type characters in the string so it should be a warning. It is simply the case that gnuplot is being generous here and interpreting in a non-C way.

Should gnuplot/gnuplot-toolkit be programmed so it too improperly decodes the utf-8 in this condition? The warning would make sense then. I'm not quite sure what the gnuplot TK is doing right now, but if you want to see the encodings, type:

If we were to place "set encoding locale" in the gnuplot scripts or "set encoding XX" based on some Octave figure property, whould that then put the warning and gnuplot behavior in sync?

Dan Sebald <sebald>
Fri 03 Mar 2017 07:18:18 PM UTC, comment #4:

I agree that running octave-cli non-interactively does not initialize the LC_CTYPE facet correctly, that's the conclusion I came to in comment #2. If the user does have a DISPLAY, or if Octave is run interactively, it is initialized correctly.

So it looks like we should handle that use case, I agree, and maybe the interpreter constructor is the right place.

The appropriate way to initialize this might be something like the following (adapted from readline):

Mike Miller <mtmiller>
Project Administrator
Fri 03 Mar 2017 11:25:31 AM UTC, comment #3:

I am running octave processing script files on a server without interactive prompt or display. The script files are supposed produce graphics. The same conditions can be reproduced on any computer by running with

with test.m being

one can see on the resulting image file that the LaTeX encoded alpha and the utf-8 encoded umlaut character are properly rendered, but nevertheless Octave prints out a warning from ft_text_renderer function referring to locale C even if the locale is set to en_US.UTF-8 for example.

I do not agree on the original posters suggestion on changing the setlocale (LC_CTYPE, 0) at libinterp/corefcn/ft-text-renderer.cc because that is effectively a "getlocale" as it should. Still, somewhere during initialization process octave should do setlocale (LC_CTYPE, ""). In libinterp/corefcn/interpreter.cc as the op suggests? Another option is to use main.cc where we have a if-else structure processing different conditions with or without gui.

Taneli Kalvas <tvkalvas>
Thu 17 Nov 2016 05:51:07 AM UTC, comment #2:

I've written a compiled oct file to test what the current locale is at the C library level, and I've tried running octave-cli with and without the --norc and --no-line-editing option. The only way I get get the LC_CTYPE facet to be uninitialized is with octave-cli --no-line-editing. Is this how you are running Octave?

If Octave runs itself as a QApplication, then the locale is initialized by Qt. If Octave runs with readline enabled, then readline initializes the locale from the environment when it is initialized. It is only in this niche corner case where Octave is run without either Qt or readline that the locale may not be initialized properly before the interpreter runs.

Mike Miller <mtmiller>
Project Administrator
Thu 17 Nov 2016 05:29:11 AM UTC, comment #1:

I actually do not get any of these warning messages when I try to reproduce this, I only get the single warning

when I set GNUTERM=dumb. This is addressed by the fscanf vs fread suggestion you made, which seems like a good change.

I only get the other warning messages you've shown if I explicitly set LC_CTYPE=C in my environment before running Octave.

Are you sure your LC_CTYPE facet is not set to C? Are you running octave-cli or octave (either with --no-gui or with the GUI)? This may have a bearing on whether the locale is being initialized correctly or not.

What version of gnuplot are you using?

Mike Miller <mtmiller>
Project Administrator
Thu 13 Oct 2016 08:14:55 PM UTC, original submission:

When printing plots containing non-ascii chars to latex files under the gnuplot graphics_toolkit, the resulting latex output is correct, but octave reports a series of false error messages, no matter which locale settings you use.

I am providing a patch solving the source of the problem too, which is spread over several files.

It is not a serious issue, but it is disturbing, and potentially missleading: in fact it made me spend a lot of time trying to see what was wrong, until I realised that nothing but the messages were really wrong.

As a sample, I use the same similar as for bug #49338, except for the graphics_toolkit:

When calling octave from the command line to run a script with the above code, octave sends the following to sdterr if we are on a X11 terminal (none of it being present if pure ascii was used):

The same script leads to even more error messages if ran from a pure text terminal:

On my computer I have LC_ALL=es_ES, but I tried all sort of settings without success. In other words, it seems that octave is currently ignoring the locale settings.

After investigation, I found the source of those false error messages, spread over several files. Here follows a short description of what is wrong:

libinterp/corefcn/ft-text-renderer.cc:

it uses
std::setlocale (LC_CTYPE, 0));
but it should be
std::setlocale (LC_CTYPE, ""));

(maybe for some compilers 0 and "" are equivalent, but it doesn't seem to be with mine, gcc 5.3.0).

libinterp/corefcn/interpreter.cc:

Octave is missing to be informed of the current locale before forcing it to
use the "C" locale for numbers (necessary to follow the matlab usage).
In other words, we need to add
setlocale (LC_CTYPE, "");
before
setlocale (LC_NUMERIC, "C");
setlocale (LC_TIME, "C");

scripts/plot/util/__gnuplot_drawnow__.m
scripts/plot/util/print.m

These two files use fscanf(fid, '%c', Inf) to read files containing text; fscanf succeds reading non-ascii chars, but report them to be out of range regardless of locale settings.

Using fread is equally successful, but has the advantage of not reporting out-of-range messages; additionally, it is faster when fscanf is wanted only to read an entire file char by char.

I already corrected print.m via the patch accompanying the description of bug #49338, so I attach two patch files correcting the remaining problems. Recompilation is necessary, as two *.cc source files are slightly modified.

Hope this helps.

Anonymous

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach File(s):
   
   
Comment:
   

Attached Files
file #38722:  latin1-cc.patch added by None (1KiB - text/x-patch)
file #38723:  latin1-gnuplot.patch added by None (817B - text/x-patch)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -unavailable- added by sebald (Posted a comment)
  • -unavailable- added by tvkalvas (Posted a comment)
  • -unavailable- added by mtmiller (Posted a comment)
  • -unavailable- added by None (Submitted the item)
  •  

    Do you think this task is very important?
    If so, you can click here to add your encouragement to it.
    This task has 0 encouragements so far.

    Only project members can vote.

     

    Please enter the title of George Orwell's famous dystopian book (it's a date):

     

     

    Follow 4 latest changes.

    Date Changed By Updated Field Previous Value => Replaced By
    Thu 17 Nov 2016 05:29:11 AM UTCmtmillerStatusNone=>Need Info
      Release4.2.0-rc2=>4.2.0
    Thu 13 Oct 2016 08:14:55 PM UTCNoneAttached File-=>Added latin1-cc.patch, #38722
      Attached File-=>Added latin1-gnuplot.patch, #38723

    Back to the top


    Powered by Savane 3.1-cleanup1