bugGNU Octave - Bugs: bug #49339, False error reports while...

 
 

bug #49339: False error reports while producing latex with non-ascii chars under gnuplot

Submitter:  None
Submitted:  Thu 13 Oct 2016 08:14:55 PM UTC
   
 
Category:  Plotting with gnuplot Severity:  3 - Normal
Priority:  1 - Later Item Group:  Inaccurate Result
Status:  Need Info Assigned to:  None
Originator Name:  R.S.Carmenes Originator Email:  -email is unavailable-
Open/Closed:  * Open Release:  * 4.2.0
Operating System:  * GNU/Linux Fixed Release:  None
Planned Release:  None
* Mandatory Fields

Add a New Comment Rich Markup
   

Jump to the original submission

Sun 14 Jan 2024 03:22:34 PM UTC, comment #8: 

Downgrading priority to 1 as gnuplot is no longer a development priority.

Rik <rik5>
Group administrator
Sat 17 Jun 2017 03:06:45 AM UTC, comment #7: 

OK, just keep in mind we should probably make the gnuplot toolkit behave the same as the core behaves.  Hopefully it is as easy as placing

"set encoding locale"

right after gnuplot is launched (and prior to the setting of the terminal type).

Dan Sebald <sebald>
Fri 16 Jun 2017 10:11:59 PM UTC, comment #6: 

This really has nothing to do with gnuplot as I understand it, this has to do with Octave not initializing the process locale state from the environment when it starts up in non-interactive non-graphical mode.

I think the untested snippet I wrote in comment #4 is the right way to solve this, I just haven't bothered to look into the best place to place this code.

Mike Miller <mtmiller>
Group Member
Thu 15 Jun 2017 11:35:49 PM UTC, comment #5: 

Any progress on this issue?  Perhaps "progress" isn't the right word, as it seems that this one isn't difficult to address, but more a question of what should be done.

One question is whether this is a gnuplot toolkit bug or a ft_text_renderer:: bug or not a bug and just a simple warning that can be ignored.

From the ft_text_renderer:: perspective, this warning does seem like the thing it should do.  If I understand correctly, the figure property is set to interpret the text as C and there is some non-C type characters in the string so it should be a warning.  It is simply the case that gnuplot is being generous here and interpreting in a non-C way.

Should gnuplot/gnuplot-toolkit be programmed so it too improperly decodes the utf-8 in this condition?  The warning would make sense then.  I'm not quite sure what the gnuplot TK is doing right now, but if you want to see the encodings, type:


sebald@moorglade ~ $ gnuplot

        G N U P L O T
        Version 5.1 patchlevel 0    last modified 2017-04-03

        Copyright (C) 1986-1993, 1998, 2004, 2007-2017
        Thomas Williams, Colin Kelley and many others

        gnuplot home:     http://www.gnuplot.info
        mailing list:     gnuplot-beta@lists.sourceforge.net
        faq, bugs, etc:   type "help FAQ"
        immediate help:   type "help"  (plot window: hit 'h')

Terminal type set to 'qt'
gnuplot> help utf
 The `set encoding` command selects a character encoding.

 Syntax:
       set encoding {<value>}
       set encoding locale
       show encoding

 Valid values are
    default     - tells a terminal to use its default encoding
    iso_8859_1  - the most common Western European encoding used by many
                  Unix workstations and by MS-Windows. This encoding is
                  known in the PostScript world as 'ISO-Latin1'.
    iso_8859_15 - a variant of iso_8859_1 that includes the Euro symbol
    iso_8859_2  - used in Central and Eastern Europe
    iso_8859_9  - used in Turkey (also known as Latin5)
    koi8r       - popular Unix cyrillic encoding
    koi8u       - Ukrainian Unix cyrillic encoding
    cp437       - codepage for MS-DOS
    cp850       - codepage for OS/2, Western Europe
    cp852       - codepage for OS/2, Central and Eastern Europe
    cp950       - MS version of Big5 (emf terminal only)
    cp1250      - codepage for MS Windows, Central and Eastern Europe
    cp1251      - codepage for 8-bit Russian, Serbian, Bulgarian, Macedonian
    cp1252      - codepage for MS Windows, Western Europe
    cp1254      - codepage for MS Windows, Turkish (superset of Latin5)
    sjis        - shift-JIS Japanese encoding
    utf8        - variable-length (multibyte) representation of Unicode
                  entry point for each character

 The command `set encoding locale` is different from the other options.
 It attempts to determine the current locale from the runtime environment.
 On most systems this is controlled by the environmental variables
 LC_ALL, LC_CTYPE, or LANG.  This mechanism is necessary, for example, to
 pass multibyte character encodings such as UTF-8 or EUC_JP to the wxt
 and cairopdf terminals.  This command does not affect the locale-specific
 representation of dates or numbers.
 See also `set locale` and `set decimalsign`.

 Generally you must set the encoding before setting the terminal type.
 Note that encoding is not supported by all terminal drivers and that
 the device must be able to produce the desired non-standard characters.


If we were to place "set encoding locale" in the gnuplot scripts or "set encoding XX" based on some Octave figure property, whould that then put the warning and gnuplot behavior in sync?

Dan Sebald <sebald>
Fri 03 Mar 2017 07:18:18 PM UTC, comment #4: 

I agree that running octave-cli non-interactively does not initialize the LC_CTYPE facet correctly, that's the conclusion I came to in comment #2. If the user does have a DISPLAY, or if Octave is run interactively, it is initialized correctly.

So it looks like we should handle that use case, I agree, and maybe the interpreter constructor is the right place.

The appropriate way to initialize this might be something like the following (adapted from readline):


std::string val = octave::sys::env::getenv ("LC_ALL");
if (val.empty ())
  val = octave::sys::env::getenv ("LC_CTYPE");
if (val.empty ())
  val = octave::sys::env::getenv ("LANG");
if (val.empty ())
  {
    char *p = setlocale (LC_CTYPE, 0);
    if (p)
      val = p;
  }
setlocale (LC_CTYPE, val.c_str ());


Mike Miller <mtmiller>
Group Member
Fri 03 Mar 2017 11:25:31 AM UTC, comment #3: 

I am running octave processing script files on a server without interactive prompt or display. The script files are supposed produce graphics. The same conditions can be reproduced on any computer by running with


export DISPLAY=
export GNUTERM=png
octave --no-window-system --no-gui -qf test.m


with test.m being


plot([1 2 3],[3 4 2])
xlabel("\\alpha Äh")
print("image.png", "-dpngcairo")


one can see on the resulting image file that the LaTeX encoded alpha and the utf-8 encoded umlaut character are properly rendered, but nevertheless Octave prints out a warning from ft_text_renderer function referring to locale C even if the locale is set to en_US.UTF-8 for example.

I do not agree on the original posters suggestion on changing the setlocale (LC_CTYPE, 0) at libinterp/corefcn/ft-text-renderer.cc because that is effectively a "getlocale" as it should. Still, somewhere during initialization process octave should do setlocale (LC_CTYPE, ""). In libinterp/corefcn/interpreter.cc as the op suggests? Another option is to use main.cc where we have a if-else structure processing different conditions with or without gui.


Taneli Kalvas <tvkalvas>
Thu 17 Nov 2016 05:51:07 AM UTC, comment #2: 

I've written a compiled oct file to test what the current locale is at the C library level, and I've tried running octave-cli with and without the --norc and --no-line-editing option. The only way I get get the LC_CTYPE facet to be uninitialized is with octave-cli --no-line-editing. Is this how you are running Octave?

If Octave runs itself as a QApplication, then the locale is initialized by Qt. If Octave runs with readline enabled, then readline initializes the locale from the environment when it is initialized. It is only in this niche corner case where Octave is run without either Qt or readline that the locale may not be initialized properly before the interpreter runs.

Mike Miller <mtmiller>
Group Member
Thu 17 Nov 2016 05:29:11 AM UTC, comment #1: 

I actually do not get any of these warning messages when I try to reproduce this, I only get the single warning


warning: range error for conversion to character value
warning: called from
    __gnuplot_drawnow__ at line 95 column 16


when I set GNUTERM=dumb. This is addressed by the fscanf vs fread suggestion you made, which seems like a good change.

I only get the other warning messages you've shown if I explicitly set LC_CTYPE=C in my environment before running Octave.

Are you sure your LC_CTYPE facet is not set to C? Are you running octave-cli or octave (either with --no-gui or with the GUI)? This may have a bearing on whether the locale is being initialized correctly or not.

What version of gnuplot are you using?

Mike Miller <mtmiller>
Group Member
Thu 13 Oct 2016 08:14:55 PM UTC, original submission:  

When printing plots containing non-ascii chars to latex files under the gnuplot graphics_toolkit, the resulting latex output is correct, but octave reports a series of false error messages, no matter which locale settings you use.

I am providing a patch solving the source of the problem too, which is spread over several files.

It is not a serious issue, but it is disturbing, and potentially missleading: in fact it made me spend a lot of time trying to see what was wrong, until I realised that nothing but the messages were really wrong.

As a sample, I use the same similar as for bug #49338, except for the graphics_toolkit:


graphics_toolkit gnuplot
plot([-10:10],[-10:10].^2,'o-')
title('Título de la gráfica')
print('-dpdflatexstandalone','-color','test')


When calling octave from the command line to run a script with the above code, octave sends the following to sdterr if we are on a X11 terminal (none of it being present if pure ascii was used):


warning: ft_text_renderer: failed to decode string `Título de la gráfica' with locale `C'
warning: called from
    title at line 53 column 3
    bug2 at line 3 column 1
warning: ft_text_renderer: failed to decode string `Título de la gráfica' with locale `C'
warning: called from
    title at line 53 column 3
    bug2 at line 3 column 1


The same script leads to even more error messages if ran from a pure text terminal:


warning: ft_text_renderer: failed to decode string `Título de la gráfica' with locale `C'
warning: called from
    title at line 53 column 3
    bug2 at line 3 column 1
warning: ft_text_renderer: failed to decode string `Título de la gráfica' with locale `C'
warning: called from
    title at line 53 column 3
    bug2 at line 3 column 1
warning: range error for conversion to character value
warning: called from
    __gnuplot_drawnow__ at line 95 column 16
    print at line 345 column 5
    bug2 at line 4 column 1


On my computer I have LC_ALL=es_ES, but I tried all sort of settings without success. In other words, it seems that octave is currently ignoring the locale settings.

After investigation, I found the source of those false error messages, spread over several files. Here follows a short description of what is wrong:

libinterp/corefcn/ft-text-renderer.cc:

it uses
  std::setlocale (LC_CTYPE, 0));
but it should be
  std::setlocale (LC_CTYPE, ""));

(maybe for some compilers 0 and "" are equivalent, but it doesn't seem to be with mine, gcc 5.3.0).

libinterp/corefcn/interpreter.cc:

Octave is missing to be informed of the current locale before forcing it to
use the "C" locale for numbers (necessary to follow the matlab usage).
In other words, we need to add
     setlocale (LC_CTYPE, "");
before
     setlocale (LC_NUMERIC, "C");
     setlocale (LC_TIME, "C");

scripts/plot/util/__gnuplot_drawnow__.m
scripts/plot/util/print.m

These two files use fscanf(fid, '%c', Inf) to read files containing text; fscanf succeds reading non-ascii chars, but report them to be out of range regardless of locale settings.

Using fread is equally successful, but has the advantage of not reporting out-of-range messages; additionally, it is faster when fscanf is wanted only to read an entire file char by char.

I already corrected print.m via the patch accompanying the description of bug #49338, so I attach two patch files correcting the remaining problems. Recompilation is necessary, as two *.cc source files are slightly modified.

Hope this helps.

Anonymous

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

Attached Files
file #38722:  latin1-cc.patch added by None (1KiB - text/x-patch)
file #38723:  latin1-gnuplot.patch added by None (817B - text/x-patch)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by rik5 (Posted a comment)
  • -email is unavailable- added by sebald (Posted a comment)
  • -email is unavailable- added by tvkalvas (Posted a comment)
  • -email is unavailable- added by mtmiller (Posted a comment)
  • -email is unavailable- added by None (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only group members can vote.

     

    Follow 5 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2024-01-14 rik5 Priority5 - Normal 1 - Later
    2016-11-17 mtmiller StatusNone Need Info
        Release4.2.0-rc2 4.2.0
    2016-10-13 None Attached File- Added latin1-cc.patch, #38722
        Attached File- Added latin1-gnuplot.patch, #38723

    Back to the top

    Powered by Savane 3.13-758e.
    Corresponding source code