bugGNU Octave - Bugs: bug #39831, New Tex interpreter requires...

 
 

bug #39831: New Tex interpreter requires delimiters around characters

Submitter:  Rik <rik5>
Submitted:  Tue 20 Aug 2013 10:44:43 PM UTC
   
 
Category:  Plotting Severity:  3 - Normal
Priority:  5 - Normal Item Group:  Matlab Compatibility
Status:  Fixed Assigned to:  goffioul
Originator Name:  Open/Closed:  * Closed
Release:  * dev Operating System:  * GNU/Linux
Fixed Release:  None Planned Release:  None
* Mandatory Fields

Add a New Comment Rich Markup
   

Jump to the original submission

Sat 24 Aug 2013 06:29:02 PM UTC, comment #10: 

Symbol lookup (outside the lexer) is not even necessary, as long as the lexer rules and the symbol array are generated from the same file. The following changeset solves the problem:

http://hg.savannah.gnu.org/hgweb/octave/rev/fe6518a1d87c

Michael Goffioul <goffioul>
Sat 24 Aug 2013 12:45:27 AM UTC, comment #9: 

Michael Goffioul,

Yes, I realized after I sent my first comment that you would probably have to list all the symbol names in the flex input file.  OTOH, you could just gperf to look up symbol definitions in the text engine function that is currently using a linear search.  For now, I'd say leave it as a linear search.  It's more important to me that we not have to maintain two separate lists, so generating these bits of code from a single file seems like the right thing to do.  If you just make it use a linear search now, I could probably modify the script that generates the code to use gperf without much trouble later on.

John W. Eaton <jwe>
Group administrator
Fri 23 Aug 2013 11:02:01 PM UTC, comment #8: 

Michael:
This was more a private joke targeted to me. I overlooked the problem of "\musec", otherwise I wouldn't have implemented the way it is at the moment. I should be more careful.

Rik:
I'm 100% aware of the inefficiency of the symbol lookup at the moment. This was not intended to be a final solution. The JHandles equivalent used a lazy-built map. But now that you raised the issue of "\musec", the symbol lookup will be moved to the lexer anyway.

John:
I admit I don't know gperf, but my understanding is that it can create an efficient hash table. However in this case, the problem is that the lexer must break "\musec" into 2 symbols. So even with using gperf, I'd have to check the hash table on every symbol character read to see whether it corresponds to a known symbol.

Everything considered, I've the impression that the easiest way to handle that is to hardcode the symbols into the lexer and let flex handle it. My plan is to have a text file with the symbol name and the corresponding font codes, and post-process the file to generate: 1) flex rules, 2) symbol->code mapping for the text renderer.

Michael Goffioul <goffioul>
Fri 23 Aug 2013 11:01:39 PM UTC, comment #7: 

If you decide to put the list of symbols directly in the lexer, it would be nice to avoid duplicating the list in both the text engine and the lexer.  I would favor putting the list of all symbol info in a separate file and generate the required code for the flex input file and the text engine using a script.  If it would help, I'll do this part.

John W. Eaton <jwe>
Group administrator
Fri 23 Aug 2013 06:36:57 PM UTC, comment #6: 

You could use gperf, same as we do for Octave keywords.

John W. Eaton <jwe>
Group administrator
Fri 23 Aug 2013 06:34:46 PM UTC, comment #5: 

Adding all the symbols to the lexer is somewhat annoying, but the list is very static and the work would only need to be done once.

As a side benefit, we currently do a linear search through the 101 symbols to go from symbol name to Unicode character set.  Computers are fast these days so this isn't exactly horrible, but it is kind of inefficient.  I had thought about using an STL map for this operation to reduce this to log2(101) operations.  But another possibility would be to assign lexer token ID for each symbol in such a way that we can just directly look up the unicode value using "symbol_codes[LEXER_ID]" in txt-eng.cc.

Rik <rik5>
Group administrator
Fri 23 Aug 2013 05:54:07 PM UTC, comment #4: 

I agree pretty much with what you said.  But TeX and
LaTeX are two different languages.  Since I use TeX
I think of it as relevant.  But, since the "tex"
interpreter is, as you point out, not really TeX,
compatibility is a minor point.

And, about " I prefer to get things right the first time :).",
the only person who has come close to that is Knuth.  I prefer
not to consider trying to match his skiils.  So, I just
try, in the end, to get it right.  The number of trys is
not important. The end result is.


Michael Godfrey <godfrey>
Group Member
Fri 23 Aug 2013 04:43:03 PM UTC, comment #3: 

There's no strong argument against it. It's just an annoyance in the sense I've to make significant changes to the lexer/parser, and I prefer to get things right the first time :).

Anyway, some other people also wants their Matlab script to work in Octave without modifications. In my opinion, the so-called "tex" interpreter is not deemed to be TeX-compatible, it's more a TeX-like/-similar syntax. It's intended to provide basic math support, without any external dependencies (which makes it faster). If you want better TeX compatibility, you should use the "latex" interpreter instead.

Michael Goffioul <goffioul>
Fri 23 Aug 2013 03:50:16 PM UTC, comment #2: 

Matlab does accept \musec.  Definitely an annoyance, and,
of course, TeX does not allow this.  It may make sense to
try harder for TeX compatibility than for Matlab.  Some
people would like the strings they give to Octave for
tex interpreter to be valid TeX.

Putting that aside, what is the argument against a table
of valid symbols? The table would be easy to maintain and
it should help to make sure it matches the documentation.
But, it should be your choice.

Michael Godfrey <godfrey>
Group Member
Fri 23 Aug 2013 02:12:35 PM UTC, comment #1: 

This is an annoyance. This means the lexer must be aware of the available symbols and break "\musec" accordingly.

My first idea would then be to hardcode all symbols directly in the lexer. That is, instead of having a generic lexer rule like:

\\[a-zA-Z]+

I must list all possible symbols.

Any better idea is welcome.

Michael Goffioul <goffioul>
Tue 20 Aug 2013 10:44:43 PM UTC, original submission:  

According to the Matlab documentation the following code should work.

Sample Code


text (0.5, 0.5, '\musec');


On Octave, this produces an error unless '\mu' is delimited by some meaningful character.


text (0.5, 0.5, '{\mu}sec');
text (0.5, 0.5, '\mu sec');
text (0.5, 0.5, '\mu\beta');


Rik <rik5>
Group administrator

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by jwe (Posted a comment)
  • -email is unavailable- added by godfrey (Posted a comment)
  • -email is unavailable- added by rik5 (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only group members can vote.

     

    Follow 2 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2013-08-24 goffioul StatusNone Fixed
        Open/ClosedOpen Closed

    Back to the top

    Powered by Savane 3.13-02a9.
    Corresponding source code