bugGNU TeXmacs - Bugs: bug #36422, bugs in tmml export : invalid xml...


You are not allowed to post comments on this tracker with your current authentication level.

bug #36422: bugs in tmml export : invalid xml (broken tags)

Submitted by:  Philippe Joyez <pjoyez>
Submitted on:  Mon 07 May 2012 08:54:13 AM UTC  
Category:  Conversion Priority:  5 - Normal
Item Group:  Error Status:  In Progress
Privacy:  Public Assigned to:  Miguel de Benito <mdbenito>
Originator Name:  Open/Closed:  Open
Release:  Release:  None
Fixed Release:  None Fixed Release: 
Keywords:  xml tmml

( Jump to the original submission)

Sat 11 Aug 2012 10:33:50 AM UTC, comment #7:

After some toying around with xml_html_parser I found it too complicated:

  • Where should utf8->cork conversion happen? Inside the xml_html_parser?
  • When is strict xml parsing mode activated and should I use this to escape the entities?
    • A few changes were needed to expand_entity() and co. whose side effects I'm not sure of. Regression tests should be made, etc.

So, I just modified a bit my previous patches:

  • I renamed the functions with "cdata" in their name to "chardata".
  • I added more functions for conversion to and from CDATA, now escaping ">" in these sections.

Your example still works and hopefully so should any containing the characters "]]>" inside any CDATA section.

(file #26348)

Miguel de Benito <mdbenito>
Project MemberIn charge of this item.
Sat 11 Aug 2012 08:52:59 AM UTC, comment #6:

A bit more on this "preferred way":

  • I think the parsing of CDATA and CharData should happen inside xml_html_parser (in parsexml.cpp). Specifically:
    • xml_html_parser::parse_until() should not always try to expand all entities, since in a CDATA section only ">" may be escaped.
    • xml_cdata_to_tm() and friends from analyze.cpp should be part of the xml_html_parser if I understand its role correctly: no extra conversions on the data in the tree (as opposed to its structure, which needs further formatting via xmlin) should be necessary after a call to (parse-xml).
  • It seems a bit asymmetrical that xml->tm is done using xml_html_parser but tm->xml is done in tmtmml.scm. Is this ok?
Miguel de Benito <mdbenito>
Project MemberIn charge of this item.
Sat 11 Aug 2012 08:29:07 AM UTC, comment #5:

Hi. Sorry for the incomplete patch. The new one I'm attaching has another function like the one proposed below, this one working and added to the glue, so importing back should work.

The problem is that although everything seems fine (except for the escaping of ">" in CDATA), there is probably some preferred way of dealing with these conversions that I'm not aware of (using all those "converter" dictionaries and classes for instance).

(file #26347)

Miguel de Benito <mdbenito>
Project MemberIn charge of this item.
Sat 11 Aug 2012 07:50:08 AM UTC, comment #4:

Hi Miguel,

Brief testing shows tmml export/import basically works with your patch (apart from re-import of htlm entities, of course). Good job Miguel!

If I want to test your proposed function where should I call it from?

As for adding to the glue, it's not imposible. I proposed a string conversion function recently and it went through.

Philippe Joyez <pjoyez>
Project Member
Tue 31 Jul 2012 08:47:18 PM UTC, comment #3:

I meant we must escape any occurences of ">" inside CDATA, not &. It really sucks comments cannot be edited.

Miguel de Benito <mdbenito>
Project MemberIn charge of this item.
Tue 31 Jul 2012 08:45:01 PM UTC, comment #2:

Hi Philippe,

thanks for the, as usual, very detailed report. Thanks to it I've been to reproduce and locate the problems quite quickly. I have a "solution" for the unescaped <, >, &, so documents won't break so easily. The assumption was made in the code that utf8 is valid as CharData in XML, which as far as I understand is not (W3C).

I've also tried a bit with images, treating raw-data tags especially. Exporting and importing back work, but there are a couple of things missing:

  • The CDATA section corresponding to an image's raw data must have any &'s escaped which occur in ]]>
  • Conversion back from tmml of strings uses utf8->cork which obviously fails for &amp;, &lt; and &gt; I'd propose a new function

which should go into the glue. But every time I did such thing it was reverted or ignored based on the arguments that there was some plan to do it better or that my idea was just bad. Because both are good arguments, I'd rather wait for some comments before I commit anything. In the meantime you can try the attached patch.

(file #26296)

Miguel de Benito <mdbenito>
Project MemberIn charge of this item.
Tue 22 May 2012 01:27:29 PM UTC, comment #1:

As a follow-up on the initial report I have found other cases when the exported tmml file is is not a valid xml structure.

See attached tm file and resulting tmml export.

As I say, the xml is invalid and most programs just refuse to process it. Even texmacs is not able to reload it (blank document).

I bumped into that bug while creating svg images of texmacs content ("copy-to-image" feature), and I was hoping to embed tmml into the svg to allow for re-edition of the initial content.

(file #25903, file #25904)

Philippe Joyez <pjoyez>
Project Member
Mon 07 May 2012 08:54:13 AM UTC, original submission:

in xml the character < is strictly reserved for tags. However, in texmacs if a chunk of text contains < , it is passed untounched in the tmml export instead of being converted to &lt;. This completely breaks xml for any use. Texmacs can't even re-open that file.

Note that part of the mechanism for converting < to &lt; is already implemented since the character & is correctly converted to &amp; when converting to tmml (but not correctly re-imported -- see bug #30801)

Philippe Joyez <pjoyez>
Project Member


Attached Files
file #26348:  bug_36422-3.patch added by mdbenito (12KiB - application/octet-stream)
file #26347:  bug_36422-2.patch added by mdbenito (7KiB - application/octet-stream)
file #26296:  bug36422.patch added by mdbenito (4KiB - application/octet-stream)
file #25903:  test.tm added by pjoyez (47KiB - text/texmacs)
file #25904:  test.tmml added by pjoyez (28KiB - application/octet-stream)


Depends on the following items: None found

Items that depend on this one: None found


Carbon-Copy List
  • -email is unavailable- added by mdbenito (Updated the item)
  • -email is unavailable- added by pjoyez (Submitted the item)

    Do you think this task is very important?
    If so, you can add your encouragement to it.
    This task has 0 encouragements so far.

    Only logged-in users can vote.


    Please enter the title of George Orwell's famous dystopian book (it's a date):



    Follow 10 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    Sat 11 Aug 2012 10:33:50 AM UTC mdbenito Attached File- => Added bug_36422-3.patch, #26348
    Sat 11 Aug 2012 08:29:07 AM UTC mdbenito Attached File- => Added bug_36422-2.patch, #26347
    Wed 08 Aug 2012 02:44:37 PM UTC mdbenito Open/ClosedAccepted => Open
    Tue 31 Jul 2012 08:45:01 PM UTC mdbenito StatusNone => In Progress
        Assigned toNone => mdbenito
        Open/ClosedOpen => Accepted
    Tue 31 Jul 2012 08:45:00 PM UTC mdbenito Attached File- => Added bug36422.patch, #26296
    Tue 22 May 2012 01:27:29 PM UTC pjoyez Attached File- => Added test.tm, #25903
        Attached File- => Added test.tmml, #25904
        Summaryless character in text breaks xml (tmml) => bugs in tmml export : invalid xml (broken tags)

    Back to the top

    Powered by Savane 3.1-cleanup1