bugGNU TeXmacs - Bugs: bug #36422, bugs in tmml export : invalid xml...

 
 

You are not allowed to post comments on this tracker with your current authentication level.

bug #36422: bugs in tmml export : invalid xml (broken tags)

Submitter:  Philippe Joyez <pjoyez>
Submitted:  Mon 07 May 2012 08:54:13 AM UTC
   
 
Category:  Conversion Priority:  5 - Normal
Item Group:  Error Status:  Duplicate
Privacy:  Public Assigned to:  sadhen
Originator Name:  Open/Closed:  Closed
Release:  None Release: 
Fixed Release:  None Fixed Release: 
Keywords:  xml tmml

Jump to the original submission

Sun 01 Nov 2020 06:04:05 PM UTC, comment #11: 

If my understanding is right, the fix was applied in revision 12234.  So I assume that this specific bug has been fixed.

Please use https://savannah.gnu.org/bugs/?57118
for further pending issues concerning the tmml format.

Joris van der Hoeven <vdhoeven>
Group administrator
Sun 20 Oct 2019 11:37:19 AM UTC, comment #10: 

comment #8:

> What is the status of this now?
> A lot of it seems to be fixed in recent releases.
>
> I noticed something that is still broken in 1.99.11:
>
> <.f> in .tm (e.g. a user-defined macro) produces <.f/> in .tmml but the dot is not a NameStartChar so it is not well-formed and XML processors refuse to load the XML document. This is in contrast to <&f>, which is correctly escaped to <_38_f/>.
>


I've fixed https://savannah.gnu.org/bugs/?30801 via https://github.com/texmacs/texmacs/commit/e388e43ffd7bd42796d23734cd3f19c94901f372 .

For other bugs related to TMML, I've planned to find it and fix it.

Thanks for your informative comments.


Darcy Shen <sadhen>
Group Member
Sat 19 Oct 2019 06:25:08 PM UTC, comment #9: 

I think that Darcy took care of pretty much issues indeed.
I just fixed the '.' problem.
I reassign the bug to Darcy for further testing.
Please let me know if you think that I can close it.

Joris van der Hoeven <vdhoeven>
Group administrator
Wed 09 Oct 2019 12:16:15 PM UTC, comment #8: 

What is the status of this now?
A lot of it seems to be fixed in recent releases.

I noticed something that is still broken in 1.99.11:

<.f> in .tm (e.g. a user-defined macro) produces <.f/> in .tmml but the dot is not a NameStartChar so it is not well-formed and XML processors refuse to load the XML document. This is in contrast to <&f>, which is correctly escaped to <_38_f/>.

Rossen Mikhov <log65536>
Sat 11 Aug 2012 10:33:50 AM UTC, comment #7: 

After some toying around with xml_html_parser I found it too complicated:

  • Where should utf8->cork conversion happen? Inside the xml_html_parser?
  • When is strict xml parsing mode activated and should I use this to escape the entities?
    • A few changes were needed to expand_entity() and co. whose side effects I'm not sure of. Regression tests should be made, etc.


So, I just modified a bit my previous patches:

  • I renamed the functions with "cdata" in their name to "chardata".
  • I added more functions for conversion to and from CDATA, now escaping ">" in these sections.


Your example still works and hopefully so should any containing the characters "]]>" inside any CDATA section.



(file #26348)

Miguel de Benito <mdbenito>
Group Member
Sat 11 Aug 2012 08:52:59 AM UTC, comment #6: 

A bit more on this "preferred way":

  • I think the parsing of CDATA and CharData should happen inside xml_html_parser (in parsexml.cpp). Specifically:
    • xml_html_parser::parse_until() should not always try to expand all entities, since in a CDATA section only ">" may be escaped.
    • xml_cdata_to_tm() and friends from analyze.cpp should be part of the xml_html_parser if I understand its role correctly: no extra conversions on the data in the tree (as opposed to its structure, which needs further formatting via xmlin) should be necessary after a call to (parse-xml).
  • It seems a bit asymmetrical that xml->tm is done using xml_html_parser but tm->xml is done in tmtmml.scm. Is this ok?


Miguel de Benito <mdbenito>
Group Member
Sat 11 Aug 2012 08:29:07 AM UTC, comment #5: 

Hi. Sorry for the incomplete patch. The new one I'm attaching has another function like the one proposed below, this one working and added to the glue, so importing back should work.

The problem is that although everything seems fine (except for the escaping of ">" in CDATA), there is probably some preferred way of dealing with these conversions that I'm not aware of (using all those "converter" dictionaries and classes for instance).


(file #26347)

Miguel de Benito <mdbenito>
Group Member
Sat 11 Aug 2012 07:50:08 AM UTC, comment #4: 

Hi Miguel,

Brief testing shows tmml export/import basically works with your patch (apart from re-import of htlm entities, of course). Good job Miguel!

If I want to test your proposed function where should I call it from?

As for adding to the glue, it's not imposible. I proposed a string conversion function recently and it went through.


Philippe Joyez <pjoyez>
Group Member
Tue 31 Jul 2012 08:47:18 PM UTC, comment #3: 

I meant we must escape any occurences of ">" inside CDATA, not &. It really sucks comments cannot be edited.

Miguel de Benito <mdbenito>
Group Member
Tue 31 Jul 2012 08:45:01 PM UTC, comment #2: 

Hi Philippe,

thanks for the, as usual, very detailed report. Thanks to it I've been to reproduce and locate the problems quite quickly. I have a "solution" for the unescaped <, >, &, so documents won't break so easily. The assumption was made in the code that utf8 is valid as CharData in XML, which as far as I understand is not (W3C).

I've also tried a bit with images, treating raw-data tags especially. Exporting and importing back work, but there are a couple of things missing:

  • The CDATA section corresponding to an image's raw data must have any &'s escaped which occur in ]]>
  • Conversion back from tmml of strings uses utf8->cork which obviously fails for &amp;, &lt; and &gt; I'd propose a new function



string
xml_cdata_to_tm (string s) {
  string r;
  int i, n= N(s);
  for (i=0; i<n; i++)
    if (s[i] != '&') r << s[i];
    else {
      int start= ++i;
      while ((i<n) && (s[i]!=';')) i++;
      string x= "<" * xml_name_to_tm (s (start, i)) * ">";
      r << x;
    }
  return r;
}


which should go into the glue. But every time I did such thing it was reverted or ignored based on the arguments that there was some plan to do it better or that my idea was just bad. Because both are good arguments, I'd rather wait for some comments before I commit anything. In the meantime you can try the attached patch.

(file #26296)

Miguel de Benito <mdbenito>
Group Member
Tue 22 May 2012 01:27:29 PM UTC, comment #1: 

As a follow-up on the initial report I have found other cases when the exported tmml file is is not a valid xml structure.

See attached tm file and resulting tmml export.

As I say, the xml is invalid and most programs just refuse to process it. Even texmacs is not able to reload it (blank document).

I bumped into that bug while creating svg images of texmacs content ("copy-to-image" feature), and I was hoping to embed tmml into the svg to allow for re-edition of the initial content.

(file #25903, file #25904)

Philippe Joyez <pjoyez>
Group Member
Mon 07 May 2012 08:54:13 AM UTC, original submission:  


in xml the character < is strictly reserved for tags. However, in texmacs if a chunk of text contains < , it is passed untounched in the tmml export instead of being converted to &lt;. This completely breaks xml for any use. Texmacs can't even re-open that file.

Note that part of the mechanism for converting < to &lt; is already implemented since the character & is correctly converted to &amp; when converting to tmml (but not correctly re-imported -- see bug #30801)

Philippe Joyez <pjoyez>
Group Member

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attached Files
file #26348:  bug_36422-3.patch added by mdbenito (12KiB - application/octet-stream)
file #26347:  bug_36422-2.patch added by mdbenito (7KiB - application/octet-stream)
file #26296:  bug36422.patch added by mdbenito (4KiB - application/octet-stream)
file #25903:  test.tm added by pjoyez (47KiB - text/texmacs)
file #25904:  test.tmml added by pjoyez (28KiB - application/octet-stream)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by sadhen (Posted a comment)
  • -email is unavailable- added by vdhoeven (Posted a comment)
  • -email is unavailable- added by log65536 (Posted a comment)
  • -email is unavailable- added by mdbenito (Updated the item)
  • -email is unavailable- added by pjoyez (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

     

    Follow 13 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2020-11-01 vdhoeven StatusIn Progress Duplicate
        Open/ClosedOpen Closed
    2019-10-19 vdhoeven Assigned tomdbenito sadhen
    2012-08-11 mdbenito Attached File- Added bug_36422-3.patch, #26348
    2012-08-11 mdbenito Attached File- Added bug_36422-2.patch, #26347
    2012-08-08 mdbenito Open/ClosedAccepted Open
    2012-07-31 mdbenito StatusNone In Progress
        Assigned toNone mdbenito
        Open/ClosedOpen Accepted
    2012-07-31 mdbenito Attached File- Added bug36422.patch, #26296
    2012-05-22 pjoyez Attached File- Added test.tm, #25903
        Attached File- Added test.tmml, #25904
        Summaryless character in text breaks xml (tmml) bugs in tmml export : invalid xml (broken tags)

    Back to the top

    Powered by Savane 3.13-cf05.
    Corresponding source code