bugGNU TeXmacs - Bugs: bug #36346, unexpected copy-verbatim character...

 
 

You are not allowed to post comments on this tracker with your current authentification level.

bug #36346: unexpected copy-verbatim character substitution

Submitted by:  Sam Liddicott <samjam>
Submitted on:  Mon 30 Apr 2012 09:10:58 AM UTC  
 
Category: ConversionPriority: 5 - Normal
Item Group: NoneStatus: In Progress
Privacy: PublicAssigned to: Miguel de Benito <mdbenito>
Originator Name: Open/Closed: Open
Release: Release: None
Fixed Release: NoneFixed Release: 
Keywords: Unicode

(Jump to the original submission Jump to the original submission)

Mon 07 Jan 2013 01:31:37 PM UTC, comment #12:

> Now the back-tick character (` ascii 0x60) in a text document is always pasted as a left quote in texmacs which in turn is pasted back as a back-tick character in the text document. This is true whether I paste normally or paste as verbatim, and whether into a top level document or into a code, verbatim or mode=prog.


I suppose this is not new. Afaik, this behavior is the same from the beginning. This is because, in the texmacs internal encoding (currently T1/Cork), the left quote is coded by a 0x60 character.

> However if I verbatim-copy this back-tick character from a texmacs document and paste to a text editor it renders as ‘ which is unicode 0x2018. I am not certain that this is desirable.


This is also not new. As I said, this is true from a semantic point of view.

> Cut-n-paste from a code/verbatim block into a text mode document is likely to need to happen as verbatim-copy, and it is in this case that preserving the common meaning is very important.


I agree.

> Current behaviour will not preserve meaning: e.g.
>
> ` paste-verbatim unto the document will produce the ascii 0x60 symbol looking like a left-quote. copy-verbatim back to the text document will produce the unicode 0x2018 symbol which is not valid in scripting or source code and not what originated.


I agree. Verbatim copy of a grave accent (ascii 0x60 `) should insert a grave accent into texmacs.

> In normal non-verbatim copy mode it may make sense to copy as unicide 0x2018 as normal copy mode would be used when copying from a document body and so we choose the symbol with the right appearance. Although in such a case I guess the ' character rendered as right quote should be converted to 0x2019 as well. But I note than I cannot enter a literal apostrophe using S-F5 ' in the same way that S-F5 " will enter an ascii quote (ascii 34)


Afaik, in T1/Cork, the apostrophe ' is a typographic apostrophe. A convertion from utf8 to Cork convert the 0x2019 apostrophe into a 0x27 character.

To summarize, currently we have the following conversions :

UTF8 T1/Cork
‘ 0x2018 <---> 0x60 left quote
’ 0x2019 ----> 0x27 apostrophe
' 0x27 <---> 0x27 apostrophe
` 0x60 <---> 0x00 grave accent

It is not a bijection, since a 0x2019 apostrophe is systematically lost in profit of a 0x27 one.

Francois Poulain <fpoulain>
Project Member
Mon 07 Jan 2013 12:49:17 AM UTC, comment #11:

We'll need François' help with the rest, but wrt. to the input of the back-tick `: a kbd-map should solve it. However, the obvious one ("text:symbol `" (insert "\0")) doesn't work for me because of the input method's rewrite for accents (this is addressed in another bug).

Miguel de Benito <mdbenito>
Project MemberIn charge of this item.
Fri 04 Jan 2013 11:36:32 AM UTC, comment #10:

Now the back-tick character (` ascii 0x60) in a text document is always pasted as a left quote in texmacs which in turn is pasted back as a back-tick character in the text document. This is true whether I paste normally or paste as verbatim, and whether into a top level document or into a code, verbatim or mode=prog.

However if I verbatim-copy this back-tick character from a texmacs document and paste to a text editor it renders as ‘ which is unicode 0x2018. I am not certain that this is desirable. (However see comment marked *** below)

Cut-n-paste from a code/verbatim block into a text mode document is likely to need to happen as verbatim-copy, and it is in this case that preserving the common meaning is very important. Current behaviour will not preserve meaning: e.g.

` paste-verbatim unto the document will produce the ascii 0x60 symbol looking like a left-quote. copy-verbatim back to the text document will produce the unicode 0x2018 symbol which is not valid in scripting or source code and not what originated.

I would prefer the other behaviour, that for copy-verbatim the character emitted by the back-tick key and represented by ascii 0x60 within the .tm document should also be represented by ascii 0x60 in the clipboard.

In normal non-verbatim copy mode it may make sense to copy as unicide 0x2018 as normal copy mode would be used when copying from a document body and so we choose the symbol with the right appearance. Although in such a case I guess the ' character rendered as right quote should be converted to 0x2019 as well. But I note than I cannot enter a literal apostrophe using S-F5 ' in the same way that S-F5 " will enter an ascii quote (ascii 34)

Also, I no longer know how to enter the grave accent character that renders with the symbol printed on the key cap of the key on my keyboard. If I edit the tm document with a text editor and insert \0 then I can enter the character, likewise I can paste the character sequence \0 from a text editor to obtain the grace accent character.

This grave character may be cut-n-pasted all over the document and keeps its represented form as the grave accent.

****
If I copy from texmacs and paste into a text editor it pastes \0, if I copy-verbatim it appears in a text editor as ` which I would be fine as it COULD mean that the proper symbol as shown in a console would be verbatim-paste into a text editor; but there is NO WAY to paste the symbol back from the text editor, it appears as the left quote instead of the grave accent; and further there is no way I know to enter that symbol from the keyboard into a verbatim/code block.

Finally, if I copy the character from http://www.unicodemap.org/details/0x2018/index.html it appears in a text editor as ‘ but if I paste into texmacs it is rendered as a ?, even with paste-from verbatim. Pasting other charcaters after that causes the ? to turn upside down.

So I think there are few problems.

1. there needs to be a way to type the grave character 0x60 (I note that within .tm, 0x60 means the left-quote 0x2019, and \0 means the grave character).
Maybe a keymap can cause the ` key to insert the right symbol in code/verbatim/mode=prog but this is a small matter.

2. if (1) is done, at least in verbatim export and verbatim-copy we should map \0 -> 0x60 and 0x60 -> 0x2019 (and reverse for verbatim import, verbatim-paste)

3. if (1) is NOT done (current) and ascii 0x60 is to be used (instead of \0) in code/verbatim/mode=prog to represent the symbol emitted by the back-tick key (grave symbol) then:
a. MAYBE it could be rendered as the grave symbol in those environments
b. it should certainly remain as 0x60 when cut-n-paste in verbatim mode and only mapped to 0x2018 0x2019 for non-verbatim cut-n-paste.

and perhaps any ` -> 0x2018 mapping should be joined with comparable mapping for ' -> 0x2019 but that would also need a different representation for a true apostrophe which is actually 0x27 which I think cannot be currently represented in a .tm document separately from the right quote.

This is confusing and I think the confusion comes from trying to represent the meaning a human attaches to the key-cap characters within the document while preserving the ascii symbols the keys generate; combined with a history of character wilful substitution due to font shortcomings as well as taking advantage of homoglyphs oferred by some specific fonts.

I think we should identify .tm representations for double quote (ascii 34) apostrophe (ascii 0x60) and left-single-quote, right-single-quote.
We should then specify how those characters should be input in the different modes, and how they should be converted in the different cut-paste modes.

I prefer that in code/verbatim/mode=prog that whatever symbols are inserted should have the same appearance as in a traditional coding environment; and that when cut-n-paste verbatim they should also have the traditional ascii codes that are generated by the same keys. When left,right quote are inserted into these modes cut-n-paste-verbatim should use the proper unicode representation. This permits cut-n-paste to preserve meaning in a text editor for scripts and source code as well as supporting those symbols in unicode strings (if the compiler supports it).

Finally; while I have talked about mapping according to if copy and paste mode is verbatim mode or not, maybe it should be according to whether or not a section is a verbatim/code/mode=prog section. But that is talk for another day

Sam Liddicott <samjam>
Project Member
Fri 04 Jan 2013 10:00:22 AM UTC, comment #9:

> the backquote key (0x60 in ascii) insert a left quotation mark in texmacs (0x60 in T1/Cork, as far as I understand (at least, it is understood as this inside texmacs))


A simple test to confirm this, since I didn't found any documentation. Consider the .tex file:

Compile it in .dvi or .ps. The result is coded via the « `' » sequence.

Francois Poulain <fpoulain>
Project Member
Thu 03 Jan 2013 02:13:01 PM UTC, comment #8:

In fact:

  • the backquote key (0x60 in ascii) insert a left quotation mark in texmacs (0x60 in T1/Cork, as far as I understand (at least, it is understood as this inside texmacs)).
  • The true "semantic conversion" (made with the verbatim converter) imply to preserve the "left quotation mark" symbol, so we convert it into an unicode 0x2018 character.
  • For source code, (made with the SourceCode converter) it is munged into an ascii 0x60 character (grave accent). This is "munging", from the semantic point of view. But from the "data flow" point of view, this is only conservation. :)

It may be relevant to change "scheme source" in favor of "code source" in the GUI.

Also, AFAIK, it is the only one char in the range 0x20 - 0x7E wich has a different semantic meaning between T1/Cork and Ascii.

Francois Poulain <fpoulain>
Project Member
Mon 31 Dec 2012 12:45:07 AM UTC, comment #7:

We fixed this during the summer, introducing a new "scheme-source" encoding. Copying and pasting of quotes and backquotes should work just fine now. Please reopen if not. If in doubt, ask our resident expert in encodings François. ;-)

Miguel de Benito <mdbenito>
Project MemberIn charge of this item.
Thu 31 May 2012 11:18:36 AM UTC, comment #6:

I note that I am not yet competent with keymap files to cause that the ` key will emit the character currently represented by \0 when in verbatim mode or code mode.

Certainly there is high risk that a left-single-quote could be cut-n-pasted from a non-code block into a code block giving rise to the problem I had of two different looking back-quote characters in code blocks.

One might like this if one was pasting a unicode string but not if one were pasting a piece of shell script.

Sam Liddicott <samjam>
Project Member
Thu 31 May 2012 11:14:27 AM UTC, comment #5:

It is not clear if the problem is really that the ` key on the keyboard is inserting the wrong character in the document.

I am sure there is also history here, to do with mis-representation of characters in fonts.

Is the ` key supposed to insert a left single quotation mark?
or is it supposed to insert the back-tick character which on some old SUN fonts appeared to be a left single quotation mark and which some old-timers may insist is truly a left single quotation mark.

In the .tm file the symbol from the ` key is the ` value, and so TeXmacs is following that old tradition of using ` to represent left single quote, because it renders it as a left single quote.

TeXmacs uses \0 to represent the actual back-tick character and presents it as a back-tick character on screen. Certainly this correctly re-produces the bad asymmetric quoting styles used for older man pages that looks like this: ``quote item''

I think in non-code non-verbatim mode, the character of the right appearance is being inserted.

I think that in verbatim mode and code mode, the ` key should insert the other character which texmacs represents as \0 which is the cork encoding for the grave accent.

Therefore, I think that the copy-paste behaviour for ` is correct, the fix will be to generate the right symbol in code or verbatim mode.

I do think that TeXmacs is using the wrong character representations in it's .tm format. I don't see that it can't use the ` symbol to represent the character traditionally produced by the ` key, and use some other symbol ("#2018) to represent left single quote. However it is probably too late to change that until texmacs moves from cork to unicode.

I'm not sure why cork \0 is being used to represent the back tick (also called back quote) unless it is because it looks the same for some fonts.

Here are the comments made by David Allouche who covered this (so very well) before:
http://lists.gnu.org/archive/html/texmacs-dev/2003-07/msg00038.html

I did raise this with Joris over a year ago but the decisions seemed not to have been his.
http://permalink.gmane.org/gmane.editors.texmacs.user/7354

(I note that the copy/paste transformations done on left single quote producing unicode 0xe2 0x80 0x98 renders quite appallingly in my terminal font, but quite well if pasted into open office).

Sam Liddicott <samjam>
Project Member
Wed 16 May 2012 10:09:46 AM UTC, comment #4:

Aaah... the joys of comments in the code...

Any progress? Did you try changing corktounicode.scm ?

Miguel de Benito <mdbenito>
Project MemberIn charge of this item.
Tue 08 May 2012 03:58:55 PM UTC, comment #3:

It may perhaps be that the bug is not in the cut-n-paste conversion but that the wrong symbol is inserted when the ` key is pressed on the keyboard.

Perhaps the symbol inserted by the ` key should be the one represented internally as \0

It's appearance certainly matches more closely the symbol on the key.

Perhaps \0 should be inserted when in code or verbatim mode and the current symbol (open-single-quote) entered otherwise.

That mapping was inserted in 2003 by Joris. The commit notes just say: " This commit was generated by cvs2svn to compensate for changes in r2, which included commits to RCS files with non-trunk default branches."

Sam Liddicott <samjam>
Project Member
Tue 08 May 2012 03:50:03 PM UTC, comment #2:

0xe2 0x80 0x98 is UTF-8 encoding for unicode character 'LEFT SINGLE QUOTATION MARK' (U+2018)

The substitution is defined at line 108 of TeXmacs/langs/encoding/corktounicode.scm:

("#5E" "#5E")
("#5F" "#5F")
("#60" "#2018")
("#61" "#61")
("#62" "#62")

(conversions defined in this file are two-ways). Obviously that particular character receives a special treatment and it would have been nice to find an explanation for this exception... I can only be sure that changing it will certainly break something elsewhere!

Can someone explain the rationale behind this and how to fix it?

Philippe Joyez <pjoyez>
Project Member
Mon 07 May 2012 09:12:27 AM UTC, comment #1:

I confirm the behavior, however I think it's more a spurious conversion of the character than an encoding problem.

If after a "copy-to-verbatim", you "Paste-from-verbatim" , the wierd character is converted back to a backtick in texmacs.

Alternatively if you do a simple copy, you get the correct backtick in the editor. but then you also copy unwanted formatting characters that texmacs uses, which is annoying.

This should be corrected.

Philippe Joyez <pjoyez>
Project Member
Mon 30 Apr 2012 09:10:58 AM UTC, original submission:

When the backtick character is copied, using the clipboard, with "Copy to verbatim" it is copied from this character:

` which is 0x60

to this character:

‘ which is 0xe2 0x80 0x98

This is unexpected, and makes cutting and pasting shell scripts impossible.

Why is the character converted, thus not being verbatim after all?

Sam Liddicott <samjam>
Project Member

 

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -unavailable- added by mdbenito (Posted a comment)
  • -unavailable- added by pjoyez (Posted a comment)
  • -unavailable- added by samjam (Submitted the item)
  •  

    Do you think this task is very important?
    If so, you can click here to add your encouragement to it.
    This task has 0 encouragements so far.

    Only logged-in users can vote.

     

    Please enter the title of George Orwell's famous dystopian book (it's a date):

     

     

    Follow 8 latest changes.

    Date Changed By Updated Field Previous Value => Replaced By
    Mon 07 Jan 2013 12:49:25 AM UTCmdbenitoOpen/ClosedClosed=>Open
    Mon 07 Jan 2013 12:49:17 AM UTCmdbenitoStatusFixed=>In Progress
    Mon 31 Dec 2012 12:45:07 AM UTCmdbenitoStatusConfirmed=>Fixed
      Assigned toNone=>mdbenito
      Open/ClosedOpen=>Closed
    Tue 08 May 2012 03:50:03 PM UTCpjoyezKeywords=>Unicode
      Summaryunexpected copy-verbatim encoding=>unexpected copy-verbatim character substitution
    Mon 07 May 2012 09:12:27 AM UTCpjoyezStatusNone=>Confirmed

    Back to the top


    Powered by Savane 3.1-cleanup