bugGNU Screen - Bugs: bug #18505, problem with caption and utf-8

 
 

bug #18505: problem with caption and utf-8

Submitted by:  kaouete <kaouete>
Submitted on:  Mon 11 Dec 2006 09:35:19 PM UTC  
 
Category: User InterfaceSeverity: 3 - Normal
Priority: 7 - HighStatus: Fixed
Privacy: PublicAssigned to: Micah Cowan <micahcowan>
Open/Closed: OpenRelease: None
Fixed Release: NonePlanned Release: Later
Work Required: 1 - Days

Add a New Comment(Rich Markup)
   

You are not logged in

Please log in, so followups can be emailed to you.

 

(Jump to the original submission Jump to the original submission)

Mon 30 Nov 2009 09:50:00 PM UTC, comment #12:

The display issue is fixed with a2f13ec78529f8e374a8af55d3aa415c2404222e. So, for example, '%M' will show up correctly with ru_RU.UTF-8.

Keeping it open until the issue with taking utf-8 inputs in COMMAND is fixed.

Sadrul Habib Chowdhury <sadrul>
Project Administrator
Wed 13 Aug 2008 09:34:45 AM UTC, comment #11:

Minor correction:
"абв" is not russian month name, there are three first letters of russian alphabet. But "Авг" is short version of "Август" that is "August". Excuse me for choosing confusing example.

My note was not about handling locale-specific strings, but about UTF-8 strings in general.

Anyway, I'm glad to know that GNU screen is not dead, thank you for investigating this issue.

Leonid Evdokimov <darkk>
Wed 13 Aug 2008 09:01:17 AM UTC, comment #10:

So, here's the current state of things, for reference, in case I decide not to address this right away.

Typing characters whose Unicode values are under 256 (u+0100) is possible, because when disp_readev_fn calls EncodeChar() on it with the default encoding, it gets mapped to its latin1 equivalent (same code point), which can then be stored in a string.

Typing most characters beyond that range will result in a mapping to '?',

few characters beyond u+0100 get transcoded as an ISO-2022 sequence (halfwidth kana, for example; or ISO 8859-15); this winds up being a worse case for colon-mode input, though, as the resulting embedded ESC character triggers exit of the colon-mode, and the remaining charset identification characters get spewed to the shell instead. That is, try to type or paste a halfwidth kana character in colon-mode, and instead of getting either the character or a '?', you'll get "(IX\033(B" spewed to the shell (where X is actually some arbitrary other character). (That's if pastefont is on.)

.

The problem with month names not being transcoded properly is a wholly separate issue, however. The individual bytes of the strings used for status lines and such are expected by PutWinMsg to represent individual characters. That works great for latin1 chars when they're input in colon-mode, but not for bytes that were copied directly to the buffer from strftime, using whatever locale happens to be current.

The result is that individual bytes of UTF-8 chars are treated as individual characters. The russian month resulted in digits because, as that poster expected, the 0x04 bytes are stripped, as they are detected as Ctrl-D characters.

.

So there are multiple problems:

- Screen needs to be aware of the current encoding when it uses locale-dependant strings, such as the name of the month, and transcode it to the one assumed by PutWinMsg (currently latin1). Perhaps until this problem is resolved, names of days and months should always use the C locale, so sensible data results (otherwise French locales could be made to work, but non-latin ones like Russian could not be).
- Layers using encoding 0, such as the overlay used for colon-command input, are not capable of handling most Unicode characters. Probably, the input overlay (and perhaps others) should use UTF-8 instead.
- PutWinMsg is not set up to handle encodings other than latin1. It should use UTF-8 (this requires the above change, plus changes to anything else that needs to know the encoding for "window message" strings (probably screenrc parsing of commands like "caption").

The underlying sources for most of these problems are architectural, and require risky changes, which I probably won't wish to make before 4.1.0 is released (so screen will most likely continue to be broken in its handling of non-ASCII characters, much the same as it always has been).

Micah Cowan <micahcowan>In charge of this item.
Tue 12 Aug 2008 10:38:27 AM UTC, comment #9:

Hm, of course, looking back, I'm seeing that my fix isn't for quite the same thing that the reporter mentioned (but rather corresponds to fixing the way another developer reported reproducing a utf8/captions issue).

I've taken care of -t, but decoding from month-names will need a little more work. Guess I'll have to play around with the date to reproduce the originally reported problem.

Micah Cowan <micahcowan>In charge of this item.
Tue 12 Aug 2008 10:31:10 AM UTC, comment #8:

I get "???" when I change title with ^Aa, if I change it with
$ echo -ne "\033kабв\033\\"
then I get "012".

Leonid Evdokimov <darkk>
Tue 12 Aug 2008 10:06:39 AM UTC, comment #7:

Addressed for 4.1.0 in 12646e3.

Note that this is only for "simple accented characters"; that is, Unicode characters that screen already decodes properly elsewhere (this patch makes characters that already work in screen via the command prompt, also work via the -t command-line option). The Russian example still fails (though I don't get 012; I get ???).

I'll keep the bug around to track the fuller issue of dealing properly with higher-valued Unicode characters, and punt it for a later release.

Micah Cowan <micahcowan>In charge of this item.
Tue 12 Aug 2008 09:11:51 AM UTC, comment #6:

The fix for this will also address the following crash: If you start screen with "-t foo", and then change the shelltitle within screen using the shelltitle command, screen attempts to free argv space, and crashes.

Micah Cowan <micahcowan>In charge of this item.
Sun 10 Aug 2008 01:30:04 AM UTC, comment #5:

Couldn't reproduce with C-a A, but could with the -t option.

Micah Cowan <micahcowan>In charge of this item.
Sun 20 Jul 2008 05:34:22 PM UTC, comment #4:

I can reproduce this bug too.

Interestingly, if I start screen with 'LC_ALL=C screen' (without the -U flag), then the utf-8 captions work correctly. (You might want to reset LC_ALL to your preferred utf-8 locale within the windows afterwards).

I am not that familiar with dealing with encodings. So I am not sure how to fix the bug.

Sadrul Habib Chowdhury <sadrul>
Project Administrator
Thu 19 Jun 2008 10:07:20 AM UTC, comment #3:

I confirm too, I checked it with ru_RU.UTF-8.

Both "абв" (u'\u0430\u0431\u0432') and "ذбв" (u'\u0630\u0431\u0432') are converted to "012", I assume only the last byte of UTF-8 symbol is taken and other bytes are thrown away.

Leonid Evdokimov <darkk>
Sun 15 Jun 2008 03:07:13 PM UTC, comment #2:

I confirm :)

Anonymous
 Spam posted by an anonymous
Mon 05 May 2008 01:35:53 PM UTC, comment #1:

https://savannah.gnu.org/bugs/index.php?16210 looks like the same bug.

Leonid Evdokimov <darkk>
 Spam posted by an anonymous
Mon 11 Dec 2006 09:35:19 PM UTC, original submission:

Hi,

i am using screen 4.0.3 on debian etch and i spotted something like a bug.

I am using locale fr_FR.UTF-8 and all is working great except this :
special Unicode characters are not showing like they should, for example accented characters.

they are displayed like if they was displayed in a ISOxxxx term.

i think that this debian bug : http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=215955
is the same problem.

this is my caption :
caption always "%{= kw}Window%{+b B}(%{W}%n%{B})(%{W}%t%{B})(%{W}%u%?-%{B}) ·%{-b w} Load%{+b B}(%{W}%l%{B}) ·%{-b w} %D %d %M %Y %{+b B}·%{W} %c"

(if you just put %M it is sufficient :)

i am using gnome-terminal

i verified that my configuration was ok with a guy on #screen named dalias.

I hope there is enough information

kaouete <kaouete>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach File(s):
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -unavailable- added by sadrul (Posted a comment)
  • -unavailable- added by micahcowan (Updated the item)
  • -unavailable- added by darkk (Posted a comment)
  • -unavailable- added by kaouete (Submitted the item)
  •  

    Do you think this task is very important?
    If so, you can click here to add your encouragement to it.
    This task has 0 encouragements so far.

    Only logged-in users can vote.

     

    Please enter the title of George Orwell's famous dystopian book (it's a date):

     

     

    Follow 16 latest changes.

    Date Changed By Updated Field Previous Value => Replaced By
    Mon 30 Nov 2009 09:50:00 PM UTCsadrulStatusConfirmed=>Fixed
    Wed 13 Aug 2008 09:01:17 AM UTCmicahcowanPriority5 - Normal=>7 - High
      StatusIn Progress=>Confirmed
      Planned Release4.1.0=>Later
    Tue 12 Aug 2008 10:38:27 AM UTCmicahcowanStatusConfirmed=>In Progress
      Planned ReleaseLater=>4.1.0
      Work Required0 - Hours=>1 - Days
    Tue 12 Aug 2008 10:06:39 AM UTCmicahcowanStatusIn Progress=>Confirmed
      Planned Release4.1.0=>Later
    Tue 12 Aug 2008 09:10:03 AM UTCmicahcowanStatusNeed Investigation=>In Progress
      Assigned toNone=>micahcowan
    Tue 24 Jun 2008 12:06:13 AM UTCmicahcowanWork RequiredNone=>0 - Hours
    Mon 23 Jun 2008 09:58:05 PM UTCmicahcowanCategoryNone=>User Interface
      Planned ReleaseNone=>4.1.0
    Sun 15 Jun 2008 07:34:56 AM UTCmicahcowanStatusNone=>Need Investigation
    Mon 22 Jan 2007 05:29:33 PM UTCnagaCarbon-Copy-=>Added naga

    Back to the top


    Powered by Savane 3.1-cleanup1