bugGNU Screen - Bugs: bug #18505, problem with caption and utf-8

 
 

bug #18505: problem with caption and utf-8

Submitter:  kaouete <kaouete>
Submitted:  Mon 11 Dec 2006 09:35:19 PM UTC
   
 
Category:  User Interface Severity:  3 - Normal
Priority:  * 7 - High Status:  Fixed
Privacy:  Public Assigned to:  micahcowan
Open/Closed:  Open Release:  None
Fixed Release:  None Planned Release:  Later
Work Required:  1 - Days
* Mandatory Fields

Add a New Comment Rich Markup
   

Jump to the original submission

Mon 30 Nov 2009 09:50:00 PM UTC, comment #14: 

The display issue is fixed with a2f13ec78529f8e374a8af55d3aa415c2404222e. So, for example, '%M' will show up correctly with ru_RU.UTF-8.

Keeping it open until the issue with taking utf-8 inputs in COMMAND is fixed.

Sadrul Habib Chowdhury <sadrul>
Group administrator
Wed 13 Aug 2008 09:34:45 AM UTC, comment #13: 

Minor correction:
"абв" is not russian month name, there are three first letters of russian alphabet. But "Авг" is short version of "Август" that is "August". Excuse me for choosing confusing example.

My note was not about handling locale-specific strings, but about UTF-8 strings in general.

Anyway, I'm glad to know that GNU screen is not dead, thank you for investigating this issue.

Leonid Evdokimov <darkk>
Wed 13 Aug 2008 09:01:17 AM UTC, comment #12: 

So, here's the current state of things, for reference, in case I decide not to address this right away.

Typing characters whose Unicode values are under 256 (u+0100) is possible, because when disp_readev_fn calls EncodeChar() on it with the default encoding, it gets mapped to its latin1 equivalent (same code point), which can then be stored in a string.

Typing most characters beyond that range will result in a mapping to '?',

few characters beyond u+0100 get transcoded as an ISO-2022 sequence (halfwidth kana, for example; or ISO 8859-15); this winds up being a worse case for colon-mode input, though, as the resulting embedded ESC character triggers exit of the colon-mode, and the remaining charset identification characters get spewed to the shell instead. That is, try to type or paste a halfwidth kana character in colon-mode, and instead of getting either the character or a '?', you'll get "(IX\033(B" spewed to the shell (where X is actually some arbitrary other character).  (That's if pastefont is on.)

.

The problem with month names not being transcoded properly is a wholly separate issue, however. The individual bytes of the strings used for status lines and such are expected by PutWinMsg to represent individual characters. That works great for latin1 chars when they're input in colon-mode, but not for bytes that were copied directly to the buffer from strftime, using whatever locale happens to be current.

The result is that individual bytes of UTF-8 chars are treated as individual characters. The russian month resulted in digits because, as that poster expected, the 0x04 bytes are stripped, as they are detected as Ctrl-D characters.

.

So there are multiple problems:

 - Screen needs to be aware of the current encoding when it uses locale-dependant strings, such as the name of the month, and transcode it to the one assumed by PutWinMsg (currently latin1). Perhaps until this problem is resolved, names of days and months should always use the C locale, so sensible data results (otherwise French locales could be made to work, but non-latin ones like Russian could not be).
 - Layers using encoding 0, such as the overlay used for colon-command input, are not capable of handling most Unicode characters. Probably, the input overlay (and perhaps others) should use UTF-8 instead.
 - PutWinMsg is not set up to handle encodings other than latin1. It should use UTF-8 (this requires the above change, plus changes to anything else that needs to know the encoding for "window message" strings (probably screenrc parsing of commands like "caption").

The underlying sources for most of these problems are architectural, and require risky changes, which I probably won't wish to make before 4.1.0 is released (so screen will most likely continue to be broken in its handling of non-ASCII characters, much the same as it always has been).

Micah Cowan <micahcowan>
Tue 12 Aug 2008 10:38:27 AM UTC, comment #11: 

Hm, of course, looking back, I'm seeing that my fix isn't for quite the same thing that the reporter mentioned (but rather corresponds to fixing the way another developer reported reproducing a utf8/captions issue).

I've taken care of -t, but decoding from month-names will need a little more work. Guess I'll have to play around with the date to reproduce the originally reported problem.

Micah Cowan <micahcowan>
Tue 12 Aug 2008 10:31:10 AM UTC, comment #10: 

I get "???" when I change title with ^Aa, if I change it with
$ echo -ne "\033kабв\033\\"
then I get "012".

Leonid Evdokimov <darkk>
Tue 12 Aug 2008 10:06:39 AM UTC, comment #9: 

Addressed for 4.1.0 in 12646e3.

Note that this is only for "simple accented characters"; that is, Unicode characters that screen already decodes properly elsewhere (this patch makes characters that already work in screen via the command prompt, also work via the -t command-line option). The Russian example still fails (though I don't get 012; I get ???).

I'll keep the bug around to track the fuller issue of dealing properly with higher-valued Unicode characters, and punt it for a later release.

Micah Cowan <micahcowan>
Tue 12 Aug 2008 09:11:51 AM UTC, comment #8: 

The fix for this will also address the following crash: If you start screen with "-t foo", and then change the shelltitle within screen using the shelltitle command, screen attempts to free argv space, and crashes.

Micah Cowan <micahcowan>
Sun 10 Aug 2008 01:30:04 AM UTC, comment #7: 

Couldn't reproduce with C-a A, but could with the -t option.

Micah Cowan <micahcowan>
Sun 20 Jul 2008 05:34:22 PM UTC, comment #6: 

I can reproduce this bug too.

Interestingly, if I start screen with 'LC_ALL=C screen' (without the -U flag), then the utf-8 captions work correctly. (You might want to reset LC_ALL to your preferred utf-8 locale within the windows afterwards).

I am not that familiar with dealing with encodings. So I am not sure how to fix the bug.

Sadrul Habib Chowdhury <sadrul>
Group administrator
Thu 19 Jun 2008 10:07:20 AM UTC, comment #5: 

I confirm too, I checked it with ru_RU.UTF-8.

Both "абв" (u'\u0430\u0431\u0432') and "ذбв" (u'\u0630\u0431\u0432') are converted to "012", I assume only the last byte of UTF-8 symbol is taken and other bytes are thrown away.

Leonid Evdokimov <darkk>
Sun 15 Jun 2008 03:07:13 PM UTC, comment #4: 

I confirm :)

Anonymous
  Spam posted by anonymous
Mon 05 May 2008 01:35:53 PM UTC, comment #2: 
Leonid Evdokimov <darkk>
  Spam posted by anonymous
Mon 11 Dec 2006 09:35:19 PM UTC, original submission:  

Hi,

i am using screen 4.0.3 on debian etch and i spotted something like a bug.

I am using locale fr_FR.UTF-8 and all is working great except this :
special Unicode characters are not showing like they should, for example accented characters.

they are displayed like if they was displayed in a ISOxxxx term.

i think that this debian bug : http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=215955
is the same problem.

this is my caption :
caption always "%{= kw}Window%{+b B}(%{W}%n%{B})(%{W}%t%{B})(%{W}%u%?-%{B}) ·%{-b w} Load%{+b B}(%{W}%l%{B}) ·%{-b w} %D %d %M %Y %{+b B}·%{W} %c"

(if you just put %M it is sufficient :)

i am using gnome-terminal

i verified that my configuration was ok with a guy on #screen named dalias.

I hope there is enough information

kaouete <kaouete>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by sadrul (Posted a comment)
  • -email is unavailable- added by micahcowan (Updated the item)
  • -email is unavailable- added by darkk (Posted a comment)
  • -email is unavailable- added by kaouete (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only logged-in users can vote.

     

    Follow 16 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2009-11-30 sadrul StatusConfirmed Fixed
    2008-08-13 micahcowan Priority5 - Normal 7 - High
        StatusIn Progress Confirmed
        Planned Release4.1.0 Later
    2008-08-12 micahcowan StatusConfirmed In Progress
        Planned ReleaseLater 4.1.0
        Work Required0 - Hours 1 - Days
    2008-08-12 micahcowan StatusIn Progress Confirmed
        Planned Release4.1.0 Later
    2008-08-12 micahcowan StatusNeed Investigation In Progress
        Assigned toNone micahcowan
    2008-06-24 micahcowan Work RequiredNone 0 - Hours
    2008-06-23 micahcowan CategoryNone User Interface
        Planned ReleaseNone 4.1.0
    2008-06-15 micahcowan StatusNone Need Investigation
    2007-01-22 naga Carbon-Copy- Added naga

    Back to the top

    Powered by Savane 3.13-f8d8.
    Corresponding source code