Wed 13 Aug 2008 09:34:45 AM UTC, comment #11:
Minor correction:
"абв" is not russian month name, there are three first letters of russian alphabet. But "Авг" is short version of "Август" that is "August". Excuse me for choosing confusing example.
My note was not about handling locale-specific strings, but about UTF-8 strings in general.
Anyway, I'm glad to know that GNU screen is not dead, thank you for investigating this issue.
|
Wed 13 Aug 2008 09:01:17 AM UTC, comment #10:
So, here's the current state of things, for reference, in case I decide not to address this right away.
Typing characters whose Unicode values are under 256 (u+0100) is possible, because when disp_readev_fn calls EncodeChar() on it with the default encoding, it gets mapped to its latin1 equivalent (same code point), which can then be stored in a string.
Typing most characters beyond that range will result in a mapping to '?',
few characters beyond u+0100 get transcoded as an ISO-2022 sequence (halfwidth kana, for example; or ISO 8859-15); this winds up being a worse case for colon-mode input, though, as the resulting embedded ESC character triggers exit of the colon-mode, and the remaining charset identification characters get spewed to the shell instead. That is, try to type or paste a halfwidth kana character in colon-mode, and instead of getting either the character or a '?', you'll get "(IX\033(B" spewed to the shell (where X is actually some arbitrary other character). (That's if pastefont is on.)
.
The problem with month names not being transcoded properly is a wholly separate issue, however. The individual bytes of the strings used for status lines and such are expected by PutWinMsg to represent individual characters. That works great for latin1 chars when they're input in colon-mode, but not for bytes that were copied directly to the buffer from strftime, using whatever locale happens to be current.
The result is that individual bytes of UTF-8 chars are treated as individual characters. The russian month resulted in digits because, as that poster expected, the 0x04 bytes are stripped, as they are detected as Ctrl-D characters.
.
So there are multiple problems:
- Screen needs to be aware of the current encoding when it uses locale-dependant strings, such as the name of the month, and transcode it to the one assumed by PutWinMsg (currently latin1). Perhaps until this problem is resolved, names of days and months should always use the C locale, so sensible data results (otherwise French locales could be made to work, but non-latin ones like Russian could not be).
- Layers using encoding 0, such as the overlay used for colon-command input, are not capable of handling most Unicode characters. Probably, the input overlay (and perhaps others) should use UTF-8 instead.
- PutWinMsg is not set up to handle encodings other than latin1. It should use UTF-8 (this requires the above change, plus changes to anything else that needs to know the encoding for "window message" strings (probably screenrc parsing of commands like "caption").
The underlying sources for most of these problems are architectural, and require risky changes, which I probably won't wish to make before 4.1.0 is released (so screen will most likely continue to be broken in its handling of non-ASCII characters, much the same as it always has been).
|
Tue 12 Aug 2008 10:38:27 AM UTC, comment #9:
Hm, of course, looking back, I'm seeing that my fix isn't for quite the same thing that the reporter mentioned (but rather corresponds to fixing the way another developer reported reproducing a utf8/captions issue).
I've taken care of -t, but decoding from month-names will need a little more work. Guess I'll have to play around with the date to reproduce the originally reported problem.
|
Tue 12 Aug 2008 10:06:39 AM UTC, comment #7:
Addressed for 4.1.0 in 12646e3.
Note that this is only for "simple accented characters"; that is, Unicode characters that screen already decodes properly elsewhere (this patch makes characters that already work in screen via the command prompt, also work via the -t command-line option). The Russian example still fails (though I don't get 012; I get ???).
I'll keep the bug around to track the fuller issue of dealing properly with higher-valued Unicode characters, and punt it for a later release.
|
Mon 11 Dec 2006 09:35:19 PM UTC, original submission:
Hi,
i am using screen 4.0.3 on debian etch and i spotted something like a bug.
I am using locale fr_FR.UTF-8 and all is working great except this :
special Unicode characters are not showing like they should, for example accented characters.
they are displayed like if they was displayed in a ISOxxxx term.
i think that this debian bug : http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=215955
is the same problem.
this is my caption :
caption always "%{= kw}Window%{+b B}(%{W}%n%{B})(%{W}%t%{B})(%{W}%u%?-%{B}) ·%{-b w} Load%{+b B}(%{W}%l%{B}) ·%{-b w} %D %d %M %Y %{+b B}·%{W} %c"
(if you just put %M it is sufficient :)
i am using gnome-terminal
i verified that my configuration was ok with a guy on #screen named dalias.
I hope there is enough information
|