Part of the problem seems to be that FromUtf8() interprets UTF8 bytes incorrectly due to a signedness issue.
For example: ¢ (U+A2)
This 2-byte UTF8 character gets interpreted as
ffffffc2
ffffffa2
instead of
c2
a2

A dirty hack to demonstrate this would be:

-static int strlen_onscreen(char *c, char *end)
+static int strlen_onscreen(unsigned char *c, unsigned char *end)

Just a response to Nicolas,

Unfortunetly I need the window title dynamic in the hardstatus line, so it can't be statically assigned.

(and the window titles themselves must be in UTF-8, otherwise the problem is just moved from the hardstatus line to the window list)

It's well known deficiency, there is quite few similar bug reports:
24208 hardcopy doesn't handle Unicode.
36172 hardstatus utf-8 padding
39330 Cannot :title ♬
44366 hardstatus lastline UTF-8 corruption
52296 Some unicode characters in hardstatus corrupt screen + hardstatus

There are things that can be done.
1. massage current encoding handling into hardstatus
2. rewrite whole encoding handling from scratch (noting where inputs come from, and where outputs are)

Personally I prefer option 2, because currently due to legacy reasons, we keep encoded character in 4 variables (image, font, fontx, mbcs), while we could just use one uin32_t (/uct4_t) and decode all input to utf-32 and then encode it to output format. In fact I already tried following this path, using libunistring helpers (https://github.com/amade/screen/tree/encoding-rewrite), but it got bit messy and I started thinking that maybe screen should have its own set of encoding handling functions like it has now, but rewritten from scratch.

Still it requires plugging a lot of input/output handling into correct places. There is quite few input points as well as output.

input points are:

• disp_readev_fn - display input function (user input)
• win_readev_fn - everything we read from windows (program output)
• Input - input prompt (confirmation etc.)
• config file - .screenrc
• all variables read from command line
• getenv

output points are:

• disp_writeev_fn - display output function (screen output - composite of

program output)

• win_writeev_fn - everything we write to windows (user input - passed to

running program)

• logfiles
• setenv
• hardcopy

and probably few that I missed, but became obvious when you start working on it.

Although now that I look again at work that I did, maybe it's not that bad, could probably use rebasing on latest code, some cleanup and then fixing rest of unfinished conversions and maybe rethinking some places where we convert stuff if it can be done in better way.

Both outside and inside screen, 'echo å | xxd' result in:
00000000: c3a5 0a           ...

Yes, in Debian 9 the default screen is 4.5.0 but since it has the hardstatus problem I compiled 4.6.2 to see if it was fixed.


Do you mean that you can't reproduce the problem if you set the hardstatus string to "räksmörgås"?

It is reproduceable on all my machines and even when I manually compile the latest version (4.6.2).

Attachments 2 and 3 were swapped in my last post, so pic 3 show the hardstatus string "å".

Those three examples were from screen v4.5.0 (Debian 8).



Continuing with the tests on screen 4.6.2 (Debian 9):

Pic 1 show the same problem as in 2.5.0. I'm also running with -U.

Pic 2 show 'screen -U' with locale.

Unfortunetly I don't have rxvt, but in all my terminal windows (xterm on Debian 8, xterm on Debian 9, mintty in Cygwin in Windows), I have the same problem, so I don't think the terminal is the problem.

PS: As you can see, åäö is working fine within the window itself. It is only when printed in the hardstatus line things go wrong.

Hej!

I am not sure I was clear enough, can you try this (using "^A :" to enter command):

hardstatus string "å"

And show us a screenshoot?

Can you try from a fresh screen started with -U?

Can you send the output of the "locale" command?

Can you try using a different terminal (rxvt-unicode)?

Thanks,

Nicolas.

Hello,

On my system, I can not fully reproduce the problem, with screen 4.5.0, 4.6.2, nor with master branch, the character are always printed correctly.

However, I can see a problem with the padding computation (%=) on the v4 branch, and a problem with width computation on the master branch.

Can you confirm that there is no problem if you set the hardstatus to a single character with 4.6.2?

I just managed to patch the 4.6.2 source.
Unfortunetly, neither of the two patches below solved my problem.
Both of them gave the same result:

Before:
Swedish chars in the statusline are shown as grey rectangular blocks.

After:
Swedish chars in the statusline are shown as Japanese chars.



So, like Mike says:
I would be grateful if someone else is able to work on this.

Hi Mike. Sorry to hear that.

This old patch doesn't just happen to work?
https://gist.github.com/saitoha/3143922

I'm no developer, so I don't know how to patch the source and try it out myself, but maybe you could just throw 20 minutes on it? It would be really nice if hardstatus support international characters, especially since they already work fine everywhere else in screen.

I have utf-8 problems in hardstatus even with the latest screen 4.6.2 from 2017.

# screen --version
Screen version 4.06.02 (GNU) 23-Oct-17

How to reproduce:
I press <ctrl-a + A> and change the window title to include a Swedish utf-8 character (like 'å' (an 'a' with a ring above it)).
If I now press <ctrl-a + "> to show the window-list, everything looks good. I see the Swedish title correct. :-)
In the screen windows themselves (the terminal windows), my UTF-8 characters work perfectly fine. :-)

Everything is working as intended as long as I don't have a hardstatus line.

If I let the hardstatus string print the window titles:

  hardstatus on
  hardstatus alwayslastline
  hardstatus string '%{=b wK}  %n  %{Kw}  %t  %=@ %H  %{wk}  %D %M %d  %c  '

...then the last line will show a garbage character instead of the Swedish 'å' character.
Also there will be some internal miscalculation of the string length, resulting in incorrect background coloring on the last lines.

In my terminal I see that the 'å' character is a two-byte-char:

# echo -n a | wc -c
1
# echo -n å | wc -c
2

(My terminal = xterm, running on Debian 9)

/Elof