bugGNU Screen - Bugs: bug #55138, Bug: UTF-8 characters in...

 
 

bug #55138: Bug: UTF-8 characters in hardstatus not working

Submitted by:  Elof <elof>
Submitted on:  Fri 30 Nov 2018 11:59:03 AM UTC  
 
Category:  User Interface Severity:  3 - Normal
Priority:  5 - Normal Status:  Duplicate
Privacy:  Public Assigned to:  None
Open/Closed:  Open Release:  4.6.2
Fixed Release:  None Planned Release:  None
Work Required:  None

Add a New Comment (Rich Markup)
   

You are not logged in

Please log in, so followups can be emailed to you.

 

( Jump to the original submission)

Thu 23 May 2019 08:38:19 AM UTC, comment #18: 

Part of the problem seems to be that FromUtf8() interprets UTF8 bytes incorrectly due to a signedness issue.
For example: ¢ (U+A2)
This 2-byte UTF8 character gets interpreted as
ffffffc2
ffffffa2
instead of
c2
a2

A dirty hack to demonstrate this would be:

-static int strlen_onscreen(char *c, char *end)
+static int strlen_onscreen(unsigned char *c, unsigned char *end)

Stefan Assmann <sassmann>
Thu 21 Feb 2019 09:31:28 PM UTC, comment #17: 

I had a look to see if I can come with a temporary solution… but I am afraid any change will break something else…

Nicolas Schodet <schodet>
Mon 18 Feb 2019 08:53:57 PM UTC, comment #16: 

Just a response to Nicolas,

Unfortunetly I need the window title dynamic in the hardstatus line, so it can't be statically assigned.

(and the window titles themselves must be in UTF-8, otherwise the problem is just moved from the hardstatus line to the window list)

Elof <elof>
Mon 18 Feb 2019 08:36:00 PM UTC, comment #15: 

Handle everything using characters instead of byte is the sane way, for sure!

Note there can still be problems with combining chars, clusters, or emoticons.

To Elof, I noticed that I do not have the problem if I define the string in .screenrc, so the problem must lie in the interactive prompt which seems to convert unicode to "latin-1".

urxvt seems to be happy with latin-1 codes, which is weird!

Nicolas Schodet <schodet>
Sun 17 Feb 2019 11:06:45 PM UTC, comment #14: 

It's well known deficiency, there is quite few similar bug reports:
24208 hardcopy doesn't handle Unicode.
36172 hardstatus utf-8 padding
39330 Cannot :title ♬
44366 hardstatus lastline UTF-8 corruption
52296 Some unicode characters in hardstatus corrupt screen + hardstatus

There are things that can be done.
1. massage current encoding handling into hardstatus
2. rewrite whole encoding handling from scratch (noting where inputs come from, and where outputs are)

Personally I prefer option 2, because currently due to legacy reasons, we keep encoded character in 4 variables (image, font, fontx, mbcs), while we could just use one uin32_t (/uct4_t) and decode all input to utf-32 and then encode it to output format. In fact I already tried following this path, using libunistring helpers (https://github.com/amade/screen/tree/encoding-rewrite), but it got bit messy and I started thinking that maybe screen should have its own set of encoding handling functions like it has now, but rewritten from scratch.

Still it requires plugging a lot of input/output handling into correct places. There is quite few input points as well as output.

input points are:

  • disp_readev_fn - display input function (user input)
  • win_readev_fn - everything we read from windows (program output)
  • Input - input prompt (confirmation etc.)
  • config file - .screenrc
  • all variables read from command line
  • getenv

output points are:

  • disp_writeev_fn - display output function (screen output - composite of

program output)

  • win_writeev_fn - everything we write to windows (user input - passed to

running program)

  • logfiles
  • setenv
  • hardcopy

and probably few that I missed, but became obvious when you start working on it.

Although now that I look again at work that I did, maybe it's not that bad, could probably use rebasing on latest code, some cleanup and then fixing rest of unfinished conversions and maybe rethinking some places where we convert stuff if it can be done in better way.

Amadeusz Sławiński <amade>
Project Administrator
Sun 17 Feb 2019 11:01:33 PM UTC, comment #13: 

OK, I just tried with xterm and I can see the problem… weird…

Now I can debug this, thanks!

Nicolas Schodet <schodet>
Sun 17 Feb 2019 10:33:27 PM UTC, comment #12: 

Both outside and inside screen, 'echo å | xxd' result in:
00000000: c3a5 0a           ...

Yes, in Debian 9 the default screen is 4.5.0 but since it has the hardstatus problem I compiled 4.6.2 to see if it was fixed.

Do you mean that you can't reproduce the problem if you set the hardstatus string to "räksmörgås"?

It is reproduceable on all my machines and even when I manually compile the latest version (4.6.2).

Elof <elof>
Sun 17 Feb 2019 09:34:29 PM UTC, comment #11: 

Thanks for your tests.

Can you run without running screen:

echo å | xxd

Where does your screen come from, I have debian 9 here and it's screen 4.5.0.

Can you run:

dpkg -l screen

Thanks,

sorry for so many questions, I just want to be able to reproduce the problem on my machine.

Nicolas Schodet <schodet>
Sun 17 Feb 2019 02:03:25 AM UTC, comment #10: 

Attachments 2 and 3 were swapped in my last post, so pic 3 show the hardstatus string "å".

Those three examples were from screen v4.5.0 (Debian 8).

Continuing with the tests on screen 4.6.2 (Debian 9):

Pic 1 show the same problem as in 2.5.0. I'm also running with -U.

Pic 2 show 'screen -U' with locale.

Unfortunetly I don't have rxvt, but in all my terminal windows (xterm on Debian 8, xterm on Debian 9, mintty in Cygwin in Windows), I have the same problem, so I don't think the terminal is the problem.

PS: As you can see, åäö is working fine within the window itself. It is only when printed in the hardstatus line things go wrong.


Elof <elof>
Sun 17 Feb 2019 01:53:20 AM UTC, comment #9: 

Hi!

Pic 1 show how everything is looking great as long as there's no åäö in the statusline.

Pic 2 show the hardstatus string "å"

Pic 3 show how the contents of the screen is moved one line up every time the clock in the right corner is updated.


Elof <elof>
Fri 15 Feb 2019 09:37:56 PM UTC, comment #8: 

Hej!

I am not sure I was clear enough, can you try this (using "^A :" to enter command):

hardstatus string "å"

And show us a screenshoot?

Can you try from a fresh screen started with -U?

Can you send the output of the "locale" command?

Can you try using a different terminal (rxvt-unicode)?

Thanks,

Nicolas.

Nicolas Schodet <schodet>
Thu 14 Feb 2019 10:42:12 PM UTC, comment #7: 

Hi Nicolas!

Yes, confirmed. As long as I use plain us-ascii chars, everything is OK everywhere. In screen windows, in the screen window-list and in the hardstatus line.

When I use Swedish characters (like: Räksmörgås), they are displayed correctly in screen windows and in the screen window-list. But the hardstatus line is now messed up.

...my current workaround is to manually exchange all 'åäö' characters with 'aao'. That way my windows don't get cluttered with "ANSI garbage" from a broken hardstatus line...

This is annoying, so if you can fix the problem I'd be really happy. :-)

/Elof

Elof <elof>
Thu 14 Feb 2019 10:11:07 PM UTC, comment #6: 

Hello,

On my system, I can not fully reproduce the problem, with screen 4.5.0, 4.6.2, nor with master branch, the character are always printed correctly.

However, I can see a problem with the padding computation (%=) on the v4 branch, and a problem with width computation on the master branch.

Can you confirm that there is no problem if you set the hardstatus to a single character with 4.6.2?

Nicolas Schodet <schodet>
Fri 28 Dec 2018 09:12:02 PM UTC, comment #5: 

No takers on this?

99.9% of screen is already UTF-8 compatible, we just need that last 0.1% fixed. :-)

Elof <elof>
Sun 02 Dec 2018 12:08:34 PM UTC, comment #4: 

I just managed to patch the 4.6.2 source.
Unfortunetly, neither of the two patches below solved my problem.
Both of them gave the same result:

Before:
Swedish chars in the statusline are shown as grey rectangular blocks.

After:
Swedish chars in the statusline are shown as Japanese chars.

So, like Mike says:
I would be grateful if someone else is able to work on this.

Elof <elof>
Sat 01 Dec 2018 02:46:26 AM UTC, comment #3: 

...or this fork made 29 Dec 2016:

https://gist.github.com/yoshikaw/10586719

Elof <elof>
Sat 01 Dec 2018 02:34:58 AM UTC, comment #2: 

Hi Mike. Sorry to hear that.

This old patch doesn't just happen to work?
https://gist.github.com/saitoha/3143922

I'm no developer, so I don't know how to patch the source and try it out myself, but maybe you could just throw 20 minutes on it? It would be really nice if hardstatus support international characters, especially since they already work fine everywhere else in screen.

Elof <elof>
Sat 01 Dec 2018 01:37:30 AM UTC, comment #1: 

Indeed, this is something that I had set out to fix a number of years back.  I did a bunch of refactoring to clean up the hardstatus code, but didn't have time to get back to work on the UTF-8 support, and don't anticipate to have the time to do so any time soon.

Screen assumes each character is one byte.

I would be grateful if someone else was able to work on this.

Mike Gerwitz <mikegerwitz>
Fri 30 Nov 2018 11:59:03 AM UTC, original submission:  

I have utf-8 problems in hardstatus even with the latest screen 4.6.2 from 2017.

# screen --version
Screen version 4.06.02 (GNU) 23-Oct-17

How to reproduce:
I press <ctrl-a + A> and change the window title to include a Swedish utf-8 character (like 'å' (an 'a' with a ring above it)).
If I now press <ctrl-a + "> to show the window-list, everything looks good. I see the Swedish title correct. :-)
In the screen windows themselves (the terminal windows), my UTF-8 characters work perfectly fine. :-)

Everything is working as intended as long as I don't have a hardstatus line.

If I let the hardstatus string print the window titles:

  hardstatus on
  hardstatus alwayslastline
  hardstatus string '%{=b wK}  %n  %{Kw}  %t  %=@ %H  %{wk}  %D %M %d  %c  '

...then the last line will show a garbage character instead of the Swedish 'å' character.
Also there will be some internal miscalculation of the string length, resulting in incorrect background coloring on the last lines.

In my terminal I see that the 'å' character is a two-byte-char:

# echo -n a | wc -c
1
# echo -n å | wc -c
2

(My terminal = xterm, running on Debian 9)

/Elof

Elof <elof>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

Attached Files
file #46291:  screen462 2 -U locale.png added by elof (54KiB - image/png)
file #46287:  screen450 1 before.png added by elof (48KiB - image/png - Screenshots from screen 4.5.0)
file #46288:  screen450 3 ticking clock.png added by elof (73KiB - image/png - Screenshots from screen 4.5.0)
file #46289:  screen450 2 after.png added by elof (54KiB - image/png - Screenshots from screen 4.5.0)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by sassmann (Posted a comment)
  • -email is unavailable- added by amade (Posted a comment)
  • -email is unavailable- added by schodet (Posted a comment)
  • -email is unavailable- added by mikegerwitz (Posted a comment)
  • -email is unavailable- added by elof (Submitted the item)
  •  

    Do you think this task is very important?
    If so, you can add your encouragement to it.
    This task has 0 encouragements so far.

    Only logged-in users can vote.

     

     

     

    Follow 6 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2019-02-17 amade StatusNone => Duplicate
    2019-02-17 elof Attached File- => Added screen462 1 -U ticking clock.png, #46290
        Attached File- => Added screen462 2 -U locale.png, #46291
    2019-02-17 elof Attached File- => Added screen450 1 before.png, #46287
        Attached File- => Added screen450 3 ticking clock.png, #46288
        Attached File- => Added screen450 2 after.png, #46289

    Back to the top


    Powered by Savane 3.5