bugGNU Screen - Bugs: bug #51890, screen randomly injects \b into...

 
 

bug #51890: screen randomly injects \b into UTF8 streams when processing combining characters

Submitter:  Mike Frysinger <vapier>
Submitted:  Tue 29 Aug 2017 09:50:37 PM UTC
   
 
Category:  None Severity:  3 - Normal
Priority:  * 5 - Normal Status:  Ready For Test
Privacy:  Public Assigned to:  None
Open/Closed:  Open Release:  4.5.0
Fixed Release:  None Planned Release:  None
Work Required:  None
* Mandatory Fields

Add a New Comment Rich Markup
   

Jump to the original submission

Thu 03 Feb 2022 12:33:40 AM UTC, comment #6: 

Same output in screen and without screen:

$ printf 'xA\U0000030Ax\n'
xÅx
$ printf 'xA\U0000030Ax\n' | hexdump
0000000 4178 8acc 0a78
0000006


Tested in xterm, Xfce Terminal and Terminology.

Please test it again. It seems fixed in 4.9.0.

Alexander Naumov <anaumov>
Group administrator
Thu 04 Jan 2018 10:34:58 AM UTC, comment #5: 

it doesn't cause a bug, but it is wasteful, especially when using a setup that is supposed to be "utf-8 sane".  i basically noticed screen emitting extraneous content when debugging terminal emulator code and i wasn't getting the exact content my program was writing (e.g. i told it to write two codepoints, but i was getting back four, and i wanted to know why my program was "broken").

Mike Frysinger <vapier>
Sun 26 Nov 2017 01:40:48 PM UTC, comment #4: 

Mike, does this behavior cause any actual bug? Looks to me that it should be harmless.

My wild guess is:

It could be related to https://bugzilla.gnome.org/show_bug.cgi?id=673981 bullet point 3, and the varying behavior of terminal emulators in that case.

When screen sees the first "A", it decides that it needs to print that, and does so.

When it encounters the combining accent, it updates its internal concept about the screen contents: it's no longer an A but an A with some combining accent.

Maybe it totally forgets that there was already an "A" printed that could be reused. Or maybe it still recalls that there's an "A" there, but didn't keep track whether the cursor was perhaps moved away (e.g. to update something on a different pane) and then moved back, according to the bullet point 3 of the aforementoned bug. Either case, it cannot be sure that simply printing the combining accent will properly go over the corresponding base character, so decides to reprint the character.

So it's not necessarily an explicit '\b' for this purpose somewhere in the source, maybe it's just the simplest way of moving the cursor to that desired place.

screen should, however, probably consume the entire input in a single step, and then update the canvas in another single one, in which case this trick wouldn't be necessary.

All these are just a random guess from me, without knowing anything about screen's internals. I just guess that the real explanation might be somewhere along these lines.

Egmont Koblinger <egmont>
Wed 20 Sep 2017 08:48:46 AM UTC, comment #3: 

Is this system is "weird" in any way? Some unusual kernel/libc?

I don't recall anything that would cause inserting '\b' bytes into a stream...

but it sounds bit similar to https://savannah.gnu.org/bugs/?31336

Amadeusz Sławiński <amade>
Group administrator
Sun 03 Sep 2017 11:29:25 PM UTC, comment #2: 

i tried a few diff terminal emulators and observed the same behavior

on a completely different system i tried screen 4.6.1 and it did not insert \b into the stream.  not sure if it's because of the upgrade or because of some other system aspect.

i guess if you know of a change that went in to change this behavior, it's fine to punt the bug.

Mike Frysinger <vapier>
Wed 30 Aug 2017 07:39:37 AM UTC, comment #1: 

Hi Mike,
thanks for bugreport.

Do you use xterm?
Can you try to reproduce it in 4.6.1 please?

The output of your example of printf-string looks same for me in xterm and in screen.

Alexander Naumov <anaumov>
Group administrator
Tue 29 Aug 2017 09:50:37 PM UTC, original submission:  

simple example:
  printf 'xA\U0000030Ax\n'

that will write out the UTF-8 byte stream (hexdump view):
  78 41 cc 8a 78 0a  |xA..x.|

when i'm not using screen, the terminal emulator sees that exactly.  however, screen will read that and then mangle it, passing along:
  xA\bA\U0000030Ax\n
  78 41 08 41 cc 8a 78 0a  |xA.A..x.|

a proper terminal emulator is able to deal with this.  but the question still stands: why is it doing this ?  i couldn't locate the logic in the screen source though.

poking it through strace shows the screen process doing the read() on the pty master (/dev/ptmx) and getting the correct UTF-8 stream, then doing a write on its slave pty with the mangled stream.  so it doesn't seem like it's an external-to-screen mangling.

my locale is set to en_US.UTF8, screen was launched with -U, and .screenrc has:
  defutf8 on
  defencoding utf8
using screen 4.05.00

since the whole pipeline is UTF-8 aware, i can't explain why screen would need to interject these things.  i might understand if it was dealing with some semi-broken systems where it tried to get slightly better output, but that doesn't apply here.

the odd thing is that when screen dumps lines from its history (e.g. when you attach or otherwise scrollback), it doesn't inject the \b logic.  only for new content.

noticed originally with a bit more pathological line:
1.001.01a अ॒ग्निमी॑ळे पु॒रोहि॑तं य॒ज्ञस्य॑ दे॒वमृ॒त्विज॑म् ।
that inserts a number of \b (all around combining chars?  didn't look super close).

Mike Frysinger <vapier>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by egmont (Posted a comment)
  • -email is unavailable- added by amade (Posted a comment)
  • -email is unavailable- added by anaumov (Posted a comment)
  • -email is unavailable- added by vapier (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only logged-in users can vote.

     

    Follows 1 latest change.

    Date Changed by Updated Field Previous Value => Replaced by
    2022-02-03 anaumov StatusNone Ready For Test

    Back to the top

    Powered by Savane 3.13-f8d8.
    Corresponding source code