bugGNU Zile (Zile is Lossy Emacs) - Bugs: bug #33908, function keys do not work

 
 

bug #33908: function keys do not work

Submitted by:  Gary V. Vaughan <gary>
Submitted on:  Sun 31 Jul 2011 01:24:38 PM UTC  
 
Category: CSeverity: 3 - Normal
Status: FixedAssigned to: None
Open/Closed: ClosedRelease: None

Add a New Comment(Rich Markup)
   

You are not logged in

Please log in, so followups can be emailed to you.

 

(Jump to the original submission Jump to the original submission)

Wed 17 Aug 2011 01:41:26 PM UTC, comment #23:

I am now satisfied this issue can be closed; could you close it if you agree, Gary?

Reuben Thomas <rrt>
Project Administrator
Thu 11 Aug 2011 11:34:31 AM UTC, comment #22:

I'm rewriting the FAQ to update its advice, and also basically inverting the order of the questions (swapping sections 1 & 2, and reversing the questions in the new section 1) to better reflect the frequency with which questions come up.

Reuben Thomas <rrt>
Project Administrator
Thu 11 Aug 2011 11:32:56 AM UTC, comment #21:

Gah! blush

http://lists.gnu.org/archive/html/bug-zile/2011-08/msg00037.html

Note to self: The computer doesn't update the paste buffer when you just look at the right page in your browser, you have to select the url and cut first!!

Gary V. Vaughan <gary>
Project Administrator
Thu 11 Aug 2011 11:19:09 AM UTC, comment #20:

Dude, that's the same one again!

Reuben Thomas <rrt>
Project Administrator
Thu 11 Aug 2011 11:18:19 AM UTC, comment #19:

That'll teach me to look at the URL for the last archived message, count the number of messages I've sent since and try to figure out what URL an as yet unarchived message will get...

Of course I should have typed:

http://lists.gnu.org/archive/html/bug-zile/2011-08/msg00036.html

Gary V. Vaughan <gary>
Project Administrator
Thu 11 Aug 2011 11:00:50 AM UTC, comment #18:

As I said, I can repro #33907, just not this one. Just to check: I am pressing Fn+F1 to get F1, and it pops up:

<f1>-

in the minibuffer (as one would expect, or at least hope!).

But as you say, I'm happy to close this in the light of the TERM setting solution, if that works for both of us.

The only other thing I can think of: are you building against a different ncurses from the one shipped with Lion? Lion's is old (5.4) but perhaps patched by Apple...

Finally, the list message you mention:

http://lists.gnu.org/archive/html/bug-zile/2011-08/msg00036.html

doesn't seem to have anything to do with this bug.

Reuben Thomas <rrt>
Project Administrator
Thu 11 Aug 2011 09:29:14 AM UTC, comment #17:

How strange that you can't reproduce!

My Lion installation is only a couple of weeks old, although I did import all the settings from my old Snow Leopard machine. So I just created a brand new user to login to. I confirm that the standard terminal setting here is xterm-color256.

I still have no problem reproducing this bug and #33907 - C-h produces a BACKSPACE (per #33907) and F1 elicits a beep when a stream of xterm F1 chars is fired at ncurses instead of the expected vt100 chars described in the xterm-color256 terminfo entry.

However, all of this is moot in light of being able to make things work for both of us with TERM=nsterm, which I've now set as my default terminal type in Terminal.app settings.

Before we close this bug and #33907, I'd like to implement something along the lines of http://lists.gnu.org/archive/html/bug-zile/2011-08/msg00036.html and to remove my earlier char-stream editing patch from Lua Zile.

(Yes it does suck that we can't send email replies to this BTS!)

Gary V. Vaughan <gary>
Project Administrator
Wed 10 Aug 2011 11:55:08 PM UTC, comment #16:

I also confirm that the function keys work OK if I set TERM=nsterm.

Reuben Thomas <rrt>
Project Administrator
Wed 10 Aug 2011 11:21:02 PM UTC, comment #15:

So, I just tried to repro this bug, and I can't. On a machine running Lion I installed XCode 4.1, and, using brew, libgc 7.1, man2html and git2cl, and then checked out git master and built it.

I added a line to .zile that bound F2 to goto-line.

I run zile, and press Fn+F2, and lo! it runs goto-line.

My TERM setting seems to be xterm-256color. I have not changed any defaults on this machine.

Help?

Reuben Thomas <rrt>
Project Administrator
Wed 10 Aug 2011 07:38:47 PM UTC, comment #14:

On 9 August 2011 06:13, Gary V. Vaughan <INVALID.NOREPLY@gnu.org> wrote:

> Follow-up Comment #13, bug #33908 (project zile):
>
> Here's a better explanation of the function key issues with a fresh
> installation of C Zile running on Apple's Terminal.app:
>
> 1. Terminal.app on Mac OS 10.7 and 10.6 sets TERM to xterm-color by
> default;
> 2. Similarly it sends vt100 compatible escape sequences to applications for
> F1-F4 key presses;


So this combination is broken: either Terminal.app should set TERM to
vt100, or it should send xterm escape sequences. It sounds as if it
may even be sending a mixture of xterm & vt100 sequences by default?

> 3. The terminfo entry for xterm-color by Apple tells applications to expect
> xterm escape sequences for F1-F4 key presses.


That sounds correct.

> b) Some users may fix their terminfo entry for xterm-color, or set TERM to
> xterm to help Zile recognise F1-F4;


That sounds as though you're saying that the Apple terminfo entry for
xterm also suggests vt100 codes...but surely that's broken, because if
you ssh into a Mac from xterm, so your TERM=xterm, then anything you
run on the Mac will expect the wrong codes for F1-F4.

Let's be clear: the codes that should be in the xterm & xterm-color
terminfo entries are not up for debate, they are specified by
xterm/xterm-color. From what you've said so far, the xterm-color entry
is correct, but the xterm one is wrong.

There's an obvious problem with trying to accept the wrong escape
sequences, of course, which is that you have to assume that they're
never sent correctly with another meaning. Hence, we'd only want to
have the filtering fix in place on known-faulty systems.

Finally, you didn't report what you found when you set TERM=nsterm.
The ncurses FAQ (http://invisible-island.net/ncurses/ncurses.faq.html)
says:

For instance, it has been reported that Mac OS X's bundled terminal
emulator uses this value. However, reliable reports of its actual
capabilities say that there are differences, which are addressed in
ncurses as the nsterm entry.

If Terminal.app does indeed send a mixture of xterm and vt100 codes,
then a custom terminal type seems the most sensible solution.

Reuben Thomas <rrt>
Project Administrator
Tue 09 Aug 2011 05:13:26 AM UTC, comment #13:

Here's a better explanation of the function key issues with a fresh installation of C Zile running on Apple's Terminal.app:

1. Terminal.app on Mac OS 10.7 and 10.6 sets TERM to xterm-color by default;
2. Similarly it sends vt100 compatible escape sequences to applications for F1-F4 key presses;
3. The terminfo entry for xterm-color by Apple tells applications to expect xterm escape sequences for F1-F4 key presses.

This has a couple of consequences for users:

a) Out of the box, F1-F4 are not recognised by Zile;
b) Some users may fix their terminfo entry for xterm-color, or set TERM to xterm to help Zile recognise F1-F4;
c) Other users may reprogram their escapes sequences in Terminal.app to send the escape sequences the xterm-color terminfo caps prescribe.

In conclusion:

* If we don't provide a fix at all, users (b) and (c) will be happy, but group (a) are left with a terminfo entry that doesn't match the behavior of Terminal.app;
* If we "fix" the xterm-color terminfo entry for all users on affected Macs, users (a) and (b) will be happy, but group (c) will be back with a terminfo that doesn't match the fix they made already;
* If we teach Zile to recognize both the "correct" and "buggy" F1-F4 escape sequences in some manner equivalent to Emacs' solution, everyone will experience a working Zile out of the box.

I've been pushing for the very last option for these reasons.

Gary V. Vaughan <gary>
Project Administrator
Mon 08 Aug 2011 08:38:47 PM UTC, comment #12:

Heck that post is is 8 years, and 6 major OS releases out of date. I didn't actually try it, because the writer is talking about entirely different problems (graphical characters, F5-F10 problems).

The default TERM setting for Terminal.app as shipped with Mac OS 10.7 is 'xterm-color', and the terminfo entry for 'xterm-color' does not match the default settings for Terminal.app (the last few versions have programmable escape sequences for function keys etc). The terminfo entry for 'xterm' does have function key escape sequences that match, but that one tells ncurses that the terminal does not support colour :(

In principle I agree that testing for OS might work, along with a collection of terminfo files... BUT they will need ongoing maintenance, especially as it's very likely that they are somewhat dependent on the OS release and patch level, the version of ncurses that is installed, whether fink/macports/homebrew is present and overriding the system ncurses. Also, while it may seem easy to fix the known problems for known terminal types -- it's equally likely that some other more subtle problem might be introduced in a more obscure terminal capability.

On reflection, decompiling the installed terminfo entry, editing with sed, and recompiling the corrected version at configure time seems like it would be less of a can of worms. At least, it should be less sensitive to local conditions than shipping and installing a whole replacement terminfo file. Even so, it might actually undo local fixes some users have applied by reprogramming the problematic key escape sequences in Terminal.app, or fixing their own terminfo files.

This all still seems like an awful lot of hoop-jumping when Emacs' proven char-stream substitution is already working for us, and which doesn't suffer from any of the problems described above.

Gary V. Vaughan <gary>
Project Administrator
Mon 08 Aug 2011 04:16:43 PM UTC, comment #11:

Please can you test the terminal definition given at:

http://osdir.com/ml/apple.fink.devel/2003-04/msg00085.html

Also, if you read that page, you'll see that TERM should be nsterm to get the correct terminal definition; is that the default setting?

Finally, you'll notice that it's possible to download the terminal definition file, and save it in a per-user file. That is going to solve the problem for 99% of users, forever, for all terminfo-using programs.

So if the solution given actually works, I see no problem with Zile's configure just saying "if I'm running on Mac OS, install the following updated terminfo file". If Apple ever fix it we can put a version check in. The terminfo file can be installed under /usr/local (I assume that will DTRT) rather than under the per-user directory (just because "make install" shouldn't write to a user's directory) or under /usr (because we shouldn't be overwriting system files).

I think this is a solution which fits all our goals: it doesn't change Zile's source, it works without any effort on the user's part, it's easy to code (just test for the OS, no need to test specifically for terminfo brokenness), and it fixes all apps not just Zile.

Reuben Thomas <rrt>
Project Administrator
Mon 08 Aug 2011 03:52:00 PM UTC, comment #10:

I agree that the Universe will certainly be a better place if we can find a transparent way for all applications to work correctly in the presence of a hosed terminfo entry. However that is certainly not the simplest possible way to fix Zile!

My point is that I (and I think, other Zile users on Mac at least) will be much happier if it works correctly out of the box. While I concede that the problem is in Apple's ncurses, I'd really rather not make every Mac Zile user (among others) choose between fiddling the terminfo database and configuration files, or else picking an editor that works already.

Whatever path we take to being able to install Zile and have it work properly right away is not terribly important, and while I rather like the easy char stream substitution algorithm I copied from Emacs, I'm certainly not opposed to having configure ship with a selection of fixed terminfo files from gnulib, compile them, and install them somewhere Zile can find them at runtime if it can be made to work reliably... or maybe even wrapping the keypad and fetch functions from nurses and editing the char stream itself before Zile ever sees it, but either seems like it is opening a can of worms to me.

Gary V. Vaughan <gary>
Project Administrator
Mon 08 Aug 2011 02:52:33 PM UTC, comment #9:

What I am after is the simplest possible way to fix terminfo for all applications. I would rather this was not tied to Zile. I don't really see why it can't just be like the web page pointed to in the Zile FAQ which says "download this file and install it in such-and-such a place". There is no problem with Zile finding a new file, the idea it to put it in a standard location. I was only suggesting making this part of Zile's build system because you seemed to think that downloading and copying a file was too much to expect of users; actually I think that for a typical Zile user that is not too much to ask until Apple fix their ncurses. This is why I also suggested making something to go in gnulib as that's precisely the sort of thing for which gnulib is designed, namely, papering over the cracks in existing OS implementations.

Emacs effectively re-does a lot of the work which gnulib does because it predates gnulib; this is gradually being changed, but understandably it's a slow process.

Reuben Thomas <rrt>
Project Administrator
Wed 03 Aug 2011 06:47:17 PM UTC, comment #8:

Is that actually any simpler though? Unless we also ship a way for configure to test for bugginess and compile and install the replacement terminfo entry, and then have Zile pick up the replacement automatically... it actually sounds like way more work than either decompiling/fixing/reinstalling your own terminfo (which makes us look like a hassle at install time compared to Emacs), or just doing a runtime substitution on the tricky sequences to workaround common buggy terminfo/terminal pairs...

I still far prefer runtime substitution, and have no problem with adding a configure time switch to udder the code out for people who know the terminal they are using is fine. I think it's really important that Zile should work out of the box without any manual fuss when Emacs and Vim etc. are able to do so.

Gary V. Vaughan <gary>
Project Administrator
Wed 03 Aug 2011 07:58:25 AM UTC, comment #7:

No, I meant a gnulib extra terminfo file for the buggy terminals. Much simpler!

Reuben Thomas <rrt>
Project Administrator
Wed 03 Aug 2011 03:38:59 AM UTC, comment #6:

You mean a gnulib replacement for curses getch() that patches up keycodes? I guess that would work too... but gnulib is of little use to Lua, and maybe a hard sell for Emacs to ditch it's own elisp solution in favour :-( Even if we add gnulib to lcurses, that doesn't help an FFI/luajit version of Zile...

Perhaps Paul Eggert would be interested in helping to guage the likely reception from the Emacs community and/or factor the relevant code from Emacs lisp into gnulib C with an eye to porting it back into Emacs afterwards?

Gary V. Vaughan <gary>
Project Administrator
Tue 02 Aug 2011 04:18:26 PM UTC, comment #5:

I'd much rather fix it for all applications, not just Zile. Hence, is it not possible to have Zile install a correct terminfo definition for Terminal.app (and potentially other incorrectly-described terminal emulators?). This is the sort of thing that could then go into gnulib.

Reuben Thomas <rrt>
Project Administrator
Tue 02 Aug 2011 08:06:25 AM UTC, comment #4:

I humbly offer the same argument as I did for #33907:

If Emacs does the right thing (and the clever chaps behind it have already determined that there are sufficient broken terminfo/ncurses installations in the wild that a system of saving the user from the distributions' stupidity is necessary) then it seems like a terrible compromise for us to frustrate our users in the first instance, and then point them at a FAQ that blames someone else and makes them jump through hoops to get things working; especially when it requires just a few lines of code to fix it. (Plus it introduces a framework we can use to fix up similar future problems that might be uncovered)

It took me less than an hour to research why things worked on Emacs and implement a super simplified version of their solution in Lua Zile, plus 30 minutes of your time to port it to C Zile versus all the time saved fielding bug reports and helping people recompile their terminfo entries to work around the issue seems like a no-brainer to me =)O|

If you'd like I'll write a configure test that #ifdefs away the extra code when passed `--disable-fix-broken-terminfo` ;)

Gary V. Vaughan <gary>
Project Administrator
Mon 01 Aug 2011 06:25:43 PM UTC, comment #3:

Thanks for this patch. I have updated the FAQs for the C branch.

However, I really don't like this way of doing things. I'd rather inform the user their system is broken. Is it possible to write a configure check for this problem? Then we could direct them to the FAQ for the workaround.

Reuben Thomas <rrt>
Project Administrator
Mon 01 Aug 2011 01:01:55 AM UTC, comment #2:

For Lua Zile only, I fixed this problem with commit v2.4.0alpha1-143-g45e1c21 - in the same way that GNU Emacs does it: by catching and replacing the erroneous key presses in the input stream.

Gary V. Vaughan <gary>
Project Administrator
Sun 31 Jul 2011 07:48:29 PM UTC, comment #1:

That's because Emacs has all the extra terminal initialisation in term/xterm.el to make sure that when typical xterm emulators (among which is Apple Terminal.app) fire \033OP when KEY_F(1) is pressed, the escape sequence is recognised and replaced with an <f1> symbol in the input stream - even when the ncurses terminfo caps say that KEY_F1 thru KEY_F4 output a different sequence.

I note that my terminfo database has the correct caps for kf1 thru kf4 for TERM=xterm, but for xterm-color (which is what I get by default) and xterm-color256 the caps do not match reality, and that's why Zile never sees an <f1> event - because curses see's it as three distinct keypresses: ESC, O, P.

I don't think it makes sense for Zile to provide the entire over-engineered key replacement lookup by terminal type (at least until we get a stream of bug reports about missing keys). But I do think we should hardcode the replacements we are aware of as a stop gap.

Gary V. Vaughan <gary>
Project Administrator
Sun 31 Jul 2011 01:24:38 PM UTC, original submission:

Emacs 24.0.50.1 can recognise all function keys F1-F12 on my standard Terminal (xterm-color compatible according to $TERM).

But Zile 2.4.0.50 bleeps and inserts the tail end of the escape sequence for F1 into the current buffer when I press F1, and otherwise ignores anything bound to the key.

Gary V. Vaughan <gary>
Project Administrator

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach File(s):
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -unavailable- added by rrt (Posted a comment)
  • -unavailable- added by gary (Submitted the item)
  •  

    Please enter the title of George Orwell's famous dystopian book (it's a date):

     

     

    Follow 2 latest changes.

    Date Changed By Updated Field Previous Value => Replaced By
    Thu 18 Aug 2011 03:02:33 AM UTCgaryStatusNone=>Fixed
      Open/ClosedOpen=>Closed

    Back to the top


    Powered by Savane 3.1-cleanup1