bugtexinfo - GNU documentation system - Bugs: bug #28359, matchtitle underscore width not...

 
 

You are not allowed to post comments on this tracker with your current authentication level.

bug #28359: matchtitle underscore width not with the title when title contain wide char

Submitted by:  Changyuan Yu <reiv>
Submitted on:  Mon 21 Dec 2009 06:42:27 AM UTC  
 
Category: makeinfoRelease: 4.13a
Priority: 5 - NormalSeverity: 3 - Normal
Item Group: bugPrivacy: Public
Open/Closed: ClosedAssigned to: Patrice Dumas <pertusus>
Status: None

(Jump to the original submission Jump to the original submission)

Fri 25 Feb 2011 05:10:54 PM UTC, comment #19:

As discussed, this should (or will) be all resolved in the new implementation.

Karl Berry <karl>
Project Administrator
Wed 20 Jan 2010 04:26:14 PM UTC, comment #18:

Width are also right, now, solving the original issue and all the issues I raised in comment #9. Line breaking is still wrong, but I don't think that it is as important. I'd like to defer it until after the merge of texi2html in texinfo.

In fact, it happens that bytes counting was quite distinct in the code from width counting. And also that character counting is not usefull at all, only the width is useful.

Patrice Dumas <pertusus>
Project MemberIn charge of this item.
Tue 19 Jan 2010 10:12:07 AM UTC, comment #17:

offset of anchors and nodes is now in bytes. Next to be fixed should be the column width.

Patrice Dumas <pertusus>
Project MemberIn charge of this item.
Mon 11 Jan 2010 08:44:22 PM UTC, comment #16:

Text::CharWidth in fact is not very good. This is simply a frontend to wcswidth and friends. But this is not what we need here, since the encoding is the document encoding, not the locale encoding. As shown by the translation issue, switching locales reliably cannot easily be done when using the system locales implementation.

Fortunately I found some potential replacements:

http://search.cpan.org/~audreyt/Unicode-EastAsianWidth-1.30/
is very simple, pure perl. It allows to find the width of east asian characters.

http://search.cpan.org/~nezumi/Unicode-LineBreak-1.007.520/
is also interesting since it additionnally handle the issue of detecting linebreaks. It is mixed C/perl, and much more complicated, however. But it could replace the current end of line cutting code.

I am still undecided regarding what to use. The most comprehensive would certainly be along

  • use a difference code when Unicode-LineBreak is installed that delegates end of line breaking to this module
  • otherwise use Unicode-EastAsianWidth and expand the already existing code to use the right length and use the additional properties to decide where to break lines. A copy of Unicode-EastAsianWidth could be shipped in texinfo, in the same way than libintl-perl to be used if needed.

I am a bit unwilling to complicate that much, but I can't decide what approach is the better at the same time...

Patrice Dumas <pertusus>
Project MemberIn charge of this item.
Wed 30 Dec 2009 10:35:25 PM UTC, comment #15:

Reading the info code it looks like the actual number of bytes is counted since this is simply a pointer shift in memory. That's not what makeinfo in perl is doing right now, since it is counting characters. I need to fix that, then.

Patrice Dumas <pertusus>
Project MemberIn charge of this item.
Wed 30 Dec 2009 10:34:27 PM UTC, comment #14:

since makeinfo doesn't know anything about "characters", i have to think that the numerical offsets in info files are in bytes, not chars. (and i hope we don't need to change that.)

sergey added some unicode/etc. support to standalone info.

Karl Berry <karl>
Project Administrator
Wed 30 Dec 2009 10:16:13 PM UTC, comment #13:

The description of the general layout was done by you, Karl.

Anyway, I think that the issue Yu report is not an issue with the emacs reader, but with the info generated by makeinfo. I think that the emacs reader does it right.

Moreover, all the readers and makeinfo must agree on what is counted for node and anchor position. Bytes ? Characters ? Do you know what info counts (I guess I'll have a look at the code anyway)?

Patrice Dumas <pertusus>
Project MemberIn charge of this item.
Wed 30 Dec 2009 07:46:51 PM UTC, comment #12:

Patrice, I'm psyched that you wrote the description. I'll take a look as soon as I have a chance. Definitely something that would be great to have in the manual.

Yu, anything regarding the Emacs info reader needs to be sent to bug-emacs@gnu.org. The Texinfo distribution doesn't include the Emacs reader and it is maintained completely separately, by the Emacs folks.

Thanks to you both.

Karl Berry <karl>
Project Administrator
Mon 28 Dec 2009 11:57:25 PM UTC, comment #11:

In mym opinion it is more defined by the info readers than by makeinfo... I made a specification based on Karl informations and what I gathered while doing the info output:

http://www.environnement.ens.fr/perso/dumas/comp_html/info_format.html

http://www.environnement.ens.fr/perso/dumas/comp_html/info_format.texi

Karl, with proper copyright notice and other headers maybe this should go in texinfo doc? Any comment?

Patrice Dumas <pertusus>
Project MemberIn charge of this item.
Mon 28 Dec 2009 12:47:07 PM UTC, comment #10:

a question:
Are there any spec or document the define the format of info file(the output file of makeinfo), or it is just directly defined by makeinfo?

Changyuan Yu <reiv>
Sun 27 Dec 2009 09:17:28 PM UTC, comment #9:

My question was in fact very unclear. I'll try to restate it more clearly.

There are other places where the number of characters or bytes are counted. At some places it is important to determine how the output looks like (like in the case that triggered this bug report), and at some other points it is even more important since it determines the document structure. For the second case, I need to know how info counts since both ways of counting must match.

Here is a list of places where the width of characters may matter, that impacts only on the output formatting:

  • Underlying of titles, as described in this bug report
  • formatting of listoffloats and menus (41 byte-width characters). My guess here is that info reader doesn't care about the precise length, what it detects is the format.
  • formatting of @printindex index entries.
  • indentation length is more than the current position on the line (for indented formats, like @itemize, @enumerate).
  • lines splitting in paragraph
  • formatting of @center and @flush*
  • formatting of @multitable

Here is the place where info readers and formatters must agree:

  • offset of anchors and nodes
Patrice Dumas <pertusus>
Project MemberIn charge of this item.
Sun 27 Dec 2009 03:15:16 AM UTC, comment #8:

What info reader do is important. As I know, info just print title and its underlining in plain text; emacs info mode try to calculate the width of title and its underlining and fontify them when they are equal.

so, I think Text::CharWidth is a good choice.

PS: I don't know the behavior of pinfo.

Changyuan Yu <reiv>
Sat 26 Dec 2009 12:25:19 AM UTC, comment #7:

It also seems that using Encode::decode can trigger length to count th enumber of bytes. It is not clear that it is what we want. My feeling is that Text::CharWidth is better, though.

Patrice Dumas <pertusus>
Project MemberIn charge of this item.
Sat 26 Dec 2009 12:15:59 AM UTC, comment #6:

Ok, seems like
http://search.cpan.org/~kubota/Text-CharWidth-0.04/CharWidth.pm
is the way to go.

It seems that the default for perl is to count characters when using a plain length(). What does the info reader do?

Patrice Dumas <pertusus>
Project MemberIn charge of this item.
Fri 25 Dec 2009 12:58:53 PM UTC, comment #5:

sorry for missing @documentencoding, the sample is in UTF-8, it's better to add "@documentencoding UTF-8" to it.

The underlining character always has the length of 1, the number of underlining characters should be equal to the length of the string above it. When `cat' the raw info file in terminal, we can see the string and its underlining decoration aligned at both side.

Or we can just ignore this issue, string and its underlining decoration not aligned is not a big problem when view the info file in terminal. When view in emacs, we can modify info.el, use `string-bytes' instead of `string-length'(or better, use both) in function `Info-fontify-node'.

and some reference for CJK character width:
http://unicode.org/reports/tr11/

Changyuan Yu <reiv>
Fri 25 Dec 2009 10:51:46 AM UTC, comment #4:

The document needs a @documentencoding...

The issue is certainly also present in the perl implementation, but hopefully can more easily be solved here.

One think I am not sure about, is what should the solution be? Suppose we can extract the real string length, how should the rounding of the number of underlining characters be done when it needs to?

Patrice Dumas <pertusus>
Project MemberIn charge of this item.
Thu 24 Dec 2009 11:03:11 PM UTC, comment #3:

Thanks for the test file. We'll try it with the new implementation. (Patrice ...?)

Karl Berry <karl>
Project Administrator
Thu 24 Dec 2009 12:42:56 PM UTC, comment #2:

a simple test file, compiled info, and what it looks like in plain text and emacs. The text file is modified from info help of Texinfo(
Sample Texinfo Files:: > Short Sample Texinfo File).

(file #19349, file #19350, file #19351, file #19352)

Changyuan Yu <reiv>
Mon 21 Dec 2009 11:43:00 PM UTC, comment #1:

Thanks for the report and patch.

We're switching to an entirely different implementation of makeinfo. I hope it will behave better with CJK. Can you please post a simple test file so we can test?

best,
karl

Karl Berry <karl>
Project Administrator
Mon 21 Dec 2009 06:42:27 AM UTC, original submission:

makeinfo assume the byte number of the title equal to its width,
however, most CJK characters are 3 bytes long in UTF-8 but only have the width of 2. Emacs use function string-width to test title width and its underscore width, so the titles contain CJK character can not be recognized by emacs.

Changyuan Yu <reiv>

 

Attached Files
file #19352:  compiled-info.png added by reiv (42KiB - image/png - sampe source, compiled info file, info file view in plain text and view in emacs)
file #19351:  view-in-emacs.png added by reiv (25KiB - image/png - sampe source, compiled info file, info file view in plain text and view in emacs)
file #19350:  sample.info added by reiv (2KiB - application/octet-stream - sampe source, compiled info file, info file view in plain text and view in emacs)
file #19349:  sample.texi added by reiv (1KiB - application/x-texinfo - sampe source, compiled info file, info file view in plain text and view in emacs)
file #19326:  sectioning.c.patch added by reiv (827B - application/octet-stream - patch for sectioning.c, just test under linux)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -unavailable- added by pertusus (Posted a comment)
  • -unavailable- added by karl (Posted a comment)
  • -unavailable- added by reiv (Submitted the item)
  •  

    Do you think this task is very important?
    If so, you can click here to add your encouragement to it.
    This task has 0 encouragements so far.

    Only logged-in users can vote.

     

    Please enter the title of George Orwell's famous dystopian book (it's a date):

     

     

    Follow 7 latest changes.

    Date Changed By Updated Field Previous Value => Replaced By
    Fri 25 Feb 2011 05:10:54 PM UTCkarlOpen/ClosedOpen=>Closed
      Assigned toNone=>pertusus
    Thu 24 Dec 2009 12:42:56 PM UTCreivAttached File-=>Added sample.texi, #19349
      Attached File-=>Added sample.info, #19350
      Attached File-=>Added view-in-emacs.png, #19351
      Attached File-=>Added compiled-info.png, #19352
    Mon 21 Dec 2009 06:42:27 AM UTCreivAttached File-=>Added sectioning.c.patch, #19326

    Back to the top


    Powered by Savane 3.1-cleanup1