bugGNU TeXmacs - Bugs: bug #60284, Hyphenation doesn’t work with...

 
 

You are not allowed to post comments on this tracker with your current authentication level.

bug #60284: Hyphenation doesn’t work with Russian language (only)

Submitter:  Vladimir <vcheese>
Submitted:  Wed 24 Mar 2021 12:17:45 PM UTC
   
 
Category:  Editor Priority:  5 - Normal
Item Group:  Error Status:  In Progress
Privacy:  Public Assigned to:  vdhoeven
Originator Name:  Open/Closed:  Open
Release:  None Release:  1.99.19
Fixed Release:  None Fixed Release: 
Keywords: 

Jump to the original submission

Thu 20 May 2021 09:55:30 PM UTC, comment #7: 

It was pointed out on the forum that hyphenation in Russian is not following the proper rules for the language [1]. Upon inspection there a few more issues with the calculation of hyphenation penalties.

The attached patch `hyphen-russian-ctd.diff` fixes the problem, but seems to make the editor sluggish in long paragraphs in Russian.  Perhaps it can be optimized further.


[1] http://forum.texmacs.cn/t/hyphenation-doesnt-work-with-russian-language-only/384/28

(file #51456)

Jeroen Wouters <jeronim>
Group Member
Mon 26 Apr 2021 02:53:37 PM UTC, comment #6: 

Thanks, that seems to fix it!

Jeroen Wouters <jeronim>
Group Member
Mon 26 Apr 2021 11:18:49 AM UTC, comment #5: 

I reviewed the Unicode hyphenation code (which I did not write) and noted that it indeed uses a different encoding, not only internally (which is reasonable), but also in the external interface (which is very annoying).

I have no time right now to fix this, but I think that I did manage to fix the present bug for now, by using an alternative way to deduce the actual position where the string is broken.

Please let me know whether my fix works for you (revision 13418).

Joris van der Hoeven <vdhoeven>
Group administrator
Thu 08 Apr 2021 09:49:56 PM UTC, comment #4: 

It seems that the problem lies in the "shorter_box_rep" class in "src/Typeset/Boxes/Modifier/modifier_boxes.cpp", which is the type of box used for the first part of a hyphenated word.

The variable "len" in that class is the length of the string in the shorter box, measured in utf8 characters, e.g. len=4 for "пере". It is, however, used in that class to increment paths, which increase in steps of 6 per utf8 character, as they are represented as "<#443>", for example.

Jeroen Wouters <jeronim>
Group Member
Thu 08 Apr 2021 01:09:00 PM UTC, comment #3: 

Thanks for including the patch. After some further testing it seems that unfortunately there is an additional problem.

With the patch the cursor cannot move properly through hyphenated words. The cursor jumps around to random positions and on the terminal I get messages like

warning: invalid cursor position [ 1, 1, 166 ]


From the output of "Debug->Status->Path" it seems that the path normally increases by 6 when moving though Cyrillic characters, as may be expect from the internal representation of utf8. When moving through a hyphenated word, jumps of various lengths occurs, not just 6.

I have tried to find out where this happens, but have been unable to find the root of the problem. Until such time, it may be better to revert the patch.

Jeroen Wouters <jeronim>
Group Member
Wed 24 Mar 2021 11:10:28 PM UTC, comment #2: 

I believe the attached patch fixes the problem. Lowercasing is also enabled for the exception case as the exception lists. I can't find a reference for the casing in the exception list, though. It would be good to verify what the convention for German is as words are written with a starting uppercase letter.

(file #51134)

Jeroen Wouters <jeronim>
Group Member
Wed 24 Mar 2021 10:15:51 PM UTC, comment #1: 

This bug seems to be due to the use of the "locase_all" function in "src/System/Language/hyphenate.cpp"

Changing the line

s= "." * locase_all (s) * ".";

to

s= "." * s * ".";

results in hyphenation of lowercase words. It seems "locase_all" can't handle Cyrillic characters.

Jeroen Wouters <jeronim>
Group Member
Wed 24 Mar 2021 12:17:45 PM UTC, original submission:  

I done everything which in Help → Notes for users of Cyrillic languages:
  * selected Russian as default language in Edit → Preferences → Language → Russian;
  * selected Russian for an entire document using Document → Language → Russian;
  * I also checked if a file with patterns exists on the path /usr/share/TeXmacs/langs/natural/hyphen/hyphen.russian;
  * I tried to force hyphenation by Format → Hyphenate as. But nothing has changed.

Despite this hyphenation doesn’t work with Russian language. But if I set any other language (I tried English and German), then everything works correctly.

At the same time, the spell check of the Russian language works.

Is it a bug? How to fix this issue?

An example file is attached to the message.


GNU/Linux
5.10.23-1-MANJARO
TeXmacs version 1.99.19

Vladimir <vcheese>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attached Files
file #51456:  hyphen-russian-ctd.diff added by jeronim (2KiB - text/x-patch)
file #51134:  hyphen-russian.diff added by jeronim (968B - text/x-patch)
file #51126:  test.tm added by vcheese (8KiB - text/x-texmacs.doc)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by vdhoeven (Posted a comment)
  • -email is unavailable- added by mgubi (Updated the item)
  • -email is unavailable- added by mgubi
  • -email is unavailable- added by jeronim (Posted a comment)
  • -email is unavailable- added by vcheese (Submitted the item)
  • -email is unavailable- added by vcheese
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

     

    Follow 8 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2021-05-20 jeronim Attached File- Added hyphen-russian-ctd.diff, #51456
    2021-04-26 vdhoeven StatusNone In Progress
    2021-04-08 mgubi Assigned toNone vdhoeven
        Carbon-Copy- Added jo
    2021-03-24 jeronim Attached File- Added hyphen-russian.diff, #51134
    2021-03-24 vcheese Attached File- Added test.tm, #51126
        Attached File- Added Screenshot_20210324_183615.png, #51127
        Carbon-Copy- Added -email is unavailable-

    Back to the top

    Powered by Savane 3.13-4448.
    Corresponding source code