bugGNU Octave - Bugs: bug #55306, Bi-directional text not displayed...

 
 

bug #55306: Bi-directional text not displayed correctly (or hidden) in the file editor

Submitter:  None
Submitted:  Thu 27 Dec 2018 09:31:04 PM UTC
   
 
Category:  GUI Severity:  3 - Normal
Priority:  5 - Normal Item Group:  None
Status:  Confirmed Assigned to:  None
Originator Name:  ahmed ghannam Originator Email:  -email is unavailable-
Open/Closed:  * Open Release:  * dev
Operating System:  * Any Fixed Release:  None
Planned Release:  None
* Mandatory Fields

Add a New Comment Rich Markup
   

Jump to the original submission

Wed 02 Jan 2019 01:34:36 PM UTC, comment #14: 

We are using QScintilla - a port of Scintilla for Qt. The version of Scintilla that QScintilla is based on doesn't support bidi text. So there isn't much we can do about this yet.

We can keep this bug report open until bidi support in Scintilla is final and it is adopted by QScintilla.

Markus Mützel <mmuetzel>
Group administrator
Wed 02 Jan 2019 12:43:55 PM UTC, comment #13: 

Re-reading the original report, I am not sure that all issues are fixed.

While with the applied changes the user should now see a meaningful error message (i.e. that some characters can't be correctly encoded), the user would still have a hard time finding these characters because they aren't displayed in the editor.

I could re-produce this issue on Ubuntu and Windows by pasting any Arabic characters from [1] at the start of a line. The characters display but overlay if they are pasted at the end of a line. Still think [2] is relevant.

Re-titling for the remaining issue.

[1]: https://de.wikipedia.org/wiki/ISO_8859-6
[2]: https://www.scintilla.org/ScintillaDoc.html#SCI_SETBIDIRECTIONAL

Markus Mützel <mmuetzel>
Group administrator
Tue 01 Jan 2019 10:38:49 PM UTC, comment #12: 

Thanks again, build and runs fine now.

NB I didn't test for the original subject of this bug report, hopefully the OP or someone else who knows the details can do that.

Philip Nienhuis <philipnienhuis>
Group Member
Tue 01 Jan 2019 10:12:56 PM UTC, comment #11: 

Thanks Markus, I'll try building again.

FWIW, in the code I didn't see references to just "u32-conv_to_encodng" but "u32-conv_to_encodng_<something>".

Philip Nienhuis <philipnienhuis>
Group Member
Tue 01 Jan 2019 12:34:51 PM UTC, comment #10: 

Sorry for that. I added the missing gnulib module in this changeset:
https://hg.savannah.gnu.org/hgweb/octave/rev/ee5ecb1bc257

Tbh, I don't understand why it worked for me (and the buildbots) without explicitly adding that module during bootstrap.

Markus Mützel <mmuetzel>
Group administrator
Tue 01 Jan 2019 12:14:39 PM UTC, comment #9: 

I get a build error:

:
_  CXXLD    src/octave-cli
/home/philip/devel/octdev/oct500/liboctave/.libs/liboctave.so: undefined reference to `u32_conv_to_encoding'-

I did run ./bootstrap and configure. At the start of the (fresh) build I didn't see a "u32... " module being pulled from gnulib, in config.log there was no mention either.

Philip Nienhuis <philipnienhuis>
Group Member
Mon 31 Dec 2018 06:15:34 PM UTC, comment #8: 

I pushed the patch to stable here:
http://hg.savannah.gnu.org/hgweb/octave/rev/5f10217b562d

Marking as ready for test.

Markus Mützel <mmuetzel>
Group administrator
Mon 31 Dec 2018 05:09:49 PM UTC, comment #7: 

@Markus: Thanks for the patch, works for me.

As far as I know, the mapping was just related to lower/upper case.

Torsten Lilge <ttl>
Group Member
Mon 31 Dec 2018 04:24:34 PM UTC, comment #6: 

If I follow the Qt sources correctly, QTextCodec::canEncode uses QTextCodec::convertFromUnicode to convert to the local encoding and checks invalidChars of its ConverterState whether it was successful. Unfortunately some implementations (e.g. for ICU [1]) don't update the invalidChars member and the check is always true.

The attached patch uses gnulib to check whether the conversion is possible. IIRC, you (Torsten) had to map some codepage names from gnulib to Qt. Do you know whether this is necessary the other way round as well?

[1]: https://code.woboq.org/qt5/qtbase/src/corelib/codecs/qicucodec.cpp.html#628

(file #45814)

Markus Mützel <mmuetzel>
Group administrator
Mon 31 Dec 2018 02:46:43 PM UTC, comment #5: 

You're right. Yet I don't see any error message when I'm saving text that cannot be encoded with the configured encoder.
Also tested on Ubuntu 18.10 with the same result.

Does it work for you?

Markus Mützel <mmuetzel>
Group administrator
Mon 31 Dec 2018 02:07:48 PM UTC, comment #4: 

canEncode () is already used in the function check_valid_codec () which is called before saving the text.

Torsten Lilge <ttl>
Group Member
Mon 31 Dec 2018 01:51:24 PM UTC, comment #3: 

The file still contains question mark characters where I think the arabic characters should be.

I think I am able to duplicate something that is similar to what you see:

1. Set the encoding to something that doesn't encode the characters you want to test with (e.g. ISO-8859-6 which doesn't contain umlauts).
2. Open a new editor tab and check that the encoding is used. Write some characters that cannot be encoded (e.g. "test äöü").
3. Save that file.
4. Close the file in the editor and re-open. The non-encodable characters are replaced by "?" (question mark).

That is confusing for a user because we don't give any indication that some of the characters were lost when saving the file.

We should probably check if the editor content can be encoded correctly before writing to the file.
This should probably be done in file_editor_tab::save_file ("libgui/src/m-editor/file-editor-tab.cc") before line 2157.
Calling "codec->canEncode (_edit_area->text ())" should de returning false in this case.

Since I don't know how to add an appropriate user interface for this, I am CC'ing Torsten. I hope you don't mind having a look at this.

I still don't know why these characters don't show up for you in the first place. Might be a Scintilla bug. Or we are not setting it up correctly. [1] might be related: "Scintilla only correctly displays bidirectional text on some platforms."

[1]: https://www.scintilla.org/ScintillaDoc.html#SCI_SETBIDIRECTIONAL

Markus Mützel <mmuetzel>
Group administrator
Sat 29 Dec 2018 01:45:53 PM UTC, comment #2: 

The Arabic characters seem to have been replaced by actual "?" (3F) characters somehow. Could you please zip the .m file and re-upload here?

Which version of Octave do you have? (Given that Octave 5 isn't released yet.)

Which Windows locale are you on? Do you know the name of the codepage that you are using?

Markus Mützel <mmuetzel>
Group administrator
Fri 28 Dec 2018 11:02:04 AM UTC, comment #1: 

That might be an issue with text direction.

Unfortunately, I don't know how to type Arabic letters. Could you please upload a file that contains those "hidden" characters?

Markus Mützel <mmuetzel>
Group administrator
Thu 27 Dec 2018 09:31:04 PM UTC, original submission:  

typed some Arabic letters by mistake .. nothing appeared on the editor. when running the editor a syntax error appeared.. when opening the file with matlab editor the Arabic letters appeared normally.

Anonymous

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

Attached Files
file #45814:  bug55306_file_editor_codec.patch added by mmuetzel (3KiB - application/octet-stream)
file #45806:  arabic_letters_error.rar added by None (117B - application/octet-stream - octave 4.4.1 windows 10)
file #45762:  arabic_letters_error.m added by None (31B - application/octet-stream - the arabic letters are in the second line ... matlab shows them as question marks...try copying the second line and pasting it several time the arabic appears over layered over english )

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by ttl (Posted a comment)
  • -email is unavailable- added by mmuetzel
  • -email is unavailable- added by mmuetzel (Posted a comment)
  • -email is unavailable- added by None (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only group members can vote.

     

    Follow 11 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2019-01-02 mmuetzel StatusReady For Test Confirmed
        Release5.0.0 dev
        Summarynon-english typing in the editor don't appear but still there hidden causing error Bi-directional text not displayed correctly (or hidden) in the file editor
    2018-12-31 mmuetzel StatusNone Ready For Test
    2018-12-31 mmuetzel Attached File- Added bug55306_file_editor_codec.patch, #45814
    2018-12-31 mmuetzel Operating SystemMicrosoft Windows Any
    2018-12-31 mmuetzel StatusNeed Info None
        Carbon-Copy- Added ttl
    2018-12-31 None Attached File- Added arabic_letters_error.rar, #45806
    2018-12-29 mmuetzel StatusNone Need Info
    2018-12-28 None Attached File- Added arabic_letters_error.m, #45762

    Back to the top

    Powered by Savane 3.13-758e.
    Corresponding source code