buglibiconv - Bugs: bug #56304, Greek, Cyrillic and Armenian...

 
 

bug #56304: Greek, Cyrillic and Armenian transliteration

Submitter:  Yury Lebedev <yuryleb>
Submitted:  Fri 10 May 2019 08:45:04 PM UTC
   
 
Category:  Transliteration Severity:  3 - Normal
Item Group:  None Status:  Not a Bug
Privacy:  Public Assigned to:  haible
Open/Closed:  Closed
* Mandatory Fields

Add a New Comment Rich Markup
   

Thu 23 Jul 2020 07:22:02 AM UTC, comment #3: 

Please look at these two bugs fixed for the glibc iconv C locale. The C locale is ASCII and so it is not the case of UTF-8 to UTF-8 conversion.
Cyrillic https://sourceware.org/bugzilla/show_bug.cgi?id=2872
Greek https://sourceware.org/bugzilla/show_bug.cgi?id=12031

At the end of the day I just had to supply the needed tables there.

Maybe you can thus already use the iconv for your needs in Cyrillic and Greek transliteration.

Maybe you will be interested in supplying the Armenian translit table as well to iconv C locale in the same way. Just write to that mailing list https://sourceware.org/mailman/listinfo/libc-locales and maybe post a traceback here so I will also be informed.

Egor Kobylkin <diegor>
Sat 11 May 2019 12:28:58 PM UTC, comment #2: 

Hi Yury,

I have developped a library focused on arbitrary transliteration.
It is not as efficient as glibc iconv or gnu libiconv but it may be enough to do what you seek to do.
https://github.com/LLyaudet/transliteration_profile_iconv
https://sourceware.org/bugzilla/show_bug.cgi?id=23076

Best regards,
   Laurent Lyaudet

Laurent Lyaudet <llyaudet>
Fri 10 May 2019 09:17:24 PM UTC, comment #1: 

Hi Yury,

Transliteration was, in the years 2000-2010, when many people were still using an 8-bit character set (most often ISO-8859-1 or ISO-8859-15), the way to represent text that used characters outside that character set in a lossy way.

Nowadays, virtually everybody uses Unicode (on Unix, mostly in the UTF-8 encoding). But iconv from UTF-8 to UTF-8 is a no-op.

Therefore, it does not make sense to implement new transliterations in iconv any more. (Neither in glibc's iconv nor in libiconv.)

The right approach, IMO, is to create a program outside the scope of iconv.

The GNU gettext package has already a filter program for transliterating Serbian from Cyrillic to Latin script (I'm saying script, not charset). It would be OK to add Greek, Cyrillic, and Armenian transliterations as separate programs in there.

For how it's done, please look at the files gettext-tools/src/filter-sr-latin.c and gettext-tools/src/recode-sr-latin.c in the GNU gettext package.

If there are different transliteration conventions for, say, Russian, Ukrainian, and Bulgarian, we can have different programs.

If there are different transliteration conventions for Western Armenian than for Armenian, it would be OK to have a command-line option to indicate the dialect of Armenian.

Alternatively, you could submit these transliterations to the Recode project https://github.com/rrthomas/recode . It's a more generic framework for such things, but you probably need a certain time in order to understand the mindset of this framework.

Bruno Haible <haible>
Group administrator
Fri 10 May 2019 08:45:04 PM UTC, original submission:  

Please look at attached translit.def and generated translit.h with added Greek, Cyrillic and Armenian transliteration.

BTW how to specify symbols in translit.def to be completely excluded from transliteration?
For example there is following record with empty translit char for accent used in Russian texts:
0301 # COMBINING ACUTE ACCENT
But conversion of following sample Russian text:
Нам любые до́роги доро́ги
is failed till:
Нам любые до
with following error:
./iconv_no_i18n -f utf-8 -t windows-1251//TRANSLIT Russian.txt >Russian_translit.txt
iconv_no_i18n: Russian.txt:1:12: cannot convert

Yury Lebedev <yuryleb>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

Attached Files
file #46895:  translit.def added by yuryleb (159KiB - text/plain)
file #46896:  translit.h added by yuryleb (106KiB - text/plain)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by diegor (Posted a comment)
  • -email is unavailable- added by llyaudet (Posted a comment)
  • -email is unavailable- added by haible (Posted a comment)
  • -email is unavailable- added by yuryleb (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only logged-in users can vote.

     

    Follow 5 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2019-05-10 haible StatusNone Not a Bug
        Assigned toNone haible
        Open/ClosedOpen Closed
    2019-05-10 yuryleb Attached File- Added translit.def, #46895
        Attached File- Added translit.h, #46896

    Back to the top

    Powered by Savane 3.13-758e.
    Corresponding source code