buglibiconv - Bugs: bug #66773, Add encoding aliases for OS/2

 
 

bug #66773: Add encoding aliases for OS/2

Submitter:  KO Myung-Hun <lvzuufx>
Submitted:  Mon 10 Feb 2025 10:56:48 AM UTC
   
 
Category:  Encodings Severity:  3 - Normal
Item Group:  None Status:  None
Privacy:  Public Assigned to:  haible
Open/Closed:  Open
* Mandatory Fields

Add a New Comment Rich Markup
   

Jump to the original submission

Wed 12 Feb 2025 01:06:50 PM UTC, comment #12: 


> > What are the better things of this approach than using name conversion tables like locale_charset()?
>
> I don't understand this question. You said yourself, in #comment5, that relying on locale_charset() is not desirable in this case.
>


What I meant is the compare of the two methods used by libiconv and by libcharset, not using libcharset. At a glance, a method of libcharset seemed to be more simple and easy than libiconv.

> libiconv does not use name conversion tables. It uses a single hash table for all encoding names and aliases, and this hash table is constructed at compile-time for maximum efficiency.


Ok, thanks for your explanation!

KO Myung-Hun <lvzuufx>
Wed 12 Feb 2025 10:05:14 AM UTC, comment #11: 


> What are the better things of this approach than using name conversion tables like locale_charset()?


I don't understand this question. You said yourself, in #comment5, that relying on locale_charset() is not desirable in this case.

libiconv does not use name conversion tables. It uses a single hash table for all encoding names and aliases, and this hash table is constructed at compile-time for maximum efficiency.

Bruno Haible <haible>
Group administrator
Wed 12 Feb 2025 09:58:21 AM UTC, comment #10: 


comment #9:

> Did I? Looking through the VCS history... Oh yes, this patch did so, but limited to specific platforms:
> https://git.savannah.gnu.org/gitweb/?p=libiconv.git;a=commitdiff;h=bcb1081e591a695f42cefd87a16e9ff6095bb094
>
>


Thanks for your kindness!

BTW, I have one more question. What are the better things of this approach than using name conversion tables like locale_charset()?

KO Myung-Hun <lvzuufx>
Wed 12 Feb 2025 07:24:53 AM UTC, comment #9: 


> I don't know that those aliases led to any interoperability problems. Your concerns seem to be too much and not proper.


I went through this in the years 2000-2005.

> This should have happend already when you added other alises.


Did I? Looking through the VCS history... Oh yes, this patch did so, but limited to specific platforms:
https://git.savannah.gnu.org/gitweb/?p=libiconv.git;a=commitdiff;h=bcb1081e591a695f42cefd87a16e9ff6095bb094

> these code page names are returned by nl_langinfo(CODESET).


Indeed, I see this in the kLIBC code: https://github.com/bitwiseworks/libc/blame/master/src/emx/src/lib/locale/__convcp.c

So, I am OK to treat OS/2 like AIX, Solaris, HP-UX, OSF/1.

You are welcome to provide a patch in the style of the commit mentioned above, that adds the following aliases (taken from the OS/2 section of gnulib/lib/localcharset.c) as system-dependent aliases:
IBM-1004
IBM-1089
IBM-1124
IBM-1125
IBM-1131
IBM-1208
IBM-1250
IBM-1251
IBM-1252
IBM-1253
IBM-1254
IBM-1255
IBM-1256
IBM-1257
IBM-1381
IBM-1383
IBM-1386
IBM-3372
IBM-367
IBM-437
IBM-4946
IBM-813
IBM-819
IBM-850
IBM-852
IBM-855
IBM-856
IBM-857
IBM-860
IBM-861
IBM-862
IBM-863
IBM-864
IBM-865
IBM-866
IBM-869
IBM-874
IBM-878
IBM-912
IBM-913
IBM-914
IBM-915
IBM-916
IBM-920
IBM-921
IBM-922
IBM-923
IBM-932
IBM-943
IBM-949
IBM-950
IBM-954
IBM-964
IBM-970

Bruno Haible <haible>
Group administrator
Wed 12 Feb 2025 06:06:33 AM UTC, comment #8: 


comment #7:

> > I don't know why another IBM-xxx aliases are not allowed.
>
> New aliases lead to interoperability problems. Say, you add IBM-949 as alias to GNU libiconv now. Then, some program uses it.
>


This should have happend already when you added other alises. Hapeened?

In addition, IBMxxx and IBM-xxx such as IBM-949 are used and supported already as aliases by Java, libicu and so on.

What you said happened?

As well as, iconv() implementation varies and some of those may support those aliases. OS/2 kLIBC iconv() is one of them.

I don't know that those aliases led to any interoperability problems. Your concerns seem to be too much and not proper.

Anyway what I want is how to add aliases to libiconv. I asked because I thought I could learn correct ways from you.

And my final target is to apply the result to OS/2 release in order to use it as drop-in replacement of OS/2 kLIBC iconv().

Whether or not to apply the patch is your choice.

KO Myung-Hun <lvzuufx>
Tue 11 Feb 2025 04:27:01 PM UTC, comment #7: 


> I don't know why another IBM-xxx aliases are not allowed.


New aliases lead to interoperability problems. Say, you add IBM-949 as alias to GNU libiconv now. Then, some program uses it.

For example, a mailer sends a mail with "Content-Type: text/plain; charset=IBM-949". The receiving MUA cannot decode this, so its users notice an interoperability problem. Some of the user files a bug report (against that MUA); then that MUA gets extended to treat IBM-949 like CP949, or files a bug to glibc to treat IBM-949 like CP949.

Then some other programmer sees IBM-949 as a possible encoding name and uses it, say, in the metadata of a calendar program. Another calendar program cannot process it, so its users notice an interoperability problem. Some of the user files a bug report against that second calendar program, and so on.

Five years later, numerous users have experienced interoperability problems and dozens of developers have had to deal with the issue. And what is the added functionality? ZERO. The same functionality could have been achieved by labeling the encoding as CP949 from the beginning.

Bruno Haible <haible>
Group administrator
Tue 11 Feb 2025 01:10:15 PM UTC, comment #6: 


comment #5:

> > Not all the programs using iconv() use Gnulib or libcharset.
> > ...
> > It would be better to use iconv() without depending on libcharset.
>
> Then the best way forward is to change the programs or scripts, to use the de-facto standardized names. For example, for CP949, glibc and GNU libiconv support the names CP949, UHC. There is no point in having programs or scripts use yet another name, "IBM-949", for the same thing.


This leads to modify many programs or scripts using libiconv. However, if libiconv supports such aliases, then they all will be happy. Practically, many IBMxxx and IBM-xxx aliases for AIX and z/OS are available, yet.

I don't know why another IBM-xxx aliases are not allowed.

KO Myung-Hun <lvzuufx>
Mon 10 Feb 2025 02:49:55 PM UTC, comment #5: 


> Not all the programs using iconv() use Gnulib or libcharset.
> ...
> It would be better to use iconv() without depending on libcharset.


Then the best way forward is to change the programs or scripts, to use the de-facto standardized names. For example, for CP949, glibc and GNU libiconv support the names CP949, UHC. There is no point in having programs or scripts use yet another name, "IBM-949", for the same thing.

Bruno Haible <haible>
Group administrator
Mon 10 Feb 2025 02:17:01 PM UTC, comment #4: 

I also know localcharset() and why it is. I had participated on that mapping.

However, localcharset() is not a standard function unlike iconv() and it requires additional dependency on libcharset.

Not all the programs using iconv() use Gnulib or libcharset.

In addition, localcharset() works only on a current locale. However, iconv() can works with arbitrary locale if aliases are added.

It would be better to use iconv() without depending on libcharset.

KO Myung-Hun <lvzuufx>
Mon 10 Feb 2025 01:54:46 PM UTC, comment #3: 


> iconv_open() does not understand some IBM-xxx code pages used on OS/2. For example, IBM-949. And these code page names are returned by nl_langinfo(CODESET).


The Gnulib module 'localcharset' was created precisely to work around this problem. It is a wrapper around nl_langinfo(CODESET). And on OS/2, it already maps IBM-949 to CP949, which is supported by the GNU libiconv implementation.

So, all you have to do is to change the program P to use locale_charset() instead of nl_langinfo(CODESET).

Bruno Haible <haible>
Group administrator
Mon 10 Feb 2025 01:41:46 PM UTC, comment #2: 

Ooops... I'm sorry that it's so ambiguous.

iconv_open() does not understand some IBM-xxx code pages used on OS/2. For example, IBM-949. And these code page names are returned by nl_langinfo(CODESET).

So I want to add those code pages to iconv's aliases so that iconv_open() can accept them. Mapping table I want to utlize is in libcharset/lib/localcharset.c.

How can I do that?

KO Myung-Hun <lvzuufx>
Mon 10 Feb 2025 01:26:33 PM UTC, comment #1: 


> I want to add encoding aliases listed in libcharset/lib/localcharset.c for OS/2.


I don't understand this problem statement.

A valid problem statement is a tuple (P, F, A, X, W) meaning "The program P invokes the function F with argument A, but instead the expected effect X it leads to the wrong effect W."
What is the list of problem statements that you want to get fixed?

Bruno Haible <haible>
Group administrator
Mon 10 Feb 2025 10:56:48 AM UTC, original submission:  

Hi/2.

I want to add encoding aliases listed in libcharset/lib/localcharset.c for OS/2. Especially IBM code pages in IBM-xxx form.

How can I do that ?

KO Myung-Hun <lvzuufx>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

Attached Files
file #56911:  0001-Add-OS-2-dependent-aliases.patch added by lvzuufx (38KiB - application/octet-stream - Patch to add OS/2 aliases)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by haible (Posted a comment)
  • -email is unavailable- added by lvzuufx (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only logged-in users can vote.

     

    Follow 7 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2025-02-17 lvzuufx Attached File- Added 0001-Add-OS-2-dependent-aliases.patch, #56911
    2025-02-12 haible Open/ClosedClosed Open
    2025-02-12 haible StatusNot a Bug None
    2025-02-10 haible Open/ClosedOpen Closed
    2025-02-10 haible StatusNeed Info Not a Bug
        Assigned toNone haible
    2025-02-10 haible StatusNone Need Info

    Back to the top

    Powered by Savane 3.15.
    Corresponding source code