buglibiconv - Bugs: bug #55609, Add support for BOM exceptions as...

 
 

bug #55609: Add support for BOM exceptions as indicated in RFC 2781 section 3.3

Submitter:  None
Submitted:  Wed 30 Jan 2019 03:24:06 PM UTC
Votes: 25
 
Category:  Encodings Severity:  3 - Normal
Item Group:  None Status:  Works For Me
Privacy:  Public Assigned to:  haible
Open/Closed:  Closed
* Mandatory Fields

Add a New Comment Rich Markup
   

Jump to the original submission

Sun 27 Jun 2021 03:02:43 PM UTC, comment #10: 


> iconv is needed to produce Little Endian, with BOM. None of the UTF-16, UTF-16BE, and UTF-16LE do just that.


Correct. The encoding name to use depends on the receiver.

If the receiver expects little-endian always, use "UTF-16LE" as the encoding specification; a BOM would be useless (and sometimes interpreted as a character, causing trouble).

If the receiver can handle both little-endian and big-endian input, it needs guidance; this guidance can be either an explicit label (out-of-band) or a BOM (in-band); in the latter case, use "UTF-16" as the encoding specification.

Bruno Haible <haible>
Group administrator
Sun 27 Jun 2021 02:55:55 PM UTC, comment #9: 


> A simple example using a "digital signature data application" DSDA.


OK, let's talk about this particular example.

> This DSDA is the recipient of UTF-16, UTF-16BE, or UTF-16LE data. It follows the RFC and so does its providers. When data comes in labeled it calls iconv to convert it to UTF-16. ... But there is a big problem, iconv cannot support this application because it cannot write the UTF-16 data in Little Endian format as an unlabeled UTF-16 file


If you are implementing such a DSDA:

If you want a DSDA that takes labeled input and computes a signature based on a UTF-16 little endian without BOM stream, you can just call iconv_open("UTF-16LE", encoding_from_label). That is, produce UTF-16 little endian without BOM as the intermediate representation.

If you are invoking / interfacing to such a DSDA:

Use iconv_open("UTF-16", ...), and use "UTF-16" as the label. This will ensure that a BOM is present, and it will resolve ambiguities.

Bruno Haible <haible>
Group administrator
Thu 31 Jan 2019 04:07:58 AM UTC, comment #8: 

I do not know how to say this more politely, but you are wrong.

The RFC envisions such a format where the BOM is required, and is on a system where the encoding is Little Endian.

A simple example using a "digital signature data application" DSDA. The SHA256 sum of a UTF-16 stream in Big Endian is different than that in Little Endian. It is allowable, per RFC, for the author of a format to REQUIRE a BOM and have a format REQUIRE Little Endian. This DSDA is the recipient of UTF-16, UTF-16BE, or UTF-16LE data. It follows the RFC and so does its providers. When data comes in labeled it calls iconv to convert it to UTF-16. If the input is not labeled, it assumes it is UTF-16 and verifies that assumption by checking the first 2 bytes.

But there is a big problem, iconv cannot support this application because it cannot write the UTF-16 data in Little Endian format as an unlabeled UTF-16 file, violating the DSDA document format which mandates a BOM in UTF-16 text, thereby requiring the use of the "UTF-16" tag only.

I do not see how following the RFC encourages anyone to not follow the RFC.

By your logic, iconv should never have comments like

/* Here we accept FFFE/FEFF marks as endianness indicators everywhere
   in the stream, not just at the beginning. (This is contrary to what
   RFC 2781 section 3.2 specifies


Further by your logic it is not desirable that iconv encourages the receiver of an UTF-16 encoded text to ignore the BOM and treat the remaining text as big-endian encoded.

I am not asking you to make the patch, I am only asking for the opportunity to have a patcho honestly considered to allow applications to facilitate internal processing, that need to convert between internal string representation and external string representation when they are doing I/O.

Does what I propose violate the RFC?

Does what I propose violate a fundamental design restriction of iconv?

Respectfully,

Jason Pyeron

Jason Pyeron <jpyeron>
Thu 31 Jan 2019 03:24:24 AM UTC, comment #7: 


> If I make a patch, could it be accepted?


No. I explained the reasons already.

The patch that you propose would encourage the receiver of an UTF-16 encoded text to ignore the BOM and treat the remaining text as little-endian encoded. This is not desirable.

Bruno Haible <haible>
Group administrator
Thu 31 Jan 2019 01:55:51 AM UTC, comment #6: 

s/RFT/RFC/

Jason Pyeron <jpyeron>
Thu 31 Jan 2019 12:20:32 AM UTC, comment #5: 

If I make a patch, could it be accepted?

There is no violation of the RFT to produce the UTF-16 in little endian, that is what is desired.

Jason Pyeron <jpyeron>
Thu 31 Jan 2019 12:14:43 AM UTC, comment #4: 


> produce Little Endian, with BOM


If you produce UTF-16 with a BOM, the receiver MUST use the BOM to determine whether they will decode in little-endian or big-endian order; that's the point of the BOM. Therefore, as the sender, you don't need to care whether you produce little-endian or big-endian output.

On the other hand, you can achieve what you want by concatenating a little-endian BOM and the iconv output produced with encoding UTF-16LE.

Bruno Haible <haible>
Group administrator
Wed 30 Jan 2019 10:57:59 PM UTC, comment #3: 

How does one reopen? It does not work for me.

Jason Pyeron <jpyeron>
Wed 30 Jan 2019 10:53:39 PM UTC, comment #2: 

Yes, we understand that.

How does iconv produce unlabeled UTF-16 in Little Endian format?

Per the RFC:


   An exception to the "SHOULD" rule of using "UTF-16BE" or "UTF-16LE"
   would occur with document formats that mandate a BOM in UTF-16 text,
   thereby requiring the use of the "UTF-16" tag only.


This is the exact case where iconv is needed to produce Little Endian, with BOM. None of the UTF-16, UTF-16BE, and UTF-16LE do just that.

Jason Pyeron <jpyeron>
Wed 30 Jan 2019 07:25:06 PM UTC, comment #1: 

iconv_open already supports UTF-16, UTF-16BE, and UTF-16LE, see https://www.gnu.org/savannah-checkouts/gnu/libiconv/documentation/libiconv-1.15/iconv_open.3.html .

It is the duty of the application that
1) calls iconv_open,
2) sends data accompanied with an encoding name
to follow the section 3.3 of RFC 2781.

Bruno Haible <haible>
Group administrator
Wed 30 Jan 2019 03:24:06 PM UTC, original submission:  

Some platforms only support unlabeled content, the same systems may have tools which are expecting Big Endian and other (native) tools expecting Little Endian.

As such it is most appropriate to use UTF-16, without labeling, with a BOM, and in the Little Endian byte order.

Anonymous

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by jpyeron (Voted in favor of this item)
  • -email is unavailable- added by jpyeron (Posted a comment)
  • -email is unavailable- added by haible (Posted a comment)
  •  

    There are 25 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only logged-in users can vote.

     

    Follow 5 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2021-06-27 haible CategoryNone Encodings
    2019-01-30 jpyeron Carbon-Copy- Added jpyeron
    2019-01-30 haible StatusNone Works For Me
        Assigned toNone haible
        Open/ClosedOpen Closed

    Back to the top

    Powered by Savane 3.13-f8d8.
    Corresponding source code