buglibiconv - Bugs: bug #39238, Fixing bad utf8.

 
 

bug #39238: Fixing bad utf8.

Submitted by:  None
Submitted on:  Wed Jun 12 11:04:32 2013  
 
Category: ConvertersSeverity: 3 - Normal
Item Group: NoneStatus: Fixed
Privacy: PublicAssigned to: Bruno Haible <haible>
Open/Closed: Closed

Add a New Comment (Rich MarkupRich Markup):
   

You are not logged in

Please log in, so followups can be emailed to you.

 

Fri Nov 18 08:47:41 2016, comment #3:

Sorry, I must have been confused since which implementation is used varies across Linux distributions. Good to see some activity here!

Milan Bouchet-Valat <nalimilan>
Thu Nov 17 23:14:05 2016, comment #2:

There are two iconv implementations in GNU: The one in GNU libc and the one in GNU libiconv. They operate in similar ways, but they are a different codebase.

As can be seen from your command 'iconv --from=UTF-8 --to=UTF-8 -c' (long options), you must be using the GNU libc one.

For GNU libc, the behaviour you reported is equivalent to https://sourceware.org/bugzilla/show_bug.cgi?id=2373

For GNU libiconv, the behaviour you reported has now been corrected: http://git.savannah.gnu.org/gitweb/?p=libiconv.git;a=commit;h=0a04404c90d6a725b8b6bbcd65e10c5fcf5993e9

Thank you for the report.

Bruno Haible <haible>
Project AdministratorIn charge of this item.
Tue Sep 10 10:02:09 2013, comment #1:

I am experiencing a similar issue: the invalid sequence \U3e3965 is not correctly removed. This is a problem since PCRE does not accept this character and generates an error.

# Correctly removed
$ echo -e "\xff" | iconv --from=UTF-8 --to=UTF-8 -c

# Not removed as it should
$ echo -e "\xf8\x8f\xa3\xa5\xa5" | iconv --from=UTF-8 --to=UTF-8 -c
���

Milan Bouchet-Valat <nalimilan>
Wed Jun 12 11:04:32 2013, original submission:

Hi,

I'm trying to clean utf8 encoded buffer using iconv and remove all invalid utf8 code points(below is my code snippet)

iconv_t conv = iconv_open("UTF-8//IGNORE","UTF-8");
iconvctl(conv, ICONV_SET_DISCARD_ILSEQ, &flag);
iconv(conv, &inbuf, &len, &outbuf, &outlen);
iconv_close(conv);

I noticed that it is not able to remove code points in range of 0xD800-0xDfff (surrogate pairs) even though these are not valid utf8 code points.

Other invalid code points like beyond 0x10FFFF and wrong/broken utf8 byte sequence is are cleaned successfully.

What might be the issue here?

Thanks,
Anurag

Anonymous

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach File(s):
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -unavailable- added by haible (Posted a comment)
  • -unavailable- added by nalimilan (Posted a comment)
  •  

    Do you think this task is very important?
    If so, you can click here to add your encouragement to it.
    This task has 0 encouragements so far.

    Only logged-in users can vote.

     

    Please enter the title of George Orwell's famous dystopian book (it's a date):

     

     

    Follow 4 latest changes.

    Date Changed By Updated Field Previous Value => Replaced By
    Wed Nov 23 23:55:25 2016haibleCategoryNone=>Converters
    Thu Nov 17 23:14:05 2016haibleStatusNone=>Fixed
      Assigned toNone=>haible
      Open/ClosedOpen=>Closed

    Back to the top


    Powered by Savane 3.1-cleanup