bugGNUstep - Bugs: bug #30172, Converting strings lossy results...

 
 

You are not allowed to post comments on this tracker with your current authentication level.

bug #30172: Converting strings lossy results to different strings than on OS X

Submitted by:  David Wetzel <dwetzel>
Submitted on:  Fri 18 Jun 2010 12:24:50 AM UTC  
 
Category: Base/FoundationSeverity: 1 - Wish
Item Group: Change RequestStatus: Postponed
Privacy: PublicAssigned to: None
Open/Closed: Analyzed

(Jump to the original submission Jump to the original submission)

Sat 05 Mar 2011 04:51:37 PM UTC, comment #7:

No patch provided for this :-(

Richard Frith-Macdonald <CaS>
Project Member
Mon 16 Aug 2010 03:00:32 PM UTC, comment #6:

GNUstep is doing the lossy conversion by using the translit flag of iconv. The documentation for iconv states:
"When the string "//TRANSLIT" is appended to tocode, transliteration is activated. This means that when a character cannot be represented in the target character set, it can be approximated through one or several characters that look similar to the original character."
This sounds just like the thing we want to do. And your example shows that this conversion is really eager to provide the best possible transliteration.

Fred Kiefer <FredKiefer>
Project Member
Fri 18 Jun 2010 10:22:57 AM UTC, comment #5:

Actually, I checked the source and GNustep generally (ie except for the gsm characterset, which generally isn't available) uses iconv() to do transliteration ... so it's using the local standard character transliteration mechanism.

Your example:
'Ortsgespräch' (with a umlaut) results in 'Ortsgespr"ach'
demonstrates that GNUstep IS performing a transliteration.

Now it's possible that something is going wrong with it, but I've seen nothing to suggest that ... the evidence simply seems to show that iconv() does a different transliteration to the one done on OSX.

Richard Frith-Macdonald <CaS>
Project Member
Fri 18 Jun 2010 09:55:37 AM UTC, comment #4:

I think the last comment rather misses the point ... the documentation on the apple site referred to does not specify the algorithm used for the conversion, or exactly what the results should be, just an example of one possible conversion. IIRC the existing GNUstep code does some mapping between charactersets, and where no specific mappings are set up uses a question mark or asterisk or somesuch.

If you can provide the mappings to use to exactly match OSX behavior, it ought to be possible to easily code things ... I'd be very happy to see a patch to implement them.

You say you aren't using any C strings ... but the method you pointed me to (-dataUsingEncoding:allowLossyConversion:) converts an NSString to a C string held inside an NSData object. Probably you are just using the term slightly differently ... I make a distinction between C strings and NSStrings, and I don't think you can meaningfully talk about the encoding of characters in an NSString (since NSString is an opaque class cluster) so any conversation about characters in a specific encoding refers to a C string rather than an NSString. Of course, avoiding confusion gets really tricky if you are talking about the internals of specific subclasses inside the class cluster ... let's not got there.

Richard Frith-Macdonald <CaS>
Project Member
Fri 18 Jun 2010 09:33:00 AM UTC, comment #3:

please read the documentation.

http://developer.apple.com/mac/library/documentation/cocoa/reference/Foundation/Classes/NSString_Class/Reference/NSString.html#//apple_ref/occ/instm/NSString/dataUsingEncoding:allowLossyConversion:

so an ä will become a
ö -> o
á -> a

etc

The implementation in base is just wrong.
And I am not using any C strings here.
See at the code I posted. its just a category on NSString, so you can create a test program from it.

Cheers,

David

David Wetzel <dwetzel>
Project Member
Fri 18 Jun 2010 09:24:44 AM UTC, comment #2:

I don't think we have a definition of what a lossy string should be (so we used our own). I agree it would be nice to have compatibility here, though given that this is the first comment I recall on it in about a decade, it can't be that people use lossy C string format much.
Perhaps you could provide some testcases to demonstrate the algorithm OSX uses to convert a unicode string to a lossy C string (possible something as crude as removing every non-ascii character) ... if so it should be easy to change the code to be compatible.

Richard Frith-Macdonald <CaS>
Project Member
Fri 18 Jun 2010 08:41:25 AM UTC, comment #1:

I tried it on OPENSTEP 4.2.
Same results as on MacOS X Snow Leopard.
See attached pic.

(file #20774)

David Wetzel <dwetzel>
Project Member
Fri 18 Jun 2010 12:24:50 AM UTC, original submission:

Hi,

feeding 'Ortsgespräch' (with a umlaut) into the following method results in

'Ortsgespr"ach' on GNUstep and 'Ortsgesprach' on OS X 10.6.4

- (NSString*) lossyString
{
NSString * myStr = nil;
NSData * myData = [self dataUsingEncoding:NSASCIIStringEncoding
allowLossyConversion:YES];

myStr = [[NSString alloc] initWithData:myData
encoding:NSASCIIStringEncoding];

[myStr autorelease];

return myStr;
}

David Wetzel <dwetzel>
Project Member

 

Attached Files
file #20774:  NeXT.png added by dwetzel (60KiB - image/png)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -unavailable- added by FredKiefer (Posted a comment)
  • -unavailable- added by CaS (Posted a comment)
  • -unavailable- added by dwetzel (Submitted the item)
  • -unavailable- added by dwetzel
  •  

    Do you think this task is very important?
    If so, you can click here to add your encouragement to it.
    This task has 0 encouragements so far.

    Only logged-in users can vote.

     

    Please enter the title of George Orwell's famous dystopian book (it's a date):

     

     

    Follow 8 latest changes.

    Date Changed By Updated Field Previous Value => Replaced By
    Sat 05 Mar 2011 04:51:37 PM UTCCaSStatusNone=>Postponed
      Open/ClosedOpen=>Analyzed
    Fri 18 Jun 2010 09:55:37 AM UTCCaSItem GroupBug=>Change Request
    Fri 18 Jun 2010 09:33:00 AM UTCdwetzelItem GroupChange Request=>Bug
    Fri 18 Jun 2010 09:24:44 AM UTCCaSSeverity3 - Normal=>1 - Wish
      Item GroupBug=>Change Request
    Fri 18 Jun 2010 08:41:25 AM UTCdwetzelAttached File-=>Added NeXT.png, #20774
    Fri 18 Jun 2010 12:24:50 AM UTCdwetzelCarbon-Copy-=>Added -unavailable-

    Back to the top


    Powered by Savane 3.1-cleanup1