bugGNUstep - Bugs: bug #30172, Converting strings lossy results...

Group
 
 

bug #30172: Converting strings lossy results to different strings than on OS X

Submitter:  David Wetzel <dwetzel>
Submitted:  Fri 18 Jun 2010 12:24:50 AM UTC
   
 
Category:  Base/Foundation Severity:  1 - Wish
Item Group:  Change Request Status:  Postponed
Privacy:  Public Assigned to:  None
Open/Closed:  Analyzed
* Mandatory Fields

Add a New Comment Rich Markup
   

Jump to the original submission

Sat 05 Mar 2011 04:51:37 PM UTC, comment #7: 

No patch provided for this :-(

Richard Frith-Macdonald <CaS>
Group Member
Mon 16 Aug 2010 03:00:32 PM UTC, comment #6: 

GNUstep is doing the lossy conversion by using the translit flag of iconv. The documentation for iconv states:
"When the string "//TRANSLIT" is appended to tocode, transliteration is activated. This means that when a character cannot be represented in the target character set, it can be approximated through one or several characters that look similar to the original character."
This sounds just like the thing we want to do. And your example shows that this conversion is really eager to provide the best possible transliteration.

Fred Kiefer <FredKiefer>
Group Member
Fri 18 Jun 2010 10:22:57 AM UTC, comment #5: 

Actually, I checked the source and GNustep generally (ie except for the gsm characterset, which generally isn't available) uses iconv() to do transliteration ... so it's using the local standard character transliteration mechanism.

Your example:
  'Ortsgespräch' (with a umlaut) results in 'Ortsgespr"ach'
demonstrates that GNUstep IS performing a transliteration.

Now it's possible that something is going wrong with it, but I've seen nothing to suggest that ... the evidence simply seems to show that iconv() does a different transliteration to the one done on OSX.


Richard Frith-Macdonald <CaS>
Group Member
Fri 18 Jun 2010 09:55:37 AM UTC, comment #4: 

I think the last comment rather misses the point ... the documentation on the apple site referred to does not specify the algorithm used for the conversion, or exactly what the results should be, just an example of one possible conversion. IIRC the existing GNUstep code does some mapping between charactersets, and where no specific mappings are set up uses a question mark or asterisk or somesuch.

If you can provide the mappings to use to exactly match OSX behavior, it ought to be possible to easily code things ... I'd be very happy to see a patch to implement them.

You say you aren't using any C strings ... but the method you pointed me to (-dataUsingEncoding:allowLossyConversion:) converts an NSString to a C string held inside an NSData object.  Probably you are just using the term slightly differently ... I make a distinction between C strings and NSStrings, and I don't think you can meaningfully talk about the encoding of characters in an NSString (since NSString is an opaque class cluster) so any conversation about characters in a specific encoding refers to a C string rather than an NSString.  Of course, avoiding confusion gets really tricky if you are talking about the internals of specific subclasses inside the class cluster ... let's not got there.

Richard Frith-Macdonald <CaS>
Group Member
Fri 18 Jun 2010 09:33:00 AM UTC, comment #3: 

please read the documentation.

http://developer.apple.com/mac/library/documentation/cocoa/reference/Foundation/Classes/NSString_Class/Reference/NSString.html#//apple_ref/occ/instm/NSString/dataUsingEncoding:allowLossyConversion:

so an ä will become a
ö -> o
á -> a

etc

The implementation in base is just wrong.
And I am not using any C strings here.
See at the code I posted. its just a category on NSString, so you can create a test program from it.

Cheers,

David

David Wetzel <dwetzel>
Group Member
Fri 18 Jun 2010 09:24:44 AM UTC, comment #2: 

I don't think we have a definition of what a lossy string should be (so we used our own).  I agree it would be nice to have compatibility here, though given that this is the first comment I recall on it in about a decade, it can't be that people use lossy C string format much.
Perhaps you could provide some testcases to demonstrate the algorithm OSX uses to convert a unicode string to a lossy C string (possible something as crude as removing every non-ascii character) ... if so it should be easy to change the code to be compatible.

Richard Frith-Macdonald <CaS>
Group Member
Fri 18 Jun 2010 08:41:25 AM UTC, comment #1: 

I tried it on OPENSTEP 4.2.
Same results as on MacOS X Snow Leopard.
See attached pic.


David Wetzel <dwetzel>
Group Member
Fri 18 Jun 2010 12:24:50 AM UTC, original submission:  

Hi,

feeding  'Ortsgespräch' (with a umlaut) into the following method results in

 'Ortsgespr"ach' on GNUstep and  'Ortsgesprach' on OS X 10.6.4

- (NSString*) lossyString
{
  NSString * myStr = nil;
  NSData * myData = [self dataUsingEncoding:NSASCIIStringEncoding
                       allowLossyConversion:YES];

  myStr = [[NSString alloc] initWithData:myData
                                encoding:NSASCIIStringEncoding];

  [myStr autorelease];
 
  return myStr;
}



David Wetzel <dwetzel>
Group Member

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

Attached Files
file #20774:  NeXT.png added by dwetzel (60KiB - image/png)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by FredKiefer (Posted a comment)
  • -email is unavailable- added by CaS (Posted a comment)
  • -email is unavailable- added by dwetzel (Submitted the item)
  • -email is unavailable- added by dwetzel
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only logged-in users can vote.

     

    Follow 8 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2011-03-05 CaS StatusNone Postponed
        Open/ClosedOpen Analyzed
    2010-06-18 CaS Item GroupBug Change Request
    2010-06-18 dwetzel Item GroupChange Request Bug
    2010-06-18 CaS Severity3 - Normal 1 - Wish
        Item GroupBug Change Request
    2010-06-18 dwetzel Attached File- Added NeXT.png, #20774
    2010-06-18 dwetzel Carbon-Copy- Added -email is unavailable-

    Back to the top

    Powered by Savane 3.13-bb6a.
    Corresponding source code