bugMIT/GNU Scheme - Bugs: bug #62618, utf8->string and...

 
 

bug #62618: utf8->string and string-foldcase do normalization by default

Submitter:  Taylor R. Campbell <riastradh>
Submitted:  Sun 12 Jun 2022 03:43:05 PM UTC
   
 
Category:  runtime Severity:  3 - Normal
Priority:  5 - Normal Item Group:  Incorrect behavior
Status:  Fixed Privacy:  Public
Assigned to:  cph Originator Name: 
Open/Closed:  Closed
Keywords: 
* Mandatory Fields

Add a New Comment Rich Markup
   

Sat 10 Dec 2022 09:58:17 AM UTC, comment #4: 

Fixed on master.

Changed the string procedures to NOT normalize by default, and tweaked the ref manual to state that it's the programmer's responsibility to normalize as needed.

Since any string composed of code points below 0x0300 is automatically in NFC, this shouldn't cause issues for most users.

Chris Hanson <cph>
Group administrator
Fri 09 Dec 2022 03:25:56 AM UTC, comment #3: 

Thanks, that is a good argument.

I'll relax the use of NFC.  I'm not sure exactly how, since that will require some careful thought.

I may also defer this to after the version 12 RC since I want to get that out this year and there's potential disruption from the change.

Chris Hanson <cph>
Group administrator
Tue 06 Dec 2022 04:27:57 PM UTC, comment #2: 

When I filed this issue, I was trying to use MIT Scheme interactively to figure out why a source code repository was getting screwed up on an Apple file system.

The reason turned out to be disagreement about normalization, but my attempts to decode the results in MIT Scheme to examine them codepoint-by-codepoint were inconclusive because it normalized everything, which was getting in my way.

What happened was:
(a) someone committed a file with a name in NFC, because that's what most of the world prefers, including MIT Scheme;
(b) someone else checked out the repository on HFS+, which noramlizes to NFD, changing the file's name as an octet string;
(c) the revision control system (cvs at the time, later converted to hg; I think the same will happen in git) saw a deleted file and a new file, and they committed the new file.

The result was a screwed-up repository, with two copies of the same file under equivalent NFC vs NFD names, that couldn't be checked out in macOS on HFS+ or APFS (which is normalization-preserving but normalization-insensitive) because of the file name collision in normal form.

In a scenario like this, even if all file names are restricted to being valid UTF-8 strings, and even if as a policy all file names must be unique up to normalization, normalizing the strings automagically turned out to be counterproductive because of disagreements about which normal form to use -- leading, amusingly, to distinct copies of the same file with two names having the same normal forms.

So what I wanted was a procedure to just do the standard Unicode UTF-8 decoding algorithm, and nothing else.  Separately it would be helpful to have an operation for computing normalization, mapping one codepoint sequence into another codepoint sequence (as strings).

In any case, even if we restrict pathnames to be Unicode codepoint sequences, it will be problematic to normalize them automagically like this.  For example:

$ mkdir tést   # NFC
$ ln -s tést nfctest
$ mkdir tést   # NFD
$ ln -s tést nfdtest

(file-soft-link? "nfctest")
;Value: "t\xe9;st"

(file-soft-link? "nfdtest")
;Value: "t\xe9;st"

The correct answer in the second case is "te\x301;st".  This causes problems for pathname operations too:

$ echo nfc > nfctest/x
$ echo nfd > nfdtest/x

(read-file "nfdtest/x")
;Value: (nfd)

(read-file (merge-pathnames "x" (pathname-as-directory (file-soft-link? "nfdtest"))))
;Value: (nfc)

In general, it seems to me that applications relying on normalization for comparison, search, and indexing should ask for that explicitly so they don't stumble over things like this unless they intend so.

(Personally I feel we should deal only in octet strings which are sometimes restricted to be valid UTF-8 sequences, rather than code point arrays, and use cursors instead of ordinal numbers of code units for indexing, but I'm not doing all that work so don't mind my kvetching about that here!)

Taylor R. Campbell <riastradh>
Group Member
Mon 05 Sep 2022 08:14:05 AM UTC, comment #1: 

Is there a use case you're thinking of? Or is this about performance?

Generally, NFC is the most useful form for strings since it is used for comparison, search, and indexing. So most strings are normalized by default.

Which is why I'd like to know what advantage there is to not normalizing.

Chris Hanson <cph>
Group administrator
Sun 12 Jun 2022 03:43:05 PM UTC, original submission:  

The UTF-8 decoding algorithm and Unicode case-folding algorithm do not involve normalization, but the utf8->string and string-foldcase procedures always normalize into NFC.

This is because they use the default mode of string-builder, which is nfc.  If they used (builder 'immutable) instead of (builder) this would fix the problem, but I tend to think that the string builder should default to no normalization unless you ask for it.

In any case there should also be automatic tests of utf8->string and string-foldcase and others to catch this.

Taylor R. Campbell <riastradh>
Group Member

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by cph (Posted a comment)
  • -email is unavailable- added by riastradh (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only logged-in users can vote.

     

    Follow 5 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2022-12-10 cph StatusConfirmed Fixed
        Open/ClosedOpen Closed
    2022-12-09 cph StatusNeed Info Confirmed
        Assigned toNone cph
    2022-12-06 cph StatusNone Need Info

    Back to the top

    Powered by Savane 3.13-02a9.
    Corresponding source code