bugGNU gettext - Bugs: bug #48481, [PATCH] Allow CLDR plural rules...

 
 

You are not allowed to post comments on this tracker with your current authentication level.

bug #48481: [PATCH] Allow CLDR plural rules lookup fallback

Submitter:  Michele Locati <mlocati>
Submitted:  Tue 12 Jul 2016 08:14:53 AM UTC
   
 
Category:  Plural forms Severity:  3 - Normal
Item Group:  None Status:  None
Privacy:  Public Assigned to:  None
Open/Closed:  Open

Jump to the original submission

Thu 04 Aug 2016 09:11:02 AM UTC, comment #7: 

What about something like this http://cpp.sh/9hov3f ?

PS: I moved the code to https://github.com/mlocati/gettext-unicode-locale-id

Michele Locati <mlocati>
Wed 03 Aug 2016 08:10:20 PM UTC, comment #6: 

Just adding some data points, you might want to look at the recent efforts to adopt CLDR script names in glibc:
https://sourceware.org/ml/libc-locales/2016-q2/msg00034.html
and some *BSD variants:
https://wiki.freebsd.org/LocaleNewApproach

Daiki Ueno <ueno>
Group administrator
Wed 03 Aug 2016 04:29:10 PM UTC, comment #5: 

I've had some time to take a look at this.

What about adding some helper functions to work with locale identifiers in Gettext and Unicode formats?

I mean, something like this: http://cpp.sh/7smfb (attached file: parse-locale-identifiers.c )


(file #38093)

Michele Locati <mlocati>
Wed 03 Aug 2016 01:46:00 PM UTC, comment #4: 

Now I understand what you mean.

BTW, I don't understand all the components of the locale identifiers as defined by http://git.savannah.gnu.org/cgit/gettext.git/tree/gettext-runtime/intl/finddomain.c#n71

For Unicode we have this definition: http://unicode.org/reports/tr35/#Unicode_language_identifier

Here's an example of an Unicode ID with all the fields:
it-Latn-IT-POSIX-NYNORSK

Where:
it: language subtag
Latn: script subtag
IT: region subtag
POSIX, NYNORSK: variant subtags

For gettext IDs, we have:
language[_territory][.codeset][@modifier]
What are the values of codeset and modifier? Do their meaning overlaps somehow with script/variant as defined by Unicode?

Michele Locati <mlocati>
Wed 03 Aug 2016 08:18:47 AM UTC, comment #3: 

Yes, that's what I understand from your code, and I wonder if it is a reasonable behaviour; shouldn't it signal an error or stop processing when it sees a '@' twice?

By the way, I think there should be a reliable conversion between CLDR locale names and gettext locale names.

Daiki Ueno <ueno>
Group administrator
Wed 03 Aug 2016 07:57:56 AM UTC, comment #2: 

My code build alternative IDs looking for these chars: '.', '_', '-', '@'.
So, if locale is "foo@bar@baz", we have this list of IDs:
- "foo@bar@baz"
- "foo@bar"
- "foo"

Instead of storing copies of the string, I save the position of the stop chars. In the above case, we'll have these values:
- locale_lengths_count = 3
- locale_lengths[0] = 11
- locale_lengths[1] = 7
- locale_lengths[2] = 3

When looking for the plural rule, the code compares the IDs using each of these locale_lengths using xmlStrncmp and stops at the first match.

So, if someone specify "en@us", the code won't convert it to "en-us", but simply looks for "en@us" and "en".

Michele Locati <mlocati>
Wed 03 Aug 2016 07:16:12 AM UTC, comment #1: 

Thank you for the patch, it makes sense.

Regarding the implementation, though I haven't tested the patch, I wonder if it accepts bogus locales like: foo@bar@baz?  Also doesn't it stop at the second '-' if the locale looks like: it-IT.UTF-8?

I would like to make it more strictly conforming to the existing logic:
http://git.savannah.gnu.org/cgit/gettext.git/tree/gettext-runtime/intl/finddomain.c#n71

Would you mind checking the above cases, and if possible add a similar comment there?

Daiki Ueno <ueno>
Group administrator
Tue 12 Jul 2016 08:14:53 AM UTC, original submission:  

Let's allow the CLDR plural rules lookup to fallback to "parent" locale ids.
For instance, looking for "it_IT.UTF-8" will also find "it_IT" and "it".

Michele Locati <mlocati>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attached Files

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by haible (Updated the item)
  • -email is unavailable- added by ueno (Posted a comment)
  • -email is unavailable- added by mlocati (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

     

    Follow 3 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2016-11-27 haible CategoryNone Plural forms
    2016-08-03 mlocati Attached File- Added parse-locale-identifiers.c, #38093
    2016-07-12 mlocati Attached File- Added 0001-Allow-CLDR-plural-rules-lookup-fallback.patch, #37823

    Back to the top

    Powered by Savane 3.13-d3ae.
    Corresponding source code