/[classpath]/classpath/doc/unicode/unicode.database.format
ViewVC logotype

Diff of /classpath/doc/unicode/unicode.database.format

Parent Directory Parent Directory | Revision Log Revision Log | View Patch Patch

revision 1.1 by rao, Sun Aug 9 23:07:47 1998 UTC revision 1.2 by ericb, Sun Feb 17 07:30:03 2002 UTC
# Line 3  GNU Classpath Unicode Attribute Database Line 3  GNU Classpath Unicode Attribute Database
3  java.lang.Character allows one to retrieve information on all 38,887  java.lang.Character allows one to retrieve information on all 38,887
4  characters of the Unicode character set.  This is a lot of data.  The  characters of the Unicode character set.  This is a lot of data.  The
5  database specification outlined here is meant to be fast, small, and  database specification outlined here is meant to be fast, small, and
6  upgradable to new versions of the Unicode 2 specification (minus  upgradable to new versions of the Unicode 3 specification (minus
7  Character.Subset information) by running a script on the data files  Character.UnicodeBlock information) by running a script on the data
8  that the Unicode Consortium distributes.  files that the Unicode Consortium distributes.
9    
10  The database consists of three files:  The database consists of three files:
11  1) character.uni (main database of character attributes)  1) character.uni (main database of character attributes)
# Line 14  The database consists of three files: Line 14  The database consists of three files:
14    
15  File sizes for Unicode 2.1.2 spec  File sizes for Unicode 2.1.2 spec
16  ---------------------------------  ---------------------------------
17  character.uni: 16359 bytes  character.uni:  8760 bytes
18  block.uni    :  4995 bytes  block.uni    :  4122 bytes
19  titlecase.uni:    16 bytes  titlecase.uni:    72 bytes
20    
21    File sizes for Unicode 3.0.0 spec
22    ---------------------------------
23    character.uni:  9776 bytes
24    block.uni    :  4938 bytes
25    titlecase.uni:    48 bytes
26    
27    File sizes for Unicode 3.2.0 spec
28    ---------------------------------
29    character.uni: 10784 bytes
30    block.uni    :  5586 bytes
31    titlecase.uni:    48 bytes
32    
33  All quantities are unsigned unless otherwise specified.  All quantities are unsigned unless otherwise specified.
34  All quantities are stored in big endian format.  All quantities are stored in big endian format.
35    
36  character.uni  character.uni
37  -------------  -------------
38  Some characters in the Unicode specification have an entry in the  All characters in the Unicode specification have an attributes entry
39  character.uni file (characters in compressed blocks do not).  Characters  in the character.uni file (alhough compression is used to share
40  are stored sequentially, based on the Unicode character number, and  entries where possible).  Characters are stored sequentially, based on
41  there are no null entries.  Each entry consists of 7 bytes.  the Unicode character number within a block, although multiple blocks
42    may point to the same character attribute location.  There are no null
43  C = Category  entries. Each entry consists of 8 bytes.  The file is limited to 4096
44  N = Numerical Decimal Value  entries (32768 bytes), without making changes to the semantics of
45      (0xFFFF if unused, 0xFFFE if not representable as nonnegative  block.uni.
46       integer value.  If category is "Nd", then this is also the  
47       decimal digit value.)  C = Category (values correspond to constants in Character. For
48  S = No-break version of a space (1/0)       example, "Nd" maps to 9.)
49  U = Uppercase mapping (0 = no mapping)  N = Numerical Decimal Value: signed.  -1 if unused, -2 if not
50  L = Lowercase mapping (0 = no mapping)       representable as nonnegative integer value, otherwise a
51         non-negative value.  (For example, this is the decimal digit
52         value of the category "Nd".  Note that Java also attaches a value
53         to some sequences of 'a'-'z'; see Character.getNumericValue(char)
54         for details.) In a compressed block, subsequent characters must
55         adjust their value by their distance from the start of the block
56         if the value is non-negative.
57    S = Non-breaking space (1/0)
58    U = Offset to uppercase mapping: signed. Characters with no mapping
59         will have 0, which maps back to itself. Otherwise, the value of
60         toUpperCase(ch) is (char) (ch - U).
61    L = Offset to lowercase mapping: signed. Characters with no mapping
62         will have 0, which maps back to itself. Otherwise, the value of
63         toLowerCase(ch) is (char) (ch - L).
64    D = Directional category: signed (Values correspond to constants in
65         Character, with -1 meaning unassigned. For example, 'a' has
66         direction "L", which maps to 0.)
67    M = Mirrored (1/0)
68  x = Empty  x = Empty
69    
70   xxSCCCCC   NNNNNNNN   NNNNNNNN   UUUUUUUU   UUUUUUUU   xMSCCCCC   NNNNNNNN   NNNNNNNN   UUUUUUUU   UUUUUUUU
71  \________/ \________/ \________/ \________/ \________/  \________/ \________/ \________/ \________/ \________/
72    byte 6     byte 5     byte 4     byte 3     byte 2    byte 7     byte 6     byte 5     byte 4     byte 3
73    
74   LLLLLLLL   LLLLLLLL   LLLLLLLL   LLLLLLLL   DDDDDDDD
75  \________/ \________/  \________/ \________/ \________/
76    byte 1     byte 0    byte 2     byte 1     byte 0
77    
78  ranges of values in Unicode 2.1.2 spec  ranges of values
79    
80  C = 0..28 (Sun uses 0..28, and skips 17, so that's what we do too)  C = 0..30 (Sun skips 17, so that's what we do too)
81  B = 1..69  N = -2..10000
82  N = 0..10000  D = -1..18
 D = 0..9  
83    
84  block.uni  block.uni
85  ---------  ---------
86  Characters within the Unicode specification tend to come in blocks --  Characters within the Unicode specification tend to come in blocks --
87  sets of sequential characters.  The Classpath Unicode database takes  sets of sequential characters.  The Classpath Unicode database takes
88  advantage of this property.  Each entry in the block.uni file consists  advantage of this property.  Each entry in the block.uni file consists
89  of 9 bytes.  Entries are stored sequentially, based on the Unicode  of 6 bytes.  Entries are stored sequentially, based on the Unicode
90  character number which starts a block.  If the compressed bit is set,  character number which starts a block.  If the compressed bit is set,
91  then there is only one entry for this block in the character.uni file.  then offset O is the only entry for this block in the character.uni
92  That entry in the character.uni file represents the attributes of all  file, and represents the attributes of all characters in the
93  the characters of that block.  block. Otherwise, the characters in the block occupy sequential
94    attributes, starting with character S at offset O.
 Note: For Unicode 2.1.2, compressed blocks are mandatory for:  
   
 U+4E00 - U+9FFF: The CJK Ideographs Area  
 U+AC00 - U+D7A3: The Hangul Syllables Area  
 U+D800 - U+DFFF: The Surrogates Area  
 U+E000 - U+F8FF: The Private Use Area  
 U+F900 - U+FAFF: CJK Compatibility Ideographs  
95    
96  S = Unicode character which represents start of block  S = Unicode character which represents start of block
97  E = Unicode character which represents end of block  E = Unicode character which represents end of block
98  O = Offset of this block within the character.uni file  O = Offset of this block within the character.uni file
99  C = Compressed  C = Compressed (1/0) If compressed, all characters in the block share
100  x = Empty       a common attribute, otherwise the characters in the block occupy
101         sequential attributes starting at offset.
102   SSSSSSSS   SSSSSSSS   EEEEEEEE   EEEEEEEE   xxxxxxxC  
103  \________/ \________/ \________/ \________/ \________/   SSSSSSSS   SSSSSSSS   EEEEEEEE   EEEEEEEE
104    byte 8     byte 7     byte 6     byte 5     byte 4  \________/ \________/ \________/ \________/
105      byte 5     byte 4     byte 3     byte 2
106    
107   OOOOOOOO   OOOOOOOO   OOOOOOOO   OOOOOOOO     COOOOOOO   OOOOOOOO
108  \________/ \________/ \________/ \________/  \________/ \________/
109    byte 3     byte 2     byte 1     byte 0      byte 1     byte 0
110    
111  titlecase.uni  titlecase.uni
112  -------------  -------------
113  Characters in which the titlecase differs from the uppercase are  Characters in which the titlecase differs from the uppercase are
114  stored in titlecase.uni.  There are only four characters in the  stored in titlecase.uni.  There are only twelve characters in the
115  Unicode 2.1.2 specification which fit this description, and it's  Unicode 3.0.0 specification which fit this description, and it's
116  doubtful that any others will ever be added to the specification.  doubtful that any others will ever be added to the specification.
117  However, we should be able to support more, without changing  However, we should be able to support more, without changing
118  java.lang.Character, and this is why we have not hardcoded these  java.lang.Character, and this is why we have not hardcoded these

Legend:
Removed from v.1.1  
changed lines
  Added in v.1.2

savannah-hackers-public@gnu.org
ViewVC Help
Powered by ViewVC 1.1.26