Internationalization/Multilingualization Torbjörn Gannholm has kindly offered to work on this. I just got back from a conference in Japan about multilingual information processing in free software, organized by the MULE folks. While there I put together a pretty clear idea of how I want Guile to work: * Guile should have a separate "byte array" and "string" types. Probably the byvect stuff in unif.c is a decent "byte array" type already, but we may need to beef up support for it. * All characters should be Unicode characters, and all strings are strings of Unicode characters. (We'll need a read syntax for Unicode characters; I think Marc Feely has a proposal for this.) * I/O ports should know what encoding they're reading or writing (ISO Latin-1 by default), and do the appropriate translations. Some structure that allows us to layer translators on top of raw ports might be nice, to decouple the character set support from the I/O source support. * Internally, strings should be represented as UTF-8 encoded strings; this is the representation that C code linked with Guile will see, and operate on. Guile should provide convenient functions to ease the complexity of handling these. * However, Scheme code will still index strings by character, not byte. The expression (string-ref s 1) will always return the second character of s, even if the first character is several bytes long. The fact that the elements of the string are of varying width will be concealed from Scheme code. This means that string-ref and string-set! will no longer be constant time operations. Oh well; people usually manipulate strings using searching, substring extraction, and concatenation anyway; the complexity of those operations is unaffected. * Each string object should record the byte position of the first non-single-byte character, so we can still index strings containing only fixed-width characters (ASCII) in constant time. This isn't perfect, but here's my rationale: My driving concern is what I'll call the "pass-through" problem. In the process of carrying out a user's request, each piece of data will pass through many different modules. For example, data might be read from an I/O port, stored in a database, and then retrieved from that database and displayed by a GUI toolkit. If any module fails to handle multilingual data correctly, the user will experience the overall system as non-multilingual. This means that it's not enough to merely have most modules handle multilingual text correctly. They must all do it, if we are to earn the user's trust. We will need to police Guile modules carefully, put pressure on the authors of non-multilingual modules, support them with plenty of helpful routines and documentation, mark entries in the public module archive as "multilingual-safe", and so on. But if we're to impose this burden on developers, it must be a reasonable burden. We don't actually have any authority; we rely on their good will. If we require them to become experts on every character set and encoding on the planet, that's too much; they simply won't bother. If we require developers to do too much, they will do very little. The proposal above presents the developer with a single character set, whose semantics are clearly documented. Developers working in C must also cope with a variable-width encoding, which does complicate code, but it has some nice properties that ameliorate the complexity somewhat. It has been suggested that Guile simply use Unicode encoded in sixteen-bit characters throughout, as Java does. However, this isn't viable; the 16-bit space for Unicode is almost full, and the Taiwanese have a ton of characters they want code points for. If you represent them using the Unicode ``surrogate characters'', then you've got a variable-width encoding again; you might as well use UTF-8 and save memory, since you're not saving any complexity. It has also been suggested that Guile use two or three different string representations, with eight, sixteen, or thirty-two bits per character. Guile could automatically select the most dense representation capable of holding the data at hand. However, this would require everyone working in C to write out three copies of their string-processing code. Each copy would be simpler than the code for handling UTF-8, because it would be working with a fixed-width encoding, but it's my sense that a single UTF-8 loop is less hair than three fixed-width loops. So, I'd love to have routines to convert text between Unicode and all the various local encodings --- the JIS standards, BIG5, ISO 8859, and so on. Guile should try to use the gconv interface in the GNU C library, then iconv, and then whatever else is available. Henry Spencer's latest regexp engine handles UTF-8, but as of this writing, it hadn't been optimized yet. Guile also needs some kind of gettext interface. We could add a new syntax for translatable strings like #"This is a translatable string."