Subject: IPP> MOD - Comment on RFC 2044: need to specify the ISO 10646 conformance level

From: Tom Hastings <[email-masked]>
Date: Wed, 8 Oct 1997 21:52:15 PDT

ISO 10646 has three conformance levels: 

1. No composed characters
2. Several code points respresenting a single character in certain Asian
languages where such composition is part of the language.
3. Composed characters using non-spacing accents.  Thus accented letters
have at least two different encodings: (1) as a single code point, and (2)
a sequence
of two code points where one is a non-spacing accent.  Characters with more
than one accent are represented by a sequence of three code points.  For
level 3,
a recipient has to fold the alternate forms into a single cannonical form for
purposes of comparing.  One of the advantages of level 3 is that new accented
letters can be invented without having to add new code points to the standard.
Another disadvantage is that characters are not fixed length.

For the purposed of Internet Protocols, we suspect that level 2 is sufficient.
But the UTF-8 definition in RFC 2144 is silent on this matter.  We suggest
that
a refision to RFC 2144 be issues that indicates that utf-8 means just level 2.
Alternatively, register a new value, say, 'utf-8-level-2'.

Tom

---

From: [email-masked]
Date: Fri, 10 Oct 1997 09:34:54 +0200

Tom,
actually I think the conformance levels are a Bad Thing.

According to 10646-1:1993, the conformance is a bit different than
you say: level 1 says to avoid combining characters and the HANGUL JAMOS
block, level 2 says to avoid a specific, short list of 30 combining 
characters, including things like the IDEOGRAPHIC LEVEL TONE MARK (don't ask 
me!) and the
COMBINING CYRILLIC PALATALIZATION, while level 3 says that you have to do
all of it.

(My sneaking suspicion is that these 30 are marks that are commonly used
together with other marks on the same character, but the standard is silent
on this point; Greek, for instance, commonly uses > 1 accent on letters,
and the standard doesn't say "no double diacritics". Would you believe
the (precomposed) Greek letter U+0390, GREEK SMALL LETTER IOTA WITH DIALYTICA
AND TONOS?)
The Unicode standard, version 2.0, seems to say nothing about levels.

All of these levels are requirements on the sender: Don't send certain
characters.

I, like you, believe that a 10646 usage that uses NO combining characters
is simply Not The Right Thing.

And wrt the 30 characters not allowed in 

The recipient is already allowed to do "magic" whenever unrecognized
characters are encountered; telling him to include the code for ignoring
unrecognized diacritcs is not a big deal, and allowing him to skip the
code for handling 30 more "unrecognized" diacritics is simply not worth it.

The coding of comparator functions is another, and complex matter; it is
with good reason that I'm far from explicit about this in the policy
document. And comparision is not a tagging function, but and end-system
operation function; that's why I thought I could get away with it.

(of course, protocols that require comparision of values and don't
address this problem are deficent - but we're not at a point where
we can make policy for those problems. See the ACAP protocol for one
example of language trying to address the issue)

BTW, the mailing list for discussion of the character set policy is
[email-masked]; mail to [email-masked]
to be added.

                           Harald A

---

From: Francois Yergeau <[email-masked]>
Date: Tue, 14 Oct 1997 09:51:59 -0400

>a refision to RFC 2144 be issues that indicates that utf-8 means just
level 2.

A restriction to level 2, which excludes most combining characters, would
severely restrict the expressive power of ISO 10646, and in consequence the
ability of the protocols that use it (in the UTF-8 form) to represent the
textual content that they need to represent for truly world-wide operation.
 IMHO, such a restriction is too serious to be entertained solely on the
basis of "we suspect that level 2 is sufficient."  ISO has not found it
sufficient, it has level 3; Unicode has *only* level 3.  And I think the
purposes of Internet protocols (at least those that carry text) are the
same as the purposes underlying ISO 10646 and Unicode: to enable
communication in all the world's languages.

>Alternatively, register a new value, say, 'utf-8-level-2'.

There is actually a revision of RFC 2044 underway.  The latest draft
(ftp://ds.internic.net/internet-drafts/draft-yergeau-utf8-rev-01.txt) has a
discussion of version-specific labels, which you may find relevant to your
proposal of a level-specific label.  Please take a look, and come back with
your proposal -- preferably to the ietf-charset list, as suggested by
Harald -- if you still think it is appropriate.

Regards,

-- 
François Yergeau <[email-masked]>
Alis Technologies inc., Montréal
Tél : +1 (514) 747-2547
Fax : +1 (514) 747-2561