going beyond Unicode

I have some food for thought, which I'm no

Perl 6 is defined in terms of Unicode in many respects, but in other respects it 
also seems to be agnostic to the character repertoire or encoding or abstraction 
level or whatever, where the latter is explicitly encoded as meta-data in Str 
values for example, allowing multiple representations of text in some manners of 
speaking.

Now, I believe that if Perl 6 truly wants to be all-encompassing, such that any 
other language can be expressed as a grammar of Perl 6, or that otherwise Perl 6 
should be more flexible to handle anyone's text needs, it can't be restricted by 
Unicode.

You see, as is known or documented in many places, numerous aspects of Unicode 
are controversial, such that while it does a lot, there are a lot of needs it 
doesn't address, and there are various complexities.

For example, there was the controversy of Han unification, where glyphs with 
very similar appearance from multiple cultures were treated as being the same 
characters in Unicode, while many Asian people want to treat them as distinct.

This behavior is also in contrast to how for some scripts Unicode provides 
various redundant codepoints for the same glyphs.

And there are other examples of characters from various scripts which are 
missing or mis-organized in Unicode, according to some users of those scripts.

One consequence of this, is that other character repertoires have been created 
or have not been dropped, and are in use alternatively to Unicode, such as by 
some cultures opposed to the Han unification, so that they can properly express 
what they mean to say.

I propose that Perl 6 extend its existing support for multiple character 
abstractions, with its meta-tagged code strings, so that character repertoires 
that extend outside of Unicode are also supported.

Examples are Mojikyo, TRON, GB18030, and several others.

See http://www.jbrowse.com/text/unij.html for some information on the matter.

See also http://www.ruby-forum.com/topic/165927 where related matters were 
discussed for Ruby, but I found that after I wrote this message.

*The idea here is that by being more flexible in what is supported, it is easier 
for Perl 6 users to express what they actually want to say in their code or in 
data processed by it.*

A corollary to there being allowed alternatives to Unicode, is that if we wanted 
to the Perl 6 spec could possibly be split more with some aspects being 
considered more core and some less so, and the support for vast or complicated 
character sets like Unicode in general could be made more optional.  The idea 
here being that it is possible to make more well-defined what a more minimal 
Perl 6 may consist of.  I suggest plain ASCII be the minimum and everything more 
is optional.  And pluggable.

Making the big complicated charsets optional and pluggable is good, I think.

-- Darren Duncan

• going beyond Unicode by Darren Duncan