develooper Front page | perl.perl5.porters | Postings from April 2008

Re: [perl #51710] utf8::valid rejects characters in \x14_FFFF - \x1F_FFFF

Thread Previous | Thread Next
Glenn Linderman
April 1, 2008 02:19
Re: [perl #51710] utf8::valid rejects characters in \x14_FFFF - \x1F_FFFF
Message ID:
On approximately 3/31/2008 9:31 AM, came the following characters from 
the keyboard of Chris Hall:
> [FWIW, this is not the only place where characters > 0x7FFF_FFFF are 
> FUBAR.  I would ditch the attempt to support anything beyond 
> 0x7FFF_FFFF before the world gets much older !]

I'm somewhat ambivalent here... supporting 64 bits could be useful on 64 
bit platforms; however, if there is ever a 128 bit platforms, there is 
no provision for further extension of the range of values.

The question is, what are the requirements? 

If it is for Unicode only, well, that was a moving target at 
implementation time.  Unicode was 16 bit, then, and ISO-646 was 32 bit, 
and no one knew which one would win.  Unicode seems to be a bit better 
defined now, but it also exceeds 16 bits now, so aren't we glad that we 
aren't locked into a 16-bit implementation?

If it is also for "v string" type things, that are not pure Unicode, but 
just lists of numbers, then where do you draw the line?  Musical 
metrical data has lots of numbers in a string, but they are generally 
small numbers, mostly < 20, so no problem.  IPv4 addresses are 4 bytes 
long, as are Windows version strings.  Dewey decimals are pretty easily 
stuck into strings also.  IPv6 addresses are longer, and the units are 
bigger.  I think they still fit.  What about a list of the number of 
words, or characters in each paragraph of the works of Shakespeare  (get 
those monkeys going on the typewriters)?  Gets long, but unlikely that 
the individual numbers would be too big for today's 72-bit range of 

Now I have no clue what sorts of "stringification" or "canonicalization" 
is done by Storable, but Data::Dumper converts everything to ASCII 
text.  For numbers that's 3.5 bits per character.  If it were hex, you 
get 4 bits per character.  Base 64 gets you 6 bits per character.  But 
you have to encode length also... "extended UTF-8" gets you 6 bits per 
character, and a dynamic length.  Could be pretty handy for that sort of 
application, if the bounds of the numbers are sufficient (and not bound 
by Unicode semantics).  But then the bounds of the "stringified 
numerics" would definitely have to cover the bounds of the numeric 
integers for the platform.

So, what are the requirements?

<imagine this> (or skip to the end of it)

Had I been consulted in the original design of Perl's "extended UTF-8", 
I'm not sure what I'd have suggested back then, but today, if the goal 
is to have extended character ranges, I would make a different trade-off.

Today's implementation (if I understand it correctly):

* being strictly UTF-8 encoding compliant for values 0-0x7FFF_FFFF
* present Perl uses a prefix byte of 0xFE to indicate 36 bit numbers? (6 
more bytes containing 6 data bits each)
* present Perl uses a prefix byte of 0xFF to indicate 72 bit numbers? 
(12 more bytes containing 6 data bits each)

Today, I would suggest a design that is different, handles larger ranges 
of numbers, and doesn't have such a huge jump in size...  Changing only 
the third point to:

* use prefix byte 0xFF to indicate that the next data byte is the number 
of additional data bytes following, containing 6 data bits each in the 
lower 6 bits, with bit 8 set and bit 7 reset.  Since 0xFE already 
implies 6 more data bytes, one could get a little more range by 
suggesting that the value in the 2nd data byte when the first is 0xFF 
should not include the 6 already accounted for, nor itself, nor the "at 
least one more that meant that it wouldn't fit in 6 bytes", so that 
means that 0xFF80 would imply 7 additional data bytes following, and 
0xFFBF would imply 70 additional data bytes following.

This would allow for up to 420 data bits.  It would jump from 7 bytes to 
9 bytes when the 37th bit is needed, but would then grow 1 byte at a 
time with increasing magnitudes, rather than having a 6 byte jump when 
the 37th bit is needed, and topping out at 72 bits.

With the current design, I have no idea why 72 bits were chosen, rather 
than 66, which would seem to cover 64 bit integers well, and only use 12 
bytes instead of that odd 13 byte value.

With my proposed alternate design, it would also be possible to 
interpret the number of additional bytes in other ways than one extra 
byte at a time... a simple scheme could suggest adding 2 bytes at a 
time, jumping from 7 to 10 bytes when the 37th bit is needed, and then 
to 12 when the 49th bit is needed.  This would give a range up to 804 
bits.  A more complex scheme could have numbers from 0-31 mean 2 more 
bytes each, but the numbers 32-63 would mean 4 more bytes each... 
trading off the number of extra bytes for bigger range.  Or a lookup 
table, with each bigger value adding even more bytes.

</imagine this>

Working within the current implementation, however, I mostly agree with 
the following proposal... an extra parameter to specify what kind of 
validity you are requesting.  Not clear if the current behavior is 
useful enough to be a default.

I wonder if an additional optional parameter, containing the Unicode 
version of interest, would be appropriate, or if it should just be 
declared that this particular version of Perl implements this particular 
version of Unicode...

> It seems to me that there's a place for an optional argument for 
> utf8::valid, bitwise:
>    * reject > 0x10_FFFF      ) strict UTF-8
>    * reject 'surrogates'     )
>    * reject 'non-characters' ) for strict UTF-8, external exchange
>    * reject > 0x7FFF_FFFF    ) to filter out Perl's non-standard stuff
> and one could make a case for:
>    * reject private use      ) for external exchange (planes 14 & 15)
>                              ) and U+E000..U+F8FF
>    * reject reserved         ) for tidy-ness... but this could be
>                              ) limited to the "large" reserved areas (?)
>    * reject U+FFFD           ) for external exchange
>    * reject controls         ) for the paranoid -- excluding whitespace
>    * reject > 0x00_FFFF      ) i.e. accept BMP (& priv. use)
>    * reject > 0x01_FFFF      ) i.e. accept BMP & SMP (& priv. use)
>    * reject > 0x02_FFFF      ) i.e. accept BMP, SMP & SIP (& priv. use)
> Something along those lines, anyway.  [Filtering by plane could be 
> made more general.]
> There are probably too many scripts to make it feasible to filter down 
> to that level, here.  Besides, an unvoiced requirement is for this to 
> be a quick "first cut" scan.
> Return value could be byte offset of first rejected object -- -1 => 
> OK. So not a true/false return.
>> Thanks for the report.
>> Steve

Thread Previous | Thread Next Perl Programming lists via nntp and http.
Comments to Ask Bjørn Hansen at | Group listing | About