develooper Front page | perl.perl6.internals.unicode | Postings from February 2001

Re: string encoding

Thread Previous | Thread Next
Simon Cozens
February 16, 2001 16:32
Re: string encoding
Message ID:
Moved to -unicode, because that's what it's *for*.

On Fri, Feb 16, 2001 at 01:17:03PM -0800, Hong Zhang wrote:
> > substr's already been mentioned.
> I have already given the counter argument. The codepoint position is useless
> in many cases. They should be deprecated.

Uh? That doesn't make sense. Codepoint position is *exactly* what people
expect when they use substr. When I say 

    $a = substr($b,10);

I want the 10th character. If I get the 10th byte, and we're using UTF-8
as you suggest, I might be cutting into the middle of a character, leaving
the resulting string malformed. That's horrific.

> I designed and implemented most of Java regular expression. I don't feel
> significant different between UTF-8 and UTF-16. Under some cases UTF-8
> are much better.

Try telling Jarkko. :)

> I agree the constant will be higher and code will not be easy. But I don't
> believe that will be a significant problem. It is just a small problem
> of dealing with Unicode.

No problem vs. problem.

I know which I'd choose.

> My understand the chop() can be very efficient under common cases, "abc\n",
> for both UTF-8 and UTF-32. 

What about in the case of "abc\x{1F1E}"? UTF-32 or UTF-16 here is *vastly*
more efficient than UTF-8.

> The s/.// case is misleading too. If you define . as [^\n], the UTF-8 and
> UTF-32 will have exactly the same performance

No, no, no, no, no, no, no.

UTF-16 case: Remove first two bytes
UTF-8  case: Examine first byte, determine character width, remove n bytes.

Now do that n times, and tell me which is more efficient.

This is not "exactly the same", by any stretch of the imagination.

> Another example is m/S/i. The Unicode case mapping is one-to-many and
> many-to-one, especially considering locale. Neither UTF-8 or UTF-32
> will save you.

That's irrelevant. The efficiency-significant part is skipping through the
string, and knowing *exactly* how far you need to skip ahead is much more
efficient than having to stop and recalculate it for each character.

I really cannot understand how to express this any simpler or any more

UTF16 : s += 2;            : O(1) : Good
UTF8  : s += UTF8WIDTH(*s) : O(n) : Bad

Going to church does not make a person religious, nor does going to school
make a person educated, any more than going to a garage makes a person a car.

Thread Previous | Thread Next Perl Programming lists via nntp and http.
Comments to Ask Bjørn Hansen at | Group listing | About