perl.unicode

perl.unicode http://www.nntp.perl.org/group/perl.unicode/ ... Copyright 1998-2008 perl.org Sun, 27 Jul 2008 09:17:04 +0000 ask@perl.org Re: /\w/ match with 'use locale' misses letters in utf8 locale by Peter Volkov В Птн, 11/07/2008 в 09:00 +0200, Juerd Waalboer пишет: > Peter Volkov skribis 2008-07-11 10:10 (+0400): > > The problem is that in Linux (Gentoo and Debian I've tried) /\w/ does > > not match Russian letter while I use locale and LC_COLLATE is set to > > ru_RU.UTF-8. > > \w should match Cyrillic letters even without "use locale". You might be > running into an annoying bug which makes \w lose its unicode support > depending on the *internal* state of a value. This behavior is reproducible with cp1251 encoding too. So... > Despite the above there's a slightly more important issue here. You're > opening a text file but you don't specify the character encoding. seems to be the answer I was looking for. But this makes me wonder why use locale exists then? I thought that it should take "default" or not specified encoding from environment... And really questionable why in FreeBSD everything works. In any case thank you Juerd for very fast answer. -- Peter. http://www.nntp.perl.org/group/perl.unicode/2008/07/msg3160.html Fri, 11 Jul 2008 02:16:23 +0000 Re: /\w/ match with 'use locale' misses letters in utf8 locale by Juerd Waalboer Peter Volkov skribis 2008-07-11 10:10 (+0400): > The problem is that in Linux (Gentoo and Debian I've tried) /\w/ does > not match Russian letter while I use locale and LC_COLLATE is set to > ru_RU.UTF-8. \w should match Cyrillic letters even without "use locale". You might be running into an annoying bug which makes \w lose its unicode support depending on the *internal* state of a value. To work around this bug, read Unicode::Semantics on CPAN and use it or utf8::upgrade. > Linux $ perl -e 'use locale; open(IN, "< test-file"); while(<IN>) { print if /\w/; }' > string with spaces (not only with [:alnum:]) > English; > hello_привет Despite the above there's a slightly more important issue here. You're opening a text file but you don't specify the character encoding. Likewise, you need to specify the encoding for output. Assuming utf8 for both: perl -le' binmode STDOUT, ":encoding(utf8)"; open my $in, "< :encoding(utf8)", "test-file"; while (<$in>) { print "match: [$1]" if /(\w+)/; } ' Which on my system prints: match: [слово] match: [строка] match: [string] match: [English] match: [hello_привет] I'm not sufficiently familiar with "use encoding" to say anything about it, but you shouldn't need it just for this. > Do I understand correctly that we should always supply encoding of > streams? Yes. > If yes, why in FreeBSD this works without supplying any encoding and is > it possible (good idea) to do the same in Linux? I have no idea. -- Met vriendelijke groet, Kind regards, Korajn salutojn, Juerd Waalboer: Perl hacker <#####@juerd.nl> <http://juerd.nl/sig> Convolution: ICT solutions and consultancy <sales@convolution.nl> 1; http://www.nntp.perl.org/group/perl.unicode/2008/07/msg3159.html Fri, 11 Jul 2008 00:00:29 +0000 /\w/ match with 'use locale' misses letters in utf8 locale by Peter Volkov Hello. Should /\w/ work with 'use locale' and correct environment set? The problem is that in Linux (Gentoo and Debian I've tried) /\w/ does not match Russian letter while I use locale and LC_COLLATE is set to ru_RU.UTF-8. The most strange thing is that in FreeBSD this works. Look: +++++++++++++++++++++++++ FreeBSD ++++++++++++++++++++++++++++++++ FreeBSD $ cat test-file слово строка с пробелами string with spaces (not only with [:alnum:]) English; hello_привет FreeBSD $ perl -e 'open(IN, "< test-file"); while(<IN>) { print if /\w/; }' string with spaces (not only with [:alnum:]) English; hello_привет FreeBSD $ perl -e 'use locale; open(IN, "< test-file"); while(<IN>) { print if /\w/; }' слово строка с пробелами string with spaces (not only with [:alnum:]) English; hello_привет FreeBSD $ ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ ++++++++++++++++++++++++++++ Linux +++++++++++++++++++++++++++++++ Linux $ perl -e 'use locale; open(IN, "< test-file"); while(<IN>) { print if /\w/; }' string with spaces (not only with [:alnum:]) English; hello_привет Linux $ ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ locale -a shows that ru_RU.utf8 locale exists on both systems and I've tried to set LANG and LC_ALL to this value with no result. Do I understand correctly that we should always supply encoding of streams? If yes, why in FreeBSD this works without supplying any encoding and is it possible (good idea) to do the same in Linux? Thank you for your time. -- Peter. http://www.nntp.perl.org/group/perl.unicode/2008/07/msg3158.html Thu, 10 Jul 2008 23:10:53 +0000 CSets 2.1 released by Mark Leisher http://www.math.nmsu.edu/~mleisher/Software/csets For those new to CSets: "The CSets collection is a set of mapping tables between various character sets and Unicode, and is intended to provide mappings not included in most character set conversion tools available today." It's been a couple years since the last release. This release features the addition of three Guarani mappings (available individually for a couple years already), and two new Serbian mappings. Future updates will be posted on freshmeat.net for those who like to track updates through subscriptions there, and I will always notify these lists of updates as well. As always, corrections, new mapping tables, information about mappings, and even pointers to things like fonts or texts with odd encodings are gladly accepted. -- Mark Leisher http://www.nntp.perl.org/group/perl.unicode/2008/05/msg3157.html Fri, 30 May 2008 15:20:40 +0000 Re: TAP YAML Diagnostics by Nicholas Clark On Sun, Apr 06, 2008 at 08:41:11AM -0700, Ovid wrote: > Currently you can shove anything in there you want, but you must use > upper-case keys for your personal use and all lower-case keys are > reserved (and it's a parse error to use an unknown lower-case key). > Are there any strange Unicode issues where we might get confused about > what is upper and lower case?) I believe that there are code points which would be considered word characters but do not have distinct upper and lower case forms (or by implication title case either), but I hope that the good folks of perl-unicode will correct me if I'm wrong. Hence I'm not sure what the most efficient way of determining if something is all lower case is. If I'm right, one can't just test if ($string eq lc $string) because these code points would mess you up, and I *assume* that they are not those which you want to consider reserved. I guess that one needs to loop over all characters in the string, and verify that if $char eq lc $char then also $char ne uc $char. (But one could first short circuit the common pass case with the test above) Nicholas Clark http://www.nntp.perl.org/group/perl.unicode/2008/04/msg3156.html Sun, 06 Apr 2008 09:33:30 +0000 Re: how to request for a new module by Darren Duncan Bayanzul, Go read http://cpan.org/modules/04pause.html and it should tell you or introduce you to everything you need to know. Read the whole thing (it isn't very long). -- Darren Duncan bayanzul lodoysamba wrote: > Dear all, > > I have a question about submitting a new module to CPAN. How can one submit a module to CPAN? > What is the procedure for it? Is there any qualification for submitting a module? > > Currently we are interested in developing a new module for converting Unicode strings in traditional mongolian script into different formats. > It will include functions that perform convertions between: > Basic Character Set <-> Presentation Set & Ligatures > Basic Character Set <-> Transliterate in Latin characters > > I guess, it is the best if the package would be under Unicode::, or Encode::. > > Please give us your suggestions. > Waiting for your reply > > > Regards, > Bayanzul.L http://www.nntp.perl.org/group/perl.unicode/2008/03/msg3155.html Thu, 27 Mar 2008 23:30:38 +0000 how to request for a new module by bayanzul lodoysamba Dear all, I have a question about submitting a new module to CPAN. How can one submit a module to CPAN? What is the procedure for it? Is there any qualification for submitting a module? Currently we are interested in developing a new module for converting Unicode strings in traditional mongolian script into different formats. It will include functions that perform convertions between: Basic Character Set <-> Presentation Set & Ligatures Basic Character Set <-> Transliterate in Latin characters I guess, it is the best if the package would be under Unicode::, or Encode::. Please give us your suggestions. Waiting for your reply Regards, Bayanzul.L ____________________________________________________________________________________ Never miss a thing. Make Yahoo your home page. http://www.yahoo.com/r/hs http://www.nntp.perl.org/group/perl.unicode/2008/03/msg3154.html Thu, 27 Mar 2008 04:03:41 +0000 Re: Pack and Unpack are Broken for > 0x7FFF_FFFF (in 5.10.0) by Chris Hall On Sun, 16 Mar 2008 I wrote >I thought 'C' worked on Octets ? Which is what 5.8.8 appears to be >doing, but not 5.10.0. I apologise... I should have read the perldelta. I now understand that: * in v5.8.8 one would say unpack('C*', ...) and get the underlying octets, if the string was a 'wide' (UTF8) string. in v5.10.0 this is no longer possible, C in a 'wide' string returns the wide character value. * in v5.10.0, in unpack 'C' and 'W' are the same as each other. (At least, I cannot tell the difference.) I cannot imagine why. * in v5.10.0, in pack 'C' and 'W' are the same as each other, except that 'C' masks the value down to 0..255. Exchanging the meaning of 'U0' and 'C0' between 5.8.X and 5.10.0 was a stroke of genius. The pack v5.10.0 documentation says: "Pack and unpack can operate in two modes, character mode (C0 mode) where the packed string is processed per character and UTF-8 mode (U0 mode) where the packed string is processed in its UTF-8-encoded Unicode form on a byte by byte basis. Character mode is the default unless the format string starts with an U . You can switch mode at any moment with an explicit C0 or U0 in the format. A mode is in effect until the next mode switch or until the end of the ()-group in which it was entered." Where UTF-8 mode appears to mean the exact opposite of UTF8 in connection with the state of a string value. The given meaning for 'C' is "An unsigned char (octet) value" doesn't help clarify things :-( Anywho, v5.10.0 will: pack('C*', 192,176,128,21) -> "\xC0\xB0\x80\x15" (byte:4) pack('W*', 192,176,128,21) -> "\xC0\xB0\x80\x15" (byte:4) pack('U*', 192,176,128,21) -> "\xC0\xB0\x80\x15" (wide:4) i.e. 'C' and 'W' produce 'byte' form strings, but 'U' produces a 'wide' character string. You might have thought that 'W' would generate a ('wide') character string... (but then it would be identical to 'U' !) pack('C0C*', 192,176,128,21) -> "\xC0\xB0\x80\x15" (byte:4) pack('C0W*', 192,176,128,21) -> "\xC0\xB0\x80\x15" (byte:4) pack('C0U*', 192,176,128,21) -> "\xC3\x80\xC2\xB0\xC2\x80\x15" (byte:7) so C0 prefix hasn't changed the result, except for C0U where we have been given a byte form string, containing the UTF-8. pack('U0C*', 192,176,128,21) -> "\x00\x00\x15" (wide:3) pack('U0W*', 192,176,128,21) -> "\x00\x00\x15" (wide:3) pack('U0U*', 192,176,128,21) -> "\xC0\xB0\x80\x15" (wide:4) which does create a ('wide') character string. This makes no difference to 'U*', but for 'C*' and 'W*', the 'byte' string produced has been decoded as UTF-8, replacing bad sequences by \x00 ! I suppose this makes sense. Though one might have imagined this would mean that the result of C* & W* would be 'utf8::upgrade'd to 'wide' characters. pack('C*', 257,359,477,91) -> "\x01\x67\xDD\x5B" (byte:4) pack('W*', 257,359,477,91) -> "\x101\x167\x1DD\x5B" (wide:4) pack('U*', 257,359,477,91) -> "\x101\x167\x1DD\x5B" (wide:4) pack('C0C*', 257,359,477,91) -> "\x01\x67\xDD\x5B" (byte:4) pack('C0W*', 257,359,477,91) -> "\x{101}\x{167}\x{1DD}\x5B" (wide:4) pack('C0U*', 257,359,477,91) -> "\xC4\x81\xC5\xA7\xC7\x9D\x5B" (byte:7) So W* and U* produce the same 'wide' character string. But 'C0W*' and 'C0U*' are quite different. I have no idea why. Almost equally amusing: $p = pack('W', 248) -> "\xF8" (byte:1) unpack('C*') -> (0xF8) unpack('C0C*') -> (0xF8) unpack('U0C*') -> (0xC3, 0xB8) So, 'U0C' seems to utf8::upgrade($p) before unpacking the raw bytes. Whereas: $p = pack('W', 257) -> "\x{101}" (wide:1) unpack('C*') -> (0x101) unpack('C0C*') -> (0x101) unpack('U0C*') -> (0xC3, 0xB8) Now, one can argue that poking around inside how characters are encoded in wide strings is a Bad Thing. But 'U0...' is exposing just that. However, there is no straightforward way to unpack a string into raw bytes without testing first for whether the string is 'wide' or not: @b = unpack(utf8::is_utf8($s) ? 'U0C*' : 'C*', $s) ; Chris -- Chris Hall highwayman.com http://www.nntp.perl.org/group/perl.unicode/2008/03/msg3153.html Sun, 16 Mar 2008 11:54:24 +0000 Re: Pack and Unpack are Broken for > 0x7FFF_FFFF (in 5.10.0) by Chris Hall On Sun, 16 Mar 2008 I wrote .... >Consider: > > use warnings ; > > sub sp { > my ($v) = @_ ; > > my $p = pack('U', $v) ; > my @t = unpack('C*', $p) ; > > printf '\x%04X_%04X: ', ($v >> 16), $v & 0xFFFF ; > print map sprintf('\x%02X', $_), @t ; > print "\n" ; > } ; ... > sp(0x7FFF_FFFF) ; ... >v5.8.8 result: > > \x7FFF_FFFD: \xFD\xBF\xBF\xBF\xBF\xBD ... >v5.10.0 result: > > \x7FFF_FFFD: \x7FFFFFFD I didn't see this in all the clutter of warnings. I thought 'C' worked on Octets ? Which is what 5.8.8 appears to be doing, but not 5.10.0. Chris -- Chris Hall highwayman.com http://www.nntp.perl.org/group/perl.unicode/2008/03/msg3152.html Sun, 16 Mar 2008 08:51:17 +0000 Pack and Unpack are Broken for > 0x7FFF_FFFF (in 5.10.0) by Chris Hall More confusion about the valid range of characters in Perl. Both v5.8.8 and v5.10.0 Perl will pack('U', $v) for values of $v which are > 0x7FFF_FFFF. The result is the (non-standard) Perl utf8 encoding for such characters. v5.8.8 Perl will unpack a string containing the non-standard encoding. v5.10.0 Perl will not. Consider: use warnings ; sub sp { my ($v) = @_ ; my $p = pack('U', $v) ; my @t = unpack('C*', $p) ; printf '\x%04X_%04X: ', ($v >> 16), $v & 0xFFFF ; print map sprintf('\x%02X', $_), @t ; print "\n" ; } ; sp(0x7FFF_FFFD) ; sp(0x8000_0000) ; sp(0xFFFF_FFFD) ; v5.8.8 result: \x7FFF_FFFD: \xFD\xBF\xBF\xBF\xBF\xBD \x8000_0000: \xFE\x82\x80\x80\x80\x80\x80 \xFFFF_FFFD: \xFE\x83\xBF\xBF\xBF\xBF\xBD v5.10.0 result: \x7FFF_FFFD: \x7FFFFFFD Malformed UTF-8 character (byte 0xfe) in unpack at tpbug.pl line 7. Malformed UTF-8 character (unexpected continuation byte 0x82, with no preceding start byte) in unpack at tpbug.pl line 7. Malformed UTF-8 character (unexpected continuation byte 0x80, with no preceding start byte) in unpack at tpbug.pl line 7. Malformed UTF-8 character (unexpected continuation byte 0x80, with no preceding start byte) in unpack at tpbug.pl line 7. Malformed UTF-8 character (unexpected continuation byte 0x80, with no preceding start byte) in unpack at tpbug.pl line 7. Malformed UTF-8 character (unexpected continuation byte 0x80, with no preceding start byte) in unpack at tpbug.pl line 7. Malformed UTF-8 character (unexpected continuation byte 0x80, with no preceding start byte) in unpack at tpbug.pl line 7. \x8000_0000: \x00\x00\x00\x00\x00\x00\x00 Malformed UTF-8 character (byte 0xfe) in unpack at tpbug.pl line 7. Malformed UTF-8 character (unexpected continuation byte 0x83, with no preceding start byte) in unpack at tpbug.pl line 7. Malformed UTF-8 character (unexpected continuation byte 0xbf, with no preceding start byte) in unpack at tpbug.pl line 7. Malformed UTF-8 character (unexpected continuation byte 0xbf, with no preceding start byte) in unpack at tpbug.pl line 7. Malformed UTF-8 character (unexpected continuation byte 0xbf, with no preceding start byte) in unpack at tpbug.pl line 7. Malformed UTF-8 character (unexpected continuation byte 0xbf, with no preceding start byte) in unpack at tpbug.pl line 7. Malformed UTF-8 character (unexpected continuation byte 0xbd, with no preceding start byte) in unpack at tpbug.pl line 7. \xFFFF_FFFD: \x00\x00\x00\x00\x00\x00\x00 And, FWIW, in 64-bit v5.8.8, pack('U', $v) appears to mask the $v value to unsigned 32-bits before attempting to pack ! -- Chris Hall highwayman.com +44 7970 277 383 http://www.nntp.perl.org/group/perl.unicode/2008/03/msg3151.html Sun, 16 Mar 2008 08:25:04 +0000 Re: UTF-8 (strict) appears borken by Chris Hall I have prepared a bug report, as below. I don't want to waste everybody's time if this is thought to be a feature... ...so if anyone thinks this is not a bug, please shout (soon). Thanks, Chris ----------------------------------------------------------------- [Please enter your report here] Encode::encode('UTF-8', $foo) and Encode::decode('UTF-8', $bar) detect the Unicode 'non-character' U+FFFF and treat it as an error. There are 65 other Unicode non-characters: U+FFFE U+01FFFE, U+02FFFE, U+03FFFE, ... U+10FFFE U+01FFFF, U+02FFFF, U+03FFFF, ... U+10FFFF U+FDD0..U+FDEF which one would expect to be treated the same as U+FFFF. They aren't. They are accepted as normal characters. This appears to be a bug. [Please do not change anything below this line] ----------------------------------------------------------------- -- Chris Hall highwayman.com http://www.nntp.perl.org/group/perl.unicode/2008/03/msg3150.html Sat, 15 Mar 2008 04:57:36 +0000 Re: utf8::valid and \x14_000 - \x1F_0000 by Chris Hall On Wed, 12 Mar 2008 Juerd Waalboer wrote >Chris Hall skribis 2008-03-12 20:49 (+0000): >> a. are you saying that characters in Perl are Unicode ? >Yes. They are called Unicode, at least. This has my preference for >explanation and documentation. >> b. or are you agreeing that characters in Perl take values >> 0..0x7FFF_FFFF (or beyond), which are generally interpreted as >> UCS, where required and possible ? >This too. This is the more technically accurate explanation, and has my >preference for implementation. 'This too' ? Goodness, superimposition ! Perl and quantum mechanics ? Suddenly it all becomes clear. Or at least as clear as the uncertainty principle will allow !-) FWIW, I have tried some of the HTTP, HTML and XML modules. The warnings that pop out every now and then about Unicode or UTF-8 or whatever are less than useful and more than irritating ! >> If (a) then characters with ordinals beyond 0x10_FFFF should throw >> warnings (at least) since they clearly are not Unicode ! >Perl just has a somewhat broad definition of "unicode", that is not >the same as the official unicode character set. BTW, in "2.14 Conforming to the Unicode Standard" I found this gem: Unacceptable Behavior It is unacceptable for a conforming implementation: - To use unassigned codes. • U+2073 is unassigned and not usable for ‘3’ (superscript 3) or any other character. This appears to say that unassigned codes should not be transmitted out, same like non-characters ! Which looks like hard work. (On the other hand, applications are supposed to cope with future defined code points...) Should 'UTF-8' be strict about unassigned codes as well ? What should chr() and "\x{...}" etc. do ? This reinforces my view that chr(n) is (a) wrong to whinge about surrogates and non-characters, and (b) wrong to return a character for n outside 0x..7FFF_FFFF. IMO: - chr() shouldn't worry about strict UCS ... - ... and doesn't, in an case, do a complete job [it does spot all non-characters and surrogates, but ignores unassigned codes.] - ... however, non-characters are perfectly legal UCS, at least for internal use. One can argue for jumping all over these when outputting (strict) UTF-8 for external exchange. - ... and 0x11_FFFE is not defined by UCS to be a non-character, it's not defined in UCS at all, any more than any other character code > U+10_FFFF ! - chr(n) doesn't whinge about characters > U+10_FFFF ! (Except for the non-characters it has invented !) - the answer to chr(-1) is 'not a character at all' -- it isn't 'the character that stands in place of some unknown character' - the utility of characters > 0x7FFF_FFFF is not worth (a) the kludge required to extend utf8, or (b) the interoperability issues. Even encode/decode 'utf8' take a dim view of chars > 0x7FFF_FFFF. I note that utf8::valid() rejects characters > 0x7FFF_FFFF ! - chr(n) accepts characters > 0x7FFF_FFFF, even though the result is not valid per utf8::valid() !! - chr(n) warns about p + 0xFFFE and p + 0xFFFF for every value of 'p', even those which are beyond the Unicode range ! >It has its own utf8, it can have its own unicode too :) And there was I thinking that things were already sufficiently confused :-} The 'utf8' decode does the Right Thing -- it decodes well-formed UTF-8 up to 0x7FFF_FFFF and handles errors and incomplete sequences and doesn't concern itself with the minutiae of UCS (surrogates, non-characters and unassigned codes). This is nicely consistent with utf8::valid(). [The only thing I would argue about is the separate treatment of each byte of an invalid sequence -- I'd be tempted to treat 0x00..0x7F and 0xC0..0xFF as terminators of an invalid sequence and 0x80..0xBF as members of an invalid sequence.] If 'unicode' were to follow that model, then chr() and friends could stop throwing (spurious) warnings around the place. Sadly, 'utf8' encode is doesn't care, and outputs whatever is in the string -- including redundant sequences, invalid sequences, incomplete sequences and Perl's extended sequences for > 0x7FFF_FFFF. That is, it will happily output something that utf8::valid would reject. Note that this "encoding" is outputting something that 'utf8' decode won't accept. If you really want what 'utf8' encode currently does you can force characters to octets (wax off) and output. The reverse is to input the octets and force to characters (wax on). Summary of Observations ----------------------- * chr(n) and friends are broken: - they winge about things that are none of their business, which is not consistent with the notion of (lax) 'unicode'. - the wingeing about not-(strict)-Unicode is, moreover, incomplete (unassigned codes and codes beyond the UCS range are allowed !) - non-characters are perfectly legal -- just not suitable for external exchange. - projecting non-characters beyond the UCS range is plain odd. - they create invalid (per utf8::valid()) strings - invalid 'n' should return an 'invalid' (i.e. undef) response * 'utf8' encode is broken: - it should not output stuff that is not at least utf8:valid() - it should be symmetrical with 'utf8' decode * characters > 0x7FFF_FFFF are not utf8::valid. I think that's a good call -- but Perl is not consistent, and will happily produce invalid strings... * 'UTF-8' is broken: - it doesn't know about all the defined non-characters. - there should be an option to allow non-characters for internal exchange of otherwise strict UTF-8. - BTW: the Unicode reference code for UTF8 to UTF32 does not trouble itself about non-characters. Nor does UTF32 to UTF8. Chris -- Chris Hall highwayman.com http://www.nntp.perl.org/group/perl.unicode/2008/03/msg3149.html Thu, 13 Mar 2008 10:16:06 +0000 UTF-8 (strict) appears borken by Chris Hall 1. 'Ill-formed' UTF-8 ===================== The Unicode Standard specifies that any UTF-8 sequence that does not correspond to this table is 'ill-formed': Code Points | 1st Byte | 2nd Byte | 3rd Byte | 4th Byte | -------------------+----------+----------+----------+----------+ U+0000..U+007F | 00..7F | -- | -- | -- | U+0080..U+07FF | C2..DF | 80..BF | -- | -- | U+0800..U+0FFF | E0 | A0..BF | 80..BF | -- | U+1000..U+CFFF | E1..EC | 80..BF | 80..BF | -- | U+D000..U+D7FF | ED | 80..9F | 80..BF | -- | U+E000..U+FFFF | EE..EF | 80..BF | 80..BF | -- | U+10000..U+3FFFF | F0 | 90..BF | 80..BF | 80..BF | U+40000..U+FFFFF | F1..F3 | 80..BF | 80..BF | 80..BF | U+100000..U+10FFFF | F4 | 80..8F | 80..BF | 80..BF | Note in particular that: - anything beyond U+10FFFF is ill-formed. - anything U+D800..U+DFFF is ill-formed. - only one encoding for each Code Point is well-formed. We'd expect UTF-8 decode to spot ill-formed sequences. Though some special handling of incomplete sequences at the end of a buffer would be handy. We'd expect UTF-8 encode to only generate well-formed sequences. 2. Extended Sequences ===================== Unicode and ISO/IEC 10646:2003 define meanings for UTF-8 compatible sequences up to 6 bytes, which allows for characters up to 0x7FFF_FFFF. The Unicode reference code for reading UTF-8 recognises these extended sequences as being single entities (though ill-formed). Perl has its own further 7 and 13 byte forms, allowing for characters up to 0xF_FFFF_FFFF and 2^72-1, respectively. These are beyond UTF-8. 3. Non-Characters ================= The only other cause for concern are non-characters. These are: * U+FFFE and U+FFFF and the last two code points in every other Unicode plane. Unicode code space is divided into 17 'planes' of 65,536 characters, each. So characters U+01_FFFE, U+01_FFFF, U+02_FFFE, U+02_FFFF, ... U+10_FFFE and U+10_FFFF are all non-characters. * U+FDD0..U+FDEF Now, Unicode 5.0.0 says: "Applications are free to use any of these noncharacter code points internally but should never attempt to exchange them. If a noncharacter is received in open interchange, an application is not required to interpret it in any way. It is good practice, however, to recognize it as a noncharacter and to take appropriate action, such as removing it from the text." "Noncharacter code points are reserved for internal use, such as for sentinel values. They should never be interchanged. They do, however, have well-formed representations in Unicode encoding forms and survive conversions between encoding forms. This allows sentinel values to be preserved internally across Unicode encoding forms, even though they are not designed to be used in open interchange." So... this is not so clear-cut. For "open interchange" UTF-8 should disallow the non-characters. However, for local storage of Unicode stuff, non-characters should be allowed. 4. What 'UTF-8' Does ==================== Ill-formed sequences -- fine (mostly): * UTF-8 decode treats these as errors, and will stop or use fallback decoding as required. The default fallback is: - errors for sequence <= 0x7FFF_FFFF -- replaced by U+FFFD *** information is being lost, here :-( - anything else: each byte which is not recognised as being part of a complete 2..6 byte sequence is replaced by U+FFFD *** so one cannot distinguish ill-formed sequences from out of range characters. The PERLQQ, HTMLCREF and XMLCREF fallbacks are: - errors for sequence <= 0x7FFF_FFFF -- replaced by the respective escape sequence for the character value. This ought to work if the data is HTML or XML, where new escape sequences fit right in if HTMLCREF or XMLCREF is used. *** PERLQQ, however, may fail if '\' appears in the input and the sender has not escaped it ! Perhaps PERLQQ should escape '\' that appear in the input ? *** In all cases, however, all that's been achieved is that non-UTF-8 characters have been transliterated. It's still a puzzle what may be done with these characters ! - anything else: each byte which is not recognised as being part of a complete up to 6 byte sequence is replaced by the respective escape sequence for the byte value. *** this is impossible to distinguish from escaped values which could exist in the input ! * UTF-8 encode will not generate ill-formed sequences and treats out of ranges character values as errors. Errors will stop encoding or cause the fallback encoding to be used. The default fallback is: - errored characters <= 0x7FFF_FFFF -- replaced by U+FFFD *** Not much one can do here. It's not clear that U+FFFD is a good thing to output -- one could argue for discarding this rubbish, instead ? - 0x8000_0000 and greater -- replaced by seven or thirteen U+FFFD, depending on the length of the Perl internal form !!! *** This is also more than a bit odd !! The PERLQQ, HTMLCREF and XMLCREF fallbacks are: - errored characters <= 0x7FFF_FFFF -- replaced by the respective escape sequence for the character value. This ought to work if the data is HTML or XML, where new escape sequences fit right in if HTMLCREF or XMLCREF is used. *** PERLQQ, however, may fail if '\' appears in the output and the sender has not escaped it ! . Perhaps PERLQQ should escape '\' that appear in the output ? *** In all cases, however, all that's been achieved is that non-UTF-8 characters have been transliterated. It's still a puzzle what may be done with these characters ! - 0x8000_0000 and greater -- replaced by the seven or thirteen bytes that comprise the Perl internal form, each as its respective escape sequence !!! *** This is also more than a bit odd !! Incomplete sequences -- fine, but not documented ! * UTF-8 decode generally treats these as ill-formed, as above. However, the STOP_AT_PARTIAL CHECK option will cause decode to stop, without error (so without invoking the fallback). Non-Character Values -- inconsistent and arguable !! As noted above, one can argue for two approaches here, depending on whether the data being en/decoded is internal or external. For internal data, non-characters are valid and should be preserved. For external data, non-characters should not be sent or received. One can debate whether they should be dropped or replaced or escaped. UTF-8 encode/decode recognise only U+FFFF as a non-character, and treat it as an error. *** This looks like a bug. If non-character values are to be treated as errors, I suggest all non-character values should be so treated. *** This caters only for external data exchange. The error handling is as for ill-formed sequences, see above. 5. Conclusion 'UTF-8' is broken =============================== * the non-character handling is incomplete. * it can be argued that there should be an option to accept/allow non- character values. * the various fallback options are all less than satisfactory in their own way. One can see why the ref:Sub CHECK argument was invented. HOWEVER: it would be handy if there was a second parameter passed to the CHECK subroutine, telling it *why* the given sequence cannot be encoded/decoded, in particular: -- out of range character value -- ill-formed sequence (and could pass in everything up to the next not invalid byte ?) -- non-character -- incomplete sequence for otherwise the subroutine has to do all the work to figure this out for itself ! ------------------------------------------------------------------------ It is clear that what data is valid, and how to deal with invalid data, is really up to the application. Trying to be helpful in Encode/Decode is apparently tricky. It is also clear that a lot of heavy duty character/byte bashing would be better if it could be provided in XS land. However, thinking about some simple but general mechanism for this is making my head hurt. [I'm going to go away now, and lie down.] -- Chris Hall highwayman.com http://www.nntp.perl.org/group/perl.unicode/2008/03/msg3148.html Wed, 12 Mar 2008 17:39:51 +0000 Re: utf8::valid and \x14_000 - \x1F_0000 by Juerd Waalboer Chris Hall skribis 2008-03-12 20:49 (+0000): > a. are you saying that characters in Perl are Unicode ? Yes. They are called Unicode, at least. This has my preference for explanation and documentation. > b. or are you agreeing that characters in Perl take values > 0..0x7FFF_FFFF (or beyond), which are generally interpreted as > UCS, where required and possible ? This too. This is the more technically accurate explanation, and has my preference for implementation. > If (a) then characters with ordinals beyond 0x10_FFFF should throw > warnings (at least) since they clearly are not Unicode ! Perl just has a somewhat broad definition of "unicode", that is not the same as the official unicode character set. It has its own utf8, it can have its own unicode too :) -- Met vriendelijke groet, Kind regards, Korajn salutojn, Juerd Waalboer: Perl hacker <#####@juerd.nl> <http://juerd.nl/sig> Convolution: ICT solutions and consultancy <sales@convolution.nl> http://www.nntp.perl.org/group/perl.unicode/2008/03/msg3147.html Wed, 12 Mar 2008 13:58:17 +0000 Re: utf8::valid and \x14_000 - \x1F_0000 by Chris Hall On Wed, 12 Mar 2008 Juerd Waalboer wrote >Chris Hall skribis 2008-03-12 13:20 (+0000): .... >> String literals are represented by UCS code points. Which >> reinforces the feeling that characters in Perl are Unicode. >Yes! OK. For the avoidance of doubt: a. are you saying that characters in Perl are Unicode ? b. or are you agreeing that characters in Perl take values 0..0x7FFF_FFFF (or beyond), which are generally interpreted as UCS, where required and possible ? If (a) then characters with ordinals beyond 0x10_FFFF should throw warnings (at least) since they clearly are not Unicode ! ....[in the context of U+D800..U+DFFF] >> "Isolated surrogate code units have no interpretation on >> their own." >> (...) >> Clearly these are illegal in UTF-8. >They have no interpretation, but this also doesn't say it's illegal. The Unicode Standard defines the set of 'Unicode scalar values' which consists of U+0000..U+D7FF and U+E000..U+10_FFFF. All Unicode encodings, including UTF-8, encode only the 'Unicode scalar values'. The code points U+D800..U+DFFF exist, but do "not contain any character assignments". Given that no Unicode encoding exists that allows these code points, it's unclear how one would ever end up with one of these things on its hands ! ....[in the context of U+FFFE, U+FFFF etc.] >> "Applications are free to use any of these noncharacter code >> points internally but should never attempt to exchange >> them. >I think it's not Perl's job to prevent exchange. Simply because the >exchange could be internal, but between processes of the same program. Well UTF-8 is jumping all over U+FFFF (at least). The warnings thrown by chr() and "\x{h...h} suggest that Perl feels that exchanging these values ain't kosher. >> I'm puzzled as to why 'UTF-8' (strict) doesn't treat U+FFFE (and >> friends) in the same way as U+FFFF (and friends). >My gut says it's out of ignorance of the "rules", and certainly not an >intentional deviation. Well... I'm running some more tests on UTF-8 to see what it thinks is illegal. ..................................... >> >The result is Unicode. >> IMHO the result of chr(n) should just be a character. >We call that a unicode character in Perl. It is true that Perl allows >ordinal values outside the currently existing range, but it is still >called unicode by Perl's documentation. OK. This is the hair which I am splitting. IMHO the things in strings and the things that chr() and ord() return or process should be plain characters (ordinal U_INT) -- so that these are general purpose. Only when it's necessary to attach meaning to the characters in a string, should Perl treat them as Unicode code points -- I accept that this is most of the time (but not *all* the time). >> FWIW I note that printf "%vX" is suggested as a means to render IPv6 >> addresses. This implies the use of a string containing eight characters >> 0..0xFFFF as the packed form of IPv6. Building one of those using >> chr(n) will generate spurious warnings about 0xFFFE and 0xFFFF ! >Interesting point. What's more, the Unicode standard suggests various *internal* uses for U+FFFE and U+FFFF (and friends), including, but not limited to, terminators and separators. This will also generate spurious warnings from chr() or "\x{...}" ! Chris -- Chris Hall highwayman.com http://www.nntp.perl.org/group/perl.unicode/2008/03/msg3146.html Wed, 12 Mar 2008 13:50:45 +0000 Re: utf8::valid and \x14_000 - \x1F_0000 by Juerd Waalboer Chris Hall skribis 2008-03-12 13:20 (+0000): > >> OK. In the meantime IMHO chr(n) should be handling utf8 and has no > >> business worrying about things which UTF-8 or UCS think aren't > >> characters. > >It should do Unicode, not any specific byte encoding, like UTF-?8. > IMHO chr(n) should do characters, which may be interpreted as per > Unicode, but may not. > When I said utf8 I was following the (sloppy) convention that utf8 means > how Perl handles characters in strings... I'm working hard to break this convention. I've changed a lot of Perl documentation, and the result was released with Perl 5.10. If in any place in Perl's official documentation, it still reads UTF-8 or UTF8 for *characters in text strings*, it's wrong. Let me know and I will fix it :) > b. in a Perl string, characters are held in a UTF-8 like form. I'd say *inside* a Perl string. This is the C implementation, but a Perl programmer should not have to know the specific *internal* encoding of a Perl string. Likewise, in Perl you don't have to know whether your number is internally encoded as a long integer or a double. > Where UTF-8 (upper case, with hyphen) means the RFC 3629 & > Unicode Consortium defined byte-wise encoding. That's the theory, but it's so often not entirely following spec. > This form is referred to as utf8 (lower case, no hyphen). Yes, but note that encoding names in Perl are case insensitive. I tend to call it UTF8 sometimes. > There is really no need to discuss this, except in the context of > messing around in guts of Perl. Exactly. > String literals are represented by UCS code points. Which > reinforces the feeling that characters in Perl are Unicode. Yes! > 'C' uses 'wide' to refer to characters that may have values > > 255. IMHO it's a shame that Perl did not follow this. It does in some places, most notably warnings about "wide characters". > d. when exchanging character data with other systems one needs to > deal with character set and encoding issues. Not just other systems. All I/O is done in bytes, even with yourself, for example if you forked. > "Isolated surrogate code units have no interpretation on > their own." > (...) > Clearly these are illegal in UTF-8. They have no interpretation, but this also doesn't say it's illegal. Compare it with the undefined behavior of multiple ++ in a single expression. There's no specification of what should happen, but it's not illegal to do it. > "Applications are free to use any of these noncharacter code > points internally but should never attempt to exchange > them. I think it's not Perl's job to prevent exchange. Simply because the exchange could be internal, but between processes of the same program. > I'm puzzled as to why 'UTF-8' (strict) doesn't treat U+FFFE (and > friends) in the same way as U+FFFF (and friends). My gut says it's out of ignorance of the "rules", and certainly not an intentional deviation. > >The result is Unicode. > IMHO the result of chr(n) should just be a character. We call that a unicode character in Perl. It is true that Perl allows ordinal values outside the currently existing range, but it is still called unicode by Perl's documentation. > OK, sure. I was using utf8 to mean any character value you like, and > UTF-8 to imply a value which is recognised in UCS -- rather than the > encoding. Please use utf8 only for naming the byte encoding that allows any character value you like, not for the ordinal values themselves. > FWIW I note that printf "%vX" is suggested as a means to render IPv6 > addresses. This implies the use of a string containing eight characters > 0..0xFFFF as the packed form of IPv6. Building one of those using > chr(n) will generate spurious warnings about 0xFFFE and 0xFFFF ! Interesting point. -- Met vriendelijke groet, Kind regards, Korajn salutojn, Juerd Waalboer: Perl hacker <#####@juerd.nl> <http://juerd.nl/sig> Convolution: ICT solutions and consultancy <sales@convolution.nl> http://www.nntp.perl.org/group/perl.unicode/2008/03/msg3145.html Wed, 12 Mar 2008 09:53:38 +0000 Re: utf8::valid and \x14_000 - \x1F_0000 by Chris Hall On Tue, 11 Mar 2008 Juerd Waalboer wrote >Chris Hall skribis 2008-03-11 21:09 (+0000): >> OK. In the meantime IMHO chr(n) should be handling utf8 and has no >> business worrying about things which UTF-8 or UCS think aren't >> characters. >It should do Unicode, not any specific byte encoding, like UTF-?8. IMHO chr(n) should do characters, which may be interpreted as per Unicode, but may not. When I said utf8 I was following the (sloppy) convention that utf8 means how Perl handles characters in strings... ...the naming is a cause of confusion. For the avoidance of doubt, this is what I understand the position to be: a. characters in Perl have integer values in 0..0x7FFF_FFFF (or more). It appears that what is actually going on is that the limit is the local unsigned Perl integer. One can debate the marginal utility of that vs the scope for confusion. b. in a Perl string, characters are held in a UTF-8 like form. Where UTF-8 (upper case, with hyphen) means the RFC 3629 & Unicode Consortium defined byte-wise encoding. Current UTF-8 defines encoding for values 0..0xD7FF and 0xE000..0x10_FFFF, which is exactly the current UCS range (less the 'surrogates'). Note that this limits UTF-8 to 4 byte sequences, explicitly excluding: * sequences that have shorter equivalents ('redundant') * 0xD800..0xDFFF -- the 'surrogates' * x11_0000..0x1F_FFFF -- beyond UCS range Older versions of the standard allowed for values 0..0x7FFF_FFFF, but also excluded the 'redundant' sequences and (I believe) the 'surrogates'. The encoding used by Perl stretches the range to 2^72-1. This is incompatible with even the older versions of UTF-8. This form is referred to as utf8 (lower case, no hyphen). There is really no need to discuss this, except in the context of messing around in guts of Perl. c. when Perl wishes to assign some meaning to a character value it interprets it as a Unicode Code Point, if it can. There are huge areas of the Unicode space that have no current meaning. There are areas which may have local meaning ("Private Use"). In addition Perl allows character values that are beyond current Unicode space. In the abstract, characters in Perl are not Unicode (UCS). But most of the time one treats them as if they were. String literals are represented by UCS code points. Which reinforces the feeling that characters in Perl are Unicode. 'C' uses 'wide' to refer to characters that may have values > 255. IMHO it's a shame that Perl did not follow this. d. when exchanging character data with other systems one needs to deal with character set and encoding issues. The 'UTF-8' encoding (character set) covers the UCS character set (values 0..0x10_FFFF, currently) and the (current) standard UTF-8 encoding. 'UTF-8' also worries about some 'suspect' (my term) UCS values, see below. The 'utf8' encoding (character set) is a super set of current UTF-8 (values 0..0x7FFF_FFFF) -- corresponding to earlier UTF-8. 'utf8' does not concern itself about any 'suspect' UCS values. [Actually, that's not entirely true. 'utf8' encode happily deals with characters all the way up to 2^64-1 (and perhaps, beyond), using Perl's extended encoding. However, 'utf8' decode treats anything > 0x7FFF_FFFF as invalid.] e. The 'suspect' UCS values. These are: * U+D800..U+DBFF and U+DC00..U+DFFF (High- and Low-surrogate, respectively). Where these are used they should appear in pairs, High followed by Low. Unicode 5.0.0 says: "Surrogate pairs are used only in UTF-16." "Isolated surrogate code units have no interpretation on their own." "Surrogate code points cannot be conformantly interchanged using Unicode encoding forms." "Unicode scalar value: Any Unicode code point except high- surrogate and low-surrogate code points." All the Unicode encodings are defined in terms of Unicode scalar value. There is by definition no way to exchange these characters, and no meaning is attached to them. Clearly these are illegal in UTF-8. * U+FFFE and U+FFFF and the last two code points in every other Unicode plane are noncharacters. [Unicode code space is divided into 17 'planes' of 65,536 characters, each. So characters U+01_FFFE, U+01_FFFF, U+02_FFFE, U+02_FFFF, ... U+10_FFFE and U+10_FFFF are all noncharacters.] The range U+FDD0..U+FDEF are also noncharacters. Unicode 5.0.0 says: "Applications are free to use any of these noncharacter code points internally but should never attempt to exchange them. If a noncharacter is received in open interchange, an application is not required to interpret it in any way. It is good practice, however, to recognize it as a noncharacter and to take appropriate action, such as removing it from the text." "Noncharacter code points are reserved for internal use, such as for sentinel values. They should never be interchanged. They do, however, have well-formed representations in Unicode encoding forms and survive conversions between encoding forms. This allows sentinel values to be preserved internally across Unicode encoding forms, even though they are not designed to be used in open interchange." So, assuming UTF-8 is used for "open interchange", these are also invalid. * U+FFFD -- the Replacement Character Unicode 5.0.0 says: "U+FFFD replacement character is the general substitute character in the Unicode Standard. It can be substituted for any 'unknown' character in another encoding that cannot be mapped in terms of known Unicode characters." This is generally legal. However on the topic of "Reserved and Private-Use Character Codes" the standard also counsels: "An implementation should not blindly delete such characters, nor should it unintentionally transform them into something else." Any corrections required would be appreciated, and may also inform any "lurkers". >Internally, a byte encoding is needed. As a programmer I don't want to >be bothered with such implementation details. >> Note that chr(n) is whingeing about 0xFFFE, which Encode::en/decode >> (UTF-8) are happy with. Unicode defines 0xFFFE and 0xFFFF as >> non-characters, not just 0xFFFF (which Encode::en/decode do deem >> invalid). >Personally, I think Perl should accept these characters without warning, >except the strict UTF-8 encoding is requested (which differs from the >non-strict UTF8 encoding). I agree -- chr(n) and 'utf8' (lax) should happily process anything 0..0x7FFF_FFFF as characters -- which may or may not be UCS. I'm puzzled as to why 'UTF-8' (strict) doesn't treat U+FFFE (and friends) in the same way as U+FFFF (and friends). >> >>In any case, is chr(n) supposed to be utf8 or UTF-8 ? AFAIKS, it's >> >>neither. >> >It's supposed to be neither on the outside. Internally, it's utf8. >> One can turn off the warnings and then chr(n) will happily take any +ve >> integer and give you the equivalent character -- so the result is utf8, >The result is Unicode. IMHO the result of chr(n) should just be a character. > The difference between Unicode and UTF8 is not >always clear, but in this case is: the character is Unicode, a single >codepoint, the internal implementation is UTF8. > >Unicode: U+20AC (one character: €) >UTF-8: E2 82 AC (three bytes) > >I am under the impression that you know the difference and made an >honest mistake. My detailed expansion is also for lurkers and archives. OK, sure. I was using utf8 to mean any character value you like, and UTF-8 to imply a value which is recognised in UCS -- rather than the encoding. >> [replacement character] >> So we'll have to differ on this :-) >Yes, although my opinion on this is not strong. undef or replacement >character - both are good options. One argument in favor of the >replacement character would be backwards compatibility. Well, having concluded that the result of chr(n) should be just a character -- to be interpreted one way or another, later -- returning "\xFFFD" for chr(-1) looks perverse ! FWIW I note that printf "%vX" is suggested as a means to render IPv6 addresses. This implies the use of a string containing eight characters 0..0xFFFF as the packed form of IPv6. Building one of those using chr(n) will generate spurious warnings about 0xFFFE and 0xFFFF ! Chris -- Chris Hall highwayman.com +44 7970 277 383 http://www.nntp.perl.org/group/perl.unicode/2008/03/msg3144.html Wed, 12 Mar 2008 06:22:40 +0000 Re: utf8::valid and \x14_000 - \x1F_0000 by Juerd Waalboer Chris Hall skribis 2008-03-11 21:09 (+0000): > OK. In the meantime IMHO chr(n) should be handling utf8 and has no > business worrying about things which UTF-8 or UCS think aren't > characters. It should do Unicode, not any specific byte encoding, like UTF-?8. Internally, a byte encoding is needed. As a programmer I don't want to be bothered with such implementation details. > Note that chr(n) is whingeing about 0xFFFE, which Encode::en/decode > (UTF-8) are happy with. Unicode defines 0xFFFE and 0xFFFF as > non-characters, not just 0xFFFF (which Encode::en/decode do deem > invalid). Personally, I think Perl should accept these characters without warning, except the strict UTF-8 encoding is requested (which differs from the non-strict UTF8 encoding). > >>In any case, is chr(n) supposed to be utf8 or UTF-8 ? AFAIKS, it's > >>neither. > >It's supposed to be neither on the outside. Internally, it's utf8. > One can turn off the warnings and then chr(n) will happily take any +ve > integer and give you the equivalent character -- so the result is utf8, The result is Unicode. The difference between Unicode and UTF8 is not always clear, but in this case is: the character is Unicode, a single codepoint, the internal implementation is UTF8. Unicode: U+20AC (one character: €) UTF-8: E2 82 AC (three bytes) I am under the impression that you know the difference and made an honest mistake. My detailed expansion is also for lurkers and archives. > [replacement character] > So we'll have to differ on this :-) Yes, although my opinion on this is not strong. undef or replacement character - both are good options. One argument in favor of the replacement character would be backwards compatibility. -- Met vriendelijke groet, Kind regards, Korajn salutojn, Juerd Waalboer: Perl hacker <#####@juerd.nl> <http://juerd.nl/sig> Convolution: ICT solutions and consultancy <sales@convolution.nl> http://www.nntp.perl.org/group/perl.unicode/2008/03/msg3143.html Tue, 11 Mar 2008 14:26:14 +0000 Re: utf8::valid and \x14_000 - \x1F_0000 by Chris Hall On Tue, 11 Mar 2008 you wrote >Chris Hall skribis 2008-03-11 18:48 (+0000): >> I'm comfortable with the notion that perl characters are unsigned >> integers that overlap UCS, and happen to be held internally as a >> superset of UTF-8. >> I wonder if perl is completely comfortable. >It isn't. There are some very unfortunate "features". >> chr(n) throws various runtime warnings where 'n' isn't kosher UCS, and >> "\x{h...h}" throws the same ones at compile time. >> (...)I'm not sure I see the point of picking on a few values to warn >> about. >I don't see the point, but Perl's warnings are arbitrary in several >ways. Abigail has a lightning talk about the "interpreted as function" >warning, that illustrates this. OK. In the meantime IMHO chr(n) should be handling utf8 and has no business worrying about things which UTF-8 or UCS think aren't characters. Note that chr(n) is whingeing about 0xFFFE, which Encode::en/decode (UTF-8) are happy with. Unicode defines 0xFFFE and 0xFFFF as non-characters, not just 0xFFFF (which Encode::en/decode do deem invalid). >> In any case, is chr(n) supposed to be utf8 or UTF-8 ? AFAIKS, it's >> neither. >It's supposed to be neither on the outside. Internally, it's utf8. One can turn off the warnings and then chr(n) will happily take any +ve integer and give you the equivalent character -- so the result is utf8, but the warnings are some (very) small subset of checking for UTF-8 :-( I wonder what happens for n >= 2^64. The encoding runs out at 2^72 ! >> If chr(-1) doesn't exist, then undef looks like a reasonable >> return value -- returning "\x{FFFD}" makes chr(-1) >> indistinguishable from chr(0xFFFD) -- where the first is >> nonsense and the second is entirely proper. >0xFFFD is the Unicode equivalent of undef. I think it makse sense in >this case. Well... Unicode says: "REPLACEMENT CHARACTER: used to represent an incoming character whose value is unknown or unrepresentable in Unicode". ...so it has plenty to do without being used to represent a value which is completely beyond the range for characters, and for which perl has a perfectly good convention already. ...besides, if I want to see if chr(n) has worked I have to check that (a) the result is not "\xFFFD" and (b) that n is not 0xFFFD. So we'll have to differ on this :-) Chris -- Chris Hall highwayman.com +44 7970 277 383 http://www.nntp.perl.org/group/perl.unicode/2008/03/msg3142.html Tue, 11 Mar 2008 14:11:05 +0000 Re: utf8::valid and \x14_000 - \x1F_0000 by Juerd Waalboer Chris Hall skribis 2008-03-11 18:48 (+0000): > I'm comfortable with the notion that perl characters are unsigned > integers that overlap UCS, and happen to be held internally as a > superset of UTF-8. > I wonder if perl is completely comfortable. It isn't. There are some very unfortunate "features". > chr(n) throws various runtime warnings where 'n' isn't kosher UCS, and > "\x{h...h}" throws the same ones at compile time. > (...)I'm not sure I see the point of picking on a few values to warn > about. I don't see the point, but Perl's warnings are arbitrary in several ways. Abigail has a lightning talk about the "interpreted as function" warning, that illustrates this. > In any case, is chr(n) supposed to be utf8 or UTF-8 ? AFAIKS, it's > neither. It's supposed to be neither on the outside. Internally, it's utf8. > If chr(-1) doesn't exist, then undef looks like a reasonable > return value -- returning "\x{FFFD}" makes chr(-1) > indistinguishable from chr(0xFFFD) -- where the first is > nonsense and the second is entirely proper. 0xFFFD is the Unicode equivalent of undef. I think it makse sense in this case. > >Could you please report this bug with perlbug? > Done. Cheers. -- Met vriendelijke groet, Kind regards, Korajn salutojn, Juerd Waalboer: Perl hacker <#####@juerd.nl> <http://juerd.nl/sig> Convolution: ICT solutions and consultancy <sales@convolution.nl> http://www.nntp.perl.org/group/perl.unicode/2008/03/msg3141.html Tue, 11 Mar 2008 12:28:19 +0000 Decode, byte codes ASCII & ISO-8859 and HTMLCREF or XMLCREF by Chris Hall Having tried: $o = Encode::decode('ascii', "abc \x80 \xFF", FB_HTMLCREF) $o = Encode::decode('ISO-8859-7', "abc \xFF", FB_HTMLCREF) $o = Encode::decode('ascii', "abc \x80 \xFF", FB_XMLCREF) $o = Encode::decode('ISO-8859-7', "abc \xFF", FB_XMLCREF) (0xFF is not a valid character value in ISO-8859-7.) I find that they produce neither &128; and &255; nor &#x80; and &#xFF, but exactly the same as: $o = Encode::decode('ascii', "abc \x80 \xFF", FB_PERLQQ) $o = Encode::decode('ISO-8859-7', "abc \xFF", FB_PERLQQ) namely \x80 and \xFF ! Almost as if there were a bug. Or have I missed the right piece of the documentation ? -- Chris Hall highwayman.com http://www.nntp.perl.org/group/perl.unicode/2008/03/msg3140.html Tue, 11 Mar 2008 12:09:34 +0000 Re: utf8::valid and \x14_000 - \x1F_0000 by Chris Hall On Tue, 11 Mar 2008 you wrote >Chris Hall skribis 2008-03-11 13:30 (+0000): >> I suggest utf8::valid() is broken. >> my $s = chr($c) ; >> my $v = utf8::valid($s) ? 1 : 0 ; >Agreed. utf8::valid(chr $foo) should ALWAYS return true. (Please note >that utf8::valid tests the internal consistency of a string - on the >outside, it has little to do with UTF8.) I'm comfortable with the notion that perl characters are unsigned integers that overlap UCS, and happen to be held internally as a superset of UTF-8. I wonder if perl is completely comfortable. chr(n) throws various runtime warnings where 'n' isn't kosher UCS, and "\x{h...h}" throws the same ones at compile time. Now there's HUGE areas of UCS code space that are essentially meaningless. There are VAST areas of perl character space that are way beyond UCS. I'm not sure I see the point of picking on a few values to warn about. In any case, is chr(n) supposed to be utf8 or UTF-8 ? AFAIKS, it's neither. I have tried the following on 5.10.0 and 5.8.8, and where these differ I have noted it: chr(-1) 5.10.0: No warning, returned "\x{FFFD}" 5.8.8: Warning 'Unicode character 0xffffffffffffffff is illegal', returned "\x{FFFF_FFFF_FFFF_FFFF}" Neither of these seem very sensible. If chr(-1) doesn't exist, then undef looks like a reasonable return value -- returning "\x{FFFD}" makes chr(-1) indistinguishable from chr(0xFFFD) -- where the first is nonsense and the second is entirely proper. chr(0xD800) Warns 'UTF-16 surrogate 0xd800', returns "\x{D800}" chr(0xFFFD) No warning, returns "\x{FFFD}" chr(0xFFFE) Warns 'Unicode character 0xfffe is illegal', returns "\x{FFFE}", NB: both Encode::encode('UTF-8', "\x{FFFE}") and Encode::decode{'UTF-8', "\xEF\xBF\xBE") are perfectly happy ! This appears inconsistent ? All the UCS planes appear to be treated like this. chr(0xFFFF) Warns 'Unicode character 0xffff is illegal', returns "\x{FFFF}", NB: both Encode::encode('UTF-8', "\x{FFFF}") and Encode::decode{'UTF-8', "\xEF\xBF\xBF") consider this to be illegal, and replace it by "\x{FFFD}" All the UCS planes appear to be treated like this. chr(0x11_0000) No warning, returns "\x11_0000" This is now outside the UCS range, so I suppose we don't care that this is no more useful than chr(0xFFFE) ? Modern (RFC 3629 & Unicode Consortium) UTF-8 is defined to exclude sequences that exceed the (current) UCS maximum of U+10_FFFF. chr(0x14_0000) No warning, returns "\x14_0000" Modern UTF-8 (RFC 3629 & Unicode Consortium) is defined to exclude any sequence containing any byte 0xC0, 0xC1, and 0xF5-0xFF. This is the first character that contains a byte 0xF5-0xFF ! chr(0xzzzz_FFFE) Warns 'Unicode character 0xzzzzfffe is illegal', returns "\x{zzzz_FFFE}" chr(0xzzzz_FFFF) Warns 'Unicode character 0xzzzzffff is illegal', returns "\x{zzzz_FFFF}" For all values of zzzz from 0x0011 onwards. Now, it's known that 0xFFFE and 0xFFFF are non-characters in all UCS planes... but we're beyond UCS here ? [I confess this baffled me at first, because 0x7FFF_FFFF generates a warning, but 0x8000_0000 doesn't.... But that's another story.] chr(0x0020_0000) No warning, returns "\x{0020_0000}" This is the first character with an encoding > 4 bytes. Modern UTF-8 (RFC 3629 & Unicode Consortium) stops at 4 bytes. chr(0x8000_0000) No warning, returns "\x{8000_0000}" This is the first character with an encoding > 6 bytes. Actually, not even 'old-style' UTF-8 supported anything longer than the 6 byte form. (Because bytes 0xFE and 0xFF were defined not to appear in a UTF-8 sequence -- to guarantee no confusion with UTF-16. Compile time warnings for "\x{h...h}" appear to complain or not complain about the same things. >Could you please report this bug with perlbug? Done. Chris -- Chris Hall highwayman.com +44 7970 277 383 http://www.nntp.perl.org/group/perl.unicode/2008/03/msg3139.html Tue, 11 Mar 2008 11:49:31 +0000 Re: utf8::valid and \x14_000 - \x1F_0000 by Juerd Waalboer Chris Hall skribis 2008-03-11 13:30 (+0000): > I suggest utf8::valid() is broken. > my $s = chr($c) ; > my $v = utf8::valid($s) ? 1 : 0 ; Agreed. utf8::valid(chr $foo) should ALWAYS return true. (Please note that utf8::valid tests the internal consistency of a string - on the outside, it has little to do with UTF8.) Could you please report this bug with perlbug? -- Met vriendelijke groet, Kind regards, Korajn salutojn, Juerd Waalboer: Perl hacker <#####@juerd.nl> <http://juerd.nl/sig> Convolution: ICT solutions and consultancy <sales@convolution.nl> http://www.nntp.perl.org/group/perl.unicode/2008/03/msg3138.html Tue, 11 Mar 2008 06:52:41 +0000 utf8::valid and \x14_000 - \x1F_0000 by Chris Hall It appears that utf8::valid() disagrees with Encode::encode('utf8', ...) do not agree for characters 0x14_0000 - 0x1F_0000. I suggest utf8::valid() is broken. The following: use strict ; use Encode qw(FB_QUIET LEAVE_SRC) ; printf "Perl v%vd & Encode %s\n", $^V, $Encode::VERSION ; my $c = 0xFFFF ; while ($c < 0x8000_0000) { my $s = chr($c) ; my $v = utf8::valid($s) ? 1 : 0 ; my $o = Encode::encode('utf8', $s, FB_QUIET() | LEAVE_SRC()) ; my $r = $o ? 1 : 0 ; if ($v != $r) { printf "0x%04X_%04X: utf8::valid=%d but Encode::encode=%d ", ($c >> 16), $c & 0xFFFF, $v, $r ; Encode::_utf8_off($s) ; print map { sprintf '\x%02X', ord($_) } split(//, $s) ; print "\n" ; } ; if ($c & 0xFFFF) { $c += 1 ; } else { $c += 0xFFFF ; } ; } ; Produces: Perl v5.8.8 & Encode 2.23 0x0014_0000: utf8::valid=0 but Encode::encode=1 \xF5\x80\x80\x80 0x0014_FFFF: utf8::valid=0 but Encode::encode=1 \xF5\x8F\xBF\xBF 0x0015_0000: utf8::valid=0 but Encode::encode=1 \xF5\x90\x80\x80 0x0015_FFFF: utf8::valid=0 but Encode::encode=1 \xF5\x9F\xBF\xBF 0x0016_0000: utf8::valid=0 but Encode::encode=1 \xF5\xA0\x80\x80 0x0016_FFFF: utf8::valid=0 but Encode::encode=1 \xF5\xAF\xBF\xBF 0x0017_0000: utf8::valid=0 but Encode::encode=1 \xF5\xB0\x80\x80 0x0017_FFFF: utf8::valid=0 but Encode::encode=1 \xF5\xBF\xBF\xBF 0x0018_0000: utf8::valid=0 but Encode::encode=1 \xF6\x80\x80\x80 0x0018_FFFF: utf8::valid=0 but Encode::encode=1 \xF6\x8F\xBF\xBF 0x0019_0000: utf8::valid=0 but Encode::encode=1 \xF6\x90\x80\x80 0x0019_FFFF: utf8::valid=0 but Encode::encode=1 \xF6\x9F\xBF\xBF 0x001A_0000: utf8::valid=0 but Encode::encode=1 \xF6\xA0\x80\x80 0x001A_FFFF: utf8::valid=0 but Encode::encode=1 \xF6\xAF\xBF\xBF 0x001B_0000: utf8::valid=0 but Encode::encode=1 \xF6\xB0\x80\x80 0x001B_FFFF: utf8::valid=0 but Encode::encode=1 \xF6\xBF\xBF\xBF 0x001C_0000: utf8::valid=0 but Encode::encode=1 \xF7\x80\x80\x80 0x001C_FFFF: utf8::valid=0 but Encode::encode=1 \xF7\x8F\xBF\xBF 0x001D_0000: utf8::valid=0 but Encode::encode=1 \xF7\x90\x80\x80 0x001D_FFFF: utf8::valid=0 but Encode::encode=1 \xF7\x9F\xBF\xBF 0x001E_0000: utf8::valid=0 but Encode::encode=1 \xF7\xA0\x80\x80 0x001E_FFFF: utf8::valid=0 but Encode::encode=1 \xF7\xAF\xBF\xBF 0x001F_0000: utf8::valid=0 but Encode::encode=1 \xF7\xB0\x80\x80 0x001F_FFFF: utf8::valid=0 but Encode::encode=1 \xF7\xBF\xBF\xBF And the same for: Perl v5.10.0 & Encode 2.23 -- Chris Hall highwayman.com http://www.nntp.perl.org/group/perl.unicode/2008/03/msg3137.html Tue, 11 Mar 2008 06:32:25 +0000 Re: Problem processing UTF-8 strings from email by Juerd Waalboer Tatsuhiko Miyagawa skribis 2008-01-12 16:42 (-0800): > use Encode; > use Encode::MIME::Header; > decode("MIME-Header", $bytes); Note that you don't have to "use Encode::MIME::Header;". Doing this or leaving it out is a matter of personal preference, of course. -- Met vriendelijke groet, Kind regards, Korajn salutojn, Juerd Waalboer: Perl hacker <#####@juerd.nl> <http://juerd.nl/sig> Convolution: ICT solutions and consultancy <sales@convolution.nl> http://www.nntp.perl.org/group/perl.unicode/2008/01/msg3136.html Sun, 13 Jan 2008 05:23:40 +0000 Re: Problem processing UTF-8 strings from email by Neil Gunton Tatsuhiko Miyagawa wrote: > use Encode; > use Encode::MIME::Header; > decode("MIME-Header", $bytes); > > to get the Unicode strings for these MIME encoded characters. > > If you want to turn them into HTML entities, you can say: > > encode("ascii", decode("MIME-Header", $bytes), Encode::FB_HTMLCREF); Wonderful, yes, that worked! Thanks VERY much. I'm slapping my forehead for failing to realize that this was MIME encoding-related. :-) /Neil http://www.nntp.perl.org/group/perl.unicode/2008/01/msg3135.html Sat, 12 Jan 2008 16:50:15 +0000 Re: Problem processing UTF-8 strings from email by Tatsuhiko Miyagawa On 1/12/08, Neil Gunton <neil@nilspace.com> wrote: > > I am somewhat experienced with Perl in general, but absolutely no > experience dealing with UTF-8. I have a community journals website which > allows updates from users via email. I'm having trouble with emails that > contain Chinese characters encoded (I think) as UTF-8. The strings look > like this: > > =?UTF-8?B?5qGQ5LmhLCBUb25neGlhbmc6IEJlaW5nIGEgJ2hhbg==?= =?UTF-8?B?dHUn?= > > When I read this text from a file, using my perl script, and then save > it into MySQL, it comes out on the website looking literally like the > above. I can't seem to get perl to "do" anything with it in terms of > conversions to a format that looks like chinese characters when > displayed on the Web. use Encode; use Encode::MIME::Header; decode("MIME-Header", $bytes); to get the Unicode strings for these MIME encoded characters. > Does anybody have any clues as to how to convert strings like this into > something more usable - e.g. HTML character entities? If you want to turn them into HTML entities, you can say: encode("ascii", decode("MIME-Header", $bytes), Encode::FB_HTMLCREF); HTH -- Tatsuhiko Miyagawa http://www.nntp.perl.org/group/perl.unicode/2008/01/msg3134.html Sat, 12 Jan 2008 16:42:12 +0000 Problem processing UTF-8 strings from email by Neil Gunton (Apologies if you see a duplicate - I think I may have originally sent this to the wrong list) Hi all, I am somewhat experienced with Perl in general, but absolutely no experience dealing with UTF-8. I have a community journals website which allows updates from users via email. I'm having trouble with emails that contain Chinese characters encoded (I think) as UTF-8. The strings look like this: =?UTF-8?B?5qGQ5LmhLCBUb25neGlhbmc6IEJlaW5nIGEgJ2hhbg==?= =?UTF-8?B?dHUn?= When I read this text from a file, using my perl script, and then save it into MySQL, it comes out on the website looking literally like the above. I can't seem to get perl to "do" anything with it in terms of conversions to a format that looks like chinese characters when displayed on the Web. Does anybody have any clues as to how to convert strings like this into something more usable - e.g. HTML character entities? I'm using stock perl 5.8.8 from Debian Etch. Thanks! /Neil http://www.nntp.perl.org/group/perl.unicode/2008/01/msg3133.html Sat, 12 Jan 2008 16:28:05 +0000 Re: Fix UTF Encoding issue by mkoegler On Tue, Dec 04, 2007 at 10:12:50AM +0200, Ismail Dönmez wrote: > I think just a better method is to use (not tested): > > if( is_utf8($str) ) > { > return decode_utf8($str); > } > else { > return decode($str); > } I already tried this function. It does not test, if a string is really UTF-8. It seems to be to intended to check, if perl stores the string internally in a multi byte encoding. mfg Martin Kögler. http://www.nntp.perl.org/group/perl.unicode/2007/12/msg3132.html Tue, 04 Dec 2007 05:21:19 +0000 Re: Fix UTF Encoding issue by Ismail Dönmez Tuesday 04 December 2007 10:47:39 Ismail DÃ¶nmez yazmÄ±ÅtÄ±: > Tuesday 04 December 2007 10:44:12 Martin Koegler yazmÄ±ÅtÄ±: > > On Tue, Dec 04, 2007 at 10:33:39AM +0200, Ismail DÃ¶nmez wrote: > > > Following to_utf8 function works for me : > > > > For me too (Debian sarge+etch). > > Thanks for testing. Use Perl built-in utf8 function for UTF-8 decoding. Signed-off-by: Ä°smail DÃ¶nmez <ismail@pardus.org.tr> diff --git a/gitweb/gitweb.perl b/gitweb/gitweb.perl index ff5daa7..db255c1 100755 --- a/gitweb/gitweb.perl +++ b/gitweb/gitweb.perl @@ -695,10 +695,9 @@ sub validate_refname { # in utf-8 thanks to "binmode STDOUT, ':utf8'" at beginning sub to_utf8 { my $str = shift; - my $res; - eval { $res = decode_utf8($str, Encode::FB_CROAK); }; - if (defined $res) { - return $res; + if (utf8::valid($str)) { + utf8::decode($str); + return $str; } else { return decode($fallback_encoding, $str, Encode::FB_DEFAULT); } -- Never learn by your mistakes, if you do you may never dare to try again. http://www.nntp.perl.org/group/perl.unicode/2007/12/msg3131.html Tue, 04 Dec 2007 05:21:18 +0000 Re: Fix UTF Encoding issue by mkoegler On Tue, Dec 04, 2007 at 10:33:39AM +0200, Ismail Dönmez wrote: > Following to_utf8 function works for me : For me too (Debian sarge+etch). > sub to_utf8 { > ·   my $str = shift; > >     if(utf8::valid($str)) >     { >         utf8::decode($str); >     } > · >     return $str; In the original thread, there was some discussion, that some people might want a different fallback endcoding. So mayme you should keep the second call to decode for the fallback encoding. > } mfg Martin Kögler http://www.nntp.perl.org/group/perl.unicode/2007/12/msg3130.html Tue, 04 Dec 2007 04:47:59 +0000 Re: Fix UTF Encoding issue by Wincent Colaiuta El 4/12/2007, a las 9:55, Ismail Dönmez escribió: > Tuesday 04 December 2007 10:47:39 Ismail Dönmez yazmıştı: >> Tuesday 04 December 2007 10:44:12 Martin Koegler yazmıştı: >>> On Tue, Dec 04, 2007 at 10:33:39AM +0200, Ismail Dönmez wrote: >>>> Following to_utf8 function works for me : >>> >>> For me too (Debian sarge+etch). >> >> Thanks for testing. > > Use Perl built-in utf8 function for UTF-8 decoding. > > Signed-off-by: İsmail Dönmez <ismail@pardus.org.tr> > > diff --git a/gitweb/gitweb.perl b/gitweb/gitweb.perl > index ff5daa7..db255c1 100755 > --- a/gitweb/gitweb.perl > +++ b/gitweb/gitweb.perl > @@ -695,10 +695,9 @@ sub validate_refname { > # in utf-8 thanks to "binmode STDOUT, ':utf8'" at beginning > sub to_utf8 { > my $str = shift; > - my $res; > - eval { $res = decode_utf8($str, Encode::FB_CROAK); }; > - if (defined $res) { > - return $res; > + if (utf8::valid($str)) { > + utf8::decode($str); > + return $str; This is good as it fixes another problem which some may have encountered. On at least one distro that I use (Red Hat Enterprise Linux 3) the Encode module is very old (it's 1.83; the latest release is 2.23), and so gitweb won't even run, dying during compilation with this: Too many arguments for Encode::decode_utf8 at gitweb.cgi line 686, near "Encode::FB_CROAK)" Of course, the workaround is to install a newer version of the module, but this patch eliminates that dependency which is IMO a good thing. Cheers, Wincent http://www.nntp.perl.org/group/perl.unicode/2007/12/msg3129.html Tue, 04 Dec 2007 04:47:58 +0000 Re: Fix UTF Encoding issue by mkoegler On Tue, Dec 04, 2007 at 09:55:04AM +0200, Ismail Dönmez wrote: > Tuesday 04 December 2007 Tarihinde 09:50:28 yazm????t??: > > The bug affects old versions of perl (Debian sarge = oldstable). > > As it works on the newer Debian etch, do you really think, that it is > > a good idea to report issue? > > Same problem here with v5.8.8 which is latest stable perl5 release. I have put together a small perl script, which tests the various ways of decoding, which have been posted on the list. The first test is wrong by design. A working decoding method should result in "#öäü#äöü". Debian sarge: #öäü#Ã€Ã¶Ã&OElig; ##äöü ##äöü ##äöü Debian etch, OpenSuSE 10.2, Fedora 7: #öäü#Ã€Ã¶Ã&OElig; #öäü#äöü #öäü#äöü #öäü#äöü mfg Martin Kögler #!/usr/bin/perl use Encode; sub t { my $str = shift; my ($res); eval { return ($res = decode_utf8($str, Encode::FB_CROAK)); }; return decode("latin1", $str, Encode::FB_DEFAULT); } sub t1 { my $str = shift; my ($res); eval { ($res = decode_utf8($str, Encode::FB_CROAK)); }; if ($@) { return decode("latin1", $str, Encode::FB_DEFAULT); } else { return $res; } } sub t2 { my $str = shift; my ($res); eval { $res = decode_utf8($str, Encode::FB_CROAK); }; if (defined $res) { return $res; } else { return decode("latin1", $str, Encode::FB_DEFAULT); } } sub t3 { my $str = shift; my $res; eval { $res = decode_utf8 ($str, 1); }; return $res || decode('latin1', $str); } print t("#öäü"); print t("#Ã€Ã¶Ã&OElig;"); print "\n"; print t1("#öäü"); print t1("#Ã€Ã¶Ã&OElig;"); print "\n"; print t2("#öäü"); print t2("#Ã€Ã¶Ã&OElig;"); print "\n"; print t3("#öäü"); print t3("#Ã€Ã¶Ã&OElig;"); print "\n"; http://www.nntp.perl.org/group/perl.unicode/2007/12/msg3128.html Tue, 04 Dec 2007 04:47:53 +0000 Re: Fix UTF Encoding issue by Ismail Dönmez Tuesday 04 December 2007 10:44:12 Martin Koegler yazmÄ±ÅtÄ±: > On Tue, Dec 04, 2007 at 10:33:39AM +0200, Ismail DÃ¶nmez wrote: > > Following to_utf8 function works for me : > > For me too (Debian sarge+etch). Thanks for testing. > > sub to_utf8 { > > Â· Â  my $str = shift; > > > > Â  Â  if(utf8::valid($str)) > > Â  Â  { > > Â  Â  Â  Â  utf8::decode($str); > > Â  Â  } > > Â· > > Â  Â  return $str; > > In the original thread, there was some discussion, that some people > might want a different fallback endcoding. So mayme you should > keep the second call to decode for the fallback encoding. Probably, I just wanted to fix this damn UTF-8 bug surfacing over and over =) Regards, ismail -- Never learn by your mistakes, if you do you may never dare to try again. http://www.nntp.perl.org/group/perl.unicode/2007/12/msg3127.html Tue, 04 Dec 2007 04:18:07 +0000 Re: Fix UTF Encoding issue by Ismail Dönmez Tuesday 04 December 2007 10:28:59 Ismail DÃ¶nmez yazmÄ±ÅtÄ±: > Tuesday 04 December 2007 10:16:34 Martin Koegler yazmÄ±ÅtÄ±: > [...] > > > print t("#Ã¶Ã¤Ã¼"); > > print t("#Ãâ¬ÃÂ¶ÃÅ"); > > print "\n"; > > How about this one, doesn't even use Encode, uses just built-in utf8 > function : > > [~]> cat test.pl > binmode STDOUT, ':utf8'; > > my $str = "#Ã¶Ã¤Ã¼"; > > if (utf8::valid($str)) > { > utf8::decode($str); > } > > print $str."\n"; > > [~]> perl test.pl > #Ã¶Ã¤Ã¼ Following to_utf8 function works for me : sub to_utf8 { Â· Â  my $str = shift; Â  Â  if(utf8::valid($str)) Â  Â  { Â  Â  Â  Â  utf8::decode($str); Â  Â  } Â· Â  Â  return $str; } Regards, ismail -- Never learn by your mistakes, if you do you may never dare to try again. http://www.nntp.perl.org/group/perl.unicode/2007/12/msg3126.html Tue, 04 Dec 2007 04:17:52 +0000 Re: Fix UTF Encoding issue by Ismail Dönmez Tuesday 04 December 2007 10:16:34 Martin Koegler yazmÄ±ÅtÄ±: [...] > print t("#Ã¶Ã¤Ã¼"); > print t("#Ãâ¬ÃÂ¶ÃÅ"); > print "\n"; How about this one, doesn't even use Encode, uses just built-in utf8 function : [~]> cat test.pl binmode STDOUT, ':utf8'; my $str = "#Ã¶Ã¤Ã¼"; if (utf8::valid($str)) { utf8::decode($str); } print $str."\n"; [~]> perl test.pl #Ã¶Ã¤Ã¼ Regards, ismail -- Never learn by your mistakes, if you do you may never dare to try again. http://www.nntp.perl.org/group/perl.unicode/2007/12/msg3125.html Tue, 04 Dec 2007 04:17:50 +0000 Re: Fix UTF Encoding issue by Ismail Dönmez Tuesday 04 December 2007 10:04:07 Martin Koegler yazmÄ±ÅtÄ±: > On Tue, Dec 04, 2007 at 08:16:24AM +1030, Benjamin Close wrote: > > Jakub Narebski wrote: > > >On Mon, 3 Dec 2007, Martin Koegler wrote: > > >>On Mon, Dec 03, 2007 at 04:06:48AM -0800, Jakub Narebski wrote: > > >>>Ismail DÃ¶nmez <ismail@pardus.org.tr> writes: > > >>>>Monday 03 December 2007 Tarihinde 12:14:43 yazm??t?: > > >>>>>Benjamin Close <Benjamin.Close@clearchain.com> writes: > > >>>>>>- eval { $res = decode_utf8($str, Encode::FB_CROAK); }; > > >>>>>>- if (defined $res) { > > >>>>>>- return $res; > > >>>>>>- } else { > > >>>>>>- return decode($fallback_encoding, $str, > > >>>>>>Encode::FB_DEFAULT); > > >>>>>>- } > > >>>>>>+ eval { return ($res = decode_utf8($str, Encode::FB_CROAK)); > > >>>>>>}; > > >>>>>>+ return decode($fallback_encoding, $str, Encode::FB_DEFAULT); > > >>>>>> } > > >> > > >>This version is broken on Debian sarge and etch