perl.unicode

perl.unicode http://www.nntp.perl.org/group/perl.unicode/ ... Copyright 1998-2013 perl.org Thu, 23 May 2013 18:56:48 +0000 ask@perl.org Re: Need info on ascii converstion. by Lars Dɪᴇᴄᴋᴏᴡ 迪拉斯 > Error that I am receiving is below : > > $VAR1 = bless( { > 'why' => 'Expected 8 or 0 byte long (19)' > }, 'Cassandra::InvalidRequestException' ); You haven't shown any code and input data to produce that error. Provide a minimal, stand-alone example that someone else can run to reproduce the problem on his own computer. <http://sscce.org/> <http://www.chiark.greenend.org.uk/~sgtatham/bugs.html#showmehow> http://www.nntp.perl.org/group/perl.unicode/2013/03/msg3352.html Tue, 19 Mar 2013 21:45:50 +0000 RE: Need info on ascii converstion. by Dhoke, Swati **CTR** Hi Team , Data I am trying to pass is getting converted into ascii , how can I avoid that in my script . I have tried below module : use utf8; use feature 'unicode_strings'; I still get ascii data even on using above modules Error that I am receiving is below : $VAR1 = bless( { 'why' => 'Expected 8 or 0 byte long (19)' }, 'Cassandra::InvalidRequestException' ); REFERENCE : http://perldoc.perl.org/Encode.html#Handling-Malformed-Data. Thanks , Swati http://www.nntp.perl.org/group/perl.unicode/2013/03/msg3351.html Thu, 14 Mar 2013 21:21:16 +0000 Re: Word boundaries by Zbigniew Łukasiak On Mon, Mar 26, 2012 at 12:57 PM, Lars Dɪᴇᴄᴋᴏᴡ 迪拉斯 <daxim@cpan.org> wrote: > Let the regex engine help you advance the character counter. > >    $ cat langs >    ΕλληνικάEnglish한국어日本語Русскийไทย > > ---- > >    $ cat langs.pl >    use 5.010; >    use strictures; >    use Unicode::UCD qw(charinfo); > >    sub script { >        return charinfo(ord substr($_[0], 0, 1))->{script} >    }; > >    # necessary because pos() magic is tracked on the scalar. >    my $copy = $_; >    while (/(\X)/g) { >        my $script = script $1; >        my ($part) = $copy =~ /(\p{$script}+)/; >        say $part; >        pos($_) = pos($_) + length($part); >    } Thanks a lot! Here is the first version of my tokenizer based on this idea: use Lingua::ZH::MMSEG; sub tokenize { my $text = shift; my @tokens; while ( $text =~ /(\X)/g ) { my $part = $1; my $script = charinfo( ord $1)->{script}; $text=~ /(\p{$script}*)/g; next if $script eq 'Common'; $part .= $1; if( $script eq 'Han' ){ push @tokens, mmseg( $part ); } else{ push @tokens, $part; } } return @tokens; } And the surprise - this works even without further splitting because space and other dots all get the 'Common' script and are not matched by \p{Latin}. -- Zbigniew Lukasiak http://brudnopis.blogspot.com/ http://perlalchemy.blogspot.com/ http://www.nntp.perl.org/group/perl.unicode/2012/03/msg3350.html Tue, 27 Mar 2012 05:21:59 +0000 Re: Word boundaries by Lars Dɪᴇᴄᴋᴏᴡ 迪拉斯 > How can I check what script a character belongs to? $ perl -Mutf8 -MUnicode::UCD=charinfo -E'say charinfo(ord "为")->{script}' Han Sanity checks: $ perl -Mutf8 -E'say "为" =~ /\p{Han}/' 1 $ uniprops -a1 为 | ack Script Script=Han Script=Hani > check if it is the same as the > previous one - i.e. back to C mode of programming. Let the regex engine help you advance the character counter. $ cat langs ΕλληνικάEnglish한국어日本語Русскийไทย ---- $ cat langs.pl use 5.010; use strictures; use Unicode::UCD qw(charinfo); sub script { return charinfo(ord substr($_[0], 0, 1))->{script} }; # necessary because pos() magic is tracked on the scalar. my $copy = $_; while (/(\X)/g) { my $script = script $1; my ($part) = $copy =~ /(\p{$script}+)/; say $part; pos($_) = pos($_) + length($part); } ---- $ perl -C -ln langs.pl < langs Ελληνικά English 한국어 Русский ไทย http://www.nntp.perl.org/group/perl.unicode/2012/03/msg3349.html Mon, 26 Mar 2012 03:58:11 +0000 Word boundaries by Zbigniew Łukasiak For our spam classifier I need to split the text into words. Unfortunately the '\b' regex does not yet work for languages with no spaces (apparently it is covered in the level 3 of unicode support http://unicode.org/reports/tr18/#Tailored_Word_Boundaries) - so I need some custom solution. This did not seem very difficult - just split the text into blocks of same unicode script and then use '\b' for most of the scripts and appropriate libraries for the rest (at least for Chinese there are some tokenizers on CPAN) - but: 1. How can I split the text into blocks of same scripts? (Wouldn't a script-boundary regex property be useful?). OK I can always loop over the characters, check their script and check if it is the same as the previous one - i.e. back to C mode of programming. But then there is still the question of: 2. How can I check what script a character belongs to? Do I need to cut and paste all the script ranges from unicode.org into a huge if-else branch in my program or is there a simpler way? Thanks in advance, Zbigniew http://www.nntp.perl.org/group/perl.unicode/2012/03/msg3348.html Mon, 26 Mar 2012 02:03:20 +0000 Re: Please help me with this perl question by Lars Dɪᴇᴄᴋᴏᴡ 迪拉斯 Yes, it is possible. This is an example: perl -C -E'say"\x{e8f4}\x{e8f5}"' Screenshot: <http://i.imgur.com/wivY9.png> The magic happens not at the Perl level, but at the rendering step. I first picked two unassigned codepoints. Unicode provides a private use area for exactly this kind of purpose. I also took care not to trample over a [registered script] (http://enwp.org/ConScript_Unicode_Registry). Then I created a font with two glyphs, `U+E8F4 ROTATED LATIN CAPITAL LETTER D` and `U+E8F5 LATIN CAPITAL LETTER A WITHOUT COUNTER`, and installed it. Last, I restarted my terminal application so that the new font gets picked up, and executed the Perl one-liner from above. http://www.nntp.perl.org/group/perl.unicode/2012/02/msg3347.html Tue, 14 Feb 2012 02:01:25 +0000 Re: Unicode on Windows Console by Michael Ludwig Lars Dɪᴇᴄᴋᴏᴡ 迪拉斯 schrieb am 12.01.2012 um 09:54 (+0100): > Run `chcp 65001`, see <http://enwp.org/chcp_(command)>. I have not > tested this. Okay, that's just the regular chcp command. > Setting the encoding to Windows-1252 and then expecting to be able to > talk variations of UTF to it seems wrong. Might seem wrong at first glance, I agree; but hey, this is Windows, and it magically works, sort of bypassing the chcp setting! You just need the C API call: _setmode(_fileno(stdout), _O_WTEXT) Or some equivalent of that, which is what Win32::Unicode seems to be doing. -- Michael Ludwig http://www.nntp.perl.org/group/perl.unicode/2012/01/msg3346.html Thu, 12 Jan 2012 11:51:13 +0000 Re: Unicode on Windows Console by Lars Dɪᴇᴄᴋᴏᴡ 迪拉斯 Run `chcp 65001`, see <http://enwp.org/chcp_(command)>. I have not tested this. Setting the encoding to Windows-1252 and then expecting to be able to talk variations of UTF to it seems wrong. http://www.nntp.perl.org/group/perl.unicode/2012/01/msg3345.html Thu, 12 Jan 2012 00:54:14 +0000 Re: question about perlunicode "Unicode Character Properties" by Lars Dɪᴇᴄᴋᴏᴡ 迪拉斯 Install <http://p3rl.org/unichars>. $ unichars -au '\p{Han}' | wc -l 75960 $ unichars -au '\p{Han}' | perl -lne'print unless $. % 1000' 㚏 U+0368F CJK UNIFIED IDEOGRAPH-368F 㩷 U+03A77 CJK UNIFIED IDEOGRAPH-3A77 㹟 U+03E5F CJK UNIFIED IDEOGRAPH-3E5F 䉇 U+04247 CJK UNIFIED IDEOGRAPH-4247 䘯 U+0462F CJK UNIFIED IDEOGRAPH-462F 䨗 U+04A17 CJK UNIFIED IDEOGRAPH-4A17 义 U+04E49 CJK UNIFIED IDEOGRAPH-4E49 刱 U+05231 CJK UNIFIED IDEOGRAPH-5231 嘙 U+05619 CJK UNIFIED IDEOGRAPH-5619 威 U+05A01 CJK UNIFIED IDEOGRAPH-5A01 巩 U+05DE9 CJK UNIFIED IDEOGRAPH-5DE9 懑 U+061D1 CJK UNIFIED IDEOGRAPH-61D1 方 U+065B9 CJK UNIFIED IDEOGRAPH-65B9 榡 U+069A1 CJK UNIFIED IDEOGRAPH-69A1 涉 U+06D89 CJK UNIFIED IDEOGRAPH-6D89 煱 U+07171 CJK UNIFIED IDEOGRAPH-7171 留 U+07559 CJK UNIFIED IDEOGRAPH-7559 祁 U+07941 CJK UNIFIED IDEOGRAPH-7941 紩 U+07D29 CJK UNIFIED IDEOGRAPH-7D29 脑 U+08111 CJK UNIFIED IDEOGRAPH-8111 蓹 U+084F9 CJK UNIFIED IDEOGRAPH-84F9 裡 U+088E1 CJK UNIFIED IDEOGRAPH-88E1 賉 U+08CC9 CJK UNIFIED IDEOGRAPH-8CC9 邱 U+090B1 CJK UNIFIED IDEOGRAPH-90B1 钙 U+09499 CJK UNIFIED IDEOGRAPH-9499 颁 U+09881 CJK UNIFIED IDEOGRAPH-9881 鱩 U+09C69 CJK UNIFIED IDEOGRAPH-9C69 礪 U+0F985 CJK COMPATIBILITY IDEOGRAPH-F985 𠊗 U+20297 CJK UNIFIED IDEOGRAPH-20297 𠙿 U+2067F CJK UNIFIED IDEOGRAPH-2067F 𠩧 U+20A67 CJK UNIFIED IDEOGRAPH-20A67 𠹏 U+20E4F CJK UNIFIED IDEOGRAPH-20E4F 𡈷 U+21237 CJK UNIFIED IDEOGRAPH-21237 𡘟 U+2161F CJK UNIFIED IDEOGRAPH-2161F 𡨇 U+21A07 CJK UNIFIED IDEOGRAPH-21A07 𡷯 U+21DEF CJK UNIFIED IDEOGRAPH-21DEF 𢇗 U+221D7 CJK UNIFIED IDEOGRAPH-221D7 𢖿 U+225BF CJK UNIFIED IDEOGRAPH-225BF 𢦧 U+229A7 CJK UNIFIED IDEOGRAPH-229A7 𢶏 U+22D8F CJK UNIFIED IDEOGRAPH-22D8F 𣅷 U+23177 CJK UNIFIED IDEOGRAPH-23177 𣕟 U+2355F CJK UNIFIED IDEOGRAPH-2355F 𣥇 U+23947 CJK UNIFIED IDEOGRAPH-23947 𣴯 U+23D2F CJK UNIFIED IDEOGRAPH-23D2F 𤄗 U+24117 CJK UNIFIED IDEOGRAPH-24117 𤓿 U+244FF CJK UNIFIED IDEOGRAPH-244FF 𤣧 U+248E7 CJK UNIFIED IDEOGRAPH-248E7 𤳏 U+24CCF CJK UNIFIED IDEOGRAPH-24CCF 𥂷 U+250B7 CJK UNIFIED IDEOGRAPH-250B7 𥒟 U+2549F CJK UNIFIED IDEOGRAPH-2549F 𥢇 U+25887 CJK UNIFIED IDEOGRAPH-25887 𥱯 U+25C6F CJK UNIFIED IDEOGRAPH-25C6F 𦁗 U+26057 CJK UNIFIED IDEOGRAPH-26057 𦐿 U+2643F CJK UNIFIED IDEOGRAPH-2643F 𦠧 U+26827 CJK UNIFIED IDEOGRAPH-26827 𦰏 U+26C0F CJK UNIFIED IDEOGRAPH-26C0F 𦿷 U+26FF7 CJK UNIFIED IDEOGRAPH-26FF7 𧏟 U+273DF CJK UNIFIED IDEOGRAPH-273DF 𧟇 U+277C7 CJK UNIFIED IDEOGRAPH-277C7 𧮯 U+27BAF CJK UNIFIED IDEOGRAPH-27BAF 𧾗 U+27F97 CJK UNIFIED IDEOGRAPH-27F97 𨍿 U+2837F CJK UNIFIED IDEOGRAPH-2837F 𨝧 U+28767 CJK UNIFIED IDEOGRAPH-28767 𨭏 U+28B4F CJK UNIFIED IDEOGRAPH-28B4F 𨼷 U+28F37 CJK UNIFIED IDEOGRAPH-28F37 𩌟 U+2931F CJK UNIFIED IDEOGRAPH-2931F 𩜇 U+29707 CJK UNIFIED IDEOGRAPH-29707 𩫯 U+29AEF CJK UNIFIED IDEOGRAPH-29AEF 𩻗 U+29ED7 CJK UNIFIED IDEOGRAPH-29ED7 𪊿 U+2A2BF CJK UNIFIED IDEOGRAPH-2A2BF 𪚧 U+2A6A7 CJK UNIFIED IDEOGRAPH-2A6A7 𪪸 U+2AAB8 CJK UNIFIED IDEOGRAPH-2AAB8 𪺠 U+2AEA0 CJK UNIFIED IDEOGRAPH-2AEA0 𫊈 U+2B288 CJK UNIFIED IDEOGRAPH-2B288 𫙰 U+2B670 CJK UNIFIED IDEOGRAPH-2B670 To get a good coverage for display, install the following font families: 文泉驿正黑 <http://wenq.org/?ZenHei> Han Nom <http://vietunicode.sf.net/fonts/fonts_hannom.html> Code200x <http://web.archive.org/web/2010/http://code2000.net/> http://www.nntp.perl.org/group/perl.unicode/2012/01/msg3344.html Thu, 12 Jan 2012 00:47:42 +0000 question about perlunicode "Unicode Character Properties" by silent in perldoc perlunicde : Unicode Character Properties : Scripts I see a Han, which can be use as $string =~/\p{Han}/; my question is how can I find out what exactly "Han" is ? I know \p{Han} can match a Chinese word, also tested it to match each word in perl-src/ext/Encode/t/gb2312.utf, but I do not know the exact range of this \p{Han}. thanks! http://www.nntp.perl.org/group/perl.unicode/2012/01/msg3343.html Wed, 11 Jan 2012 23:10:34 +0000 Re: Unicode on Windows Console by Michael Ludwig Michael Ludwig schrieb am 07.01.2012 um 18:30 (+0100): > There's a WinAPI function that sets stdout to Unicode so you can > read Cyrillic and Greek characters in the cmd.exe console window: > Can I get the same feature in Perl? Yes: https://metacpan.org/module/Win32::Unicode Printing twelve hearts: perl -MWin32::Unicode -lwe "printW qq( \x{2665}) x 12" &hearts; &hearts; &hearts; &hearts; &hearts; &hearts; &hearts; &hearts; &hearts; &hearts; &hearts; &hearts; -- Michael Ludwig http://www.nntp.perl.org/group/perl.unicode/2012/01/msg3342.html Wed, 11 Jan 2012 14:55:44 +0000 Unicode on Windows Console by Michael Ludwig There's a WinAPI function that sets stdout to Unicode so you can read Cyrillic and Greek characters in the cmd.exe console window: \,,,/ (o o) ------oOOo-(_)-oOOo------ // http://msdn.microsoft.com/en-us/library/tw4k6df8.aspx - _setmode // crt_setmodeunicode.c // This program uses _setmode to change // stdout to Unicode. Cyrillic and Ideographic // characters will appear on the console (if // your console font supports those character sets). #include <fcntl.h> #include <io.h> #include <stdio.h> int main(void) { _setmode(_fileno(stdout), _O_U16TEXT); wprintf(L"\x043a\x043e\x0448\x043a\x0430 \x65e5\x672c\x56fd\n"); return 0; } ------------------------- Compile it and will print "кошка 日本国", so a Russian name and three fancy ideograms. (The ideograms aren't supported by my font, but that is an unrelated problem; and I can't read them anyway; still they look nice.) My console codepage is just 1252, Western European. Can I get the same feature in Perl? I left the codepage at 1252 (can be changes using CHCP) and tried the following: * binmode STDOUT, ':encoding(UTF-16LE)' * binmode STDOUT, ':encoding(UTF-16BE)' * binmode STDOUT, ':encoding(UTF-8)' None of these produced the desired effect. Any ideas? I know I could use a Linux UTF-8 terminal or Cygwin/MinTTY, which is what I'm using to write this mail, by the way; but the question is specific to the Windows console in cmd.exe and how to make Perl use its features. -- Michael Ludwig http://www.nntp.perl.org/group/perl.unicode/2012/01/msg3341.html Sat, 07 Jan 2012 09:31:01 +0000 Re: Please help me with this perl question by Dean Hoover I don't see how you can do that... you are dependent on the capabilities of the terminal. If its an old school text terminal, you can't just draw arbitrary stuff like a rotated D. That is pretty fancy and even in graphics mode has nothing to do with perl, or do I misunderstand your question? On Fri, Dec 30, 2011 at 2:12 PM, Michael Ludwig <milu71@gmx.de> wrote: > FORREST COPLEY schrieb am 29.12.2011 um 10:48 (-0700): > > Is it possible to write a perl script to print a completely custom > > character on a console text terminal? > > Say a D rotated 90 degrees or something. > > or an A with the innards filled in. > > Here's how to print alpha to omega: > > perl -C2 -lwe 'print join q(, ), map chr, 0x391 .. 0x3a9' > > Α, Β, Γ, Δ, Ε, Ζ, Η, Θ, Ι, Κ, Λ, Μ, Ν, Ξ, Ο, Π, Ρ, ΢, Σ, Τ, Υ, Φ, Χ, Ψ, Ω > > If you know the Unicode codepoints you can certainly print the characters. > For Perl, it's essentially numbers. > > Whether your particular character comes out nicely or not depends on > whether the font you're using has the glyph in question. > > Next time you might want to spend two seconds or even ten thinking > about a suitable subject line … > > -- > Michael Ludwig > http://www.nntp.perl.org/group/perl.unicode/2011/12/msg3340.html Fri, 30 Dec 2011 17:39:34 +0000 Re: Please help me with this perl question by Karl Williamson On 12/29/2011 10:48 AM, FORREST COPLEY wrote: > Is it possible to write a perl script to print a completely custom > character on a console text terminal? > Say a D rotated 90 degrees or something. > or an A with the innards filled in. > -- > > http://search.cpan.org/~bdfoy/Unicode-Tussle-1.03/lib/Unicode/Tussle.pm contains the Perl script: leo - uʍopəpᴉsdn sƃuᴉɥʇ əʇᴉɹʍ oʇ ɹəʇlᴉɟ That's the closest thing I know about http://www.nntp.perl.org/group/perl.unicode/2011/12/msg3339.html Fri, 30 Dec 2011 11:47:21 +0000 Re: Please help me with this perl question by Michael Ludwig FORREST COPLEY schrieb am 29.12.2011 um 10:48 (-0700): > Is it possible to write a perl script to print a completely custom > character on a console text terminal? > Say a D rotated 90 degrees or something. > or an A with the innards filled in. Here's how to print alpha to omega: perl -C2 -lwe 'print join q(, ), map chr, 0x391 .. 0x3a9' Α, Β, Γ, Δ, Ε, Ζ, Η, Θ, Ι, Κ, Λ, Μ, Ν, Ξ, Ο, Π, Ρ, ΢, Σ, Τ, Υ, Φ, Χ, Ψ, Ω If you know the Unicode codepoints you can certainly print the characters. For Perl, it's essentially numbers. Whether your particular character comes out nicely or not depends on whether the font you're using has the glyph in question. Next time you might want to spend two seconds or even ten thinking about a suitable subject line … -- Michael Ludwig http://www.nntp.perl.org/group/perl.unicode/2011/12/msg3338.html Fri, 30 Dec 2011 11:12:44 +0000 Please help me with this perl question by FORREST COPLEY Is it possible to write a perl script to print a completely custom character on a console text terminal? Say a D rotated 90 degrees or something. or an A with the innards filled in. -- Forrest Copley | Senior Technical Support Engineer, Solaris OS Support , Global Systems Support Email: forrest.copley@oracle.com <mailto:rob.hulme@oracle.com> Phone: 303.272.6716 OracleGlobal Customer Services Log, update, and monitor your Service Request online using https://support.oracle.com <https://support.oracle.com/> http://www.nntp.perl.org/group/perl.unicode/2011/12/msg3337.html Fri, 30 Dec 2011 10:55:25 +0000 New API available to access Unicode DB, and RFC on changes to it. by Karl Williamson Repeat of last message, but now the attachment should be correct. Perl 5.15.5, now available, has additions to Unicode::UCD in it to allow unfettered programmatic access to the Unicode character data base. The API is quite similar to what was sent out for comment on this list several months ago; several changes were required as a result of lessons learned during implementation. This email has an attachment that is an html file giving (with a yellow background) the additions since 5.14 to the pod. As a result of this API, it is deprecated to read the files in lib/unicore directly. These may change, and the API will be stable as of 5.16. In the meantime, I'd be happy to have people use this, and give me get feedback on any problems with the API or bugs in the code. And, I do wish to change the API already for certain of the outputs in prop_invmap() in order to make them more compact. For example, take the uc() property. What it currently returns is this (taken from the attached pod): @$uppers_ranges_ref @$uppers_maps_ref Note 0 "<code point>" 97 65 'a' maps to 'A' 98 66 'b' => 'B' 99 67 'c' => 'C' ... 120 88 'x' => 'X' 121 89 'y' => 'Y' 122 90 'z' => 'Z' 123 "<code point>" 181 924 MICRO SIGN => Greek Cap MU 182 "<code point>" ... 0x0149 [ 0x02BC 0x004E ] 0x014A "<code point>" 0x014B 0x014A ... That could be more compactly represented as: @$uppers_ranges_ref @$uppers_maps_ref Note 0 0 97 -32 'a-z' maps to 'A'-'Z' 123 0 181 743 MICRO SIGN => Greek Cap MU 182 0 ... 0x0149 [ 0x02BC 0x004E ] 0x014A 0 0x014B -1 ... where the map is to be added to the code point to get the final result. Thus only one entry is needed to represent all 26 ASCII lower case character mappings, instead of 26 entries. This makes such tables significantly smaller. The Perl core currently does a linear search through them looking for mappings. Using the more compact versions would speed that up significantly. The percentage gain is 30-40%, and with the mapping for decimal digits the result is a full order of magnitude smaller, making the search much much faster. Returning the delta only makes sense on a few tables, ones that whose map is code points, or the decimal digits. As you can see in the example for 0x0149, I wouldn't propose to make deltas of the lists, even though that is inconsistent. They generally require special handling. http://www.nntp.perl.org/group/perl.unicode/2011/11/msg3336.html Mon, 21 Nov 2011 13:19:46 +0000 New API available to access Unicode DB, and RFC on changes to it. by Karl Williamson Perl 5.15.5, now available, has additions to Unicode::UCD in it to allow unfettered programmatic access to the Unicode character data base. The API is quite similar to what was sent out for comment on this list several months ago; several changes were required as a result of lessons learned during implementation. This email has an attachment that is an html file giving (with a yellow background) the additions since 5.14 to the pod. As a result of this API, it is deprecated to read the files in lib/unicore directly. These may change, and the API will be stable as of 5.16. In the meantime, I'd be happy to have people use this, and give me get feedback on any problems with the API or bugs in the code. And, I do wish to change the API already for certain of the outputs in prop_invmap() in order to make them more compact. For example, take the uc() property. What it currently returns is this (taken from the attached pod): @$uppers_ranges_ref @$uppers_maps_ref Note 0 "<code point>" 97 65 'a' maps to 'A' 98 66 'b' => 'B' 99 67 'c' => 'C' ... 120 88 'x' => 'X' 121 89 'y' => 'Y' 122 90 'z' => 'Z' 123 "<code point>" 181 924 MICRO SIGN => Greek Cap MU 182 "<code point>" ... 0x0149 [ 0x02BC 0x004E ] 0x014A "<code point>" 0x014B 0x014A ... That could be more compactly represented as: @$uppers_ranges_ref @$uppers_maps_ref Note 0 0 97 -32 'a-z' maps to 'A'-'Z' 123 0 181 743 MICRO SIGN => Greek Cap MU 182 0 ... 0x0149 [ 0x02BC 0x004E ] 0x014A 0 0x014B -1 ... where the map is to be added to the code point to get the final result. Thus only one entry is needed to represent all 26 ASCII lower case character mappings, instead of 26 entries. This makes such tables significantly smaller. The Perl core currently does a linear search through them looking for mappings. Using the more compact versions would speed that up significantly. The percentage gain is 30-40%, and with the mapping for decimal digits the result is a full order of magnitude smaller, making the search much much faster. Returning the delta only makes sense on a few tables, ones that whose map is code points, or the decimal digits. As you can see in the example for 0x0149, I wouldn't propose to make deltas of the lists, even though that is inconsistent. They generally require special handling. http://www.nntp.perl.org/group/perl.unicode/2011/11/msg3335.html Mon, 21 Nov 2011 12:42:40 +0000 Re: RFC: API to access Unicode db files by Karl Williamson Here's a new version of the API for comment, with the addition of 2 extra functions: prop_invlist() "prop_invlist" returns an inversion list (described below) that defines all the code points for the Unicode property given by the input parameter string: use Unicode::UCD 'prop_invlist'; say join ", ", prop_invlist("Any"); 0, 1114112 An empty list is returned if the given property is unknown; the number of elements in the list is returned if called in scalar context. perluniprops gives the list of properties that this function accepts, as well as all the possible forms for them (loose matching rules are used on the parameter). Note that many properties can be specified in a compound form, such as say join ", ", prop_invlist("Script=Shavian"); 66640, 66688 say join ", ", prop_invlist("ASCII_Hex_Digit=No"); 0, 48, 58, 65, 71, 97, 103 say join ", ", prop_invlist("ASCII_Hex_Digit=Yes"); 48, 58, 65, 71, 97, 103 Inversion lists are a compact way of specifying Unicode properties. The 0th item in the list is the lowest code point that has the property-value. The next item is the lowest code point after that one that does NOT have the property-value. And the next item after that is the lowest code point after that one that has the property-value, and so on. Put another way, each element in the list gives the beginning of a range that has the property-value (for even numbered elements), or doesn't have the property-value (for odd numbered elements). In the final example above, the first ASCII Hex digit is code point 48, the character "0", and all code points from it through 57 (a "9") are ASCII hex digits. Code points 58 through 64 aren't, but 65 (an "A") through 70 (an "F") are, as are 97 ("a") through 102 ("f"). 103 starts a range of code points that aren't ASCII hex digits. That range extends to infinity, which on your computer can be found in the variable $Unicode::UCD::MAX_CP. (This variable is as close to infinity as Perl can get on your platform, and may be too high for some operations to work; you may wish to use a smaller number for your purposes.) The name for this data structure stems from the fact that each element in the list toggles (or inverts) whether the corresponding range is or isn't on the list. It is a simple matter to expand out an inversion list to a full list of all code points that have the property-value: my @invlist = prop_invlist("My Property"); die "empty" unless @invlist; my @full_list; for (my $i = 0; $i < @invlist; $i += 2) { my $upper = ($i + 1) < @invlist ? $invlist[$i+1] - 1 # In range : $Unicode::UCD::MAX_CP; # To infinity. You may want # to stop much much earlier; # going this high may expose # perl bugs with very large # numbers. for my $j ($invlist[$i] .. $upper) { push @full_list, $j; } } prop_aliases() use Unicode::UCD 'prop_aliases'; my $full_name = prop_value_aliases("White Space"); my @all_names = prop_value_aliases("White Space"); my $short_name = $all_names[0]; print join ", ", @all_names, "\n"; XXX Most Unicode properties have several synonymous names. Typically, there is at least a short name, convenient to type, and a long name that more fully describes the property, and hence is more easily understood. If you know one name for a property, you can use "prop_aliases" to find either the long name (when called in scalar context), or a list of all of the names, somewhat ordered so that the short name is in the 0th element, the long name in the next element, and any other synonyms in the remaining elements, in no particular order. The long name is returned in a form nicely capitalized, suitable for printing. White space, hyphens, and underscores are ignored in the input parameter name. If the name is unknown, "undef" is returned. prop_value_aliases() use Unicode::UCD 'prop_value_aliases'; my $full_name = prop_value_aliases("Gc", "Punct"); my @all_names = prop_value_aliases("Gc", "Punct"); my $short_name = $all_names[0]; print "The aliases are: ", join ", ", @all_names, "\n"; print "The fullname is $full_name\n"; The aliases are: P, Punctuation, Punct The fullname is Punctuation Some Unicode properties have a restricted set of legal values. For example, all binary properties are restricted to just "true" or "false"; and there are only a few dozen possible General Categories. For such properties, there are usually several synonyms for each possible value. For example, in binary properties, truth can be represented by any of the strings, "Y", "Yes", "T", or "True"; and the General Category "Punctuation" by that string, or "Punct", or simply "P". Like property names, there is typically at least a short name for each such property-value, and a long name. If you know any name of the property-value, you can use "prop_value_aliases"() to get the long name (when called in scalar context), or a list of all the names, with the short name in the 0th element, the long name in the next element, and any other synonyms in the remaining elements, in no particular order, except that any all-numeric synonyms will be last. The long name is returned in a form nicely capitalized, suitable for printing. White space, hyphens, and underscores are ignored in the input parameters. If either name is unknown, "undef" is returned. If called with a property that doesn't have synonyms for its values, it returns the input value, possibly normalized with capitalization and underscores. For the block property, new-style block names are returned (see "Old-style versus new-style block names"). prop_invmap() "prop_invmap" is used to get the complete mapping definition for a property, in the form of an inversion map. An inversion map consists of two parallel arrays. One is an ordered list of code points that mark range beginnings, and the other gives the value (or mapping) that all code points in the corresponding range have. "prop_invmap" is called with the name of the desired property. The name is loosely matched, meaning that differences in case, white-space, hyphens, and underscores are not meaningful. Many Unicode properties have more than one name (or alias). "prop_invmap" understands all of these. "undef" is returned if the property name is unknown. It is a fatal error to call this function except in list context. In addition to the the two arrays that form the inversion map, "prop_invmap" returns two other values, one is a scalar that gives some details as to the format of the entries of the map array; the other is used for specialized purposes, described at the end of this section. This means that "prop_invmap" returns a 4 element list. For example, my ($blocks_ranges_ref, $blocks_maps_ref, $format, $default) = prop_invmap("Block"); In this call, the two arrays will be populated as shown below (for Unicode 6.0): Index @blocks_ranges @blocks_maps 0 0x0000 Basic Latin 1 0x0080 Latin-1 Supplement 2 0x0100 Latin Extended-A 3 0x0180 Latin Extended-B 4 0x0250 IPA Extensions 5 0x02B0 Spacing Modifier Letters 6 0x0300 Combining Diacritical Marks 7 0x0370 Greek and Coptic 8 0x0400 Cyrillic ... 233 0x2B820 No_Block 234 0x2F800 CJK Compatibility Ideographs Supplement 235 0x2FA20 No_Block 236 0xE0000 Tags 237 0xE0080 No_Block 238 0xE0100 Variation Selectors Supplement 239 0xE01F0 No_Block 240 0xF0000 Supplementary Private Use Area-A 241 0x100000 Supplementary Private Use Area-B 242 0x110000 No_Block The first line (with Index 0) means that the value for code point 0 is "Basic Latin". The entry "0x0080" in the @blocks_ranges column in the second line means that the value from the first line, "Basic Latin", extends to all code points in the range up to but not including 0x0080, that is, to 255. In other words, the code points from 0 to 255 are all in the "Basic Latin" block. Similarly, all code points in the range from 0x0080 up to (but not including) 0x0100 are in the block named "Latin-1 Supplement", etc. (Notice that the return is the old-style block names; see "Old-style versus new-style block names"). The final line (with Index 242) means that the value for all code points above the legal Unicode maximum code point have the value "No_Block", which is the term Unicode uses for a non-existing block. The arrays completely specify the mappings for all possible code points. The final element in an inversion map returned by this function will always be for the range that consists of all the code points that aren't legal Unicode, but that are expressible on the platform. (That is, it starts with code point 0x110000, the first code point above the legal Unicode maximum, and extends to infinity.) The value for that range will be the same that any normal unassigned code point has for the specified property. (Certain unassigned code points are not "normal"; for example the non-character code points, or those in blocks that are to be written right-to- left. The range value will not necessarily be the same as those code points have.) It could be argued that, instead of treating these as unassigned Unicode code points, the value for this range should be "undef". You can make that decision and change the returned array accordingly. The maps are almost always simple scalars that should be interpreted as-is. These values are those given in the Unicode data files, which may be inconsistent as to capitalization and which synonym for a property-value is given. The results may be normalized by using the "prop_value_aliases()" function. There are exceptions to the simple scalar maps. Some properties have some elements in their map list that are themselves lists of scalars; and some special strings are returned that are not to be interpreted as-is. Element [2] (placed into $format in the example above) of the returned 4 element list tells you if the map has any of these special elements, as follows: "s" means all the elements of the map array are simple scalars. Almost all properties are like this, like the "block" example above. "sl" means that some of the map array elements have the form given by "s", and the rest are lists of scalars. For example, here is a portion of the output of calling "prop_invmap"() with the "Script Extensions" property: @scripts_ranges @scripts_maps ... 0x0953 Deva 0x0964 [ Beng Deva Guru Orya ] 0x0966 Deva 0x0970 Common Here, the code points 0x964 and 0x965 are used in the Bengali, Devanagari, Gurmukhi, and Oriya scripts. "r" means that all the elements of the map array are either rational numbers or the string "NaN", meaning "Not a Number". A rational number is either an integer, or two integers separated by a solidus ("/"). The second integer represents the denominator of the division implied by the solidus, and is guaranteed not to be 0. If you want to convert them to scalar numbers, you can use something like this: my ($format, $invlist_ref, $invmap_ref) = prop_invmap($property); if ($format && $format eq "r") { map { $_ = eval $_ } @$invmap_ref; } Here's some entries from the output of the property "Nv", which has format "r". @numerics_ranges @numerics_maps Note 0x00 "NaN" 0x30 0 DIGIT 0 0x31 1 0x32 2 ... 0x37 7 0x38 8 0x39 9 DIGIT 9 0x3A "NaN" 0xB2 2 SUPERSCRIPT 2 0xB3 3 SUPERSCRIPT 2 0xB4 "NaN" 0xB9 1 SUPERSCRIPT 1 0xBA "NaN" 0xBC 1/4 VULGAR FRACTION 1/4 0xBD 1/2 VULGAR FRACTION 1/2 0xBE 3/4 VULGAR FRACTION 3/4 0xBF "NaN" 0x660 0 ARABIC-INDIC DIGIT ZERO "c" is like "s" in that all the map array elements are scalars, but some of them are the special string "<code point>", meaning that the map of each code point in the corresponding range in the inversion list is the code point itself. For example, in: my ($format, $uppers_ranges_ref, $uppers_maps_ref) = prop_invmap("Simple_Uppercase_Mapping"); the returned arrays look like this: @$uppers_ranges_ref @$uppers_maps_ref Note 0 "<code point>" 97 65 'a' maps to 'A' 98 66 'b' => 'B' 99 67 'c' => 'C' ... 120 88 'x' => 'X' 121 89 'y' => 'Y' 122 90 'z' => 'Z' 123 "<code point>" 181 924 MICRO SIGN => Greek Cap MU 182 "<code point>" ... The first line means that the uppercase of code point 0 is 0; the uppercase of code point 1 is 1; ... of code point 96 is 96. Without the "<code_point>" notation, every code point would have to have an entry. This would mean that the arrays would each have more than a million entries to list just the legal Unicode code points! "cl" means that some of the map array elements have the form given by "c", and the rest are ordered lists of code points. For example, in: my ($format, $uppers_ranges_ref, $uppers_maps_ref) = prop_invmap("Uppercase_Mapping"); the returned arrays look like this: @$uppers_ranges_ref @$uppers_maps_ref Note 0 "<code point>" 97 65 ... 122 90 123 "<code point>" 181 924 182 "<code point>" ... 0x0149 [ 0x02BC 0x004E ] This is the full Uppercase_Mapping property (as opposed to the Simple_Uppercase_Mapping given in the example for "c"). The only difference between the two in the ranges shown is that the code point at 0x0149 (LATIN SMALL LETTER N PRECEDED BY APOSTROPHE) maps to a string of two characters, 0x02BC (MODIFIER LETTER APOSTROPHE) followed by 0x004E (LATIN CAPITAL LETTER N). "n" means the Name property. All the elements of the map array are simple scalars, but some of them contain special strings that require more work to get the actual name. Entries such as: CJK UNIFIED IDEOGRAPH-<code point> mean that the name for the code point is "CJK UNIFIED IDEOGRAPH-" with the code point (expressed in hexadecimal) appended to it (similarly for "CJK COMPATIBILITY IDEOGRAPH-<code point>"). Also, entries like <hangul syllable> means that the name is algorithmically calculated. This is easily done by the function charnames::viacode(). Note that for control characters ("Gc=cc"), Unicode's data files have the string ""control"", but the real name of each of these characters is the empty string. This function returns the real name. "d" means the Decomposition_Mapping property. Like "n", this property uses <hangul syllable> for those code points whose decomposition is algorithmically calculated. These can be generated via the function Unicode::Normalize::NFD(). Otherwise, this property is like "cl" properties. Note that the mapping is the one that is specified in the Unicode data files, and to get the final decomposition, it may need to be applied recursively. A binary search can be used to quickly find a code point in the inversion list, and hence its corresponding mapping. The final element ([3], assigned to $default in the "block" example) in the list returned by this function may be useful for applications that wish to convert the returned inversion map data structure into some other, such as a hash. It gives the mapping that most code points map to under the property. If you establish the convention that any code point not explicitly listed in your data structure maps to this value, you can potentially make your data structure much smaller. As you construct your data structure from the one returned by this function, simply ignore those ranges that map to this value, generally called the "default" value. One internal Perl property is accessible by this function. "Perl_Decimal_Digit" returns an inversion map in which all the Unicode decimal digits map to their numeric values, and everything else to the empty string, like so: @digits @values 0x0000 "" 0x0030 0 0x0031 1 0x0032 2 0x0033 3 0x0034 4 0x0035 5 0x0036 6 0x0037 7 0x0038 8 0x0039 9 0x003A "" 0x0660 0 0x0661 1 ... Old-style versus new-style block names Unicode publishes the names of blocks in two different styles, though the two are equivalent under Unicode's loose matching rules. The original style uses blanks and hyphens in the block names (except for "No_Block"), like so: Miscellaneous Mathematical Symbols-B The newer style replaces these with underscores, like this: Miscellaneous_Mathematical_Symbols_B This newer style is consistent with the values of other Unicode properties. To preserve backward compatibility, all the functions in Unicode::UCD that return block names (except one) return the old-style ones. That one function, "prop_value_aliases"() can be used to convert from old-style to new-style: my $new_style = prop_values_aliases("block", $old_style); http://www.nntp.perl.org/group/perl.unicode/2011/08/msg3334.html Wed, 17 Aug 2011 13:42:17 +0000 Re: RFC: API to access Unicode db files by Zefram Karl Williamson wrote: > prop_invmap("Numeric_Value", \@numerics_ranges, \@numerics_maps); Looks fine to me, except for the convention for returning the arrays. It would be neater to return a list of items rather than modify arrays in place: ($status, $ranges, $maps) = prop_invmap("Numeric_Value"); -zefram http://www.nntp.perl.org/group/perl.unicode/2011/07/msg3333.html Sun, 24 Jul 2011 20:26:21 +0000 RFC: API to access Unicode db files by Karl Williamson Some applications are finding it necessary to read in the Unicode files that mktables generates. For example, grepping through CPAN indicates that Text::Unicode::Equivalents reads Decomposition.pl. This, and most of the other generated files are marked for internal use only, because we wish to reserve the right to change them around, etc. But applications currently have no feasible alternative. Prior to 5.14, we delivered the full Unicode db files that the Unicode consortium publishes, and whose format is guaranteed not to change. But we dropped those files in 5.14 to save disk space. I'm proposing a new function Unicode::UCD::prop_invmap() to return the contents of those files in a Unicode-centric way, so that applications can use it and we can deprecate non-core use of our generated files. The function returns an inversion map, which is a data structure more used in the Unicode world than the Perl world. It consists of two parallel arrays. I suppose a more Perl-centric data structure would be an array of hashes, but the inversion map seems simpler to me to manipulate. (This function would be in addition to the previously rfc'd function Unicode::UCD::prop_invlist() which would return a list of all code points that match a property-value.) =pod =head2 prop_invmap C<prop_invmap> is used to get the complete mapping definition for the input property, in the form of an inversion map. An inversion map consists of two parallel arrays. One is an ordered list of code points that mark range beginnings, and the other gives the value that all code points in the corresponding range have. C<prop_invmap> is called with the name of the desired property, and references to the two arrays, which it fills. For example, prop_invmap("Numeric_Value", \@numerics_ranges, \@numerics_maps); will populate the arrays as shown below: @numerics_ranges @numerics_maps Note 0x00 "NaN" NaN stands for "Not a Number" 0x30 0 DIGIT 0 0x31 1 0x32 2 ... 0x37 7 0x38 8 0x39 9 DIGIT 9 0x3A "NaN" 0xB2 2 SUPERSCRIPT 2 0xB3 3 SUPERSCRIPT 2 0xB4 "NaN" 0xB9 1 SUPERSCRIPT 1 0xBA "NaN" 0xBC 0.25 VULGAR FRACTION 1/4 0xBD 0.5 VULGAR FRACTION 1/2 0xBE 0.75 VULGAR FRACTION 3/4 0xBF "NaN" 0x660 0 ARABIC-INDIC DIGIT ZERO ... ... 0x110000 undef The second line means that the value for the code point 0x30 (which is "DIGIT 0") is 0. The first line means that all code points in the range from 0x00 to 0x2F (which is 0x30 (from the second line) - 1) have the value "NaN". The final line means that the value for all code points above the legal Unicode maximum code point have the value C<undef> (not the string "u-n-d-e-f"). The arrays completely specify the mappings for all possible code points. The special string S<C<"E<lt>code pointE<gt>">> is used to specify that the value of a code point is itself. For example, the beginnings of the arrays for prop_invmap("Uppercase_Mapping", \@uppers_ranges, \@uppers_maps); look like this: @uppers_ranges @uppers_maps Note 0 "<code point>" 97 65 'a' maps to 'A' 98 66 'b' => 'B' 99 67 'c' => 'C' ... 120 88 'x' => 'X' 121 89 'y' => 'Y' 122 90 'z' => 'Z' 123 "<code point>" 181 924 MICRO SIGN => Greek Cap MU 182 "<code point>" 223 [ 83 83 ] SHARP S => 'SS' 224 192 The first line means that the uppercase of code point 0 is 0, of 1 is 1, ... of 96 is 96. Without the C<"E<lt>code_pointE<gt>"> notation, every code point would have to have an entry. This would mean that the arrays would each have more than a million entries to list just the legal Unicode code points! In some properties some code points map to a sequence of multiple code points. For those, the corresponding entries in the map array are not scalars, but references to anonymous arrays containing the ordered list of code points mapped to, as shown in the example above for 223. The "Name" property map includes entries such as CJK UNIFIED IDEOGRAPH-<code point> This means that the name for the code point is "CJK UNIFIED IDEOGRAPH-" with the code point (expressed in hexadecimal) appended to it. Also, the notation "E<lt>hangul syllableE<gt>" occurs in this property, meaning that the name is algorithmically calculated. These names can be generated via the function C<charnames::viacode>(). The "Decomposition_Mapping" property also uses "E<lt>hangul syllableE<gt>" for those code points whose decomposition is algorithmically calculated. These can be generated via the function C<Unicode::Normalize::NFD>(). This property contains many occurrences of code points whose mappings are ordered lists of other code points. The return value is C<undef> if the property is unknown; C<s> if all the elements of the map array are simple scalars; C<n> for the Name property, which has the complications described above; C<d> for the Decomposition_Mapping property (complications already described); otherwise C<c> if some of map array elements are S<C<"E<lt>code pointE<gt>">>; and C<l> if additionally some are lists of code points. A binary search can be used to quickly find a code point in the inversion list, and hence its corresponding mapping. =cut http://www.nntp.perl.org/group/perl.unicode/2011/07/msg3332.html Thu, 21 Jul 2011 08:04:27 +0000 Re: Encode question by Karl Williamson On 07/07/2011 01:17 AM, Dave Saunders wrote: > Dear Encode Developers, > > I am migrating a perl application from Solaris 2.10 to Linux Fedora Core > 14 (2.6.35.13-92.fc14.x86_64), which is running perl 5.12.3. The app > uses SDBM and I'm encountering a problem which looks related to the > Encode module (which is at 2.39). > > The error message is: > > > panic: sv_setpvn called with negative strlen at p.pl line 21. > > > and this URL led me to Encode: > > > https://rt.cpan.org/Public/Bug/Display.html?id=65541 > > > I ran the simple test at the bottom of the web page: > > > binmode STDOUT, ':encoding(cp1250)'; > print( ( "a" x 1023 ) . "\x{0378}" ); > > > and got the same output: > > > "\x{0340}" does not map to cp1250 at a.pl line 2. > panic: sv_setpvn called with negative strlen at a.pl line 2. > > > I looked at CPAN: > http://search.cpan.org/~dankogai/Encode-2.43/ > > > and read the Changes file, but don't see Bug 65541 discussed. > > > Has this problem been solved, or is it being worked on? > Is there a work-around for SDBM files (other than not using SDBM)? > > > Thanks very much for your help, > David Saunders > ITS/UVa > It happens in today's latest development version of Perl. It looks like an Encode bug, and you'd have to contact the maintainer, Dan, directly to find out its status. http://www.nntp.perl.org/group/perl.unicode/2011/07/msg3331.html Thu, 07 Jul 2011 10:19:05 +0000 Encode question by Dave Saunders Dear Encode Developers, I am migrating a perl application from Solaris 2.10 to Linux Fedora Core 14 (2.6.35.13-92.fc14.x86_64), which is running perl 5.12.3. The app uses SDBM and I'm encountering a problem which looks related to the Encode module (which is at 2.39). The error message is: panic: sv_setpvn called with negative strlen at p.pl line 21. and this URL led me to Encode: https://rt.cpan.org/Public/Bug/Display.html?id=65541 I ran the simple test at the bottom of the web page: binmode STDOUT, ':encoding(cp1250)'; print( ( "a" x 1023 ) . "\x{0378}" ); and got the same output: "\x{0340}" does not map to cp1250 at a.pl line 2. panic: sv_setpvn called with negative strlen at a.pl line 2. I looked at CPAN: http://search.cpan.org/~dankogai/Encode-2.43/ and read the Changes file, but don't see Bug 65541 discussed. Has this problem been solved, or is it being worked on? Is there a work-around for SDBM files (other than not using SDBM)? Thanks very much for your help, David Saunders ITS/UVa http://www.nntp.perl.org/group/perl.unicode/2011/07/msg3330.html Thu, 07 Jul 2011 00:31:52 +0000 Re: Need: list of Unicode characters that have canonical decompositions. by Karl Williamson On 07/01/2011 11:49 AM, Karl Williamson wrote: > On 07/01/2011 10:40 AM, BobH wrote: >> Karl Williamson wrote: >>>> >>> >>> I'm trying to think of a good name. Best so far is >>> UCD::get_prop_invlist() >> >> >> Hm, "get" normally isn't needed. >> >> How about something simpler such as UCD::charlist() >> >> Bob >> > > I think not having prop in the name is potentially misleading, and it > actually isn't a list of the chars. It's an inversion list that is > readily convertible into such a list. > I've mostly written and tested it. But here is my proposed API to see how people like it (or not); (I'm still open to a better name, but I do thing that the name needs to have the requirements I mentioned above): =pod =head2 prop_invlist C<prop_invlist> returns an inversion list (see below) that defines all the code points for the Unicode property given by the input parameter string: say join ", ", prop_invlist("Any"); 0, 1114112 An empty list is returned if the given property is unknown. L<perluniprops|perluniprops/Properties accessible through \p{} and \P{}> gives the list of properties that this function accepts, as well as all the possible forms for them. Note that many properties can be specified in a compound form, such as say join ", ", prop_invlist("Script=Shavian"); 66640, 66688 say join ", ", prop_invlist("ASCII_Hex_Digit=No"); 0, 48, 58, 65, 71, 97, 103 say join ", ", prop_invlist("ASCII_Hex_Digit=Yes"); 48, 58, 65, 71, 97, 103 Inversion lists are a compact way of specifying Unicode properties. The 0th item in the list is the lowest code point that has the property-value. The next item is the lowest code point after that one that does NOT have the property-value. And the next item after that is the lowest code point after that one that has the property-value, and so on. Put another way, each element in the list gives the beginning of a range that has the property-value (for even numbered elements), or doesn't have the property-value (for odd numbered elements). In the final example above, the first ASCII Hex digit is code point 48, the character "0", and all code points from it through 57 (a "9") are ASCII hex digits. Code points 58 through 64 aren't, but 65 (an "A") through 70 (an "F") are, as are 97 ("a") through 102 ("f"). 103 starts a range of code points that aren't ASCII hex digits. That range extends to infinity, which on your computer can be found in the variable C<$Unicode::UCD::MAX_CP>. It is a simple matter to expand out an inversion list to a full list of all code points that have the property-value: my @invlist = prop_invlist("My Property"); die "empty" unless @invlist; my @full_list; for (my $i = 0; $i < @invlist; $i += 2) { my $upper = ($i + 1) < @invlist ? $invlist[$i+1] - 1 # In range : $Unicode::UCD::MAX_CP; # To infinity. You may want # to stop earlier for my $j ($invlist[$i] .. $upper) { print $upper, ": ", $j, "\n"; push @full_list, $j; } } =cut http://www.nntp.perl.org/group/perl.unicode/2011/07/msg3329.html Wed, 06 Jul 2011 13:43:18 +0000 Re: Need: list of Unicode characters that have canonical decompositions. by Karl Williamson On 07/01/2011 10:40 AM, BobH wrote: > Karl Williamson wrote: >>> >> >> I'm trying to think of a good name. Best so far is >> UCD::get_prop_invlist() > > > Hm, "get" normally isn't needed. > > How about something simpler such as UCD::charlist() > > Bob > I think not having prop in the name is potentially misleading, and it actually isn't a list of the chars. It's an inversion list that is readily convertible into such a list. http://www.nntp.perl.org/group/perl.unicode/2011/07/msg3328.html Fri, 01 Jul 2011 10:50:24 +0000 Re: Need: list of Unicode characters that have canonical decompositions. by BobH Karl Williamson wrote: >> > > I'm trying to think of a good name. Best so far is > UCD::get_prop_invlist() Hm, "get" normally isn't needed. How about something simpler such as UCD::charlist() Bob http://www.nntp.perl.org/group/perl.unicode/2011/07/msg3327.html Fri, 01 Jul 2011 09:40:34 +0000 Re: Need: list of Unicode characters that have canonical decompositions. by Karl Williamson On 06/29/2011 09:06 AM, BobH wrote: > Karl Williamson wrote: > >> If I did this, I would be tempted to have it return an inversion >> list, instead of an array of every code point that matches the >> property. ... >> >> My question to you is would that be acceptable to you, do you think? >> I hate to return an enormous array by default when the application >> doesn't really need it. > > Yes, that kind of representation would be sufficient and reasonably > compact. > > Thanks. > > Bob > I'm trying to think of a good name. Best so far is UCD::get_prop_invlist() Any ideas http://www.nntp.perl.org/group/perl.unicode/2011/07/msg3326.html Fri, 01 Jul 2011 08:38:22 +0000 Re: Need: list of Unicode characters that have canonical decompositions. by BobH Karl Williamson wrote: > If I did this, I would be tempted to have it return an inversion > list, instead of an array of every code point that matches the > property. ... > > My question to you is would that be acceptable to you, do you think? > I hate to return an enormous array by default when the application > doesn't really need it. Yes, that kind of representation would be sufficient and reasonably compact. Thanks. Bob http://www.nntp.perl.org/group/perl.unicode/2011/06/msg3325.html Wed, 29 Jun 2011 08:06:21 +0000 Re: Need: list of Unicode characters that have canonical decompositions. by Karl Williamson On 06/27/2011 08:04 PM, BobH wrote: > Karl Williamson wrote: > > > I'm presuming you need this not for a one-time only thing, but to be > > able to run this program over and over. > > Yes -- this is for a module that will be usable in a number of > situations. See > http://search.cpan.org/~bhallissy/Text-Unicode-Equivalents-0.05/. > > The current implementation cheats by accessing unicore/Decomposition.pl > exactly the same way Unicode::UCD does. > > > You can always download UnicodeData.txt from the Unicode web site. > > Yes I can -- and certainly have done for my personal use. But including > that file (or some derivative) in a general purpose module would mean > that it wouldn't necessarily have the same Unicode version as the Perl > installation into which my module might be installed. And besides, the > information I need is already in the Perl core -- though supposedly not > usable. > > > In a regular expression, > > \p{Dt= can} (Decomposition_Type=Canonical) will match all characters > > that you want. > > Yes, I understand that I can test a character to see if it has a > particular decomposition, but I'm not sure I understand how to use a > regex to generate a complete list of characters with decompositions. > > > I'm thinking that 5.16 will have the stringification > > of that regex include the list you want, but not in 5.14, and > > stringification is not necessarily fixed either. > > > > I could easily write a new function for UCD that returns a list of > > all code points that have a given property. > > That is an interesting offer, and I think this should be given serious > consideration. I'm sure my little module isn't the only one that, as we > go into the future, would benefit from such a function. > > Thanks for your reply, Karl. > > Bob > If I did this, I would be tempted to have it return an inversion list, instead of an array of every code point that matches the property. Such an array could be potentially length 1,114,112. The largest possible inversion list is potentially half that, but the largest one that matches a Unicode property is around length 700, and yours would be somewhat over 200 entries. That is why inversion lists are often used for Unicode because they compactly represent the Unicode properties. An inversion list is an array. An example is: 5, 101, 116, 120, ... This represents 5..100, 116..119 ... The 0th element gives the first code point that is in the property; the next element gives the first code point after that one that's not in the property, and so forth. Each succeeding element marks the beginning of a range that is/isn't in the property, inverting the is/isnt each time. It is a simple matter to convert an inversion list into a true array or hash of every code point that matches. My question to you is would that be acceptable to you, do you think? I hate to return an enormous array by default when the application doesn't really need it. http://www.nntp.perl.org/group/perl.unicode/2011/06/msg3324.html Tue, 28 Jun 2011 10:31:51 +0000 Re: Need: list of Unicode characters that have canonical decompositions. by BobH Karl Williamson wrote: > I'm presuming you need this not for a one-time only thing, but to be > able to run this program over and over. Yes -- this is for a module that will be usable in a number of situations. See http://search.cpan.org/~bhallissy/Text-Unicode-Equivalents-0.05/. The current implementation cheats by accessing unicore/Decomposition.pl exactly the same way Unicode::UCD does. > You can always download UnicodeData.txt from the Unicode web site. Yes I can -- and certainly have done for my personal use. But including that file (or some derivative) in a general purpose module would mean that it wouldn't necessarily have the same Unicode version as the Perl installation into which my module might be installed. And besides, the information I need is already in the Perl core -- though supposedly not usable. > In a regular expression, > \p{Dt= can} (Decomposition_Type=Canonical) will match all characters > that you want. Yes, I understand that I can test a character to see if it has a particular decomposition, but I'm not sure I understand how to use a regex to generate a complete list of characters with decompositions. > I'm thinking that 5.16 will have the stringification > of that regex include the list you want, but not in 5.14, and > stringification is not necessarily fixed either. > > I could easily write a new function for UCD that returns a list of > all code points that have a given property. That is an interesting offer, and I think this should be given serious consideration. I'm sure my little module isn't the only one that, as we go into the future, would benefit from such a function. Thanks for your reply, Karl. Bob http://www.nntp.perl.org/group/perl.unicode/2011/06/msg3323.html Mon, 27 Jun 2011 19:04:19 +0000 Re: Need: list of Unicode characters that have canonical decompositions. by Karl Williamson On 06/27/2011 08:26 AM, BobH wrote: > A project I'm working on needs to build a list of all Unicode characters > that have canonical decompositions. The most efficient ways I can think > of to get such a list are from unicore/Decomposition.pl or by scanning > unicore/UnicodeData.txt. However: > > Re unicore/Decomposition.pl, the header of this says: > >> # !!!!!!! INTERNAL PERL USE ONLY !!!!!!! >> # This file is for internal use by the Perl program only. The format >> and even >> # the name or existence of this file are subject to change without >> notice. >> # Don't use it directly. > > Re unicore/UnicodeData.txt, I've recently posted a version of my module > that uses unicore/UnicodeData.txt to CPAN, and from Perl 5.14 testers > I've received only failure notices which indicate that the file cannot > be found :-( > > Unicode::UCD can tell me if a specific character has a decomposition, > but can't give me a list of characters that have decompositions. > > Any suggestions would be appreciated. > > Bob > I'm presuming you need this not for a one-time only thing, but to be able to run this program over and over. You can always download UnicodeData.txt from the Unicode web site. In a regular expression, \p{Dt= can} (Decomposition_Type=Canonical) will match all characters that you want. I'm thinking that 5.16 will have the stringification of that regex include the list you want, but not in 5.14, and stringification is not necessarily fixed either. I could easily write a new function for UCD that returns a list of all code points that have a given property. http://www.nntp.perl.org/group/perl.unicode/2011/06/msg3322.html Mon, 27 Jun 2011 13:02:09 +0000 Re: Need: list of Unicode characters that have canonical decompositions. by BobH BobH wrote: > Re unicore/UnicodeData.txt, I've recently posted a version of my module > that uses unicore/UnicodeData.txt to CPAN, and from Perl 5.14 testers > I've received only failure notices which indicate that the file cannot > be found :-( > Just installed ActivePerl 5.14 and, indeed, this file no longer exists -- guess that forces me to use unicore/Decomposition.pl in spite of its included warning. Bob http://www.nntp.perl.org/group/perl.unicode/2011/06/msg3321.html Mon, 27 Jun 2011 08:10:47 +0000 Need: list of Unicode characters that have canonical decompositions. by BobH A project I'm working on needs to build a list of all Unicode characters that have canonical decompositions. The most efficient ways I can think of to get such a list are from unicore/Decomposition.pl or by scanning unicore/UnicodeData.txt. However: Re unicore/Decomposition.pl, the header of this says: > # !!!!!!! INTERNAL PERL USE ONLY !!!!!!! > # This file is for internal use by the Perl program only. The format and even > # the name or existence of this file are subject to change without notice. > # Don't use it directly. Re unicore/UnicodeData.txt, I've recently posted a version of my module that uses unicore/UnicodeData.txt to CPAN, and from Perl 5.14 testers I've received only failure notices which indicate that the file cannot be found :-( Unicode::UCD can tell me if a specific character has a decomposition, but can't give me a list of characters that have decompositions. Any suggestions would be appreciated. Bob http://www.nntp.perl.org/group/perl.unicode/2011/06/msg3320.html Mon, 27 Jun 2011 07:27:06 +0000 Enumerating all canonically equivalent strings by BobH Does there exist a standard module or function that, given a Combining Character Sequence (or, more generally, an arbitrary Unicode text string), will generate a list of all canonically equivalent strings? For example, if given the character U+1EAD, I'd like to get back a list of all these canonically equivalent sequences: 0061 0302 0323 0061 0323 0302 00E2 0323 1EA1 0302 1EAD (I don't particularly care whether the interface is in terms of arrays of USVs or utf strings.) Some years ago I created such a module for my own use (I called it Unicode::MakeEquivalents), and am now wondering whether there exists a standard solution to this problem (so I can abandon my own stuff), or whether I should pursue adding this functionality to CPAN somewhere. Suggestions? Bob http://www.nntp.perl.org/group/perl.unicode/2011/06/msg3319.html Mon, 20 Jun 2011 16:22:12 +0000 Given 5.014 features, is encoding::warnings still needed? by Lars Dɪᴇᴄᴋᴏᴡ 迪拉斯 In <http://stackoverflow.com/q/6281049#comment-7334585>, tchrist asks: | Is `encoding::warnings` actually still needed given the `/dual` modifiers | and the `unicode_strings` feature? http://www.nntp.perl.org/group/perl.unicode/2011/06/msg3318.html Wed, 08 Jun 2011 12:57:46 +0000 Re: Unicode::Collate string replacements and case sensitivity by SADAHIRO Tomoyuki On Thu, 28 Apr 2011 10:06:58 -0700 (PDT) Frank MÃ¼ller <pottwal1@freenet.de> wrote: > dear all, > I'm trying to do some string replacements with Unicode::Collate which > usually work very well, but these replacements seem to be case > insensitive by default - how can I change this? look at this simple > example: > > my $myCollator = Unicode::Collate->new( normalization => undef, level > => 1 ); > my $str = "Camel camel donkey zebra came\x{301}l CAMEL horse > cAmEL..."; > $myCollator->gsubst($str, "camel", sub { "#$_[0]#" }); > > which makes the following replacements: > > #Camel# #camel# donkey zebra #camÃ©l# #CAMEL# horse #cAmEL#... > > what I would love to see is the following result: > > Camel #camel# donkey zebra #camÃ©l# CAMEL horse cAmEL... > > As there doesn't seem to be gsubst for case sensitive and gisubst for > case insensitive string replacements, what would a solution look like? > > Thanks a lot for any suggestions, > Frank As (level => 1) is not default, (level => 3) is also allowed for case sensitive matching. But UCA thinks accent difference (level 2) is more important than case difference (level 3), then camÃ©l won't match camel when (level => 3). level 1: camel matches camÃ©l and Camel. level 2: camel matches Camel but not camÃ©l. level 3: camel matches neither Camel nor camÃ©l. --Even at level 3, it isn't so strict: camel matches "c-a-m-e-l", "ca mel", etc. since punctuation difference is level 4. To make camel match camÃ©l but not Camel, other workwround is need. In next release, a new parameter (ignore_level2) will allow it. (However the behavior of ignore_level2 is quite different from so-called caseLevel in UCA etc.) Regards, SADAHIRO Tomoyuki http://www.nntp.perl.org/group/perl.unicode/2011/05/msg3317.html Thu, 05 May 2011 06:07:02 +0000 Unicode::Collate string replacements and case sensitivity by Frank Müller dear all, I'm trying to do some string replacements with Unicode::Collate which usually work very well, but these replacements seem to be case insensitive by default - how can I change this? look at this simple example: my $myCollator = Unicode::Collate->new( normalization => undef, level => 1 ); my $str = "Camel camel donkey zebra came\x{301}l CAMEL horse cAmEL..."; $myCollator->gsubst($str, "camel", sub { "#$_[0]#" }); which makes the following replacements: #Camel# #camel# donkey zebra #camél# #CAMEL# horse #cAmEL#... what I would love to see is the following result: Camel #camel# donkey zebra #camél# CAMEL horse cAmEL... As there doesn't seem to be gsubst for case sensitive and gisubst for case insensitive string replacements, what would a solution look like? Thanks a lot for any suggestions, Frank http://www.nntp.perl.org/group/perl.unicode/2011/04/msg3316.html Fri, 29 Apr 2011 02:21:42 +0000 Re: encoding(UTF16-LE) on Windows by Erland Sommarskog Michael Ludwig (milu71@gmx.de) writes: >> For instance, I use Windows exclusively, so Unicode in file names is >> no problem. > > Did a quick test: > > (v5.12.1) built for MSWin32-x86-multi-thread (so ActiveState) > > * aÃ¢â¬Â¦b.txt > * not correct > * doesn't have anything with "uni" or "utf" in "perl -V" OK, so the implementation would have to know that on this platform filenames are in UTF-16, on this it is UTF-8 and so on. Not that it is a terribly big deal. In the program where I want to support Unicode names, I've already written a module around Win32API::File, which permits to open a file in Windows, and the associate it with a file handle. -- Erland Sommarskog, Stockholm, esquel@sommarskog.se http://www.nntp.perl.org/group/perl.unicode/2011/02/msg3315.html Wed, 02 Feb 2011 01:06:21 +0000 Re: encoding(UTF16-LE) on Windows by Michael Ludwig Erland Sommarskog schrieb am 31.01.2011 um 23:42 (+0100): > Michael Ludwig (milu71@gmx.de) writes: > > Erland Sommarskog schrieb am 29.01.2011 um 14:02 (+0100): > > > >> Yes, there certainly seems to be some more stuff to do in the > >> Unicode support in Perl. For instance, support for Unicode > >> filenames in open or opendir. > > > > I think there is no portable answer here, as it depends on the > > filesystem's support for Unicode. > > Did I say it have to be portable? :-) No … but Perl did. :-) > For instance, I use Windows exclusively, so Unicode in file names is > no problem. Did a quick test: \,,,/ (o o) ------oOOo-(_)-oOOo------ use strict; use warnings; use utf8; my $fn = 'a…b.txt'; # mit Unicode-Zeichen open my $fh, '>:encoding(UTF-8)', $fn or die "open $fn: $!"; print $fh "$fn\n"; close $fh; ------------------------- v5.10.1 (*) built for i686-cygwin-thread-multi-64int * a…b.txt * correct (in Explorer, cmd.exe, MinTTY) * has: CYG17 utf8-paths (which might be responsible) (v5.12.1) built for MSWin32-x86-multi-thread (so ActiveState) * aâ€¦b.txt * not correct * doesn't have anything with "uni" or "utf" in "perl -V" -- Michael Ludwig http://www.nntp.perl.org/group/perl.unicode/2011/01/msg3314.html Mon, 31 Jan 2011 17:32:59 +0000 Re: encoding(UTF16-LE) on Windows by Erland Sommarskog Michael Ludwig (milu71@gmx.de) writes: > Erland Sommarskog schrieb am 29.01.2011 um 14:02 (+0100): > >> Yes, there certainly seems to be some more stuff to do in the Unicode >> support in Perl. For instance, support for Unicode filenames in open >> or opendir. > > I think there is no portable answer here, as it depends on the > filesystem's support for Unicode. Did I say it have to be portable? :-) Obviously, Unicode cannot happen on systems which do not support Unicode. For instance, I use Windows exclusively, so Unicode in file names is no problem. On the other hand, it's a dead case for system() and backticks as far as I can make out. (That is, I have not been able to run Unicode BAT files.) -- Erland Sommarskog, Stockholm, esquel@sommarskog.se http://www.nntp.perl.org/group/perl.unicode/2011/01/msg3313.html Mon, 31 Jan 2011 16:22:29 +0000