develooper Front page | perl.perl5.porters | Postings from January 2014

[perl #121106] run/locale.t fails all tests on Win32

Thread Previous
From:
bulk88
Date:
January 28, 2014 21:25
Subject:
[perl #121106] run/locale.t fails all tests on Win32
Message ID:
rt-4.0.18-12339-1390944341-1970.121106-75-0@perl.org
# New Ticket Created by  bulk88 
# Please include the string:  [perl #121106]
# in the subject line of all future correspondence about this issue. 
# <URL: https://rt.perl.org/Ticket/Display.html?id=121106 >


This is a bug report for perl from bulk88@hotmail.com,
generated with the help of perlbug 1.39 running under perl 5.19.7.


-----------------------------------------------------------------
[Please describe your issue here]

Some time in the last couple days this test on blead started failing on 
me. WinXP, 32 bit, VC 2003.

------------------------------------------------------------------
Test Summary Report
-------------------
run/locale.t                                                    (Wstat: 
0 Tests:
  16 Failed: 16)
   Failed tests:  1-16
Files=2402, Tests=694552, 2421 wallclock secs (77.34 usr +  4.25 sys = 
81.59 CPU
)
Result: FAIL
NMAKE : fatal error U1077: '..\perl.exe' : return code '0x10'
Stop.

C:\perl519\src\win32>
------------------------------------------------------------------

http://perl5.git.perl.org/perl.git/commit/d369fd5b026662cae636af9907037fdd15fc4adb 
maybe this is related?

-----------------------------------------------------------------
C:\perl519\src\t>..\perl  -I..\lib  harness run/locale.t
run/locale.t .. # Failed test 1 - no locales where LC_NUMERIC breaks at 
./test.p
l line 954
#      got undef
# expected ""
# PROG:
# for (qw(Afrikaans Albanian American Basque Bulgarian C Croatian Czech 
Danish D
utch English Estonian Finnish Galician German Greek Hungarian Indonesian 
Italian
  Latvian Lithuanian Norwegian Polish Russian Serbian Slovak Slovenian 
Swedish Tu
rkish Welsh af da en fi fy ga hu it la mo no no\@nynorsk ro ru sl)) {
#     use POSIX qw(locale_h);
#     use locale;
#     setlocale(LC_NUMERIC, "$_") or next;
#     my $s = sprintf "%g %g", 3.1, 3.1;
#     next if $s eq '3.1 3.1' || $s =~ /^(3.+1) \1$/;
#     print "$_ $s\n";
# }
# STATUS: -1
# Failed test 2 - LC_NUMERIC without environment nor setlocale() has no 
effect i
n any locale at ./test.pl line 954
#      got undef
run/locale.t .. 1/16 # expected ""
# PROG:
# for (qw(Afrikaans Albanian American Basque Bulgarian C Croatian Czech 
Danish D
utch English Estonian Finnish Galician German Greek Hungarian Indonesian 
Italian
  Latvian Lithuanian Norwegian Polish Russian Serbian Slovak Slovenian 
Swedish Tu
rkish Welsh af da en fi fy ga hu it la mo no no\@nynorsk ro ru sl)) {
#         use POSIX qw(locale_h);
#         use locale;
#         my $in = 4.2;
#         my $s = sprintf "%g", $in; # avoid any constant folding bugs
#         next if $s eq "4.2";
#         print "$_ $s\n";
#     }
# STATUS: -1
# Failed test 3 - format() does not look at LC_NUMERIC without 'use 
locale' at .
/test.pl line 954
#      got undef
# expected "4.2"
# PROG:
# format STDOUT =
# @.#
# 4.179
# .
# write;
# STATUS: -1
# Failed test 4 - format() looks at LC_NUMERIC with 'use locale' at 
./test.pl li
ne 954
#      got undef
# expected "4,2"
# PROG:
# use locale;
# format STDOUT =
# @.#
# 4.179
# .
# write;
# STATUS: -1
# Failed test 5 - format() looks at LC_NUMERIC with 'use locale 
":not_characters
"' at ./test.pl line 954
#      got undef
# expected "4,2"
# PROG:
# use locale ":not_characters";
# format STDOUT =
# @.#
# 4.179
# .
# write;
# STATUS: -1
# Failed test 6 - too late to look at the locale at write() time at 
./test.pl li
ne 954
#      got undef
# expected "4.2"
# PROG:
# format STDOUT =
# @.#
# 4.179
# .
# { require locale; import locale; write; }
# STATUS: -1
# Failed test 7 - too late to ignore the locale at write() time at 
./test.pl lin
e 954
#      got undef
# expected "4,2"
# PROG:
# use locale;
# format STDOUT =
# @.#
# 4.179
# .
# { no locale; write; }
# STATUS: -1
# Failed test 8 - version does not clobber version at ./test.pl line 954
#      got undef
# expected ""
# PROG:
#             BEGIN {
#                 if("define") {
#                     require locale; import locale;
#                 }
#             }
#             use POSIX;
#             my $i = 0.123;
#             POSIX::setlocale(POSIX::LC_NUMERIC(),"Albanian");
#             $a = sprintf("%.2f", $i);
#             require version;
#             $b = sprintf("%.2f", $i);
#             print ".$a $b" unless $a eq $b
# STATUS: -1
# Failed test 9 - version does not clobber version (via eval) at 
./test.pl line
954
#      got undef
# expected ""
# PROG:
#             use locale;
#             use POSIX;
#             my $i = 0.123;
#             POSIX::setlocale(POSIX::LC_NUMERIC(),"Albanian");
#             $a = sprintf("%.2f", $i);
#             eval "use v5.0.0";
#             $b = sprintf("%.2f", $i);
#             print "$a $b" unless $a eq $b
# STATUS: -1
# Failed test 10 - sprintf() and printf() look at LC_NUMERIC regardless 
of const
ant folding at ./test.pl line 954
#      got undef
# expected "4,2 4,2 4,2 4,2 "
# PROG:
#         use locale;
#           use POSIX qw(locale_h);
#           setlocale(LC_NUMERIC, "");
#           my $in = 4.2;
#           printf("%g %g %s %s ", $in, 4.2, sprintf("%g", $in), 
sprintf("%g", 4
.2));
# STATUS: -1
# Failed test 11 - No compile error on v-strings when setting the locale 
to non-
dot radix at compile time when default environment has non-dot radix at 
./test.p
l line 954
#      got undef
# expected "C"
# PROG:
#           use POSIX qw(locale_h);
#
#             BEGIN { setlocale(LC_NUMERIC, "Albanian"); };
#             setlocale(LC_ALL, "C");
#             use 5.008;
#             print setlocale(LC_NUMERIC);
# STATUS: -1
# Failed test 12 - Passed in LC_NUMERIC is valid at compilation time at 
./test.p
l line 954
#      got undef
# expected "Albanian"
# PROG:
#           use POSIX qw(locale_h);
#
#             BEGIN { print setlocale(LC_NUMERIC), "
# "; };
# STATUS: -1
# Failed test 13 - Radix print properly in locale scope, and without at 
./test.p
l line 954
#      got undef
# expected "1,5\n1.5"
# PROG:
#             my $i = 1.5;
#             {
#                 use locale;
#                 use POSIX;
#                 POSIX::setlocale(POSIX::LC_NUMERIC(),"Albanian");
#                 print $i, "
# ";
#             }
#             print $i, "
# ";
# STATUS: -1
# Failed test 14 - Can do math when radix is a comma at ./test.pl line 954
#      got undef
# expected "1,5\n2,5"
# PROG:
#             my $i = 1.5;   # Should be exactly representable as a base 2
#                             # fraction, so can use 'eq' below
#             use locale;
#             use POSIX;
#             POSIX::setlocale(POSIX::LC_NUMERIC(),"Albanian");
#             print $i, "
# ";
#             $i += 1;
#             print $i, "
# ";
# STATUS: -1
# Failed test 15 - POSIX::strtod() uses underlying locale at ./test.pl 
line 954
#      got undef
# expected "1.5"
# PROG:
#                 use POSIX;
#                 POSIX::setlocale(POSIX::LC_NUMERIC(),"Albanian");
#                 my $one_point_5 = POSIX::strtod("1,5");
#                 $one_point_5 =~ s/0+$//;  # Remove any trailing zeros
#                 print $one_point_5, "
# ";
# STATUS: -1
# Failed test 16 - /il matching of [bracketed] doesn't skip POSIX class 
if fails
  individ char at ./test.pl line 954
#      got undef
# expected "1"
# PROG:
#                 use locale;
#                 use POSIX;
#                 POSIX::setlocale(POSIX::LC_CTYPE(),"C");
#                 print "h" =~ /[g\w]/i || 0;
#                 print "\n";
# STATUS: -1
run/locale.t .. Failed 16/16 subtests

Test Summary Report
-------------------
run/locale.t (Wstat: 0 Tests: 16 Failed: 16)
   Failed tests:  1-16
Files=1, Tests=16,  1 wallclock secs ( 0.03 usr +  0.00 sys =  0.03 CPU)
Result: FAIL

C:\perl519\src\t>
-----------------------------------------------------------------
C:\perl519\src\t>git log -n 4
WARNING: terminal is not fully functional
commit c02d66974dbe736ac96e0ee464b2ee7948935ffe
Author: Daniel Dragan <bulk88@hotmail.com>
Date:   Fri Jan 10 16:32:37 2014 -0500

     b88 bld

commit fb29cc72fe9ef39100123477fffae4fad34866b7
Author: Karl Williamson <public@khwilliamson.com>
Date:   Mon Jan 27 22:42:25 2014 -0700

     regcomp.c: Change a variable and flag bit names

     The meaning of these was expanded two commits ago, so update the 
name to
     reflect this, to prevent future confusion

commit baa60164aa329dbdc2e418a7743d51a081db237e
Author: Karl Williamson <public@khwilliamson.com>
Date:   Mon Jan 27 22:30:29 2014 -0700

     White-space, comments only

     This mostly indents and outdents base on blocks added or removed by the
     previous commit.  But there are a few comment changes and vertical
     alignment of macro backslash continuation characters, and other
     white-space changes
:
commit 31f05a37c4e9c37a7263491f2fc0237d836e1a80
Author: Karl Williamson <public@khwilliamson.com>
Date:   Mon Jan 27 15:35:00 2014 -0700

     Work properly under UTF-8 LC_CTYPE locales

     This large (sorry, I couldn't figure out how to meaningfully split it
     up) commit causes Perl to fully support LC_CTYPE operations (case
     changing, character classification) in UTF-8 locales.

     As a side effect it resolves [perl #56820].

     The basics are easy, but there were a lot of details, and one
     troublesome edge case discussed below.

     What essentially happens is that when the locale is changed to a UTF-8
     one, a global variable is set TRUE (FALSE when changed to a non-UTF-8
     locale).  Within the scope of 'use locale', this variable is checked,
     and if TRUE, the code that Perl uses for non-locale behavior is used
     instead of the code for locale behavior.  Since Perl's internal
     representation is UTF-8, we get UTF-8 behavior for a UTF-8 locale.

     More work had to be done for regular expressions.  There are three
     cases.

     1) The character classes \w, [[:punct:]] needed no extra work, as
     the changes fall out from the base work.

     2) Strings that are to be matched case-insensitively.  These form
     EXACTFL regops (nodes).  Notice that if such a string contains only
     characters above-Latin1 that match only themselves, that the node 
can be
     downgraded to an EXACT-only node, which presents better optimization
     possibilities, as we now have a fixed string known at compile time 
to be
     required to be in the target string to match.  Similarly if all
     characters in the string match only other above-Latin1 characters
     case-insensitively, the node can be downgraded to a regular EXACTFU 
node
     (match, folding, using Unicode, not locale, rules).  The code changes
     for this could be done without accepting UTF-8 locales fully, but there
     were edge cases which needed to be handled differently if I stopped
     there, so I continued on.

     In an EXACTFL node, all such characters are now folded at compile time
     (just as before this commit), while the other characters whose 
folds are
     locale-dependent are left unfolded.  This means that they have to be
     folded at execution time based on the locale in effect at the moment.
     Again, this isn't a change from before.  The difference is that now 
some
     of the folds that need to be done at execution time (in regexec) are
     potentially multi-char.  Some of the code in regexec was trivial to
     extend to account for this because of existing infrastructure, but the
     part dealing with regex quantifiers, had to have more work.

     Also the code that joins EXACTish nodes together had to be expanded to
     account for the possibility of multi-character folds within locale
     handling.  This was fairly easy, because it already has infrastructure
     to handle these under somewhat different circumstances.

     3) In bracketed character classes, represented by ANYOF nodes, a new
     inversion list was created giving the characters that should be matched
     by this node when the runtime locale is UTF-8.  The list is ignored
     except under that circumstance.  To do this, I created a new ANYOF type
     which has an extra SV for the inversion list.

     The edge case that caused the most difficulty is folding involving the
     MICRO SIGN, U+00B5.  It folds to the GREEK SMALL LETTER MU, as does the
     GREEK CAPITAL LETTER MU.  The MICRO SIGN is the only 0-255 range
     character that folds to outside that range.  The issue is that it
     doesn't naturally fall out that it will match the CAP MU.  If we 
let the
     CAP MU fold to the samll mu at compile time (which it can because both
     are above-Latin1 and so the fold is the same no matter what locale 
is in
     effect), it could appear that the regnode can be downgraded away from
     EXACTFL to EXACTFU, but doing so would cause the MICRO SIGN to not case
     insensitvely match the CAP MU.  This could be special cased in regcomp
     and regexec, but I wanted to avoid that.  Instead the mktables tables
     are set up to include the CAP MU as a character whose presence forbids
     the downgrading, so the special casing is in mktables, and not in the C
     code.
(END)
C:\perl519\src\t>
---------------------------------------------------------------------

I tried stepping locale.t but fresh_perl_is() blocked my debugging 
efforts a bit.

---------------------------------------------------------------------
my ($base, $different, $comma, $difference);
for ("C", @locales) { # prefer C for the base if available
     BEGIN {
         if($Config{d_setlocale}) {
             require locale; import locale;
         }
     }
     setlocale(LC_NUMERIC, $_) or next;
     my $in = 4.2; # avoid any constant folding bugs
     if ((my $s = sprintf("%g", $in)) eq "4.2")  {
	$base ||= $_;
     } else {
	$different ||= $_;
	$difference ||= $s;
         $comma ||= $_ if localeconv()->{decimal_point} eq ',';
     }

     last if $base && $different && $comma;
}

---------------------------------------------------------------------
sprintf did use "," instead of "." occasionally.



here is the 1st fresh_perl_is,
------------------------------------------------------------------------
1..16
not ok 1 - no locales where LC_NUMERIC breaks
# Failed test 1 - no locales where LC_NUMERIC breaks at ./test.pl line 954
#      got undef
# expected ""
# PROG:
# for (qw(Afrikaans Albanian American Basque Bulgarian C Croatian Czech 
Danish Dutch English Estonian Finnish Galician German Greek Hungarian 
Indonesian Italian Latvian Lithuanian Norwegian Polish Russian Serbian 
Slovak Slovenian Swedish Turkish Welsh af da en fi fy ga hu it la mo no 
no\@nynorsk ro ru sl)) {
#     use POSIX qw(locale_h);
#     use locale;
#     setlocale(LC_NUMERIC, "$_") or next;
#     my $s = sprintf "%g %g", 3.1, 3.1;
#     next if $s eq '3.1 3.1' || $s =~ /^(3.+1) \1$/;
#     print "$_ $s\n";
# }
# STATUS: -1
------------------------------------------------------------------------

I saved the tmp file and ran it manually,
------------------------------------------------------------------------
C:\perl519\src\t>..\perl -I..\lib  test1.pl

C:\perl519\src\t>
------------------------------------------------------------------------

next I tried a different test,
------------------------------------------------------------------------
not ok 15 - POSIX::strtod() uses underlying locale
# Failed test 15 - POSIX::strtod() uses underlying locale at ./test.pl 
line 954
#      got undef
# expected "1.5"
# PROG:
#                 use POSIX;
#                 POSIX::setlocale(POSIX::LC_NUMERIC(),"Albanian");
#                 my $one_point_5 = POSIX::strtod("1,5");
#                 $one_point_5 =~ s/0+$//;  # Remove any trailing zeros
#                 print $one_point_5, "
# ";
-------------------------------------------------------------------------
C:\perl519\src\t>..\perl -I..\lib  test1.pl 
         1.5 
 
                           C:\perl519\src\t> 

-------------------------------------------------------------------------

stepping t/test.pl in runperl(), $runperl is "C:\perl519\src\perl.exe 
"-I../lib" "tmp7516A" 2>&1", no outer quotes, result of backticks is 
undef or empty string according to my watch window. Cwd of perl process 
is "C:\perl519\src\t\" when it does backticks. $? after backticks is -1.

[Please do not change anything below this line]
-----------------------------------------------------------------
---
Flags:
         category=core
         severity=medium
---
Site configuration information for perl 5.19.7:

Configured by Owner at Thu Nov 28 02:32:44 2013.

Summary of my perl5 (revision 5 version 19 subversion 7) configuration:
       Derived from: 8f47723e28b75530b743941cdd8b07f849ec48e2
       Ancestor: 1061065f7a09399eefb50e9a035502621722bcc0
       Platform:
         osname=MSWin32, osvers=5.1, archname=MSWin32-x86-multi-thread
         uname=''
         config_args='undef'
         hint=recommended, useposix=true, d_sigaction=undef
         useithreads=define, usemultiplicity=define
         useperlio=define, d_sfio=undef, uselargefiles=define, 
usesocks=undef
         use64bitint=undef, use64bitall=undef, uselongdouble=undef
         usemymalloc=n, bincompat5005=undef
       Compiler:
         cc='cl', ccflags ='-nologo -GF -W3 -O1 -MD -Zi -DNDEBUG -G7 -GL
-DWIN32 -D_CONSOLE -DNO_STRICT  -DPERL_TEXTMODE_SCRIPTS
-DPERL_HASH_FUNC_ONE_AT_A_TIME -DPERL_IMPLICIT_CONTEXT
-DPERL_IMPLICIT_SYS -DUSE_PERLIO -D_USE_32BIT_TIME_T',
         optimize='-O1 -MD -Zi -DNDEBUG -G7 -GL',
         cppflags='-DWIN32'
         ccversion='13.10.6030', gccversion='', gccosandvers=''
         intsize=4, longsize=4, ptrsize=4, doublesize=8, byteorder=1234
         d_longlong=undef, longlongsize=8, d_longdbl=define, longdblsize=8
         ivtype='long', ivsize=4, nvtype='double', nvsize=8, 
Off_t='__int64',
lseeksize=8
         alignbytes=8, prototype=define
       Linker and Libraries:
         ld='link', ldflags ='-nologo -nodefaultlib -debug -opt:ref,icf
-ltcg  -libpath:"c:\perl519\lib\CORE"  -machine:x86'
         libpth="C:\Program Files\Microsoft Visual Studio .NET 2003\VC7\lib"
         libs=oldnames.lib kernel32.lib user32.lib gdi32.lib winspool.lib
comdlg32.lib advapi32.lib shell32.lib ole32.lib oleaut32.lib
netapi32.lib uuid.lib ws2_32.lib mpr.lib winmm.lib  version.lib
odbc32.lib odbccp32.lib comctl32.lib msvcrt.lib
         perllibs=oldnames.lib kernel32.lib user32.lib gdi32.lib
winspool.lib  comdlg32.lib advapi32.lib shell32.lib ole32.lib
oleaut32.lib  netapi32.lib uuid.lib ws2_32.lib mpr.lib winmm.lib
version.lib odbc32.lib odbccp32.lib comctl32.lib msvcrt.lib
         libc=msvcrt.lib, so=dll, useshrplib=true, libperl=perl519.lib
         gnulibc_version=''
       Dynamic Linking:
         dlsrc=dl_win32.xs, dlext=dll, d_dlsymun=undef, ccdlflags=' '
         cccdlflags=' ', lddlflags='-dll -nologo -nodefaultlib -debug
-opt:ref,icf -ltcg  -libpath:"c:\perl519\lib\CORE"  -machine:x86'

Locally applied patches:
         uncommitted-changes
         8f47723e28b75530b743941cdd8b07f849ec48e2

---
@INC for perl 5.19.7:
         C:/perl519/site/lib
         C:/perl519/lib
         .

---
Environment for perl 5.19.7:
         HOME (unset)
         LANG (unset)
         LANGUAGE (unset)
         LD_LIBRARY_PATH (unset)
         LOGDIR (unset)
         PATH=C:\perl519\bin;C:\Program Files\Microsoft Visual Studio .NET
2003\Common7\IDE;C:\Program Files\Microsoft Visual Studio .NET
2003\VC7\BIN;C:\Program Files\Microsoft Visual Studio .NET
2003\Common7\Tools;C:\Program Files\Microsoft Visual Studio .NET
2003\Common7\Tools\bin\prerelease;C:\WINDOWS\system32;C:\WINDOWS;C:\WINDOWS\system32\wbem;
         PERL_BADLANG (unset)
         SHELL (unset)







Thread Previous


nntp.perl.org: Perl Programming lists via nntp and http.
Comments to Ask Bjørn Hansen at ask@perl.org | Group listing | About