Front page | perl.perl5.porters |
Postings from October 2013
[perl #120330] [PATCH] parallel testing on Win32, now with working prototype, needs work
Thread Next
From:
bulk88
Date:
October 24, 2013 00:35
Subject:
[perl #120330] [PATCH] parallel testing on Win32, now with working prototype, needs work
Message ID:
rt-3.6.HEAD-26210-1382574923-1540.120330-75-0@perl.org
# New Ticket Created by bulk88
# Please include the string: [perl #120330]
# in the subject line of all future correspondence about this issue.
# <URL: https://rt.perl.org:443/rt3/Ticket/Display.html?id=120330 >
This is a bug report for perl from bulk88@hotmail.com,
generated with the help of perlbug 1.39 running under perl 5.19.6.
-----------------------------------------------------------------
[Please describe your issue here]
Reposting
http://www.nntp.perl.org/group/perl.perl5.porters/2013/10/msg208739.html
to RT so it doesn't get lost.
Earlier post (not to P5P) I made about the idea that got no response
http://www.nntp.perl.org/group/perl.qa/2013/07/msg13473.html
Original P5P post below.
__________________________________________________________
I decided to get parallel testing working on Win32 Perl.It has never
worked before because Win32 C select is sockets only (its exported by
ws2_32.dll, winsock library). Since TAP::Harness suite is an OOP for the
sake of OOP design wise, I couldn't figure out how to modify it or
plugin it. My next idea was to fork and rewrite IO::Select to work. The
standard stream kernel handles that the child perl writes to and the
parent perl (./t/harness) reads are "Named Pipe Filing System" (an NT
Kernel driver called npfs.sys) handles. Someone once tried to get
socketpair to be the standard stream handles at
http://www.nntp.perl.org/group/perl.perl5.porters/2009/09/msg150546.html
so Win32 C select would recognize the handles but it didn't work and the
discussion died out.
The commit disabling use of IO::Select in TAP::Parser::Multiplexer is
https://github.com/Perl-Toolchain-Gang/Test-Harness/commit/6d27f808824c3592e7089e7438f1ca18c860d307
and
https://github.com/Perl-Toolchain-Gang/Test-Harness/commit/ccffd0283e1820165ec307ffbbc4d3ee16fca6b3
.
NPFS I think is capable of supplying all information/APIs necessary to
implement posix select, but not through the Win32 API, only NT Native
API. FSCTL_PIPE_ASSIGN_EVENT is used to assign a kernel Event object
that is tripped whenever the other end of the pipe interacts with pipe,
you also assign a opaque pointer of your choice you will get back later.
FSCTL_PIPE_QUERY_EVENT takes an Event Object, and writes a log of all
events (reads, writes, and state changes (disconnect, accept, etc)) that
happened to all pipes that that Event Object was registered with. You
also get back your opaque pointer that you set hopefully per pipe. I
didn't look at the implementation of Interix/SFU to see if their posix
select uses these 2 ioctls or not (not meaning Interix implements its
own pipe API using IPC back to its interix service process).
For my first try, I decided to just do the bare minimum for harness to
be parallel, and do it in a Win32-only way, no Native API. Some select
for Win32 implementations I saw used PeekNamedPipe which gives the
number of bytes in the read buffer in the pipe instance in the kernel
driver and polling through the pipes with a sleep timer after going
through each set of handles to get readability status. For writeability,
they had to use Native API to poll the write buffer max size and current
write buffer fill. I won't describe how they got sockets to also work
with their select for all handles.
I dont see myself implementing writability or exceptions parts of select
ATM. harness never uses them. I might rewrite this to use Native API
described above and WaitForMultipleObjects to avoid wasting CPU on polling.
So I am attaching 2 files. 1 is a patch to TAP::Parser::Multiplexer to
use my IO::Select fork. The other is the IO::Select fork that I've
temporarily named IO::Select::Win32Pipes.
The code *barely* works on Win32 with TEST_JOBS=8 and I get 8 advancing
test counters on the bottom of the console window.
The code has a number of problems I dont know what to do about and I
need help on.
TAP::Parser::Multiplexer and TAP::Parser::Iterator::Process may have
some issues/bugs that need to be fixed that I can't figure out how to debug.
To get this into perl git, is another problem. It uses use Win32::API
(not core) right now and is mostly Pure Perl. For clarity of the error
numbers it tests, I'll probably add Win32::WinError to the mostly Pure
Perl implementation. I could also rewrite the Win32Pipes implementation
of select in XS. I think TAP:: and harness are pure perl ATM, I'm not
sure if they ever load an XS module right now (didn't research). If they
don't use any XS, then would it be okay to make them load XS or not?
Would there be any issues with running harness on miniperl and static perl?
I've also found out, very very very often (if I turn on the reporting,
the console will flooded with warnings), can_read() is being called on
pipes that have "ERROR_BROKEN_PIPE" or in POSIX EPIPE. I decided as a
wild ass guess to convert ERROR_BROKEN_PIPE to mean readable according
to this terse post
http://osdir.com/ml/bug-gnulib-gnu/2010-08/msg00141.html . This got
harness running in parallel finally.
Another problem is, TAP::Parser::Multiplexer is INCAPABLE of processing
ANY FAILURE of select from IO::Select. If Win32Pipes select returns -1,
it goes up the callstack and I get
__________________________________________________________________
Can't use an undefined value as an ARRAY reference at
../lib/TAP/Parser/Multiple
xer.pm line 149, <GEN0> line 1.
_________________________________________________________________
line 149 is
__________________________________________________________________
sub _iter {
my $self = shift;
my $sel = $self->{select};
my $avid = $self->{avid};
my @ready = ();
return sub {
# Drain all the non-selectable parsers first
if (@$avid) {
my ( $parser, $stash ) = @{ $avid->[0] };
my $result = $parser->next;
shift @$avid unless defined $result;
return ( $parser, $stash, $result );
}
unless (@ready) {
return unless $sel->count;
@ready = $sel->can_read;
}
my ( $h, $parser, $stash, @handles ) = @{ shift @ready
};<<<<<<<<<<<<<<<<<<<<
my $result = $parser->next;
unless ( defined $result ) {
$sel->remove(@handles);
$self->{count}--;
# Force another can_read - we may now have removed a handle
# thought to have been ready.
@ready = ();
}
return ( $parser, $stash, $result );
};
}
________________________________________________________________
IDK what to do about this.
Also TAP is constantly doing selects on closed file descriptors.
_______________________________________________________________
sub _fileno
{
my($self, $f) = @_;
return unless defined $f;
$f = $f->[0] if ref($f) eq 'ARRAY';
my $fn = ($f =~ /^\d+$/) ? $f : fileno($f);
#validate that the handle is a pipe
if(defined $fn) {
my ($hnd, $ret) = Win32API::File::FdGetOsFHandle($fn);
if($hnd != Win32API::File::INVALID_HANDLE_VALUE()) {
my $err;
$ret = Win32API::File::GetFileType($hnd);
$err = Win32::GetLastError();
if($ret == Win32API::File::FILE_TYPE_UNKNOWN && $err != 0)
#{ return undef;}
{ die "GetFileType failed"}
if($ret != Win32API::File::FILE_TYPE_PIPE) {
die "not a pipe";
}
$ret = GetNamedPipeInfo($hnd, undef, undef, undef, undef);
if(!$ret) {
die "not a pipe GNPI";
}
#else {
# die "is a pipe";
#}
}
else { warn "no handle $fn";
}
}
return $fn;
}
______________________________________________________________
______________________________________________________________
no handle 8 at C:\perl519\site\lib/IO/Select/Win32Pipes.pm line 170.
Use of uninitialized value $f in numeric eq (==) at
C:\perl519\site\lib/IO/Selec
t/Win32Pipes.pm line 202.
Use of uninitialized value $f in numeric eq (==) at
C:\perl519\site\lib/IO/Selec
t/Win32Pipes.pm line 202.
Use of uninitialized value $f in numeric eq (==) at
C:\perl519\site\lib/IO/Selec
t/Win32Pipes.pm line 202.
Use of uninitialized value $f in numeric eq (==) at
C:\perl519\site\lib/IO/Selec
t/Win32Pipes.pm line 202.
Use of uninitialized value $f in numeric eq (==) at
C:\perl519\site\lib/IO/Selec
t/Win32Pipes.pm line 202.
Use of uninitialized value $f in numeric eq (==) at
C:\perl519\site\lib/IO/Selec
t/Win32Pipes.pm line 202.
Use of uninitialized value $f in numeric eq (==) at
C:\perl519\site\lib/IO/Selec
t/Win32Pipes.pm line 202.
______________________________________________________________
C:\Documents and Settings\Owner\Desktop\cpan libs\p519\perl\t>perl
-I../lib harn
ess
No saved state, selection will be empty
base/cond.t ....................................................... 1/4
no handl
e 3 at C:\perl519\site\lib/IO/Select/Win32Pipes.pm line 170.
base/cond.t ....................................................... ok
base/if.t ......................................................... 1/2
no handl
e 3 at C:\perl519\site\lib/IO/Select/Win32Pipes.pm line 170.
base/if.t ......................................................... ok
base/lex.t ........................................................ 1/91
no hand
le 3 at C:\perl519\site\lib/IO/Select/Win32Pipes.pm line 170.
base/lex.t ........................................................ ok
base/num.t ........................................................ 1/53
no hand
le 3 at C:\perl519\site\lib/IO/Select/Win32Pipes.pm line 170.
base/num.t ........................................................ ok
base/pat.t ........................................................ 1/2
no handl
e 3 at C:\perl519\site\lib/IO/Select/Win32Pipes.pm line 170.
base/pat.t ........................................................ ok
Terminating on signal SIGINT(2)
Terminating on signal SIGINT(2)
_______________________________________________________________
I can't figure out what is causing all the "Use of uninitialized value
$f in numeric eq (==)" or what to do about it
_______________________________________________________________
sub _update
{
my $vec = shift;
my $add = shift eq 'add';
my $bits = $vec->[VEC_BITS];
$bits = '' unless defined $bits;
my $count = 0;
my $f;
foreach $f (@_)
{
my $fn = $vec->_fileno($f);
if ($add) {
next unless defined $fn;
my $i = $fn + FIRST_FD;
if (defined $vec->[$i]) {
$vec->[$i] = $f; # if array rest might be different, so we update
next;
}
$vec->[FD_COUNT]++;
vec($bits, $fn, 1) = 1;
$vec->[$i] = $f;
} else { # remove
if ( ! defined $fn ) { # remove if fileno undef'd
defined($_) && $_ == $f and do { $vec->[FD_COUNT]--; $_ = undef; }
for @{$vec}[FIRST_FD .. $#$vec];
###################<<<<<<<<<<<<<<<<< line 202
next;
}
______________________________________________________________
I also attached a screenshot of console window during parallel testing.
IO::Select::Win32Pipes needs a new name also. I dont have the tuits to
ever make it a general purpose select/async IO module. I see its future
being only for TAP parallel testing.
Also the fork of IO::Select isn't the most efficient design. I kept
IO::Select almost intact except for fileno method tweaks, since that
method is used to validate a FD during an add procedure, and renaming
all calls from (CORE::) select to _select. In theory the select method
could do a caller or blessed check on $_[0] to see whether its being
called as a use subs override or as a public API of IO::Select method.
The other choice is to integrate the Win32Pipes code into the official
IO::Select. I dont understand the internal design of IO::Select enough
to really do anything with it. Its uses @rrays with constant subs for
indexes instead of %ashes for what I guess is efficiency. C inspired
code :-)
Another problem is, Multiplexer/TAP has a problem servicing the child
processes. IDK if its my code or TAP, but what happens is, TAP gets
stuck on reading a rapidly ok()ing child process, and it will stay on
that rapidly ok() process until it runs dry or ends. Meanwhile other
child processes fill their pipe write buffers and C write() blocks for
10 to 30 seconds with zero CPU used and zero context switches in Process
Monitor (I took a C debugger to one of these frozen/blocked childs and
it was doing a C/PerlIO write to stdout).
Need help. Lots of it.
[Please do not change anything below this line]
-----------------------------------------------------------------
---
Flags:
category=core
severity=low
---
Site configuration information for perl 5.19.6:
Configured by Owner at Wed Oct 23 00:23:23 2013.
Summary of my perl5 (revision 5 version 19 subversion 6) configuration:
Derived from:
Platform:
osname=MSWin32, osvers=5.1, archname=MSWin32-x86-multi-thread
uname=''
config_args='undef'
hint=recommended, useposix=true, d_sigaction=undef
useithreads=define, usemultiplicity=define
useperlio=define, d_sfio=undef, uselargefiles=define, usesocks=undef
use64bitint=undef, use64bitall=undef, uselongdouble=undef
usemymalloc=n, bincompat5005=undef
Compiler:
cc='cl', ccflags ='-nologo -GF -W3 -O1 -MD -Zi -DNDEBUG -DWIN32
-D_CONSOLE -DNO_STRICT -DPERL_TEXTMODE_SCRIPTS -DPERL_IMPLICIT_CONTEXT
-DPERL_IMPLICIT_SYS -DUSE_PERLIO -D_USE_32BIT_TIME_T',
optimize='-O1 -MD -Zi -DNDEBUG',
cppflags='-DWIN32'
ccversion='13.10.6030', gccversion='', gccosandvers=''
intsize=4, longsize=4, ptrsize=4, doublesize=8, byteorder=1234
d_longlong=undef, longlongsize=8, d_longdbl=define, longdblsize=8
ivtype='long', ivsize=4, nvtype='double', nvsize=8,
Off_t='__int64', lseeksize=8
alignbytes=8, prototype=define
Linker and Libraries:
ld='link', ldflags ='-nologo -nodefaultlib -debug -opt:ref,icf
-libpath:"c:\perl519\lib\CORE" -machine:x86'
libpth="C:\Program Files\Microsoft Visual Studio .NET 2003\VC7\lib"
libs=oldnames.lib kernel32.lib user32.lib gdi32.lib winspool.lib
comdlg32.lib advapi32.lib shell32.lib ole32.lib oleaut32.lib
netapi32.lib uuid.lib ws2_32.lib mpr.lib winmm.lib version.lib
odbc32.lib odbccp32.lib comctl32.lib msvcrt.lib
perllibs=oldnames.lib kernel32.lib user32.lib gdi32.lib
winspool.lib comdlg32.lib advapi32.lib shell32.lib ole32.lib
oleaut32.lib netapi32.lib uuid.lib ws2_32.lib mpr.lib winmm.lib
version.lib odbc32.lib odbccp32.lib comctl32.lib msvcrt.lib
libc=msvcrt.lib, so=dll, useshrplib=true, libperl=perl519.lib
gnulibc_version=''
Dynamic Linking:
dlsrc=dl_win32.xs, dlext=dll, d_dlsymun=undef, ccdlflags=' '
cccdlflags=' ', lddlflags='-dll -nologo -nodefaultlib -debug
-opt:ref,icf -libpath:"c:\perl519\lib\CORE" -machine:x86'
Locally applied patches:
uncommitted-changes
---
@INC for perl 5.19.6:
C:/perl519/src/lib
.
---
Environment for perl 5.19.6:
HOME (unset)
LANG (unset)
LANGUAGE (unset)
LD_LIBRARY_PATH (unset)
LOGDIR (unset)
PATH=C:\perl519\bin;C:\Program Files\Microsoft Visual Studio .NET
2003\Common7\IDE;C:\Program Files\Microsoft Visual Studio .NET
2003\VC7\BIN;C:\Program Files\Microsoft Visual Studio .NET
2003\Common7\Tools;C:\Program Files\Microsoft Visual Studio .NET
2003\Common7\Tools\bin\prerelease;C:\WINDOWS\system32;C:\WINDOWS;C:\WINDOWS\system32\wbem;
PERL_BADLANG (unset)
PERL_DEBUG_FULL_TEST=1
SHELL (unset)
Thread Next
-
[perl #120330] [PATCH] parallel testing on Win32, now with working prototype, needs work
by bulk88