develooper Front page | perl.perl5.porters | Postings from October 2013

[perl #120330] [PATCH] parallel testing on Win32, now with working prototype, needs work

Thread Next
From:
bulk88
Date:
October 24, 2013 00:35
Subject:
[perl #120330] [PATCH] parallel testing on Win32, now with working prototype, needs work
Message ID:
rt-3.6.HEAD-26210-1382574923-1540.120330-75-0@perl.org
# New Ticket Created by  bulk88 
# Please include the string:  [perl #120330]
# in the subject line of all future correspondence about this issue. 
# <URL: https://rt.perl.org:443/rt3/Ticket/Display.html?id=120330 >



This is a bug report for perl from bulk88@hotmail.com,
generated with the help of perlbug 1.39 running under perl 5.19.6.


-----------------------------------------------------------------
[Please describe your issue here]


Reposting 
http://www.nntp.perl.org/group/perl.perl5.porters/2013/10/msg208739.html 
to RT so it doesn't get lost.

Earlier post (not to P5P) I made about the idea that got no response 
http://www.nntp.perl.org/group/perl.qa/2013/07/msg13473.html

Original P5P post below.
__________________________________________________________

I decided to get parallel testing working on Win32 Perl.It has never 
worked before because Win32 C select is sockets only (its exported by 
ws2_32.dll, winsock library). Since TAP::Harness suite is an OOP for the 
sake of OOP design wise, I couldn't figure out how to modify it or 
plugin it. My next idea was to fork and rewrite IO::Select to work. The 
standard stream kernel handles that the child perl writes to and the 
parent perl (./t/harness) reads are "Named Pipe Filing System" (an NT 
Kernel driver called npfs.sys) handles. Someone once tried to get 
socketpair to be the standard stream handles at 
http://www.nntp.perl.org/group/perl.perl5.porters/2009/09/msg150546.html 
so Win32 C select would recognize the handles but it didn't work and the 
discussion died out.

The commit disabling use of IO::Select in TAP::Parser::Multiplexer is 
https://github.com/Perl-Toolchain-Gang/Test-Harness/commit/6d27f808824c3592e7089e7438f1ca18c860d307 
and 
https://github.com/Perl-Toolchain-Gang/Test-Harness/commit/ccffd0283e1820165ec307ffbbc4d3ee16fca6b3 
.

NPFS I think is capable of supplying all information/APIs necessary to 
implement posix select, but not through the Win32 API, only NT Native 
API.  FSCTL_PIPE_ASSIGN_EVENT is used to assign a kernel Event object 
that is tripped whenever the other end of the pipe interacts with pipe, 
you also assign a opaque pointer of your choice you will get back later. 
FSCTL_PIPE_QUERY_EVENT takes an Event Object, and writes a log of all 
events (reads, writes, and state changes (disconnect, accept, etc)) that 
happened to all pipes that that Event Object was registered with. You 
also get back your opaque pointer that you set hopefully per pipe. I 
didn't look at the implementation of Interix/SFU to see if their posix 
select uses these 2 ioctls or not (not meaning Interix implements its 
own pipe API using IPC back to its interix service process).

For my first try, I decided to just do the bare minimum for harness to 
be parallel, and do it in a Win32-only way, no Native API. Some select 
for Win32 implementations I saw used PeekNamedPipe which gives the 
number of bytes in the read buffer in the pipe instance in the kernel 
driver and polling through the pipes with a sleep timer after going 
through each set of handles to get readability status. For writeability, 
they had to use Native API to poll the write buffer max size and current 
write buffer fill. I won't describe how they got sockets to also work 
with their select for all handles.

I dont see myself implementing writability or exceptions parts of select 
ATM. harness never uses them. I might rewrite this to use Native API 
described above and WaitForMultipleObjects to avoid wasting CPU on polling.

So I am attaching 2 files. 1 is a patch to TAP::Parser::Multiplexer to 
use my IO::Select fork. The other is the IO::Select fork that I've 
temporarily named IO::Select::Win32Pipes.

The code *barely* works on Win32 with TEST_JOBS=8 and I get 8 advancing 
test counters on the bottom of the console window.

The code has a number of problems I dont know what to do about and I 
need help on.

TAP::Parser::Multiplexer and TAP::Parser::Iterator::Process may have 
some issues/bugs that need to be fixed that I can't figure out how to debug.

To get this into perl git, is another problem. It uses use Win32::API 
(not core) right now and is mostly Pure Perl. For clarity of the error 
numbers it tests, I'll probably add Win32::WinError to the mostly Pure 
Perl implementation. I could also rewrite the Win32Pipes implementation 
of select in XS. I think TAP:: and harness are pure perl ATM, I'm not 
sure if they ever load an XS module right now (didn't research). If they 
don't use any XS, then would it be okay to make them load XS or not? 
Would there be any issues with running harness on miniperl and static perl?

I've also found out, very very very often (if I turn on the reporting, 
the console will flooded with warnings), can_read() is being called on 
pipes that have "ERROR_BROKEN_PIPE" or in POSIX EPIPE. I decided as a 
wild ass guess to convert ERROR_BROKEN_PIPE to mean readable according 
to this terse post 
http://osdir.com/ml/bug-gnulib-gnu/2010-08/msg00141.html . This got 
harness running in parallel finally.

Another problem is, TAP::Parser::Multiplexer is INCAPABLE of processing 
ANY FAILURE of select from IO::Select. If Win32Pipes select returns -1, 
it goes up the callstack and I get
__________________________________________________________________
Can't use an undefined value as an ARRAY reference at 
../lib/TAP/Parser/Multiple
xer.pm line 149, <GEN0> line 1.
_________________________________________________________________
line 149 is
__________________________________________________________________
sub _iter {
     my $self = shift;

     my $sel   = $self->{select};
     my $avid  = $self->{avid};
     my @ready = ();

     return sub {

         # Drain all the non-selectable parsers first
         if (@$avid) {
             my ( $parser, $stash ) = @{ $avid->[0] };
             my $result = $parser->next;
             shift @$avid unless defined $result;
             return ( $parser, $stash, $result );
         }

         unless (@ready) {
             return unless $sel->count;
             @ready = $sel->can_read;
         }

         my ( $h, $parser, $stash, @handles ) = @{ shift @ready 
};<<<<<<<<<<<<<<<<<<<<
         my $result = $parser->next;

         unless ( defined $result ) {
             $sel->remove(@handles);
             $self->{count}--;

             # Force another can_read - we may now have removed a handle
             # thought to have been ready.
             @ready = ();
         }

         return ( $parser, $stash, $result );
     };
}
________________________________________________________________

IDK what to do about this.

Also TAP is constantly doing selects on closed file descriptors.

_______________________________________________________________
sub _fileno
{
  my($self, $f) = @_;
  return unless defined $f;
  $f = $f->[0] if ref($f) eq 'ARRAY';
  my $fn = ($f =~ /^\d+$/) ? $f : fileno($f);
  #validate that the handle is a pipe
  if(defined $fn) {
     my ($hnd, $ret) = Win32API::File::FdGetOsFHandle($fn);
     if($hnd != Win32API::File::INVALID_HANDLE_VALUE()) {
         my $err;
         $ret = Win32API::File::GetFileType($hnd);
         $err = Win32::GetLastError();
         if($ret == Win32API::File::FILE_TYPE_UNKNOWN && $err != 0)
         #{ return undef;}
         { die "GetFileType failed"}
         if($ret != Win32API::File::FILE_TYPE_PIPE) {
             die "not a pipe";
         }
         $ret = GetNamedPipeInfo($hnd, undef, undef, undef, undef);
         if(!$ret) {
             die "not a pipe GNPI";
         }
         #else {
         #    die "is a pipe";
         #}
     }
     else { warn "no handle $fn";
           }
  }
  return $fn;
}
______________________________________________________________

______________________________________________________________
no handle 8 at C:\perl519\site\lib/IO/Select/Win32Pipes.pm line 170.
Use of uninitialized value $f in numeric eq (==) at 
C:\perl519\site\lib/IO/Selec
t/Win32Pipes.pm line 202.
Use of uninitialized value $f in numeric eq (==) at 
C:\perl519\site\lib/IO/Selec
t/Win32Pipes.pm line 202.
Use of uninitialized value $f in numeric eq (==) at 
C:\perl519\site\lib/IO/Selec
t/Win32Pipes.pm line 202.
Use of uninitialized value $f in numeric eq (==) at 
C:\perl519\site\lib/IO/Selec
t/Win32Pipes.pm line 202.
Use of uninitialized value $f in numeric eq (==) at 
C:\perl519\site\lib/IO/Selec
t/Win32Pipes.pm line 202.
Use of uninitialized value $f in numeric eq (==) at 
C:\perl519\site\lib/IO/Selec
t/Win32Pipes.pm line 202.
Use of uninitialized value $f in numeric eq (==) at 
C:\perl519\site\lib/IO/Selec
t/Win32Pipes.pm line 202.
______________________________________________________________
C:\Documents and Settings\Owner\Desktop\cpan libs\p519\perl\t>perl 
-I../lib harn
ess
No saved state, selection will be empty
base/cond.t ....................................................... 1/4 
no handl
e 3 at C:\perl519\site\lib/IO/Select/Win32Pipes.pm line 170.
base/cond.t ....................................................... ok
base/if.t ......................................................... 1/2 
no handl
e 3 at C:\perl519\site\lib/IO/Select/Win32Pipes.pm line 170.
base/if.t ......................................................... ok
base/lex.t ........................................................ 1/91 
no hand
le 3 at C:\perl519\site\lib/IO/Select/Win32Pipes.pm line 170.
base/lex.t ........................................................ ok
base/num.t ........................................................ 1/53 
no hand
le 3 at C:\perl519\site\lib/IO/Select/Win32Pipes.pm line 170.
base/num.t ........................................................ ok
base/pat.t ........................................................ 1/2 
no handl
e 3 at C:\perl519\site\lib/IO/Select/Win32Pipes.pm line 170.
base/pat.t ........................................................ ok
Terminating on signal SIGINT(2)
Terminating on signal SIGINT(2)
_______________________________________________________________

I can't figure out what is causing all the "Use of uninitialized value 
$f in numeric eq (==)" or what to do about it

_______________________________________________________________
sub _update
{
  my $vec = shift;
  my $add = shift eq 'add';

  my $bits = $vec->[VEC_BITS];
  $bits = '' unless defined $bits;

  my $count = 0;
  my $f;
  foreach $f (@_)
   {
    my $fn = $vec->_fileno($f);
    if ($add) {
      next unless defined $fn;
      my $i = $fn + FIRST_FD;
      if (defined $vec->[$i]) {
      $vec->[$i] = $f;  # if array rest might be different, so we update
      next;
      }
      $vec->[FD_COUNT]++;
      vec($bits, $fn, 1) = 1;
      $vec->[$i] = $f;
    } else {      # remove
      if ( ! defined $fn ) { # remove if fileno undef'd
          defined($_) && $_ == $f and do { $vec->[FD_COUNT]--; $_ = undef; }
            for @{$vec}[FIRST_FD .. $#$vec]; 
###################<<<<<<<<<<<<<<<<< line 202
          next;
      }
______________________________________________________________

I also attached a screenshot of console window during parallel testing.


IO::Select::Win32Pipes needs a new name also. I dont have the tuits to 
ever make it a general purpose select/async IO module. I see its future 
being only for TAP parallel testing.

Also the fork of IO::Select isn't the most efficient design. I kept 
IO::Select almost intact except for fileno method tweaks, since that 
method is used to validate a FD during an add procedure, and renaming 
all calls from (CORE::) select to _select. In theory the select method 
could do a caller or blessed check on $_[0] to see whether its being 
called as a use subs override or as a public API of IO::Select method. 
The other choice is to integrate the Win32Pipes code into the official 
IO::Select. I dont understand the internal design of IO::Select enough 
to really do anything with it. Its uses @rrays with constant subs for 
indexes instead of %ashes for what I guess is efficiency. C inspired 
code :-)

Another problem is, Multiplexer/TAP has a problem servicing the child 
processes. IDK if its my code or TAP, but what happens is, TAP gets 
stuck on reading a rapidly ok()ing child process, and it will stay on 
that rapidly ok() process until it runs dry or ends. Meanwhile other 
child processes fill their pipe write buffers and C write() blocks for 
10 to 30 seconds with zero CPU used and zero context switches in Process 
Monitor (I took a C debugger to one of these frozen/blocked childs and 
it was doing a C/PerlIO write to stdout).

Need help. Lots of it.



[Please do not change anything below this line]
-----------------------------------------------------------------
---
Flags:
     category=core
     severity=low
---
Site configuration information for perl 5.19.6:

Configured by Owner at Wed Oct 23 00:23:23 2013.

Summary of my perl5 (revision 5 version 19 subversion 6) configuration:
   Derived from:
   Platform:
     osname=MSWin32, osvers=5.1, archname=MSWin32-x86-multi-thread
     uname=''
     config_args='undef'
     hint=recommended, useposix=true, d_sigaction=undef
     useithreads=define, usemultiplicity=define
     useperlio=define, d_sfio=undef, uselargefiles=define, usesocks=undef
     use64bitint=undef, use64bitall=undef, uselongdouble=undef
     usemymalloc=n, bincompat5005=undef
   Compiler:
     cc='cl', ccflags ='-nologo -GF -W3 -O1 -MD -Zi -DNDEBUG -DWIN32 
-D_CONSOLE -DNO_STRICT  -DPERL_TEXTMODE_SCRIPTS -DPERL_IMPLICIT_CONTEXT 
-DPERL_IMPLICIT_SYS -DUSE_PERLIO -D_USE_32BIT_TIME_T',
     optimize='-O1 -MD -Zi -DNDEBUG',
     cppflags='-DWIN32'
     ccversion='13.10.6030', gccversion='', gccosandvers=''
     intsize=4, longsize=4, ptrsize=4, doublesize=8, byteorder=1234
     d_longlong=undef, longlongsize=8, d_longdbl=define, longdblsize=8
     ivtype='long', ivsize=4, nvtype='double', nvsize=8, 
Off_t='__int64', lseeksize=8
     alignbytes=8, prototype=define
   Linker and Libraries:
     ld='link', ldflags ='-nologo -nodefaultlib -debug -opt:ref,icf 
-libpath:"c:\perl519\lib\CORE"  -machine:x86'
     libpth="C:\Program Files\Microsoft Visual Studio .NET 2003\VC7\lib"
     libs=oldnames.lib kernel32.lib user32.lib gdi32.lib winspool.lib 
comdlg32.lib advapi32.lib shell32.lib ole32.lib oleaut32.lib 
netapi32.lib uuid.lib ws2_32.lib mpr.lib winmm.lib  version.lib 
odbc32.lib odbccp32.lib comctl32.lib msvcrt.lib
     perllibs=oldnames.lib kernel32.lib user32.lib gdi32.lib 
winspool.lib  comdlg32.lib advapi32.lib shell32.lib ole32.lib 
oleaut32.lib  netapi32.lib uuid.lib ws2_32.lib mpr.lib winmm.lib 
version.lib odbc32.lib odbccp32.lib comctl32.lib msvcrt.lib
     libc=msvcrt.lib, so=dll, useshrplib=true, libperl=perl519.lib
     gnulibc_version=''
   Dynamic Linking:
     dlsrc=dl_win32.xs, dlext=dll, d_dlsymun=undef, ccdlflags=' '
     cccdlflags=' ', lddlflags='-dll -nologo -nodefaultlib -debug 
-opt:ref,icf  -libpath:"c:\perl519\lib\CORE"  -machine:x86'

Locally applied patches:
     uncommitted-changes

---
@INC for perl 5.19.6:
     C:/perl519/src/lib
     .

---
Environment for perl 5.19.6:
     HOME (unset)
     LANG (unset)
     LANGUAGE (unset)
     LD_LIBRARY_PATH (unset)
     LOGDIR (unset)
     PATH=C:\perl519\bin;C:\Program Files\Microsoft Visual Studio .NET 
2003\Common7\IDE;C:\Program Files\Microsoft Visual Studio .NET 
2003\VC7\BIN;C:\Program Files\Microsoft Visual Studio .NET 
2003\Common7\Tools;C:\Program Files\Microsoft Visual Studio .NET 
2003\Common7\Tools\bin\prerelease;C:\WINDOWS\system32;C:\WINDOWS;C:\WINDOWS\system32\wbem;
     PERL_BADLANG (unset)
     PERL_DEBUG_FULL_TEST=1
     SHELL (unset)

Thread Next


nntp.perl.org: Perl Programming lists via nntp and http.
Comments to Ask Bjørn Hansen at ask@perl.org | Group listing | About