Front page | perl.perl5.porters |
Postings from January 2014
[perl #121119] [PATCH] speed up miniperl @INC searching, buildcustomize
Thread Previous
|
Thread Next
From:
bulk88 via RT
Date:
January 30, 2014 19:49
Subject:
[perl #121119] [PATCH] speed up miniperl @INC searching, buildcustomize
Message ID:
rt-4.0.18-17418-1391111381-475.121119-15-0@perl.org
On Thu Jan 30 05:30:31 2014, davem wrote:
> On Thu, Jan 30, 2014 at 12:13:16AM -0800, bulk88 via RT wrote:
> > > See attached patch. Each dir being searched that fails on Win32
> > > results
> > > in 10 I/O sys calls which I counted at 17 ms wall time, times 16
> > > build
> > > customize dirs that didn't match is an extra 272 ms of wall time
> > > dealing
> > > with the not found dirs during an @INC search.
>
> That does sound extraordinarily slow. On my (admittedly fast) Linux
> laptop,
> the following code takes approx 0.9 microsec per -f test:
>
> my @path = (
> "home/davem/perl5/git/bleed-davem/cpan/AutoLoader/lib",
> "home/davem/perl5/git/bleed-davem/dist/Carp/lib",
> "home/davem/perl5/git/bleed-davem/dist/PathTools",
> "home/davem/perl5/git/bleed-davem/dist/PathTools/lib",
> "home/davem/perl5/git/bleed-davem/dist/ExtUtils-Command/lib",
> "home/davem/perl5/git/bleed-davem/dist/ExtUtils-Install/lib",
> "home/davem/perl5/git/bleed-davem/cpan/ExtUtils-MakeMaker/lib",
> "home/davem/perl5/git/bleed-davem/dist/ExtUtils-Manifest/lib",
> "home/davem/perl5/git/bleed-davem/cpan/File-Path/lib",
> "home/davem/perl5/git/bleed-davem/ext/re",
> "home/davem/perl5/git/bleed-davem/dist/Term-ReadLine/lib",
> "home/davem/perl5/git/bleed-davem/dist/Exporter/lib",
> "home/davem/perl5/git/bleed-davem/ext/File-Find/lib",
> "home/davem/perl5/git/bleed-davem/cpan/Text-Tabs/lib",
> "home/davem/perl5/git/bleed-davem/dist/constant/lib",
> "home/davem/perl5/git/bleed-davem/lib",
> );
>
> my $i = 0;
> for my $file ("aaa" .."zzz") {
> for (@path) {
> $i++;
> next unless -f "$_/NoSuchFile-$file.pmc" or -f
> "$_/NoSuchFile-$file.pm";
> }
> }
> print "i=$i\n";
>
> Perhaps this implies that's there's something sub-optimal in the way
> perl
> does its @INC scanning under Win32?
Go to the RT ticket through web interface, and look at the bad_inc_procmon.txt attachment, it contains strace like data with wall time on the very left. The calls are sniffed by a filter driver that gets first dibs on all I/O packets (since NT Kernel uses a packetized (or transactional) async I/O model) coming from user mode.
A failed @INC test, per dir looks like this after hand trimming of the right side of each line (look at the raw data if you want to see a dump of the args to the syscalls). QueryDirectory is related to enumerating a dir. It requires a dir handle, hence the CreateFile on C:\p519\src\cpan\Test-Harness. "QueryOpen" I'm not sure what C function generates it, or if its called internally by the FS driver, but it fetches this struct http://msdn.microsoft.com/en-us/library/windows/hardware/ff545822%28v=vs.85%29.aspx (from later research it is GetFileAttributesA)
-------------------------------------------------------
9:08:36.3922806 PM miniperl.exe 1208 CreateFile C:\p519\src\cpan\Test-Harness\DynaLoader.pmc NAME NOT FOUND
9:08:36.3937813 PM miniperl.exe 1208 CreateFile C:\p519\src\cpan\Test-Harness SUCCESS
9:08:36.3949223 PM miniperl.exe 1208 QueryDirectory C:\p519\src\cpan\Test-Harness\DynaLoader.pmc NO SUCH FILE
9:08:36.3960337 PM miniperl.exe 1208 CloseFile C:\p519\src\cpan\Test-Harness SUCCESS
9:08:36.3976316 PM miniperl.exe 1208 QueryOpen C:\p519\src\cpan\Test-Harness\DynaLoader.pmc NAME NOT FOUND
9:08:36.3992307 PM miniperl.exe 1208 CreateFile C:\p519\src\cpan\Test-Harness\DynaLoader.pm NAME NOT FOUND
9:08:36.4007414 PM miniperl.exe 1208 CreateFile C:\p519\src\cpan\Test-Harness SUCCESS 9:08:36.4019127 PM miniperl.exe 1208 QueryDirectory C:\p519\src\cpan\Test-Harness\DynaLoader.pm NO SUCH FILE
9:08:36.4030441 PM miniperl.exe 1208 CloseFile C:\p519\src\cpan\Test-Harness SUCCESS
9:08:36.4046092 PM miniperl.exe 1208 QueryOpen C:\p519\src\cpan\Test-Harness\DynaLoader.pm NAME NOT FOUND
-------------------------------------------------------
By setting a BP at the enter kernel mode func, and doing "miniperl -Ilib -e "system('pause'); require FooLoader;"" I'll try to figure out what the syscall were. First one
from win32_stat path 0x00a0b660 "C:\perl519\src\cpan\AutoLoader\lib/FooLoader.pmc" const char *
---------------------------------------------------
ntdll.dll!_KiFastSystemCall@0() + 0x2
ntdll.dll!_NtCreateFile@44() + 0xc
kernel32.dll!_CreateFileW@28() + 0x1b6
kernel32.dll!_CreateFileA@28() + 0x2b
miniperl.exe!win32_stat(const char * path=0x00100080, stat * sbuf=0x0012fe50) Line 1499 + 0xe C
miniperl.exe!S_doopen_pm(sv * name=0x00070023) Line 3630 + 0x2a C
miniperl.exe!Perl_pp_require() Line 4012 C
miniperl.exe!Perl_runops_standard() Line 42 + 0x3 C
miniperl.exe!S_run_body(long oldscope=0x00000001) Line 2448 C
miniperl.exe!perl_run(interpreter * my_perl=0x0012ffc0) Line 2362 + 0x8 C
miniperl.exe!main(int argc=0x00000004, char * * argv=0x00912478, char * * env=0x00912c90) Line 118 + 0x5 C
miniperl.exe!mainCRTStartup() Line 398 + 0xe C
kernel32.dll!_BaseProcessStart@4() + 0x23
---------------------------------------------------
+ path 0x00a0b660 "C:\perl519\src\cpan\AutoLoader\lib/FooLoader.pmc" const char *
---------------------------------------------------
ntdll.dll!_KiFastSystemCall@0() + 0x2
ntdll.dll!_ZwOpenFile@24() + 0xc
kernel32.dll!_FindFirstFileExW@24() + 0x179
kernel32.dll!_FindFirstFileA@8() + 0x3e
msvcr71.dll!_stati64(const char * name=0x0012f810, _stati64 * buf=0x00000000) Line 184 C
miniperl.exe!win32_stat(const char * path=0x00a0b660, _stati64 * sbuf=0x0012fd48) Line 1510 + 0xe C
miniperl.exe!S_doopen_pm(interpreter * my_perl=0x009e3fb0, sv * name=0x009e7b30) Line 3630 + 0x43 C
miniperl.exe!Perl_pp_require(interpreter * my_perl=0x009e3fb0) Line 4011 + 0xd C
miniperl.exe!Perl_runops_standard(interpreter * my_perl=0x009e3fb0) Line 42 + 0xa C
miniperl.exe!S_run_body(interpreter * my_perl=0x009e3fb0, long oldscope=0x00000001) Line 2446 + 0xd C
miniperl.exe!perl_run(interpreter * my_perl=0x009e3fb0) Line 2365 C
miniperl.exe!main(int argc=0x00000004, char * * argv=0x009e2478, char * * env=0x009e2c90) Line 118 + 0xb C
miniperl.exe!mainCRTStartup() Line 398 + 0xe C
kernel32.dll!_BaseProcessStart@4() + 0x23
---------------------------------------------------------------------------
+ path 0x00a0b660 "C:\perl519\src\cpan\AutoLoader\lib/FooLoader.pmc" const char *
------------------------------------------------
ntdll.dll!_KiFastSystemCall@0() + 0x2
ntdll.dll!_ZwQueryDirectoryFile@44() + 0xc
kernel32.dll!_FindFirstFileExW@24() + 0x245
kernel32.dll!_FindFirstFileA@8() + 0x3e
msvcr71.dll!_stati64(const char * name=0x0012f810, _stati64 * buf=0x00000000) Line 184 C
miniperl.exe!win32_stat(const char * path=0x00a0b660, _stati64 * sbuf=0x0012fd48) Line 1510 + 0xe C
miniperl.exe!S_doopen_pm(interpreter * my_perl=0x009e3fb0, sv * name=0x009e7b30) Line 3630 + 0x43 C
miniperl.exe!Perl_pp_require(interpreter * my_perl=0x009e3fb0) Line 4011 + 0xd C
miniperl.exe!Perl_runops_standard(interpreter * my_perl=0x009e3fb0) Line 42 + 0xa C
miniperl.exe!S_run_body(interpreter * my_perl=0x009e3fb0, long oldscope=0x00000001) Line 2446 + 0xd C
miniperl.exe!perl_run(interpreter * my_perl=0x009e3fb0) Line 2365 C
miniperl.exe!main(int argc=0x00000004, char * * argv=0x009e2478, char * * env=0x009e2c90) Line 118 + 0xb C
miniperl.exe!mainCRTStartup() Line 398 + 0xe C
kernel32.dll!_BaseProcessStart@4() + 0x23
-------------------------------------------------------------------
+ path 0x00a0b660 "C:\perl519\src\cpan\AutoLoader\lib/FooLoader.pmc" const char *
-------------------------------------------------------------------
ntdll.dll!_KiFastSystemCall@0() + 0x2
ntdll.dll!_NtClose@4() + 0xc
kernel32.dll!_FindFirstFileExW@24() + 0x497
kernel32.dll!_FindFirstFileA@8() + 0x3e
msvcr71.dll!_stati64(const char * name=0x0012f810, _stati64 * buf=0x00000000) Line 184 C
miniperl.exe!win32_stat(const char * path=0x00a0b660, _stati64 * sbuf=0x0012fd48) Line 1510 + 0xe C
miniperl.exe!S_doopen_pm(interpreter * my_perl=0x009e3fb0, sv * name=0x009e7b30) Line 3630 + 0x43 C
miniperl.exe!Perl_pp_require(interpreter * my_perl=0x009e3fb0) Line 4011 + 0xd C
miniperl.exe!Perl_runops_standard(interpreter * my_perl=0x009e3fb0) Line 42 + 0xa C
miniperl.exe!S_run_body(interpreter * my_perl=0x009e3fb0, long oldscope=0x00000001) Line 2446 + 0xd C
miniperl.exe!perl_run(interpreter * my_perl=0x009e3fb0) Line 2365 C
miniperl.exe!main(int argc=0x00000004, char * * argv=0x009e2478, char * * env=0x009e2c90) Line 118 + 0xb C
miniperl.exe!mainCRTStartup() Line 398 + 0xe C
kernel32.dll!_BaseProcessStart@4() + 0x23
--------------------------------------------------------------------------
+ path 0x00a0b660 "C:\perl519\src\cpan\AutoLoader\lib/FooLoader.pmc" const char *
---------------------------------------------------------------------------
ntdll.dll!_KiFastSystemCall@0() + 0x2
ntdll.dll!_NtQueryAttributesFile@8() + 0xc
kernel32.dll!_GetFileAttributesW@4() + 0x67
kernel32.dll!_GetFileAttributesA@4() + 0x1d
miniperl.exe!win32_stat(const char * path=0x00a0b660, _stati64 * sbuf=0x0012fd48) Line 1521 + 0xa C
miniperl.exe!S_doopen_pm(interpreter * my_perl=0x009e3fb0, sv * name=0x009e7b30) Line 3630 + 0x43 C
miniperl.exe!Perl_pp_require(interpreter * my_perl=0x009e3fb0) Line 4011 + 0xd C
miniperl.exe!Perl_runops_standard(interpreter * my_perl=0x009e3fb0) Line 42 + 0xa C
miniperl.exe!S_run_body(interpreter * my_perl=0x009e3fb0, long oldscope=0x00000001) Line 2446 + 0xd C
miniperl.exe!perl_run(interpreter * my_perl=0x009e3fb0) Line 2365 C
miniperl.exe!main(int argc=0x00000004, char * * argv=0x009e2478, char * * env=0x009e2c90) Line 118 + 0xb C
miniperl.exe!mainCRTStartup() Line 398 + 0xe C
kernel32.dll!_BaseProcessStart@4() + 0x23
----------------------------------------------------------------------------------
*************************now switch to checking .pm, not .pmc anymore****************************************************
+ path 0x00a0a388 "C:\perl519\src\cpan\AutoLoader\lib/FooLoader.pm" const char *
---------------------------------------------------------------------------------
ntdll.dll!_KiFastSystemCall@0() + 0x2
ntdll.dll!_NtCreateFile@44() + 0xc
kernel32.dll!_CreateFileW@28() + 0x1b6
kernel32.dll!_CreateFileA@28() + 0x2b
miniperl.exe!win32_stat(const char * path=0x00a0a388, _stati64 * sbuf=0x0012fce8) Line 1499 + 0x16 C
miniperl.exe!S_check_type_and_open(interpreter * my_perl=0x009e3fb0, sv * name=0x009e7b30) Line 3594 + 0xd C
miniperl.exe!S_doopen_pm(interpreter * my_perl=0x009e3fb0, sv * name=0x009e7b30) Line 3633 + 0xd C
miniperl.exe!Perl_pp_require(interpreter * my_perl=0x009e3fb0) Line 4011 + 0xd C
miniperl.exe!Perl_runops_standard(interpreter * my_perl=0x009e3fb0) Line 42 + 0xa C
miniperl.exe!S_run_body(interpreter * my_perl=0x009e3fb0, long oldscope=0x00000001) Line 2446 + 0xd C
miniperl.exe!perl_run(interpreter * my_perl=0x009e3fb0) Line 2365 C
miniperl.exe!main(int argc=0x00000004, char * * argv=0x009e2478, char * * env=0x009e2c90) Line 118 + 0xb C
miniperl.exe!mainCRTStartup() Line 398 + 0xe C
kernel32.dll!_BaseProcessStart@4() + 0x23
--------------------------------------------------------------------------------
+ path 0x00a0a388 "C:\perl519\src\cpan\AutoLoader\lib/FooLoader.pm" const char *
--------------------------------------------------------------------------------
ntdll.dll!_KiFastSystemCall@0() + 0x2
ntdll.dll!_ZwOpenFile@24() + 0xc
kernel32.dll!_FindFirstFileExW@24() + 0x179
kernel32.dll!_FindFirstFileA@8() + 0x3e
msvcr71.dll!_stati64(const char * name=0x7c927784, _stati64 * buf=0x00000000) Line 184 C
miniperl.exe!win32_stat(const char * path=0x00a0a388, _stati64 * sbuf=0x0012fce8) Line 1510 + 0xe C
miniperl.exe!S_check_type_and_open(interpreter * my_perl=0x009e3fb0, sv * name=0x009e7b30) Line 3594 + 0xd C
miniperl.exe!S_doopen_pm(interpreter * my_perl=0x009e3fb0, sv * name=0x009e7b30) Line 3633 + 0xd C
miniperl.exe!Perl_pp_require(interpreter * my_perl=0x009e3fb0) Line 4011 + 0xd C
miniperl.exe!Perl_runops_standard(interpreter * my_perl=0x009e3fb0) Line 42 + 0xa C
miniperl.exe!S_run_body(interpreter * my_perl=0x009e3fb0, long oldscope=0x00000001) Line 2446 + 0xd C
miniperl.exe!perl_run(interpreter * my_perl=0x009e3fb0) Line 2365 C
miniperl.exe!main(int argc=0x00000004, char * * argv=0x009e2478, char * * env=0x009e2c90) Line 118 + 0xb C
miniperl.exe!mainCRTStartup() Line 398 + 0xe C
kernel32.dll!_BaseProcessStart@4() + 0x23
---------------------------------------------------------------------------------
+ path 0x00a0a388 "C:\perl519\src\cpan\AutoLoader\lib/FooLoader.pm" const char *
---------------------------------------------------------------------------------
ntdll.dll!_KiFastSystemCall@0() + 0x2
ntdll.dll!_ZwQueryDirectoryFile@44() + 0xc
kernel32.dll!_FindFirstFileExW@24() + 0x245
kernel32.dll!_FindFirstFileA@8() + 0x3e
msvcr71.dll!_stati64(const char * name=0x7c927784, _stati64 * buf=0x00000000) Line 184 C
miniperl.exe!win32_stat(const char * path=0x00a0a388, _stati64 * sbuf=0x0012fce8) Line 1510 + 0xe C
miniperl.exe!S_check_type_and_open(interpreter * my_perl=0x009e3fb0, sv * name=0x009e7b30) Line 3594 + 0xd C
miniperl.exe!S_doopen_pm(interpreter * my_perl=0x009e3fb0, sv * name=0x009e7b30) Line 3633 + 0xd C
miniperl.exe!Perl_pp_require(interpreter * my_perl=0x009e3fb0) Line 4011 + 0xd C
miniperl.exe!Perl_runops_standard(interpreter * my_perl=0x009e3fb0) Line 42 + 0xa C
miniperl.exe!S_run_body(interpreter * my_perl=0x009e3fb0, long oldscope=0x00000001) Line 2446 + 0xd C
miniperl.exe!perl_run(interpreter * my_perl=0x009e3fb0) Line 2365 C
miniperl.exe!main(int argc=0x00000004, char * * argv=0x009e2478, char * * env=0x009e2c90) Line 118 + 0xb C
miniperl.exe!mainCRTStartup() Line 398 + 0xe C
kernel32.dll!_BaseProcessStart@4() + 0x23
----------------------------------------------------------------------------------
+ path 0x00a0a388 "C:\perl519\src\cpan\AutoLoader\lib/FooLoader.pm" const char *
-----------------------------------------------------------------------------------
ntdll.dll!_KiFastSystemCall@0() + 0x2
ntdll.dll!_NtClose@4() + 0xc
kernel32.dll!_FindFirstFileExW@24() + 0x497
kernel32.dll!_FindFirstFileA@8() + 0x3e
msvcr71.dll!_stati64(const char * name=0x7c927784, _stati64 * buf=0x00000000) Line 184 C
miniperl.exe!win32_stat(const char * path=0x00a0a388, _stati64 * sbuf=0x0012fce8) Line 1510 + 0xe C
miniperl.exe!S_check_type_and_open(interpreter * my_perl=0x009e3fb0, sv * name=0x009e7b30) Line 3594 + 0xd C
miniperl.exe!S_doopen_pm(interpreter * my_perl=0x009e3fb0, sv * name=0x009e7b30) Line 3633 + 0xd C
miniperl.exe!Perl_pp_require(interpreter * my_perl=0x009e3fb0) Line 4011 + 0xd C
miniperl.exe!Perl_runops_standard(interpreter * my_perl=0x009e3fb0) Line 42 + 0xa C
miniperl.exe!S_run_body(interpreter * my_perl=0x009e3fb0, long oldscope=0x00000001) Line 2446 + 0xd C
miniperl.exe!perl_run(interpreter * my_perl=0x009e3fb0) Line 2365 C
miniperl.exe!main(int argc=0x00000004, char * * argv=0x009e2478, char * * env=0x009e2c90) Line 118 + 0xb C
miniperl.exe!mainCRTStartup() Line 398 + 0xe C
kernel32.dll!_BaseProcessStart@4() + 0x23
----------------------------------------------------------------------------------
+ path 0x00a0a388 "C:\perl519\src\cpan\AutoLoader\lib/FooLoader.pm" const char *
----------------------------------------------------------------------------------
ntdll.dll!_KiFastSystemCall@0() + 0x2
ntdll.dll!_NtQueryAttributesFile@8() + 0xc
kernel32.dll!_GetFileAttributesW@4() + 0x67
kernel32.dll!_GetFileAttributesA@4() + 0x1d
miniperl.exe!win32_stat(const char * path=0x00a0a388, _stati64 * sbuf=0x0012fce8) Line 1521 + 0xa C
miniperl.exe!S_check_type_and_open(interpreter * my_perl=0x009e3fb0, sv * name=0x009e7b30) Line 3594 + 0xd C
miniperl.exe!S_doopen_pm(interpreter * my_perl=0x009e3fb0, sv * name=0x009e7b30) Line 3633 + 0xd C
miniperl.exe!Perl_pp_require(interpreter * my_perl=0x009e3fb0) Line 4011 + 0xd C
miniperl.exe!Perl_runops_standard(interpreter * my_perl=0x009e3fb0) Line 42 + 0xa C
miniperl.exe!S_run_body(interpreter * my_perl=0x009e3fb0, long oldscope=0x00000001) Line 2446 + 0xd C
miniperl.exe!perl_run(interpreter * my_perl=0x009e3fb0) Line 2365 C
miniperl.exe!main(int argc=0x00000004, char * * argv=0x009e2478, char * * env=0x009e2c90) Line 118 + 0xb C
miniperl.exe!mainCRTStartup() Line 398 + 0xe C
kernel32.dll!_BaseProcessStart@4() + 0x23
----------------------------------------------------------------------------------
*****************now on a different dir*************************************************
+ path 0x00a049e8 "C:\perl519\src\dist\Carp\lib/FooLoader.pmc" const char *
----------------------------------------------------------------------------------
ntdll.dll!_KiFastSystemCall@0() + 0x2
ntdll.dll!_NtCreateFile@44() + 0xc
kernel32.dll!_CreateFileW@28() + 0x1b6
kernel32.dll!_CreateFileA@28() + 0x2b
miniperl.exe!win32_stat(const char * path=0x00a049e8, _stati64 * sbuf=0x0012fd48) Line 1499 + 0x16 C
miniperl.exe!S_doopen_pm(interpreter * my_perl=0x009e3fb0, sv * name=0x009e7b30) Line 3630 + 0x43 C
miniperl.exe!Perl_pp_require(interpreter * my_perl=0x009e3fb0) Line 4011 + 0xd C
miniperl.exe!Perl_runops_standard(interpreter * my_perl=0x009e3fb0) Line 42 + 0xa C
miniperl.exe!S_run_body(interpreter * my_perl=0x009e3fb0, long oldscope=0x00000001) Line 2446 + 0xd C
miniperl.exe!perl_run(interpreter * my_perl=0x009e3fb0) Line 2365 C
miniperl.exe!main(int argc=0x00000004, char * * argv=0x009e2478, char * * env=0x009e2c90) Line 118 + 0xb C
miniperl.exe!mainCRTStartup() Line 398 + 0xe C
kernel32.dll!_BaseProcessStart@4() + 0x23
----------------------------------------------------------------------------------
and the loop continues. Note that by calling all the "SomethingFileMoreSomethingA" function calls, each A call does a ASCII to UTF16 conversion. There goes a lil bit of CPU. Maybe win32_stat should do a utf16 conversion and use only kernel32W() and wclib() calls afterwards which dont convert.
On Thu Jan 30 03:50:16 2014, nicholas wrote:
>
> Thanks for ensuring that the patch gets to the list.
>
> No, specifically it's completely unsafe for a parallel make.
>
> You end up with race conditions where the build fails because one
> process
> process loads from lib/ a partially written module and aborts, because
> it
> happens just as another process is copying that file there.
> (And the rest of the time you don't have a problem because the file is
> either
> not in lib, so loaded from the original dist/... etc, or it's fully
> copied to
> lib, so loaded from there)
>
> Which is exactly what the comment in the file tries to explain.
make_ext has no provisions for parallel building. No select(), no open3. Can you explain?
>
> However, as Win32 doesn't have parallel makes *yet*, and this seems to
> be a
> big speed hit on Win32, I think that it would be reasonable to do the
> proposed re-ordering on Win32, with a comment that it needs to be
> rethought
> once anyone starts getting parallel makes working on Win32.
What would that process be?
--
bulk88 ~ bulk88 at hotmail.com
---
via perlbug: queue: perl5 status: open
https://rt.perl.org/Ticket/Display.html?id=121119
Thread Previous
|
Thread Next