develooper Front page | perl.perl5.porters | Postings from February 2015

[perl #123854] parallel building with Win32 dmake problems

Thread Next
From:
bulk88
Date:
February 17, 2015 04:49
Subject:
[perl #123854] parallel building with Win32 dmake problems
Message ID:
rt-4.0.18-22995-1424148568-1241.123854-75-0@perl.org
# New Ticket Created by  bulk88 
# Please include the string:  [perl #123854]
# in the subject line of all future correspondence about this issue. 
# <URL: https://rt.perl.org/Ticket/Display.html?id=123854 >


This is a bug report for perl from bulk88@hotmail.com,
generated with the help of perlbug 1.40 running under perl 5.21.4.


-----------------------------------------------------------------
[Please describe your issue here]

This really shouldn't be a ticket, but the mailing list server has a 
250KB message size limit, and I dont want to send 22 posts to the ML. 
The log files will be in a different post for this ticket since that 
post will bounce from the ML server when RT forwards it to the ML server.

I decided to implement parallel building with dmake since nmake's clone 
with parallelism, QT's jom make tool, has serious bugs, and I dont think 
it is worth my time to get them fixed when dmake parallelism doesn't 
have straight up "build will absolutely fail" bugs. Jom's specific bug 
IIRC is doesn't support having a dep/target with ".." in it. A target 
with ".." in its name is silently skipped.

I am using dmake that came with strawberry perl 5.18. It calls itself 
dmake 4.12
----------------------------------------------------------------
C:\perl521>dmake -V
dmake - Version 4.12 (Windows / MinGW)
Copyright (c) 1990,...,1997 by WTI Corp.

Default Configuration:
         MAXLINELENGTH := 32766
         MAXPROCESSLIMIT := 64
         .IMPORT .IGNORE: DMAKEROOT
         .MAKEFILES : makefile.mk makefile
         .SOURCE    : .NULL
         DMAKEROOT *= $(ABSMAKECMD:d)startup
         MAKESTARTUP := $(DMAKEROOT)\startup.mk

Please read the NEWS file for the latest release notes.

C:\perl521>
----------------------------------------------------------------

But dmake parallel ("dmake -P2 all") has very bad design in finding 
"next shell line to run". A "dmake -P2 all" does around 
"!unsatisfied_deps_of_target_all" IO stat/file test calls to MS 
CRT/kernel, for each recipe line. With procmon profiling, in 1 minute, 
it made 839447 IO calls (839447 lines, 99% of which are IO calls). I've 
attached the IO call log in the 2nd post to this ticket. It contains 
about 1 minute (and remember, each IO calls is slower because of the 
procmon profiler) of IO calls of dmake.exe. Yes it is 150 MB long. With 
file IO profiling off, and just process events turned on in procmon, it 
takes 3 SECONDS to run each "echo something >> config.h" line on the 
profilers. Without the procmon profiler, it takes 30-35 seconds to 
generate config.h with 1 core maxed out at nearly 100% kernel time for 
that core. So without procmon, it takes 350 ms to do each echo shell 
call. Below a console log of what is happening.

C:\perl521\srcnewb4opt\win32>dmake -P2 all
dmake:  makefile.mk:  line 1682:  Warning: -- Macro `TESTPREPGCC' 
redefined afte
r use
if not exist ".\mini" mkdir ".\mini"
if exist config.h del /f config.h
copy config_H.vc config.h
rem. > .\mini\.exists
lib /def:ntdllundec.def /machine:x86
copy config.vc config.w32
Microsoft (R) Libra        1 file(s) copied.
ry Manager Version 7.10.6030
Copyright (C) Microsoft Corporation.  All rights reserved.

    Creating library ntdllundec.lib and object ntdllundec.exp
lib /def:ntdlldec.def /machine:X86 ntdllundec.lib
         1 file(s) copied.
Microsoft (R) Library Manager Version 7.10.6030
Copyright (C) Microsoft Corporation.  All rights reserved.

LIB : warning LNK4001: no object files specified; libraries used
    Creating library ntdlldec.lib and object ntdlldec.exp
copy runperl.c perlmain.c
rc -i.. -DINCLUDE_MANIFEST perlexe.rc
         1 file(s) copied.
cl -c  -nologo -GF -W3 -I..\lib\CORE -I.\include -I. -I.. -DWIN32 
-D_CONSOLE -DN
O_STRICT -DPERLDLL -DPERL_CORE           -O1 -MD -Zi -DNDEBUG 
-DPERL_TEXTMODE_S
CRIPTS -DPERL_IMPLICIT_CONTEXT -DPERL_IMPLICIT_SYS -D_USE_32BIT_TIME_T 
-Foperlgl
ob.obj -Fdperlglob.pdb perlglob.c
perlglob.c
link  -nologo -nodefaultlib -debug -opt:ref,icf -ltcg 
-libpath:"c:\per
l\lib\CORE"             -machine:x86 oldnames.lib kernel32.lib 
user32.lib gdi32.
lib winspool.lib                comdlg32.lib advapi32.lib shell32.lib 
ole32.lib
oleaut32.lib            netapi32.lib uuid.lib ws2_32.lib mpr.lib winmm.lib
         version.lib odbc32.lib odbccp32.lib comctl32.lib msvcrt.lib 
-out:..\perl
glob.exe -subsystem:console \
             perlglob.obj setargv.obj
dmake:  Warning: -- Target [..\perly.c] was made but the time stamp has 
not been
  updated.
if exist ..\perlglob.exe.manifest mt -nologo -manifest 
..\perlglob.exe.manifest
-outputresource:..\perlglob.exe;1 &&              if exist 
..\perlglob.exe.manif
est del ..\perlglob.exe.manifest
LINK : warning LNK4235: /LTCG specified but no code generation required; 
remove
/LTCG from the link command line to improve linker performance
setargv.obj : warning LNK4099: PDB 'libc.pdb' was not found with 
'C:\Program Fil
es\Microsoft Visual Studio .NET 2003\VC7\LIB\setargv.obj' or at 
'C:\perl521\srcn
ewb4opt\libc.pdb'; linking object as if no debug info
********everything above took 3 seconds***********************
*******30 to 35 second pause with no console output here, this is 
generating miniperl's config.h with many echo >> config.h shell calls 
*******************
cl -c -nologo -GF -W3 -I..\lib\CORE -I.\include -I. -I.. -DWIN32 
-D_CONSOLE -DNO
_STRICT -DPERLDLL -DPERL_CORE            -O1 -MD -Zi -DNDEBUG 
-DPERL_EXTERNAL_G
LOB -DPERL_IS_MINIPERL -Fomini\av.obj -Fdmini\av.pdb ..\av.c
cl -c -nologo -GF -W3 -I..\lib\CORE -I.\include -I. -I.. -DWIN32 
-D_CONSOLE -DNO
_STRICT -DPERLDLL -DPERL_CORE            -O1 -MD -Zi -DNDEBUG 
-DPERL_EXTERNAL_G
LOB -DPERL_IS_MINIPERL -Fomini\caretx.obj -Fdmini\caretx.pdb ..\caretx.c
cl -c -nologo -GF -W3 -I..\lib\CORE -I.\include -I. -I.. -DWIN32 
-D_CONSOLE -DNO
_STRICT -DPERLDLL -DPERL_CORE            -O1 -MD -Zi -DNDEBUG 
-DPERL_EXTERNAL_G
LOB -DPERL_IS_MINIPERL -Fomini\deb.obj -Fdmini\deb.pdb ..\deb.c
av.c
caretx.c
cl -c -nologo -GF -W3 -I..\lib\CORE -I.\include -I. -I.. -DWIN32 
-D_CONSOLE -DNO
_STRICT -DPERLDLL -DPERL_CORE            -O1 -MD -Zi -DNDEBUG 
-DPERL_EXTERNAL_G
LOB -DPERL_IS_MINIPERL -Fomini\doio.obj -Fdmini\doio.pdb ..\doio.c
deb.c
cl -c -nologo -GF -W3 -I..\lib\CORE -I.\include -I. -I.. -DWIN32 
-D_CONSOLE -DNO
_STRICT -DPERLDLL -DPERL_CORE            -O1 -MD -Zi -DNDEBUG 
-DPERL_EXTERNAL_G
LOB -DPERL_IS_MINIPERL -Fomini\doop.obj -Fdmini\doop.pdb ..\doop.c
doio.c
cl -c -nologo -GF -W3 -I..\lib\CORE -I.\include -I. -I.. -DWIN32 
-D_CONSOLE -DNO
_STRICT -DPERLDLL -DPERL_CORE            -O1 -MD -Zi -DNDEBUG 
-DPERL_EXTERNAL_G
LOB -DPERL_IS_MINIPERL -Fomini\dump.obj -Fdmini\dump.pdb ..\dump.c
doop.c
cl -c -I..\ -nologo -GF -W3 -I..\lib\CORE -I.\include -I. -I.. -DWIN32 
-D_CONSOL
E -DNO_STRICT -DPERLDLL -DPERL_CORE              -O1 -MD -Zi -DNDEBUG 
-DPERL_TE
XTMODE_SCRIPTS -DPERL_IMPLICIT_CONTEXT -DPERL_IMPLICIT_SYS 
-D_USE_32BIT_TIME_T -
Fo..\generate_uudmap.obj -Fd..\generate_uudmap.pdb ..\generate_uudmap.c
dump.c
cl -c -nologo -GF -W3 -I..\lib\CORE -I.\include -I. -I.. -DWIN32 
-D_CONSOLE -DNO
_STRICT -DPERLDLL -DPERL_CORE            -O1 -MD -Zi -DNDEBUG 
-DPERL_EXTERNAL_G
LOB -DPERL_IS_MINIPERL -Fomini\gv.obj -Fdmini\gv.pdb ..\gv.c
generate_uudmap.c
cl -c -nologo -GF -W3 -I..\lib\CORE -I.\include -I. -I.. -DWIN32 
-D_CONSOLE -DNO
_STRICT -DPERLDLL -DPERL_CORE            -O1 -MD -Zi -DNDEBUG 
-DPERL_EXTERNAL_G
LOB -DPERL_IS_MINIPERL -Fomini\mro.obj -Fdmini\mro.pdb ..\mro.c
gv.c
cl -c -nologo -GF -W3 -I..\lib\CORE -I.\include -I. -I.. -DWIN32 
-D_CONSOLE -DNO
_STRICT -DPERLDLL -DPERL_CORE            -O1 -MD -Zi -DNDEBUG 
-DPERL_EXTERNAL_G
LOB -DPERL_IS_MINIPERL -Fomini\hv.obj -Fdmini\hv.pdb ..\hv.c
cl -c -nologo -GF -W3 -I..\lib\CORE -I.\include -I. -I.. -DWIN32 
-D_CONSOLE -DNO
_STRICT -DPERLDLL -DPERL_CORE            -O1 -MD -Zi -DNDEBUG 
-DPERL_EXTERNAL_G
LOB -DPERL_IS_MINIPERL -Fomini\locale.obj -Fdmini\locale.pdb ..\locale.c
mro.c
hv.c
******killed******

By my eye, "dmake -P2 ../mini/av.obj" is faster than "dmake -P2 
../miniperl.exe" is faster than "dmake -P2 Extensions" which is faster 
than "dmake -P2 all", all because of that delay before the first .obj 
for miniperl is produced. The narrower the build target, or the less 
total targets, the shorter that pause before the first miniperl .obj is 
compiled.

I am thinking that the only way to hack fix the performance problem with 
the echos is turn all the echo calls in .\config.h target 
http://perl5.git.perl.org/perl.git/blob/HEAD:/win32/makefile.mk#l1054 
into 1 very long shell line filled with &&s. The other solution is for 
miniperl.exe and miniperl's config.h, and buildcustomize.pl to be built 
from a 2nd dmake lang makefile, to break the IO file existence test 
chain tree up. Anyone disagree to consolidating all the echos into 1 
shell call or have any other opinions?

[Please do not change anything below this line]
-----------------------------------------------------------------
---
Flags:
                category=core
                severity=low
---
Site configuration information for perl 5.21.4:

Configured by Owner at Thu Sep 18 12:08:58 2014.

Summary of my perl5 (revision 5 version 21 subversion 4) configuration:
              Derived from: 7d2b2edb94ab56333b9049a3e26d15ea18445512
              Ancestor: 19be3be6968e2337bcdfe480693fff795ecd1304
              Platform:
                osname=MSWin32, osvers=5.1,
archname=MSWin32-x86-multi-thread
                uname=''
                config_args='undef'
                hint=recommended, useposix=true, d_sigaction=undef
                useithreads=define, usemultiplicity=define
                use64bitint=undef, use64bitall=undef, uselongdouble=undef
                usemymalloc=n, bincompat5005=undef
              Compiler:
                cc='cl', ccflags ='-nologo -GF -W3 -O1 -MD -Zi -DNDEBUG
-DWIN32
-D_CONSOLE -DNO_STRICT  -DPERL_TEXTMODE_SCRIPTS
-DPERL_HASH_FUNC_ONE_AT_A_TIME -DPERL_IMPLICIT_CONTEXT
-DPERL_IMPLICIT_SYS -DUSE_PERLIO -D_USE_32BIT_TIME_T',
                optimize='-O1 -MD -Zi -DNDEBUG',
                cppflags='-DWIN32'
                ccversion='12.00.8168', gccversion='', gccosandvers=''
                intsize=4, longsize=4, ptrsize=4, doublesize=8,
byteorder=1234
                d_longlong=undef, longlongsize=8, d_longdbl=define,
longdblsize=8,
longdblkind=0
                ivtype='long', ivsize=4, nvtype='double', nvsize=8,
Off_t='__int64',
lseeksize=8
                alignbytes=8, prototype=define
              Linker and Libraries:
                ld='link', ldflags ='-nologo -nodefaultlib -debug
-opt:ref,icf
-libpath:"c:\perl521\lib\CORE"  -machine:x86'
                libpth=C:\PROGRA~1\MIAF9D~1\VC98\lib
                libs=oldnames.lib kernel32.lib user32.lib gdi32.lib
winspool.lib
comdlg32.lib advapi32.lib shell32.lib ole32.lib oleaut32.lib
netapi32.lib uuid.lib ws2_32.lib mpr.lib winmm.lib  version.lib
odbc32.lib odbccp32.lib comctl32.lib msvcrt.lib
                perllibs=oldnames.lib kernel32.lib user32.lib gdi32.lib
winspool.lib  comdlg32.lib advapi32.lib shell32.lib ole32.lib
oleaut32.lib  netapi32.lib uuid.lib ws2_32.lib mpr.lib winmm.lib
version.lib odbc32.lib odbccp32.lib comctl32.lib msvcrt.lib
                libc=msvcrt.lib, so=dll, useshrplib=true,
libperl=perl521.lib
                gnulibc_version=''
              Dynamic Linking:
                dlsrc=dl_win32.xs, dlext=dll, d_dlsymun=undef, ccdlflags=' '
                cccdlflags=' ', lddlflags='-dll -nologo -nodefaultlib -debug
-opt:ref,icf  -libpath:"c:\perl521\lib\CORE"  -machine:x86'

Locally applied patches:
                uncommitted-changes
                a0fe7a7e75de29e59f1da0d6822dc06e5be658fe
                a261faffee83d0145642ab5d1d046c9f813bc497
                6506ab86ad1602a9ca720fcd30446dce1461d23d
                7d2b2edb94ab56333b9049a3e26d15ea18445512

---
@INC for perl 5.21.4:
                lib
                C:/perl521/srcnew/lib
                .

---
Environment for perl 5.21.4:
                HOME (unset)
                LANG (unset)
                LANGUAGE (unset)
                LD_LIBRARY_PATH (unset)
                LOGDIR (unset)
                PATH=
                PERL_BADLANG (unset)
                PERL_JSON_BACKEND=Cpanel::JSON::XS
                PERL_YAML_BACKEND=YAML
                SHELL (unset)























Thread Next


nntp.perl.org: Perl Programming lists via nntp and http.
Comments to Ask Bjørn Hansen at ask@perl.org | Group listing | About