develooper Front page | perl.perl5.porters | Postings from December 2014

[perl #123482] parallel building of core .c files on VC

Thread Previous
December 22, 2014 23:02
[perl #123482] parallel building of core .c files on VC
Message ID:
# New Ticket Created by  bulk88 
# Please include the string:  [perl #123482]
# in the subject line of all future correspondence about this issue. 
# <URL: >

This is a bug report for perl from,
generated with the help of perlbug 1.40 running under perl 5.21.6.

[Please describe your issue here]

I tried playing around with parallel building (-MP[number]) option of cl.exe. -MP was added in VC 2008. A long time ago I tried using jmake or cmake (I dont remember) with VC 2003 and telling it to parallel build, IIRC that failed since all the cl.exe'es tried to get file locks on "vc70.pdb" (a symbol file), and all but 1 cl.exe fatally exited. In that case, 2 unapatizing solutions exist, disabling .pdb creation (absolutly no for P5P, unless its a smoke machine), or copying each .c file to its own folder and changing cwd before calling cl.exe. That way each cl.exe has their own vc70.pdb file simultanously. But these 2 solutions are for DIY parallism, not MS sponsored parallelism. -MP on VC 2008 does produce multiple simultanous cl.exe processes but they dont seem to mind sharing 1 vc90.pdb file or 1 cwd, so maybe MS fixed the file lock on a .pdb problem when they created -MP or -MP uses IPC or something to tell the child cl.exe'es to not lock the file exclusivly and share it 
 with other child cl.exe'es but not foreign cl.exe processes (not lauched with -MP).

First attempt attempt failed and nothing was built. /Fo is global for some stupid reason instead of being bound to the .c right ahead of it on the command line.

C:\p521\srcnew\win32>cl -c -nologo -GF -W3 -I..\lib\CORE -I.\include -I. -I.. -D
recise -DPERL_EXTERNAL_GLOB -DPERL_IS_MINIPERL  -MP8 -Fomini\av.obj ..\av.c -Fom
ini\caretx.obj ..\caretx.c
cl : Command line warning D9025 : overriding '/Fomini\av.obj' with '/Fomini\care
cl : Command line warning D9026 : options apply to entire command line
cl : Command line error D8036 : '/Fomini\caretx.obj' not allowed with multiple s
ource files

Attempt 2, use -Fo in directory mode

C:\p521\srcnew\win32>cl -c -nologo -GF -W3 -I..\lib\CORE -I.\include -I. -I.. -D
recise -DPERL_EXTERNAL_GLOB -DPERL_IS_MINIPERL  -MP8 -Fomini ..\av.c ..\caretx.c

cl : Command line error D8036 : '/Fomini' not allowed with multiple source files

Attempt 3 I picked up on google (and an MS employee says they will try to fix "the bug" but put a backslash in the meantime) , make -Fo end with a \

C:\p521\srcnew\win32>cl -c -nologo -GF -W3 -I..\lib\CORE -I.\include -I. -I.. -D
fp:precise -DPERL_EXTERNAL_GLOB -DPERL_IS_MINIPERL  -MP8  -Fomini\ ..\av.c  ..\c
aretx.c  ..\deb.c  ..\doio.c  ..\doop.c  ..\dump.c  ..\globals.c  ..\gv.c  ..\mr
o.c  ..\hv.c  ..\locale.c  ..\keywords.c  ..\mathoms.c  ..\mg.c  ..\numeric.c  .
.\op.c  ..\pad.c  ..\perl.c  ..\perlapi.c  ..\perly.c  ..\pp.c  ..\pp_ctl.c  ..\
pp_hot.c  ..\pp_pack.c  ..\pp_sort.c  ..\pp_sys.c  ..\reentr.c  ..\regcomp.c  ..
\regexec.c  ..\run.c  ..\scope.c  ..\sv.c  ..\taint.c  ..\toke.c  ..\universal.c
  ..\utf8.c  ..\util.c  ..\miniperlmain.c  ..\perlio.c

Successful build log from console attached as attachment.

The big problem now is, how to implement this in make language? Since now many .c targets have to be built with 1 rule invocation so 1 cl.exe process runs. Also the dependency on     ..\generate_uudmap.exe  and ..\uudmap.h ..\bitcount.h ..\mg_data.h has to be tweaked so those 4 things are built before any core .c files are ever compiled. the nmake makefile currently has


so only global.c depends on those 3 files, for parallel building an ifdef will be needed (remember we still support non-MP option VC) so all core .c files depend on those 3 headers.

Below is a "nmake -n" log

    del /f config.h
    copy config.h
    echo #ifndef _config_h_footer_>>.\config.h
    echo #define _config_h_footer_>>.\config.h
    echo #undef Off_t>>.\config.h
    echo #undef LSEEKSIZE>>.\config.h
    echo #undef Off_t_size>>.\config.h
    echo #undef PTRSIZE>>.\config.h
    echo #undef SSize_t>>.\config.h
    echo #undef HAS_ATOLL>>.\config.h
    echo #undef HAS_STRTOLL>>.\config.h
    echo #undef HAS_STRTOULL>>.\config.h
    echo #undef IVTYPE>>.\config.h
    echo #undef UVTYPE>>.\config.h
    echo #undef IVSIZE>>.\config.h
    echo #undef UVSIZE>>.\config.h
    echo #undef NV_PRESERVES_UV>>.\config.h
    echo #undef NV_PRESERVES_UV_BITS>>.\config.h
    echo #undef IVdf>>.\config.h
    echo #undef UVuf>>.\config.h
    echo #undef UVof>>.\config.h
    echo #undef UVxf>>.\config.h
    echo #undef UVXf>>.\config.h
    echo #undef USE_64_BIT_INT>>.\config.h
    echo #undef Size_t_size>>.\config.h
    echo #define Off_t __int64>>.\config.h
    echo #define LSEEKSIZE ^8>>.\config.h
    echo #define Off_t_size ^8>>.\config.h
    echo #define PTRSIZE ^8>>.\config.h
    echo #define SSize_t __int64>>.\config.h
    echo #define HAS_ATOLL>>.\config.h
    echo #define HAS_STRTOLL>>.\config.h
    echo #define HAS_STRTOULL>>.\config.h
    echo #define Size_t_size ^8>>.\config.h
    echo #define IVTYPE __int64>>.\config.h
    echo #define UVTYPE unsigned __int64>>.\config.h
    echo #define IVSIZE ^8>>.\config.h
    echo #define UVSIZE ^8>>.\config.h
    echo #undef NV_PRESERVES_UV>>.\config.h
    echo #define NV_PRESERVES_UV_BITS 53>>.\config.h
    echo #define IVdf "I64d">>.\config.h
    echo #define UVuf "I64u">>.\config.h
    echo #define UVof "I64o">>.\config.h
    echo #define UVxf "I64x">>.\config.h
    echo #define UVXf "I64X">>.\config.h
    echo #define USE_64_BIT_INT>>.\config.h
    echo #endif>>.\config.h
    if not exist ".\mini" mkdir ".\mini"
    link -subsystem:console -out:..\generate_uudmap.exe @C:\DOCUME~1\ADMINI~1\LOCALS~1\Temp\nmD65.tmp
    if exist ..\generate_uudmap.exe.manifest mt -nologo -manifest ..\generate_uudmap.exe.manifest -outputresource:..\generate_uudmap.exe;1 &&  if exist ..\generate_uudmap.exe.manifest del ..\generate_uudmap.exe.manifest
    ..\generate_uudmap.exe ..\uudmap.h ..\bitcount.h ..\mg_data.h
    cl -c -nologo -GF -W3 -I..\lib\CORE -I.\include -I. -I.. -DWIN32 -D_CONSOLE -DNO_STRICT -DWIN64 -DCONSERVATIVE -D_CRT_SECURE_NO_DEPRECATE -D_CRT_NONSTDC_NO_DEPRECATE -DPERLDLL -DPERL_CORE   -O1 -MD -Zi -DDEBUGGING -fp:precise -DPERL_EXTERNAL_GLOB -DPERL_IS_MINIPERL -Fo.\mini\miniperlmain.obj ..\miniperlmain.c
    cl -c -nologo -GF -W3 -I..\lib\CORE -I.\include -I. -I.. -DWIN32 -D_CONSOLE -DNO_STRICT -DWIN64 -DCONSERVATIVE -D_CRT_SECURE_NO_DEPRECATE -D_CRT_NONSTDC_NO_DEPRECATE -DPERLDLL -DPERL_CORE   -O1 -MD -Zi -DDEBUGGING -fp:precise -DPERL_IS_MINIPERL -Fo.\mini\win32sck.obj win32sck.c
    cl -c -nologo -GF -W3 -I..\lib\CORE -I.\include -I. -I.. -DWIN32 -D_CONSOLE -DNO_STRICT -DWIN64 -DCONSERVATIVE -D_CRT_SECURE_NO_DEPRECATE -D_CRT_NONSTDC_NO_DEPRECATE -DPERLDLL -DPERL_CORE   -O1 -MD -Zi -DDEBUGGING -fp:precise -DPERL_IS_MINIPERL -Fo.\mini\win32thread.obj win32thread.c
    cl -c -nologo -GF -W3 -I..\lib\CORE -I.\include -I. -I.. -DWIN32 -D_CONSOLE -DNO_STRICT -DWIN64 -DCONSERVATIVE -D_CRT_SECURE_NO_DEPRECATE -D_CRT_NONSTDC_NO_DEPRECATE -DPERLDLL -DPERL_CORE   -O1 -MD -Zi -DDEBUGGING -fp:precise -DPERL_IS_MINIPERL -Fo.\mini\fcrypt.obj fcrypt.c
    cl -c -nologo -GF -W3 -I..\lib\CORE -I.\include -I. -I.. -DWIN32 -D_CONSOLE -DNO_STRICT -DWIN64 -DCONSERVATIVE -D_CRT_SECURE_NO_DEPRECATE -D_CRT_NONSTDC_NO_DEPRECATE -DPERLDLL -DPERL_CORE   -O1 -MD -Zi -DDEBUGGING -fp:precise -DPERL_IS_MINIPERL -Fo.\mini\win32io.obj win32io.c
    link -subsystem:console -out:..\miniperl.exe @C:\DOCUME~1\ADMINI~1\LOCALS~1\Temp\nmD66.tmp

[Please do not change anything below this line]
Site configuration information for perl 5.21.6:

Configured by Administrator at Sat Oct 25 06:12:11 2014.

Summary of my perl5 (revision 5 version 21 subversion 6) configuration:
  Derived from: 
    osname=MSWin32, osvers=5.2, archname=MSWin32-x86-multi-thread
    hint=recommended, useposix=true, d_sigaction=undef
    useithreads=define, usemultiplicity=define
    use64bitint=undef, use64bitall=undef, uselongdouble=undef
    usemymalloc=n, bincompat5005=undef
    optimize='-O1 -MD -Zi -DNDEBUG -GS- -GL',
    ccversion='15.00.30729.01', gccversion='', gccosandvers=''
    intsize=4, longsize=4, ptrsize=4, doublesize=8, byteorder=1234
    d_longlong=undef, longlongsize=8, d_longdbl=define, longdblsize=8, longdblkind=0
    ivtype='long', ivsize=4, nvtype='double', nvsize=8, Off_t='__int64', lseeksize=8
    alignbytes=8, prototype=define
  Linker and Libraries:
    ld='link', ldflags ='-nologo -nodefaultlib -debug -opt:ref,icf -ltcg  -libpath:"c:\p521\lib\CORE"  -machine:x86 "/manifestdependency:type='Win32' name='Microsoft.Windows.Common-Controls' version='' processorArchitecture='*' publicKeyToken='6595b64144ccf1df' language='*'"'
    libs=oldnames.lib kernel32.lib user32.lib gdi32.lib winspool.lib  comdlg32.lib advapi32.lib shell32.lib ole32.lib oleaut32.lib  netapi32.lib uuid.lib ws2_32.lib mpr.lib winmm.lib  version.lib odbc32.lib odbccp32.lib comctl32.lib msvcrt.lib
    perllibs=oldnames.lib kernel32.lib user32.lib gdi32.lib winspool.lib  comdlg32.lib advapi32.lib shell32.lib ole32.lib oleaut32.lib  netapi32.lib uuid.lib ws2_32.lib mpr.lib winmm.lib  version.lib odbc32.lib odbccp32.lib comctl32.lib msvcrt.lib
    libc=msvcrt.lib, so=dll, useshrplib=true, libperl=perl521.lib
  Dynamic Linking:
    dlsrc=dl_win32.xs, dlext=dll, d_dlsymun=undef, ccdlflags=' '
    cccdlflags=' ', lddlflags='-dll -nologo -nodefaultlib -debug -opt:ref,icf -ltcg  -libpath:"c:\p521\lib\CORE"  -machine:x86 "/manifestdependency:type='Win32' name='Microsoft.Windows.Common-Controls' version='' processorArchitecture='*' publicKeyToken='6595b64144ccf1df' language='*'"'

Locally applied patches:

@INC for perl 5.21.6:

Environment for perl 5.21.6:
    HOME (unset)
    LANG (unset)
    LANGUAGE (unset)
    LOGDIR (unset)
    PATH=C:\p521\bin;C:\WINDOWS\system32;C:\Perl\site\bin;C:\Perl\bin;C:\WINDOWS;C:\WINDOWS\System32\Wbem;C:\Program Files (x86)\Microsoft Visual Studio 9.0\VC\bin;C:\Program Files (x86)\Microsoft Visual Studio 9.0\VC;C:\Program Files\TortoiseGit\bin;
    PERL_BADLANG (unset)
    SHELL (unset)


Thread Previous Perl Programming lists via nntp and http.
Comments to Ask Bjørn Hansen at | Group listing | About