develooper Front page | perl.perl5.porters | Postings from December 2014

[perl #123482] parallel building of core .c files on VC

Thread Previous | Thread Next
From:
bulk88 via RT
Date:
December 23, 2014 02:43
Subject:
[perl #123482] parallel building of core .c files on VC
Message ID:
rt-4.0.18-7873-1419302625-790.123482-15-0@perl.org
More thoughts.

It seems the tool I tried a long time ago was not jmake or cmake but "jom" http://qt-project.org/wiki/jom . The specific VC error is "fatal error C1033: cannot open program database" http://msdn.microsoft.com/en-us/library/4ks5fw35.aspx . This internet post https://social.msdn.microsoft.com/Forums/vstudio/en-US/a722c6b6-53ab-41a4-b6c1-5cb02b2623d4/visual-studio-2005-professional-mspdbsrv-failure-sporadic-fatal-error-c1033-cannot-open-program?forum=vcgeneral says to fix the conflict use -Fd http://msdn.microsoft.com/en-us/library/9wst99a9%28v=vs.71%29.aspx . This is interesting since it means VC 6 and VC 2003 can be parallel without MS support, and those 2 compilers are as fast as a mouse click in compiling C code (VC 2003 is the perfect blend of speed of compiling, C lang support, and good machine code emission, for me)

Another issue is that the various .c file take different times to build. Building takes about 20 seconds on my 8 core Win64 machine with VC 2008 -O1 and DEBUGGING/asserts on but half (10 sec) of those 20 seconds are spent <= 50% cpu on all 8 cores (1 thread can max out at 12.5%), so the parallelism isn't very great. It is 1:30 to 2:00 minutes without any parallelism (current way).

    C:\perl\bin\perl.exe -E" @a = qw(  ..\av.c  ..\caretx.c  ..\deb.c  ..\doio.c  ..\doop.c  ..\dump.c  ..\globals.c  ..\gv.c  ..\mro.c  ..\hv.c  ..\locale.c  ..\keywords.c  ..\mathoms.c  ..\mg.c  ..\numeric.c  ..\op.c  ..\pad.c  ..\perl.c  ..\perlapi.c  ..\perly.c  ..\pp.c  ..\pp_ctl.c  ..\pp_hot.c  ..\pp_pack.c  ..\pp_sort.c  ..\pp_sys.c  ..\reentr.c  ..\regcomp.c  ..\regexec.c   ..\run.c  ..\scope.c  ..\sv.c  ..\taint.c  ..\toke.c  ..\universal.c  ..\utf8.c  ..\util.c  ..\miniperlmain.c  ..\perlio.c win32.c win32sck.c win32thread.c fcrypt.c win32io.c); foreach(@a) {my $t = time; system('cl -c -nologo -GF -W3 -I..\lib\CORE -I.\include -I. -I.. -DWIN32 -D_CONSOLE -DNO_STRICT -DWIN64 -DCONSERVATIVE -D_CRT_SECURE_NO_DEPRECATE -D_CRT_NONSTDC_NO_DEPRECATE -DPERLDLL -DPERL_CORE   -O1 -MD -Zi -DDEBUGGING -fp:precise -DPERL_EXTERNAL_GLOB -DPERL_IS_MINIPERL   -Fomini '.$_.' 1>NUL'); say $_.' '.(time-$t);}"

There are 3 very long to compile files.

They specifically are 

..\regcomp.c 11
..\regexec.c 9
..\toke.c 15

raw data:

..\av.c 1
..\caretx.c 0
..\deb.c 1
..\doio.c 1
..\doop.c 1
..\dump.c 1
..\globals.c 1
..\gv.c 2
..\mro.c 1
..\hv.c 1
..\locale.c 0
..\keywords.c 2
..\mathoms.c 1
..\mg.c 1
..\numeric.c 1
..\op.c 3
..\pad.c 1
..\perl.c 1
..\perlapi.c 1
..\perly.c 1
..\pp.c 2
..\pp_ctl.c 3
..\pp_hot.c 1
..\pp_pack.c 4
..\pp_sort.c 1
..\pp_sys.c 1
..\reentr.c 1
..\regcomp.c 11
..\regexec.c 9
..\run.c 0
..\scope.c 1
..\sv.c 4
..\taint.c 1
..\toke.c 15
..\universal.c 1
..\utf8.c 1
..\util.c 1
..\miniperlmain.c 1
..\perlio.c 1
win32.c 1
win32sck.c 1
win32thread.c 0
fcrypt.c 0
win32io.c 1

So that means unless someone will split ..\regcomp.c ..\regexec.c ..\toke.c into more .c files, those 3 hold up every other .c file. I looked specifically in toke.c, Perl_yylex is the largest function inside, but there are a ton of S_ static functions, so to split toke.c means creating static by use by not static by C language functions.

Top 5 largest by machine code from toke.c (big to small)
Perl_yylex
S_scan_const
S_scan_str
S_scan_heredoc
S_scan_ident

Top 8 largest by machine code from regcomp.c (big to small)
S_study_chunk
S_reg
S_make_trie
S_regclass
S_regatom
S_join_exact
S_dumpuntil
Perl_regprop

Top 5 largest by machine code from regexec.c (big to small)
S_find_byclass
Perl_re_intuit_start
S_regrepeat
S_setup_EXACTISH_ST_c1_c2
S_reg_set_capture_string

With the pigs moved to the front of the cmd line

C:\p521\srcnew\win32>    C:\perl\bin\perl.exe -E" say time" && cl -c -nologo -GF
 -W3 -I..\lib\CORE -I.\include -I. -I.. -DWIN32 -D_CONSOLE -DNO_STRICT -DWIN64 -
DCONSERVATIVE -D_CRT_SECURE_NO_DEPRECATE -D_CRT_NONSTDC_NO_DEPRECATE -DPERLDLL -
DPERL_CORE   -O1 -MD -Zi -DDEBUGGING -fp:precise -DPERL_EXTERNAL_GLOB -DPERL_IS_
MINIPERL  -MP8  -Fomini\ ..\regcomp.c  ..\regexec.c  ..\toke.c ..\av.c  ..\caret
x.c  ..\deb.c  ..\doio.c  ..\doop.c  ..\dump.c  ..\globals.c  ..\gv.c  ..\mro.c
 ..\hv.c  ..\locale.c  ..\keywords.c  ..\mathoms.c  ..\mg.c  ..\numeric.c  ..\op
.c  ..\pad.c  ..\perl.c  ..\perlapi.c  ..\perly.c  ..\pp.c  ..\pp_ctl.c  ..\pp_h
ot.c  ..\pp_pack.c  ..\pp_sort.c  ..\pp_sys.c  ..\reentr.c    ..\run.c  ..\scope
.c  ..\sv.c  ..\taint.c    ..\universal.c  ..\utf8.c  ..\util.c  ..\miniperlmain
.c  ..\perlio.c win32.c win32sck.c win32thread.c fcrypt.c win32io.c 1>NUL && C:\
perl\bin\perl.exe -E" say time"
1419301655
1419301673

18 seconds

C:\p521\srcnew\win32>    C:\perl\bin\perl.exe -E" say time" && cl -c -nologo -GF
 -W3 -I..\lib\CORE -I.\include -I. -I.. -DWIN32 -D_CONSOLE -DNO_STRICT -DWIN64 -
DCONSERVATIVE -D_CRT_SECURE_NO_DEPRECATE -D_CRT_NONSTDC_NO_DEPRECATE -DPERLDLL -
DPERL_CORE   -O1 -MD -Zi -DDEBUGGING -fp:precise -DPERL_EXTERNAL_GLOB -DPERL_IS_
MINIPERL  -MP8  -Fomini\ ..\regcomp.c  ..\regexec.c  ..\toke.c ..\av.c  ..\caret
x.c  ..\deb.c  ..\doio.c  ..\doop.c  ..\dump.c  ..\globals.c  ..\gv.c  ..\mro.c
 ..\hv.c  ..\locale.c  ..\keywords.c  ..\mathoms.c  ..\mg.c  ..\numeric.c  ..\op
.c  ..\pad.c  ..\perl.c  ..\perlapi.c  ..\perly.c  ..\pp.c  ..\pp_ctl.c  ..\pp_h
ot.c  ..\pp_pack.c  ..\pp_sort.c  ..\pp_sys.c  ..\reentr.c    ..\run.c  ..\scope
.c  ..\sv.c  ..\taint.c    ..\universal.c  ..\utf8.c  ..\util.c  ..\miniperlmain
.c  ..\perlio.c win32.c win32sck.c win32thread.c fcrypt.c win32io.c 1>NUL && C:\
perl\bin\perl.exe -E" say time"
1419301928
1419301947

19 seconds


Now putting the fat ones on the end of the cmd line

C:\p521\srcnew\win32>    C:\perl\bin\perl.exe -E" say time" && cl -c -nologo -GF
 -W3 -I..\lib\CORE -I.\include -I. -I.. -DWIN32 -D_CONSOLE -DNO_STRICT -DWIN64 -
DCONSERVATIVE -D_CRT_SECURE_NO_DEPRECATE -D_CRT_NONSTDC_NO_DEPRECATE -DPERLDLL -
DPERL_CORE   -O1 -MD -Zi -DDEBUGGING -fp:precise -DPERL_EXTERNAL_GLOB -DPERL_IS_
MINIPERL  -MP8  -Fomini\   ..\av.c  ..\caretx.c  ..\deb.c  ..\doio.c  ..\doop.c
 ..\dump.c  ..\globals.c  ..\gv.c  ..\mro.c  ..\hv.c  ..\locale.c  ..\keywords.c
  ..\mathoms.c  ..\mg.c  ..\numeric.c  ..\op.c  ..\pad.c  ..\perl.c  ..\perlapi.
c  ..\perly.c  ..\pp.c  ..\pp_ctl.c  ..\pp_hot.c  ..\pp_pack.c  ..\pp_sort.c  ..
\pp_sys.c  ..\reentr.c     ..\run.c  ..\scope.c  ..\sv.c  ..\taint.c    ..\unive
rsal.c  ..\utf8.c  ..\util.c  ..\miniperlmain.c  ..\perlio.c win32.c win32sck.c
win32thread.c fcrypt.c win32io.c ..\toke.c ..\regcomp.c  ..\regexec.c 1>NUL && C
:\perl\bin\perl.exe -E" say time"
1419301793
1419301816

23 seconds

C:\p521\srcnew\win32>    C:\perl\bin\perl.exe -E" say time" && cl -c -nologo -GF
 -W3 -I..\lib\CORE -I.\include -I. -I.. -DWIN32 -D_CONSOLE -DNO_STRICT -DWIN64 -
DCONSERVATIVE -D_CRT_SECURE_NO_DEPRECATE -D_CRT_NONSTDC_NO_DEPRECATE -DPERLDLL -
DPERL_CORE   -O1 -MD -Zi -DDEBUGGING -fp:precise -DPERL_EXTERNAL_GLOB -DPERL_IS_
MINIPERL  -MP8  -Fomini\   ..\av.c  ..\caretx.c  ..\deb.c  ..\doio.c  ..\doop.c
 ..\dump.c  ..\globals.c  ..\gv.c  ..\mro.c  ..\hv.c  ..\locale.c  ..\keywords.c
  ..\mathoms.c  ..\mg.c  ..\numeric.c  ..\op.c  ..\pad.c  ..\perl.c  ..\perlapi.
c  ..\perly.c  ..\pp.c  ..\pp_ctl.c  ..\pp_hot.c  ..\pp_pack.c  ..\pp_sort.c  ..
\pp_sys.c  ..\reentr.c     ..\run.c  ..\scope.c  ..\sv.c  ..\taint.c    ..\unive
rsal.c  ..\utf8.c  ..\util.c  ..\miniperlmain.c  ..\perlio.c win32.c win32sck.c
win32thread.c fcrypt.c win32io.c ..\toke.c ..\regcomp.c  ..\regexec.c 1>NUL && C
:\perl\bin\perl.exe -E" say time"
1419301868
1419301892

24 seconds


Further optimizing the list of .c files by compile time yields no benefit (sv.c, op.c and pp_ctl.c)

C:\p521\srcnew\win32>    C:\perl\bin\perl.exe -E" say time" && cl -c -nologo -GF
 -W3 -I..\lib\CORE -I.\include -I. -I.. -DWIN32 -D_CONSOLE -DNO_STRICT -DWIN64 -
DCONSERVATIVE -D_CRT_SECURE_NO_DEPRECATE -D_CRT_NONSTDC_NO_DEPRECATE -DPERLDLL -
DPERL_CORE   -O1 -MD -Zi -DDEBUGGING -fp:precise -DPERL_EXTERNAL_GLOB -DPERL_IS_
MINIPERL  -MP8  -Fomini\ ..\regcomp.c  ..\regexec.c  ..\toke.c ..\sv.c ..\pp_pac
k.c  ..\pp_ctl.c  ..\op.c  ..\av.c  ..\caretx.c  ..\deb.c  ..\doio.c  ..\doop.c
 ..\dump.c  ..\globals.c  ..\gv.c  ..\mro.c  ..\hv.c  ..\locale.c  ..\keywords.c
  ..\mathoms.c  ..\mg.c  ..\numeric.c  ..\op.c  ..\pad.c  ..\perl.c  ..\perlapi.
c  ..\perly.c  ..\pp.c    ..\pp_hot.c  ..\pp_pack.c  ..\pp_sort.c  ..\pp_sys.c
..\reentr.c    ..\run.c  ..\scope.c    ..\taint.c    ..\universal.c  ..\utf8.c
..\util.c  ..\miniperlmain.c  ..\perlio.c win32.c win32sck.c win32thread.c fcryp
t.c win32io.c 1>NUL && C:\perl\bin\perl.exe -E" say time"
1419302166
1419302185

19 seconds

C:\p521\srcnew\win32>    C:\perl\bin\perl.exe -E" say time" && cl -c -nologo -GF
 -W3 -I..\lib\CORE -I.\include -I. -I.. -DWIN32 -D_CONSOLE -DNO_STRICT -DWIN64 -
DCONSERVATIVE -D_CRT_SECURE_NO_DEPRECATE -D_CRT_NONSTDC_NO_DEPRECATE -DPERLDLL -
DPERL_CORE   -O1 -MD -Zi -DDEBUGGING -fp:precise -DPERL_EXTERNAL_GLOB -DPERL_IS_
MINIPERL  -MP8  -Fomini\ ..\regcomp.c  ..\regexec.c  ..\toke.c ..\sv.c ..\pp_pac
k.c  ..\pp_ctl.c  ..\op.c  ..\av.c  ..\caretx.c  ..\deb.c  ..\doio.c  ..\doop.c
 ..\dump.c  ..\globals.c  ..\gv.c  ..\mro.c  ..\hv.c  ..\locale.c  ..\keywords.c
  ..\mathoms.c  ..\mg.c  ..\numeric.c  ..\op.c  ..\pad.c  ..\perl.c  ..\perlapi.
c  ..\perly.c  ..\pp.c    ..\pp_hot.c  ..\pp_pack.c  ..\pp_sort.c  ..\pp_sys.c
..\reentr.c    ..\run.c  ..\scope.c    ..\taint.c    ..\universal.c  ..\utf8.c
..\util.c  ..\miniperlmain.c  ..\perlio.c win32.c win32sck.c win32thread.c fcryp
t.c win32io.c 1>NUL && C:\perl\bin\perl.exe -E" say time"
1419302206
1419302225

19 seconds

-- 
bulk88 ~ bulk88 at hotmail.com

---
via perlbug:  queue: perl5 status: new
https://rt.perl.org/Ticket/Display.html?id=123482

Thread Previous | Thread Next


nntp.perl.org: Perl Programming lists via nntp and http.
Comments to Ask Bjørn Hansen at ask@perl.org | Group listing | About