develooper Front page | perl.perl5.porters | Postings from July 2016

[perl #128564] increase parallelization for GNU make builds on Win32

From:
bulk88 via RT
Date:
July 16, 2016 03:40
Subject:
[perl #128564] increase parallelization for GNU make builds on Win32
Message ID:
rt-4.0.18-4526-1468640414-1810.128564-15-0@perl.org
On Wed Jul 06 21:58:22 2016, tonyc wrote:
> The attached patch improves parallization of perl build on Win32 when
> using GNU make.
> 
> It builds on top of my patch in 128438.
> 
> Build times, before and after:
> 
> gmake -j6   :  219  128
> gmake -j2   :  266  232
> gmake       :  321  346
> 
> Single job builds take 8% longer, but parallel builds improve
> significantly.
> 
> Tony

Use $(PLMAKE) not $(MAKE), otherwise a "gmake -n" is impossible to use for debugging.

Math/BigInt/FastCalc on List-Utils dep needs to be removed in Unix makefile and in , Im refering to the IRC discussion with you.

You have exts.mk target as

exts.mk : $(CONFIGPM) FindExt.pm $(HAVEMINIPERL) mkexts.pl $(HAVE_COREDIR)

it can be simplified to just

exts.mk : FindExt.pm mkexts.pl $(CONFIGPM)

$(CONFIGPM) requires miniperl already. C headers only ("$(HAVE_COREDIR)") make sense only for XS code. Extensions_nonxs target in theory can be started before/in parallel with COREDIR copying over.

There is also this patch https://rt.perl.org/Ticket/Display.html?id=126686 which competes with this patch, since your patch *claims to* increase total CPU but also increases parallelism, that patch in https://rt.perl.org/Ticket/Display.html?id=126686 decreases CPU time. I havent benchmarked your patch but I did build it.

Your GNUMakefile patch can't be used with dmake since dmake procs dont have a "job server" and dont use IPC to negotiate between different gmake procs how many child procs are launched. I see you didnt try to make the dmake makefile do anything. I am curious why serial gmake build increases CPU usage according to your claims. 

Even though this would decrease parallelism opportunity, the lower CPU might pay off by running 1 gmake proc instead of 4 of them, IDK, only benchmarking can tell.

	$(PLMAKE) -j8 -f exts.mk Extensions_nonxs Extensions_static Extensions_normalize Extensions

The dep list for the mega-target would have to be longer though, since Extensions is the most complicated target in its deps and Extensions_nonxs is the simplest (least deps). Maybe FindExt.pm needs to be NYTProfed and optimized so it, I am guessing without reading the code, does 1 pass over the master module list and sorts it into 3-4 categories, instead of generating master list 4 times then filtering it.

-- 
bulk88 ~ bulk88 at hotmail.com

---
via perlbug:  queue: perl5 status: new
https://rt.perl.org/Ticket/Display.html?id=128564



nntp.perl.org: Perl Programming lists via nntp and http.
Comments to Ask Bjørn Hansen at ask@perl.org | Group listing | About