develooper Front page | perl.perl5.porters | Postings from July 2016

[perl #128564] increase parallelization for GNU make builds on Win32

bulk88 via RT
July 16, 2016 03:40
[perl #128564] increase parallelization for GNU make builds on Win32
Message ID:
On Wed Jul 06 21:58:22 2016, tonyc wrote:
> The attached patch improves parallization of perl build on Win32 when
> using GNU make.
> It builds on top of my patch in 128438.
> Build times, before and after:
> gmake -j6   :  219  128
> gmake -j2   :  266  232
> gmake       :  321  346
> Single job builds take 8% longer, but parallel builds improve
> significantly.
> Tony

Use $(PLMAKE) not $(MAKE), otherwise a "gmake -n" is impossible to use for debugging.

Math/BigInt/FastCalc on List-Utils dep needs to be removed in Unix makefile and in , Im refering to the IRC discussion with you.

You have target as : $(CONFIGPM) $(HAVEMINIPERL) $(HAVE_COREDIR)

it can be simplified to just : $(CONFIGPM)

$(CONFIGPM) requires miniperl already. C headers only ("$(HAVE_COREDIR)") make sense only for XS code. Extensions_nonxs target in theory can be started before/in parallel with COREDIR copying over.

There is also this patch which competes with this patch, since your patch *claims to* increase total CPU but also increases parallelism, that patch in decreases CPU time. I havent benchmarked your patch but I did build it.

Your GNUMakefile patch can't be used with dmake since dmake procs dont have a "job server" and dont use IPC to negotiate between different gmake procs how many child procs are launched. I see you didnt try to make the dmake makefile do anything. I am curious why serial gmake build increases CPU usage according to your claims. 

Even though this would decrease parallelism opportunity, the lower CPU might pay off by running 1 gmake proc instead of 4 of them, IDK, only benchmarking can tell.

	$(PLMAKE) -j8 -f Extensions_nonxs Extensions_static Extensions_normalize Extensions

The dep list for the mega-target would have to be longer though, since Extensions is the most complicated target in its deps and Extensions_nonxs is the simplest (least deps). Maybe needs to be NYTProfed and optimized so it, I am guessing without reading the code, does 1 pass over the master module list and sorts it into 3-4 categories, instead of generating master list 4 times then filtering it.

bulk88 ~ bulk88 at

via perlbug:  queue: perl5 status: new Perl Programming lists via nntp and http.
Comments to Ask Bjørn Hansen at | Group listing | About