develooper Front page | perl.perl5.porters | Postings from August 2012

[RFC-PATCH 0/3] automatic -Dmksymlinks with incremental action

Thread Next
Jim Cromie
August 3, 2012 14:22
[RFC-PATCH 0/3] automatic -Dmksymlinks with incremental action
Message ID:
For past year, Ive been keeping a set of build trees of various
configs, much like a smoker setup, except my trees are persistent - an
orchard vs a forest.

[jimc@groucho perl]$ ls -d ../build-*
../build-dbg    ../build-do0    ../build-slabdbg      ../build-thrdbg
../build-dg3    ../build-do1    ../build-thr
../build-dg3o0  ../build-plain  ../build-thrclang
../build-dg3o1  ../build-slab   ../build-thrclangdbg

Back then, I worked up several patches to help maintain the orchard.
They've been mostly collecting dust, except for the script which adds
new symlinks to build-tree when a new file is added to src-tree.  I
use it regularly, via a shell function;

mklink () {
    ( [ -f MANIFEST ] && cd ..;
	for d in build-*; do
	    (cd $d; echo $d; perl ../

Anyway, I've just dusted off the -Dmksymlink patches and fixed some
bugs and usage problems in make and configure parts.

I want to see if they could be made suitable for inclusion.  They
include patches against Configure, which isnt ideal, but Ive had
trouble in the past trying to use the config-gen, so I concentrated on
getting the Configure file to do what I wanted.

the attached 3 patches:

1- adds Porting/, alters Makefile to invoke it.

script looks at MANIFEST, if its a symlink we assume we're in a
build-tree previously set up with -Dmksyumlinks, so opens MANIFEST,
and adds any missing symlinks to the files named in MANIFEST.

Ive used the script manually for the past year to maintain an orchard
of build trees.

The makefile and configure parts are far less tested.

The Makefile addition creates a catch-22 situation (solved by patch 3)

[jimc@groucho build-slab]$ make
perl Porting/
Can't open perl script "Porting/": No such file or directory
make: *** [mksymlinks] Error 2

That could be fixed, but it feels a little grody to do so in the makefile
[ -f Porting/ ] && perl Porting/
or finding the srcdir from inside the Makefile, and invoking it with relative path.

So I added patch 3 instead.

Note also that the script could replace the Configure-unit which does
": Make symlinks util" but theres potential bootstrapping issues with
doing so.

2- alters Configure to add -Dmksymlinks to its invocation when appropriate.

The new "proto-unit" looks at $0 to see how it was invoked: anything
other than ./Configure or Configure sets mksymlinks=true.  This isnt
always correct - sh $PWD/Configure will fool it, but it is harmless,
make && make test complete, despite the incorrect mksymlinks value.

Again, patch 3 is needed to fix the same catch-22 as above:

lrwxrwxrwx. 1 jimc jimc 44 Jul 20 14:57 Configure -> /home/jimc/projects/perl-core/perl/Configure
[jimc@groucho build-plain]$ sh ./Configure -des -Dusedevel
src is 
First let's make sure your kit is complete.  Checking...
ls: cannot access Porting/ No such file or directory


You have the option of continuing the configuration process, despite the
distinct possibility that your kit is damaged, by typing 'y'es.  If you
do, don't blame me if something goes wrong.  I advise you to type 'n'o
and contact the author (

manually invoking the added script fixes it...

[jimc@groucho build-plain]$ perl ../perl/Porting/ 
adding symlink for Porting/
./Porting/ is now a symlink

thereafter, Configure -des -Dusedevel && make works.

3 - patch moves the package-complete unit to after the mksymlinks unit.
this fixes the catch-22s mentioned above

This is a 1st step towards keeping an orchard of build trees, ie
persistent dirs that rarely require a full git clean -dxf ; Configure
; make ; make test cycle.

there are several circumstances that do still require a full rebuild:

- moving subdirs - from ext to dist forex - these arent happening
  often these days.  these manifest as cant make foo, or cant make
  foo/Makefile also possibly dangling symlinks (though patch doesnt
  need to do anything here, theyre either handled elsewhere, or are
  only a trasient thing (I cant recall))

- new configure items get added (these error early in the make, iirc
  are easily diagnosed)

- probably other things too.

Advantages of orchards:

- incremental rebuilds - changes other than regens will often only
  require 1 object recompile (doesnt help to avoid subdir remakes, but
  it couldnt anyway)

- parallel builds
  on my 4 core AMD, I typically run make -j4 test_harness on 4 trees
  simultaneously this gives faster testing (and saturates load) (with
  8gb, I average 60% mem, 0% swap)

[jimc@groucho build-slab]$ ps -efH |grep perl | cut -c50-
     /usr/bin/perl ./ -vsj4 -- test_harness
       sh -c mkdir -p build-thrdbg && cd build-thrdbg && HARNESS_TIMER=/home/jimc/projects/perl-core/timings time make test_harness  > /home/jimc/projects/perl-core/smoke-runs/make-mksymlinks-23720-thrdbg 2>&1
               ./perl harness
                 ./perl -I.. -MTestInit io/through.t
                 ./perl -I.. -MTestInit io/crlf_through.t
                   /home/jimc/projects/perl-core/build-thrdbg/perl -we binmode STDOUT, ':crlf';eval qq(sub w {syswrite STDOUT, \x24_} 1) or die; w() for grep length, split /(.{1,1})/s, qq(x\n\n yy\n\nz\n\n)
                 ./perl -I.. -MTestInit io/eintr.t
                 ./perl -I.. -MTestInit re/regexp_qr_embed_thr.t
       sh -c mkdir -p build-thrclangdbg && cd build-thrclangdbg && HARNESS_TIMER=/home/jimc/projects/perl-core/timings time make test_harness  > /home/jimc/projects/perl-core/smoke-runs/make-mksymlinks-23720-thrclangdbg 2>&1
               ./perl harness
                 ./perl -I.. -MTestInit re/regexp_qr_embed_thr.t
                 ./perl -I.. -MTestInit porting/podcheck.t
       sh -c mkdir -p build-slabdbg && cd build-slabdbg && HARNESS_TIMER=/home/jimc/projects/perl-core/timings time make test_harness  > /home/jimc/projects/perl-core/smoke-runs/make-mksymlinks-23720-slabdbg 2>&1
               ./perl harness
                 ./perl -I.. -MTestInit io/through.t
                 ./perl -I.. -MTestInit io/crlf_through.t
                 ./perl -I.. -MTestInit io/inplace.t
                 ./perl -I.. -MTestInit io/fflush.t
       sh -c mkdir -p build-slab && cd build-slab && HARNESS_TIMER=/home/jimc/projects/perl-core/timings time make test_harness  > /home/jimc/projects/perl-core/smoke-runs/make-mksymlinks-23720-slab 2>&1
               ./perl harness
                 ./perl -I.. -MTestInit re/uniprops.t
                 ./perl -I.. -MTestInit re/pat_psycho.t
                 ./perl -I.. -MTestInit re/fold_grind.t
                 ./perl -I.. -MTestInit re/reg_eval_scope.t

times for a full rebuild

finished: elapsed:3239 usr:0.09 sys:0.04 chuser:10135.47 chsys:1391.23

times for a retest

finished: elapsed:2437 usr:0.06 sys:0.03 chuser:6905.53 chsys:942.67

Thread Next Perl Programming lists via nntp and http.
Comments to Ask Bjørn Hansen at | Group listing | About