NWCLARK TPF grant report #34

[Hours]		[Activity]
2012/04/23	Monday
 1.00		ID 20020513.013 (#9319)
 3.25		RT #108286
=====
 4.25

2012/04/24	Tuesday
 0.25		RT #112536
 8.50		build process [autodoc, perlmodlib, $(Icwd)]
 0.25		installman
 0.75		reading/responding to list mail
 0.25		the todo list
=====
10.00

2012/04/25	Wednesday
 5.25		Makefile-norecurse
 5.25		installman
=====
10.50

2012/04/26	Thursday
 0.75		Makefile-norecurse
 4.50		minitest cleaner
=====
 5.25

2012/04/27	Friday
 1.50		AIX make bug
 0.25		ID 20010801.037 (#7425)
 2.00		LDLIBPTH
 1.00		Makefile-norecurse
 2.00		minitest cleaner
 0.75		reading/responding to list mail
=====
 7.50

2012/04/28	Saturday
 1.00		AIX make bug (xlc -DDEBUGGING segv)
 0.25		LDLIBPTH
 3.25		Makefile.SH miniperl rule unification
=====
 4.50

2012/04/29	Sunday
 1.25		AIX make bug (ccache)
 0.75		AIX make bug (https://bugzilla.samba.org/show_bug.cgi?id=8906)
 0.75		AIX make bug (xlc -DDEBUGGING segv)
 0.25		LDLIBPTH
 0.25		RT #108286
 0.50		Solaris bisect.
 0.25		Solaris bisect. (-xO3)
=====
 4.00

Which I calculate is 46.00 hours

The connecting theme for the week was working to simplify the generated
Makefile, by removing duplication and self-recursion.

On platforms that run Configure (so Linux, Unix, Cygwin and OS/2), the
Makefile is extracted from a shell script Makefile.SH. Most of it is written
out verbatim, but there is some shell logic to generate OS or configuration
specific code.

In turn that Makefile is used by makedepend to generate a second Makefile
containing dependency information (named makefile on most systems, and
GNUMakefile on OS X - both names chosen so that the make utility picks the
newer makefile in preference).

The makefile compiles all the object files and links them into miniperl,
which is pretty much the real perl, only without any form of dynamic linking.
Between 5.005 and 5.6.0 things got a bit more complex, because perl switched
to using File::Glob to implement globbing [using the BSD glob code, written
by that well-known Perl contributor Guido van Rossum :-), safely quarantined
in its own file for licensing safety]. As File::Glob needs miniperl to
build, and miniperl isn't built yet, to solve the bootstrapping problem
op.c is conditionally compiled as opmini.o, and that instead calls the old
"shell out to csh" globbing code. This need to link with opmini.o instead
of op.o becomes relevant later.

miniperl bootstraps the rest of the build, and is used as much as possible
to avoid writing any subsequent build system 3 (or more) times - shell,
batch files and DCL scripts. In particular, miniperl builds DynaLoader
and perlmain.c, and links both with all those already compiled object files
to make the real perl.

There is a configuration option to build a shared perl library. In this case,
the C code is compiled to be position independent, and linked into a
shared libperl, with the perl binary just a small stub linked against it.
This also becomes relevant later, as libperl.so (or .dylib etc) contains
the code for op.o, but not opmini.o.


Currently in blead, the Makefile has 20 places where it calls $(MAKE) to run
a different target in the top level directory. Of these, this one is the
most troubling:

	$(LDLIBPTH) $(RUN) ./miniperl$(HOST_EXE_EXT) -w -Ilib -MExporter -e '<?>' || $(MAKE) minitest

as it can cause a fork bomb with a parallel make if miniperl builds but fails
to work. Having been bitten by that fork bomb one time too many, I decided to
eliminate it. However, that above line is actually replicated in 4 different
places, 3 OS specific and the generic fallback. Hence the first digression
into yak-shaving - reduce that number.

So, we have AIX (and BeOS), NeXT, OS X and "general". 3 of those are very
much alive, so we can't just kill the code...

Firstly, AIX.

On AIX, when building with a shared perl library, one needs the list symbols
that library should export when building it. For 5.005 and earlier, that list
was generated by a shell script, so no bootstrapping problem.

However, the export list had to be manually updated in the shell script, so
with commit 549a6b102c2ac8c4 in Jul 1999, the Win32 solution was generalised
to work on both AIX and Win32. This uses Perl script, makedef.pl to parse
various files and generate the current export list automatically. This
introduces a bootstrapping problem - miniperl is needed to generate
libperl.a, but libperl.a is needed to generate miniperl. This commit solves
the problem by introducing a new target, MINIPERL_NONSHR, which builds a
special staticly-linked miniperl (named miniperl_nonshr) in order to run
makedef.pl, which in turn permits libperl.a and the regular miniperl to be
built, and the build to proceed.

All was well until commits 52bb0670c0d245e3 and 8d55947163edbb9f (Dec 1999)
changed the default for CORE::glob() to use the XS module File::Glob, and
linked miniperl against an opmini.o, built from op.c but with compiler flags
to use the old glob-via-csh code. The change made for AIX was to build
*miniperl_nonshr* with the bootstrapping glob code, but leave the build of
miniperl unchanged. This broke the build on AIX - miniperl would build just
fine, but would fail to build any XS extensions, as the ExtUtils::MakeMaker
code requires working globbing.

The AIX build was fixed with commit 18c4b137c9980e71 (Feb 2000) by changing
Makefile.SH so that AIX used the same rules to build miniperl as NeXT. The
rules for NeXT generated miniperl from an explicit list of object files,
instead of using libperl.a. The result of this change was that on AIX,
miniperl was now identical to miniperl_nonshr. Both correctly use the csh
globbing code, but now neither require the shared libperl.a to work.

This makes miniperl_nonshr redundant. So I eliminated it. This starts to
converge the code for AIX with the other platforms.

The default case:

Curiously, as a side effect of commit 908fcb8bef8cbab8 (Dec 2006) which
moved DynaLoader.o into libperl.so, the default platform build rules for
miniperl were changed to use an explicit list of object files, instead of
C<-lperl>, which had the side effect of building miniperl non-shared. Hence
all platforms now have a non-shared miniperl when building perl with a
shared perl library.

Next, OS X:

Commit cb3fc4263509f28c (May 2003) removed the use of -flat_namespace from
the link flags, but added it specially in Makefile.SH for miniperl, so that
symbols from opmini.o overrode those in libperl.dylib. However, a side effect
of commit 908fcb8bef8cbab8 (Dec 2006) was to change the linker line to use
explicit object files, meaning that op.o was no longer part of linking,
meaning that the override is no longer needed. Hence darwin's link does not
need special-casing.

Lastly, NeXT:

Whilst it's not clear whether anyone is still using NeXT, since the
previous changes have refactored the other 3 cases to use an explicit list
of object files, the only difference remaining between the makefile rule
for NeXT and the rest is that next doesn't use $(CLDFLAGS) when linking
miniperl, whereas all the other do. It's not clear if this difference is
significant or accidental, but lacking any NeXT system to test on, it was
simple enough to preserve the difference but use simpler code to implement
it.

The result - all 4 cases are merged. However, I've not yet actually
eliminated that particular troublesome C<|| $(MAKE) ...> rule, as to test
it properly requires C<make minitest> to pass first time, without first
building all the non-XS extensions - work I've made progress on, but not
yet completed.

In turn, I also eliminated a lot of redundancy in the various variant
install targets (each of which had been implemented with a call to $(MAKE)
in the same directory), and the pre-requisites for various debugging
targets.

One of these needs to pass a flag to installperl to instruct it to run strip
on the installed binaries. installman doesn't need to strip anything, so
doesn't accept such a flag. As it's the only difference between the
invocations of the two, I decided that the simplest option was actually to
make installman accept a --strip flag and do nothing. This, as ever, isn't
as simple as it seems. installperl's strip flag is currently -s, but
installman uses Getopt::Long so accepts -s as an abbreviation for --silent
Hence the best solution is to have both accept a long-option --strip.  This
means refactoring installperl to use Getopt::Long, which in turn is "fun"
because it accepts both +v and -v as distinct options, which Getopt::Long
can't support. Or, more pragmatically, can't *directly* support, in the
general case. Fortunately installperl isn't the general case, as it takes
no non-option arguments, which permits a reasonably simple solution.

With this, more duplication died. En route, I spotted and eliminated some
dead code in installperl related to a 5.000->5.001 change - specifically
where in @INC autosplit installs files. Small instances of this sort of
cruft accumulate all over the source tree, but generally the code is never
annotated sufficiently as to the purpose to make it obvious that it had a
specific time-limited purpose. And of course, it's maybe only 1% of the
slightly "look twice and wonder why" code that is actually redundant - most
is subtly useful on some platform or configuration corner case that is hard
to test, but likely still needed somewhere.

This work is in the branches smoke-me/Makefile-miniperl-unification,
smoke-me/Makefile-norecurse, smoke-me/make_ext and smoke-me/perlmodlib

As well as fixing minitest, still to do are some further simplifications
of how ./miniperl is invoked to run various utilities. Most of the Makefile
command lines have -Ilib -Idist/Cwd -Idist/Cwd/lib, dating back to the
time when Cwd was detangled from ext/ and lib/ into a single directory in
dist/ with the same layout as the CPAN distribution. However, since then
I refactored miniperl to use the sitecustomize code to set up the build
@INC automatically, meaning that everything after that first -Ilib is now
redundant. So that's still to clean.

Nicholas Clark

• NWCLARK TPF grant report #34 by Nicholas Clark