develooper Front page | perl.perl5.porters | Postings from July 2013

NWCLARK TPF grant report #94

Nicholas Clark
July 26, 2013 14:05
NWCLARK TPF grant report #94
Message ID:
[Hours]		[Activity]
2013/06/17	Monday
 7.50		known_extensions, unbuilt non-XS extensions

2013/06/18	Tuesday
 6.25		known_extensions, unbuilt non-XS extensions

2013/06/19	Wednesday
 5.00		Makefile target pruning
 1.00		RT #118509
 1.50		RT #118549
 4.00		known_extensions, unbuilt non-XS extensions

2013/06/20	Thursday
 6.75		Makefile target pruning
 0.25		RT #47467

2013/06/21	Friday
 2.50		Makefile target pruning
 2.00		RT #118549
 0.75		RT #38812
 0.25		RT #40403
 1.00		RT #67114
 0.50		reading/responding to list mail

2013/06/22	Saturday
 1.00		ASAN causing Makefile loop
 2.00		RT #118603
 0.25		RT #38812
 0.25		reading/responding to list mail

Which I calculate is 42.75 hours

So, what prevents us from having a pure-Perl extension in ext/ but not
building it? And how did it happen?

The situation we had reached was that there were 5 configuration variables:

dynamic_ext:        built dynamically linked XS modules
static_ext:         built statically linked XS modules
nonxs_ext:          built pure-Perl modules (from ext/, dist/ and cpan/)
extensions:         "$dynamic_ext $static_ext $nonxs_ext"
known_extensions:   *just* the XS modules shipped in ext/, dist/ and cpan/ 

with the upshot that "extensions" is typically much larger than
"known_extensions". Daft.

This situation has come about through "organic growth", rather than design.
I guess it's summarised as

0) Perl 5 predates CPAN
1) Originally ext/ only held XS code
2) Originally there was no concept of dual-life - if you wanted the
   extensions in ext/, you had to build them with perl
   (There wasn't even a toolchain - you could add other extensions into ext/
    and they would be build)
3) 15 years ago was patched to add nonxs_ext (commit 4318d5a0158916ac) ready
   to support Errno
   (Errno was added about two weeks later in commit eab60bb1f2e96e20)
   [curiously that commit adds Errno to known_extensions but not to
4) A few days later commit bfb7748a896459cc updates Configure so that
   nonxs_ext *are* in extensions, but are *not* in known_extensions.
   The description of the change is:   
    Explicitly split list of extensions into 3 kinds:  dynamic, static,
    and non-xs.  The Configure variable $extensions now holds all three.
    (The only current non-xs extension is Errno).

    It also updates Porting/Glossary, explicitly changing the description
    of known_extensions from "list of all extensions included" to
    "list of all XS extensions included", and extensions from
    "all extension files linked into the package" to
    "all extension files (both XS and non-xs linked into the package."

[Note that Errno *is* architecture specific, so gets installed into
the same directory subtree as all the shared objects]

Fast forward from 1998 to 2006

5) Commit 1d8961043b9b86e1 (or thereabouts) in April 2006 regenerates the
   sample to this:

       nonxs_ext='Compress/IO/Base Compress/IO/Zlib Compress/Zlib Errno'

   at which point, we have 3 more non-XS extensions, all of which are
   architecture independent. 

Subsequent re-arranging of dual-life modules in 2009 means that we've got a
lot more.

Effectively, the term "extensions" has been meaning "things we build via
Makefile.PL" for at least 7 years, if not 15, despite what all the
documentation tries to claim.

So after a lot of figuring out the why and how, and what it would likely
break (answer, nothing), I patched the *nix and Win32 build systems to fix
this. (I chickened out of figuring out enough DCL to deal with VMS. Craig
Berry was kind enough to deal with that.)

So why did this even matter? Because whilst the build system was quite happy
not building a pure-Perl module, all the tests for it would still be run
(and fail), due to implementation details of how t/TEST (and thus also
t/harness) decides what to skip. It refuses to skip anything unless it's in
"known_extensions" but missing from "extensions". As Andy Dougherty observed
after I submitted the patches to fix the build, nothing after Configure
should actually use known_extensions. Hence t/TEST is arguably buggy and
needs fixing. Maybe I could have used a smaller hammer if I had spotted the
correct problem to hit. :-)

However, it's done now, and the distribution is saner for it. And it permitted
the tests for FindExt to be made more comprehensive (and have fewer special
cases and skips).

Whilst looking at the *nix Makefile a lot trying to figure out how to
resolve the problems above, I noticed that there are quite a lot of
short-cut targets. These are targets added to simplify running various
commands, and I don't think that anyone uses. For example there were targets
related to profiling and testing tools for Tru64 and Irix (pixie and Third
Degree), for purify, quantify and purecov, targets to run the tests through
B::Deparse, to convert the tests to UTF-8 or UTF-16 before running them, and
to run the tests with -t to flag up taint warnings. (Plus, in some cases
targets to combine two of the above actions.)

It's still perfectly possible to *run* any of the above programs by "hand" -
no underlying functionality has been removed from the Makefile. It's just
got a little bit shorter and a little bit clearer.

We also discovered a problem with the previous week's refactoring of the
initial build rules. While Father Chrysostomos was trying something out
(which seriously broke the ability of miniperl to even parse code), his make
went into an infinite loop calling itself recursively. Effectively, a fork
bomb. This isn't supposed to happen - a build failure is supposed to stop,
not take out one's machine.

The problem is that the *nix Makefile contains a lot of places where it
calls back to itself *in the same directory* to build a different target.
I'd been bitten by these some time ago. If things don't go as intended, you
can end up with an infinite loop as each recursive invocation of make
decides that the same thing needs doing first, and calling make again with
the same arguments. It gets even worse running make in parallel.

I think that historically things had been done this way as a means to have
various little utility commands or command sequences available, without
having to clutter the build directory with a shell script for each desired
"program", or repeating the same commands in multiple places in the
Makefile. Even if you get it right (ie avoid the above problems) then I feel
that it actually makes the build *less* clear, because you have to scan back
through the same Makefile, and then work out if the target requested is
stand alone, or going to have more side effects. Hence I'd considered these
as a pain point some time ago, and had tried to work to eliminate them.

They actually even directly work against correctness. The miniperl build
rules used to be this:

	$(LDLIBPTH) $(RUN) ./miniperl$(HOST_EXE_EXT) -w -Ilib -MExporter -e '<?> || $(MAKE) minitest

The intent is to be "helpful" and automatically run minitest if miniperl
fails a basic sanity test. The problem is that minitest then looks like

# Can't depend on lib/ because that might be where miniperl
# is crashing.
minitest: $(MINIPERL_EXE) minitest.prep
        - cd t && (rm -f $(PERL_EXE); $(LNS) ../$(MINIPERL_EXE) $(PERL_EXE)) \
                && $(RUN_PERL) TEST base/*.t comp/*.t cmd/*.t run/*.t io/*.t re/*.t opbasic/*.t op/*.t uni/*.t </dev/tty

with a dependency on minitest.prep, which looks like this:

        -@test -f lib/ || $(MAKE) lib/ $(unidatafiles)
        @echo " "
        @echo "You may see some irrelevant test failures if you have been unable"
        @echo "to build lib/, or the Unicode data files."
        @echo " "

Hence to avoid a recursive loop when attempting to helpfully run minitest
automatically, it needs to recurse to a third level, and to skip doing so if
lib/ already exists. Note, *exists*, not "is up to date".

ie correctness has been sacrificed, although it's not immediately obvious.
The reason is that done this way, if you update the pre-requisites for
lib/, make *won't* automatically re-build it. Meaning that you may
get bogus results if you edit them, and then re-run minitest to check your

The problem here seemed to be that my other changes made things more
fragile, and the fork bomb a lot more likely to trip. For now, I've removed
the automatic recursion (with make) to run minitest, as it removes the
fragility. Given that running minitest is one line, albeit rather long (as
shown above) I think that it should be possible to have that run directly by
the Makefile (without calling back to make to do it), but I can't quite see
how to do it. It feels like it ought to be possible to merge it with the
shell script that runs the regular tests, but I can't yet see a way that
merges the two without using *more* code than doing it separately. Something
is eluding me.

Nicholas Clark Perl Programming lists via nntp and http.
Comments to Ask Bjørn Hansen at | Group listing | About