NWCLARK TPF grant report #86

[Hours]		[Activity]
2013/04/22	Monday
 2.00		process, scalability, mentoring
 5.75		reading/responding to list mail
=====
 7.75

2013/04/23	Tuesday
 0.50		HvFILL
 0.50		RT #116989
 2.00		process, scalability, mentoring
 4.25		reading/responding to list mail
=====
 7.25

2013/04/24	Wednesday
 0.25		ExtUtils::CBuilder
 0.25		RT #113794
 6.00		RT #116989, toolchain bootstrap, probing.
 0.50		RT #117743
=====
 7.00

2013/04/25	Thursday
 5.25		RT #116989
 2.25		RT #117501 (Open64 compiler)
=====
 7.50

2013/04/26	Friday
 0.50		CPAN #83167
 2.50		RT #116989
 1.75		RT #117501 (Open64 compiler)
=====
 4.75

2013/04/27	Saturday
 0.50		pp_pack.c
=====
 0.50

2013/04/28	Sunday
 3.00		TAP::Parser
=====
 3.00

Which I calculate is 37.75 hours

The main activity of the week was dealing with RT #116989, a bug originally
related to S_croak_memory_wrap(), but which turned out to be the tip a
considerably more pervasive iceberg.

The perl source code uses a lot of macros. Some of these were used to
implement inline functions long before C compilers could. Now that the
semantics of "static inline" functions are settled and widely supported,
we've started to replace the (still-working) macros with inline functions.

However, this threw up an unexpected surprise. You'd think that replacing
well-behaved macros (those that only use their parameters, and not more than
once) with inline functions would have no functional change. But we found
an interesting gotcha...

Unreferenced macros disappear at the end of pre-processing. Unreferenced
inline functions stick around for a while longer. Some compilers only remove
them if the optimiser is enabled. (Certainly some versions of gcc), whereas
others remove them always (for example clang).

The problem was with macros (and now functions) that reference global
variables or other functions. If the dead code isn't removed by link time,
it causes unsatisfied dependencies, and the build fails. The effect is that
the header files have changed such that they can now only be included by
code which links to the perl library.

Why is this a problem?

Probing code which wants to figure out if a function is present, or compiler
flags work, will compile a small test program using the perl header files,
and if it runs assume that the probed-for thing can be used. The assumption
is that "not running" is caused by the probed-for thing's absence, and not
some problem in the probe. So the first problem is that this stopped being
true, and the second problem is that we didn't realise this because none of
the probing code is verifying this assumption by testing the null hypothesis
- that a test program without anything changed does run. So when the header
files change such that including them now makes the test program fail to
link, all the probing code silently stops working - it simply thinks that it
can't find anything, routinely reports this, with results such as
Time::HiRes failing to find platform-specific functions that it uses, and
hence disabling optional functionality. And we didn't realise this.

Probing code such as that in Time::HiRes could likely resolve this by
linking its probe programs with the perl runtime library. But the code in
cflags.SH wants to work out what extra compile flags it can use locally to
*build* the perl library - ie which known potentially useful flags can't be
added to the build command line locally because they cause compilation
failures on this particular system. This is a bootstrapping problem, as we
want to know what we can compile it with, but if the probe now only works
once we can link with the perl codebase, then we can't answer that until
after we've compiled it.

As well as me, Andy Dougherty and Tony Cook had both already worked on this,
so we had a couple of proposed patches, along with a new regression test to
actually test that null hypothesis of probe code. However, it still didn't
quite feel complete. After thinking about the implications for a while, and
experimenting with various approaches I can't see a way to have
everything. ie

1) static inline functions to replace macros

combined with

2) defaulting to use them

3) not screwing up the "traditional" use without linking


I think that the least worst option is to keep (1) and (2), and force (3) to
work by adding a define that disables inclusion of "inline.h". Of course,
that means that you can't *use* anything from it. But as all it contains are
functions *declarations* that are of no use without data structures
initialised from the perl library, it's no loss. Hence we added a macro
PERL_NO_INLINE_FUNCTIONS for that purpose - explicitly disabling the
declaration of inline functions, for probes and similar code which needs it.
Having tested it on Solaris and HP-UX (as well as the more common
platforms), I pushed it to a smoke-me branch expecting it to work
everywhere. Fortunately I left it until "next week" to let as many machines
as possible catch up with it. Which revealed one last silly little surprise.

Nicholas Clark

• NWCLARK TPF grant report #86 by Nicholas Clark