develooper Front page | perl.perl5.porters | Postings from September 2012

NWCLARK TPF grant report #53

Nicholas Clark
September 22, 2012 01:24
NWCLARK TPF grant report #53
Message ID:
[Hours]		[Activity]
2012/09/03	Monday
 2.00		PL_main_start/PL_main_root
 0.50		Perl_magic_setdbline
 0.75		bootstrapping Will Braswell
 1.00		failing smoke reports
 0.25		process, scalability, mentoring
 0.50		reading/responding to list mail
 1.75		readonly ops

2012/09/04	Tuesday
 6.25		PL_main_start/PL_main_root
 0.75		reading/responding to list mail
 2.25		readonly ops

2012/09/05	Wednesday
 0.50		$^L
 1.75		ByteLoader
 3.00		PL_main_start/PL_main_root
 0.75		bootstrapping Will Braswell
 3.00		reading/responding to list mail
 0.25		readonly ops

2012/09/06	Thursday
 1.50		PL_main_start/PL_main_root
 3.50		reading/responding to list mail

2012/09/07	Friday
 5.00		reading/responding to list mail

2012/09/08	Saturday
 1.00		method_named
 0.75		reading/responding to list mail

2012/09/09	Sunday
 2.75		method_named

This week I finally emerged from the rabbit hole of PL_main_start, that I
entered into at some point in mid July. Father Chrysostomos has refactored
OP allocation to always use a slab allocator, as part of fixing some very
long standing bugs to do with OPs. A at the start of parsing, a new slab is
allocated. While parsing is ongoing, the slab is "owned" by the CV itself.
To avoid the need to create another pointer in each and every CV, he took
advantage of the fact that during compilation CvROOT() and CvSTART() are
both unused, and stored a pointer to the slab in CvSTART(). [Once compiled,
respectively they point to the root of the subroutine's optree, and to the
first OP to run. At that point, the pointer to the slab can be retrieved
from the OP pointed to by CvROOT()]. As he notes:

   When a CV has a reference count on its slab (CvSLABBED), it is
   responsible for making sure it is freed.  (Hence, no two CVs should ever
   have a reference count on the same slab.)  The CV only needs to reference
   the slab during compilation. Once it is compiled and CvROOT attached, it
   has finished its job, so it can forget the slab.

For nearly every subroutine out there, this little internal game is
completely unnoticeable. The only point that it becomes exposed is with the
(hidden, internal) subroutine that represents the main program, PL_main_cv.
From the description above you would think that the first OP to run from the
main program would be found in CvSTART(PL_main_cv). Well, to be fair, from
all this build up you'd be pretty sure that it's *not*, and you'd be
right. PL_main_cv is a relatively late addition, added in 5.001 (as part of
adding closures), whereas PL_main_root and PL_main_start date from 5.000.
Hence the main program's optree root and first OP are stored in special
variables in the interpreter structure. Which means that when the
compilation of the main program finishes, nothing overwrites
CvSTART(PL_main_cv) with a pointer to the first OP to execute, so it's
non-NULL and pointing to an OP slab, not an OP.

So, I tried to work out (roughly)

a) whether this was a bad thing
b) whether it's possible to ensure that it's NULLed out
c) whether it's possible to finish the work of 5.001 and abolish
   PL_main_root and PL_main_start to save some space.

Given that the answer to (b) is "no", the answer to (a) is obviously "no,
it's not", because any other answer would be depressing :-)

(c) seems to be possible, and is currently on the branch
smoke-me/abolish-PL_main_start, but relies on a test refactoring for
t/op/gv.t that fails on Win32 in ways that I can't determine from logs
supplied by George Greer's smoke-me server.

As to why the answer to (b) is "no" - well, that's a bit more involved.

To be clear, global destruction cleans up everything, so nothing leaks.
It's just that it's not cleared up quite as soon as it might have been.
But could it be cleaned up earlier?

The obvious place to start looking is the location that sets PL_main_start.

    else {
        if (o->op_type == OP_STUB) {
        PL_main_root = op_scope(sawparens(scalarvoid(o)));
        PL_curcop = &PL_compiling;
        PL_main_start = LINKLIST(PL_main_root);
        PL_main_root->op_private |= OPpREFCOUNTED;
        OpREFCNT_set(PL_main_root, 1);
        PL_main_root->op_next = 0;
        PL_compcv = 0;

That works just fine, until you encounter a program such as

    perl -e 'BEGIN {1}'

because that turns out to enter that block (o->op_type == OP_STUB) above.
The code for that is:

            PL_comppad_name = 0;
            PL_compcv = 0;
            S_op_destroy(aTHX_ o);

No comments explained anything.

There are now three paragraphs of comments, as a result of all my digging.
Here are the first two, added by commit 22e660b408c16433:

This block is entered if nothing is compiled for the main program. This will
be the case for an genuinely empty main program, or one which only has BEGIN
blocks etc, so already run and freed.

Historically (5.000) the guard above was !o. However, commit
f8a08f7b8bd67b28 (Jun 2001), integrated to blead as c71fccf11fde0068,
changed perly.y so that newPROG() is now called with the output of
block_end(), which returns a new OP_STUB for the case of an empty
optree. ByteLoader (and maybe other things) also take this path, because
they set up PL_main_start and PL_main_root directly, without generating an

So that's fine - add some more code there to fix up CvSTART(PL_main_cv),
and job's a good'un.

Only it's not. Whack-a-mole continues. The arms race expands.

    perl -e 'BEGIN {exit 1}'

doesn't even pass through Perl_newPROG(). The exit opcode is called during
the running of the BEGIN block, triggering a stack unwinding which flies
straight up all the routines involved in the top level parse phase, only
stopping in their caller, somewhat confusingly named perl_parse().

But that shouldn't make a difference to the main program (although code
running during global destruction might be surprised), because perl_run()
isn't called? After all, main() looks like this:

    exitstatus = perl_parse(my_perl, xs_init, argc, argv, (char **)NULL);
    if (!exitstatus)

Well, no, actually it can still matter. Because this program enters

    perl -e 'BEGIN {exit 0}'

If a call to my_exit() was caught by perl_parse, then value it returns is
the exit status. Which in this case is indistinguishable from a successful
parse. So perl_run() is entered, but returns pretty quickly because
PL_main_start is still NULL.

("Can you tell what it is yet?")

Well, in that case PL_main_start is NULL in blead. But if you try to
s/PL_main_start/CvSTART(PL_main_sv)/ then the wheels fall off. You've merged
the two previously disjoint pointers. The (original) CvSTART(PL_main_sv) is
non-NULL because it's pointing to an OP slab, now the former PL_main_start
is the same thing, so now enter perl_run() with it non-NULL, and hilarity
(well, a SEGV) rapidly ensues.

So the moral of this story is that it's rather too complex to ensure that
in the corner cases the slab is freed prior to global destruction.

And the practical take-away is that when CvROOT() is NULL, "CvSTART() is not
the pointer you were looking for". Remember that, and you'll be fine.

Nicholas Clark Perl Programming lists via nntp and http.
Comments to Ask Bjørn Hansen at | Group listing | About