develooper Front page | perl.perl5.porters | Postings from July 2012

NWCLARK TPF grant report #46

Nicholas Clark
July 25, 2012 06:00
NWCLARK TPF grant report #46
Message ID:
[Hours]		[Activity]
2012/07/16	Monday
 0.25		RT #114128
 0.50		given/when/smartmatch
 4.75		magicflags
 3.00		reading/responding to list mail

2012/07/17	Tuesday
 1.50		BBC (8be227ab5eaa23f2)
 2.75		magicflags
 1.00		reading/responding to list mail

2012/07/18	Wednesday
 1.50		PL_main_start/PL_main_root
 0.50		reading/responding to list mail

2012/07/19	Thursday
 0.50		NetWare
 3.25		fold_constants
 2.25		reading/responding to list mail

2012/07/20	Friday
 0.25		RT #114128
 0.50		RT #114142
 2.25		fold_constants
 0.25		i5 query
 1.50		process, scalability, mentoring
 0.50		reading/responding to list mail
 1.00		investigating security tickets

Which I calculate is 28.00 hours

I spent a while digging into the pre-history of the various scalar flags,
trying to make sense of how we got to where we are, and why Chip's patch
to magic flags makes sense. The full conclusions are here

but the question comes down to an inconsistency - there are both "public"
and "private" flag bits for integers (I), floating point values (N) and
strings (P), but there is only one flag for references (R). This seems wrong
- why is this?

It turns out that public and private flags were added by 5.000 alpha 4, as
part of implementing magic on scalars. Prior to that version, tainting was
implemented by building a separate taintperl binary. Magic enabled tainting
to be implemented at runtime (with the -T command line option) in the same
binary as the regular perl, without a significant speed hit. Magic also
permitted the implementation of tie and untie. However at that time there
was no SVf_ROK(), or SvROK(). References could *only* be in SVs of type
SVt_REF, and the code in sv_setsv() downgrades the destination SV to type
to SVt_REF if needed. Note that one *can't* get a reference from the
environment, so a reference can never be tainted.

Once the alpha went out into the wild, people discovered that this meant
that also a reference could not be assigned to a tied variable, as noted in
this thread from 1994:!msg/comp.lang.perl/TlLd6ttq4o4/-3YuF4n9UysJ

to which Larry replies "I'll fix it.  Sounds like we'll want an alpha 5
pretty quick."

And so alpha 5 appeared, and changed SVt_REF to SVt_RV, added SVf_ROK,
SvROK() and and SvRV(), thus (pretty much) promoting references to first
class scalars with the same semantics as I, N and P.

Alpha 5 also contained a file internals,

which describes the public flags like this:

    These tell whether an integer, double or string value is
    immediately available without further consideration.  All tainting
    and magic (but not objecthood) works by turning off these bits and
    forcing a routine to be executed to discover the real value.  The
    SvIV(), SvNV() and SvPV() macros that fetch values are smart about
    all this, and should always be used if possible.

and the private flags:

    These shadow the bits in sv_flags for tainted variables, indicated that
    there really is a valid value available, but you have to set the global
    tainted flag if you acces them.

which suggests that the lack of public and private flags for references was
a mistake. The scheme was designed for tainting and tie, or designed for
tainting and extended to tie, and references weren't quite first class then.
References became first class one alpha too late, and that's why they never
had the proper split public and private flags. Probably it wasn't noticed
because references weren't tainted, and most early uses of references were
effectively idempotent, with the result that as long as code was called, it
didn't notice if it was called multiple times instead of once.

And yes, this does mean that every version from 5.000 to maint-5.16 has
been subtly buggy.

I worked further on fold_constants(). It looked fairly easy to write tests
for the documented behaviour, and add it to the public API. It seemed pretty
clear that it can return two types of OPs, so both would need testing:

    if (type == OP_RV2GV)
        newop = newGVOP(OP_GV, 0, MUTABLE_GV(sv));
        newop = newSVOP(OP_CONST, OPpCONST_FOLDED<<8, MUTABLE_SV(sv));
    return newop;

It's clear that the OP_CONST is the common case, and it's obvious how to
test it, but what about that newGVOP? I had no idea what called that, so
took the brute force approach of replacing it with an abort(), and running
a full build and test cycle. (Parallel build and tests mean this takes less
than 5 minutes. It's often faster than any other approach if it's not
immediately obvious how to reach some code).

Nothing failed.


So what's going on here? type is the type of the original op was folded.
So the newGVOP route can only be reached if fold_constants() completes
for an op of type OP_RV2GV. But fold_constants() will never complete for
an op of type OP_RV2GV, as it will return almost immediately:

    if (!(PL_opargs[type] & OA_FOLDCONST))
        return o;

as only ops with the OA_FOLDCONST bit set can be folded. That is set if the
op is flagged as 'f' in regen/opcodes, and rv2gv doesn't have the flag. So,
did it use to? It turns out that it never had it. The opcode data has moved
around a bit in the history of perl, but even back in the earliest revision
of perl 5 in git, alpha 2, rv2gv isn't flagged as 'f':

So the code to return newGVOP, also added in alpha 2:

has always been dead code. So on the branch, it's gone.

Which only leaves the obviously testable code. "obvious" - always a danger

Turns out that the first problem with testing the folding of constants is
that if you try to build an optree ready to fold, the op constructor
functions such as newBINOP() spot this and helpfully fold it for you,
returning a single OP_CONST, instead of the tree you were hoping for.
So you have to subvert their efficiency by lying to them - build OP_NULL
instead of the op you really want to fold, then replace the op_type and
op_ppaddr values after its returned.

So now you have your tree ready to fold, and you pass it to fold_constants().
At which point you hit the second problem - nothing happens. It turns out
that when it executes the ops in order to get the result, the BINOP I was
using (OP_MULTIPLY) panics because it doesn't have a target allocated.
OP_NULL doesn't need a target, so newBINOP() doesn't create one needlessly.
However, no error report escapes, because constant folding runs with all
warnings and exceptions trapped, and if anything goes wrong, constant
folding is abandoned and the original optree remains. So, also allocate
a pad slot, and all is happy.

Except that writing more tests reveals that it's not. SEGVs, wrong numbers
of tests run, and "interesting" things like that, which valgrind reveals is
due to a read from freed memory in pp_iternext(), the implementation of the
looping part of for(). The problem turns out to be allocating that pad slot,
however it's done, newBINOP() or the XS test code. It's all because XS code
doesn't have its own pad - so at the time of the C calls the current pad is
that of the calling Perl subroutine. The *running* subroutine. The problems
happen when the pad gets moved as a side effect of being extended to
accommodate allocating another slot in it, because the runtime for for() has
taken the address of a location within the pad, never expecting it to
move. This is a totally reasonable assumption, because the pad moving at
runtime simply doesn't happen within the perl interpreter itself - once a
subroutine is compiled to ops, neither the optree nor the pad changes
again. I don't know how much of the runtime code makes assumptions such as
these, but it suggests that the level at which the optree construction
functions act doesn't make a good API to call near directly from Perl space.

Nicholas Clark Perl Programming lists via nntp and http.
Comments to Ask Bjørn Hansen at | Group listing | About