[PATCH] v4: fix blead usage of OK flags vs. all magic

Front page | perl.perl5.porters | Postings from June 2012

[PATCH] v4: fix blead usage of OK flags vs. all magic

Thread Next

From:

Reverend Chip

Date:

June 11, 2012 23:46

Subject:

[PATCH] v4: fix blead usage of OK flags vs. all magic

Message ID:

4FD6E5C6.5050308@gmail.com

This patch has been updated post-5.16 and pushed as branch
"chip/magicflags4".  I still think it's good, and the below
description still holds.

Given the long time before 5.18 will be out, I hope we can mostly
agree this is an improvement to the core so we can merge it early.

==============================================================

Turns out Perl has been cheating.  (Surprised?  Sure.)

First realize that the way get magic is made to work at all -- the way
Perl knows when to call FETCH, for instance -- is by imposing a burden
on code that uses the value.  Any code that wants to use an SV must
somehow arrange that if SVs_GMG is set (a.k.a. SvGMAGICAL()), mg_get()
must be called on that value (usually via SvGETMAGIC()) -- but, and
this is important, *EXACTLY ONCE*.  And then the value can be used.
Lots of Perl's public API handles this automatically, so it's not
_much_ of a burden, but it is definitely the responsibility of value
users to keep track of magic count.  There's a parallel requirement
for set magic.

Second, realize that the primary way the public and private OK flags
work -- POK vs. POKp, for example -- is that the public flags identify
authoritative values, and the private flags identify non-authoritative
values.  e.g. if NOK and IOKp are set, but not IOK, then SvIVX()
contains the result of (IV)SvNVX(), which may be truncated and thus
not authoritative.  It's just a cached conversion.

Third realize that from 5.000 until now, after magic processing --
well really in restore_magic(), which is called after any magic
processing -- all of the public OK flags were shifted into private OK
flags.  The reasons are lost in the mists of time.  But as a side
effect any value with get magic would never have public OK flags set.
In Perl 5.000, the shifting didn't happen, but instead all the public
flags were just *cleared* after get processing.

Weird, right.  But now's the cheat, and it's a doozy.

it turns out the way the core has been deciding whether it can use
no-magic short cuts to get values -- e.g. how SvPV() decides whether
to just grab SvPVX(), or how addition knows it can use SvIVX(), or
whether mg_get() might need to be called first -- is not the REAL
CORRECT way, by checking SVs_GMG, but the fake kludgy way, by just
happening to know that the public OK flags would always be cleared on
any magical value.  For example: SvPV() and company were checking
*only* POK() to decide whether to use SvPVX().  You may think that's
OK, but notice I did not say SvPV_nomg(), for which it would be OK,
but rather SvPV(), for which it is *not* OK, because it has to check
SVs_GMG also; if that bit is set, then mg_get() is needed, and using
SvPVX() immediately is very wrong.The core has been cheating this way
for a LONG, LONG TIME this way.

Guess what else does this?  SvTRUE().  Yup.

You may think we should leave well enough alone.  Well, that's not
possible.  The clearing or shifting of the public OK values actually
breaks all kinds of logic that tests what kind of value it's been
given.  Essentially, it means that when magic is involved you simply
can't count on SvOK(), SvPOK(), SvIOK(), etc. telling the truth, and
so you have to count on SvIOKp(), SvPOKp(), etc. and those simply are
the wrong flags for the job.  That's not acceptable.  It's just nobody
noticed for a while.  Apparently it's rare to pass magic values
directly to sophisticated XS code, or we'd have heard more yelling.

BTW, almost all XS code works fine with the change because it follows
the rules.  A few bits of the compression modules don't work right
because they keep calling the functions that invoke magic over and
over, and expect the returned pointers not to move.   This patch
includes fixes.  As you can see they are simple.

Share & Enjoy!

Thread Next