develooper Front page | perl.perl5.porters | Postings from August 2023

Re: PPC Elevator Pitch for Perl::Types

Oodler 577 via perl5-porters
August 22, 2023 16:48
Re: PPC Elevator Pitch for Perl::Types
Message ID:
* Dave Mitchell <> [2023-08-22 11:02:53 +0100]:

First, thank you for the reply. I got to your conclusion and understand
your stance. I'd like to still address a few techincal things inline.

> On Tue, Aug 22, 2023 at 04:19:04AM +0000, Oodler 577 via perl5-porters wrote:
> > Yes you are correct, the current mechanism for enabling type-checking
> > for subroutine calls is essentially a type of source filter.  As
> > mentioned in our first reply:
> > 
> > > > The Perl compiler currently supports type enforcement for subroutine calls, so that is our starting point for Perl::Types.
> > 
> > This source-filter-like functionality is currently contained in
> > the Perl compiler's file ``
> So just for the avoidance of doubt, is the proposal that Perl::Types,
> when running in a perl (rather than RPerl) environment, will use perl's
> actual source filter mechanism, or that it will use some other
> "essentially a type of source filter" thing?

Yes, it uses a source filter to start. Lots of things use source filters,
just like lots of things other parts of Perl. Short of carrying along
patches to core, I am not sure how you would really hook into the process

> If the latter, please expand.
> If the former, note that we generally regard the perl source filter
> mechanism as deeply flawed, and wouldn't recommend its use in production
> code. The basic problem comes down to the old mantra that "only perl can
> parse Perl". A source filter may correctly parse the source 99.9% of the
> time, but on those 0.1% occasions, it will get something wrong. Maybe it
> will get the start of a // operator confused with the start of a pattern.
> Or maybe it will trip over some complex code embeded into a pattern:
> /...(?{  $x = '/'; }) .../. Or whatever. At that point, code will get
> injected at the wrong point into the source, and the end user gets a
> compile error from code that isn't even in their source, which is
> completely mystifying and hard to debug.
> used source filters, and it was the cause of an endless stream
> of tickets. It was eventually deprecated and removed from core in 5.14.0.


> > Regarding your `my number $x; $x = 'foo';` example, this will
> > require the use of a `tie` or similar mechanism
> Again, this is very vague. Do you actually mean perl's tie / magic
> mechanism? If not, what "similar mechanism" are you proposing to use?

It was an speculative answer to how we'd do something. It is not currently
possible so we simply do not know.

> The problem with using actual ties, is that they make the variable very
> slow, even if the actual tie methods such as FETCH() are XS code. That's

> because a lot of code paths in the perl interpreter have optimisations for
> non-magic values. For example, look at pp_add() in pp_hot.c in the perl
> source. This implements perl's '+' operator. Obviously in general the two
> operands can be anything - ints, ,nums, strings, magic values, overloaded
> objects etc, and mixtures thereof. But in pp_add(), the first thing it
> does it check the flags of the two operands, and says:
>     If they are both non magical and non ref, and if both are either
>     simple ints or simple nums, then just add the two values, check they
>     haven't overflowed, set the return value and return. Otherwise go
>     through the complex 200-lines-of-code path which handles magic, mixed
>     types etc.
> So if, for example, an integer variable was marked with get magic, it
> would become a lot slower when being added. Similar considerations apply
> in many places.
> Also, once tie magic starts being applied to arrays and hashes, you start
> to hit edges cases. It's very had to make a tied aggregate behave *exactly*
> like a plain array/hash in all circumstances, even ignoring the slowdown.

Again, what recourse do we have? This is the mechanism by which perl gives
us ways to "implement" our own high level data types (SCALAR, HASH, ARRAY).

This is and the source filter being our only options to hook into fundamental
perl "things" is exactly the pain points we as a community need for there
to be some fundamental improvements in. Perhaps through this effort there
will be some motivation to add a more reliable stage that in the past was
only able to be achieved via filters. And with `tie`, it'd be super if there
was a "better" way to Tie there is not. This is a good role for core developers
to fill.

It is the role of the core development to provide proper programming interfaces
for external parties to propose improvements or integrations without dealing
with gobs of very tedious magic; Perl provides these things currently in 3
fundamental ways:

* source filters
* tie interfaces for SCALAR, ARRAY, HASH
* subroutine prototypes (which are IMO great for creating "keywords" UX)

We've asked before, and we will ask again: what mechanism would you suggest;
and if it doesn't exist or isn't mature enough, what changes would be required
to core to more cleanly/appropriately handle what it is we WANT to do here?
Hiring core developers and internal process people is not an option of us, 
we're just volunteers like most, with limited time and zero money to throw at

> > So, as mentioned in the original Elevator Pitch, we can "utilize
> > Perl data types to achieve a number of benefits including but not
> > limited to":
> > 
> > * increased performance
> So how would you get increased performance? I've already pointed out that
> magic tends to slow things down, and the cost of calling a check routine
> will slow things down even further.

Performance increased for type checking because we're taking advantage of the
native type checking in the underlying C or C++, give the source filters enable
more or less a "source to source" translation in differing degrees. This is
in contrast to, e.g., creating a framework that simply allows one to apply
regular expressions all over the place. On one hand regexps are slow, on the
otherhand, we still submit - what options do we really have? 

It is also important to get this out front, performance does matter; any costs
for setting up the types constraints should be able to be amortized over the
lifetime of the running program to justify its use. Just like one can make
a program run arbitrarily fast if it is not "correct", it can also become
unacceptibly slow if there is too much overhead in the set up. It's the same
trade off we all see every day in other programming projects. It's no different

So performance is absolution a critical point to consider - be it the overhead
induced but the constraints or the benefits of using real (as in C or C++) based
types and checking, which is where this step actually belongs.

> > * memory safety (bounds checking)
> Can you give an example of where currently perl isn't memory safe, but
> Perl::Types would make it so?

Again, this leverages the underlying C or C++ data structures and capabilities
provided by these compilers and runtimes. It's all about passing it to
the right level. In our case, it is the fundamental C or C++ data type capabilities
present in the compiler and in their runtimes.

> > * potential for polymorphism
> Please explain further.

The same way it's facilitated everywhere, by dispatching on what the programmer
is passing into the function - in this case, based on real types.

> Ok, that was the technical side of things. Now on to the policy side.
> 20 years or so ago, perl went through a phase of adding lots of 'useful'
> modules to the perl core. This was widely seen as a mistake.
> First, it made a particular module seem to be officially endorsed.
> Hypothetically, "among several XML parser modules, we chosen this one as
> the best and the one you should use". When design flaws in XML::Parser
> (say) are discovered, and the general recommendation is to use
> XML::BetterParser instead, XML::Parser is still sitting in the perl core
> getting an unfair endorsement. And removing it from core is hard, because
> now lots of people used it, and they're using it because, well, we
> endorsed it!

Are you suggesting some prudence be applied in endorsing one thing over another?
What is the process in place for vetting such modules or subsystems? It is
a sticky situation, i.e., vouching for things.

> Second, if modules stop being actively maintained by their author(s) for
> whatever reason, then suddenly we become responsible for maintaining it.
> It's enough work just maintaining the core perl interpreter, without
> taking on extra responsibilities.

This is already an issue for a lot of things and for a lot of people, literally
anywhere in tech.

> Third, in these days of package managers etc, it's relatively easy for
> people to install the CPAN modules they want. They will also get the
> newest and best version by default, rather than using the old buggy version
> bundled with perl. So there is less good reason to bundle a package with
> perl.

This is a fair point.

> So these days the policy is not to bundle CPAN modules with perl unless
> they are necessary, usually because they are needed as part of the tool
> chain. There have been very few new modules added to the perl core in the
> last 15 years or so that are capable of acting as standalone CPAN
> distributions instead.

Fair enough. I'd like to ask, what recourse do we have to hook into the perl
interpreter in the way that we are, if not to use the limited options present
in perl? Criticizing source filters and any mention of tie is not fair because
that's all we have. The alternative is also not feasible or fair. So what
do you suggest we do here?

> So the short answer is that even if Perl::Types really appealed to us, it
> is very unlikely that we would agree to bundle it. And I think we're 
> far from a consensus yet that it appeals.
> On a purely individual note, I am finding Perl::Types very unappealing so
> far.

Fair enough, thank you for your time in replying. What technical avenue would
you take if you were in our situation, knowing as much as you know and having
taken the time to understand what we are doing here? Like I said above, beyond
source filters and tie, perl doesn't give us many options that don't include
paying a core developer (or more) to incrementally work on the support necessary
to get this working "fast" or "reliably". Because at the end of the day, that's
what it is going to take - a way to work on both sides of the process in order
to simultaneously deliver an advanced Perl module that is unincumbered by the
traditional limitations and dirth of options to hook deeply into what magick
is happening inside of the perl interpreter. And keep in mind, your reply here
will not just help us. 

Thank you again.


> -- 
> I took leave and went to hear Mrs Turner's daughter play on the harpsicon,
> but Lord, it was enough to make any man sick to hear her; yet I was forced
> to commend her highly.
>     -- The Diary of Samuel Pepys, 1 May 1663

SDF-EU Public Access UNIX System - #openmp #pdl #native Perl Programming lists via nntp and http.
Comments to Ask Bjørn Hansen at | Group listing | About