develooper Front page | perl.perl5.porters | Postings from December 2017

Re: Request For Context - about the 5.22->5.24 context stack rework

Thread Previous | Thread Next
Paul "LeoNerd" Evans
December 13, 2017 14:59
Re: Request For Context - about the 5.22->5.24 context stack rework
Message ID:
On Wed, 13 Dec 2017 15:36:06 +0200
Sawyer X <> wrote:

> > So if at some point in the future I make a change to the internals
> > of the context stack implementation which breaks your module, who
> > is is responsible for fixing it?  
> That's easy: It's Paul.
> However, there are two situations this doesn't solve:
> * If Paul is unavailable, the users are doomed, unless someone adopts
> it (or co-maintains).
> * If Paul is unwilling, the users are forced to contact p5p and ask we
> fix it.

I'm sure the same is true for any of my CPAN modules, or indeed
equivalent for anyone else's for that matter. If core changes something
that any CPAN module makes use of, it always comes down to trying to
decide what's best to do about that.

The only real difference with something like Future::AsyncAwait is the
degree to which it digs into internals of perl's implementation. I
happily accept that to implement what I've done I've had to read
through many (undocumented) .h and .c files that came with perl core

It's true that p5p provide no guarantee on changing details in there,
however I don't think it fundamentally changes the nature of support of
my module. I have many other modules whose correct behaviour depends on
external factors - Amazon webservices, Google, many external C
libraries. These are all liabilities to future breakage - p5p is not
unique among them.

> While it is clearly Paul's responsibility, I sympathize the general
> concept Dave is raising here: The end result is that everyone (both
> the users and p5p as an extension of the 2nd clause) depends on Paul
> being available and willing to continue updating. We had to deal with
> this in the past and still do. I won't mention examples because we
> all know them and the incredible frustration involved.
> While Paul is a far more cooperative person than others, the first
> clause still applies. "Unavailable" could also mean "not (currently?)
> capable." The next internal change might be much harder to deal with
> or might not be possible even. It's a lot of responsibility to take
> on.

I thank you for your kind words Sawyer :)

Yes I'll happily accept there might be an occasion when I'm not
available to fix whatever needs fixing, but again want to point out
that really the situation is the same for any other module.

> What bothers me is not the change we make. If it's unreasonable, we
> revert it. If it's reasonable (and covered), we shouldn't. But when
> the author of a module that needs updating doesn't feel like fixing
> it, they can (and some have) simply directed their users at p5p.
> "They broke my code by changing internals that were never documented
> or provided as public. Go tell them to change it."
> (This should be a warning to all users of whether you want to use
> modules that have an author who is unwilling to update it when they
> use something undocumented or non-public, but that's for another day.)
> > Which is why I feel users of your module should be warned that it
> > could break in the future without notice.  
> I wish we could also provide a clear statement of modules that use
> internals and that *we* expect they might break in the future. "No
> liability" kind of sticker, but that's just asking to argue.

Dave suggests adding some wording to the documentation in
Future::AsyncAwait, and yes I'm happy to add that. I thought the overall
status of the module of "Frankly I'm surprised this thing currently
works at all; Good Luck and Have Fun" came out clearly enough but
perhaps not. I can add more words to that effect.

> >> Perhaps one day I'll even come up with a set of APIs/suggestions
> >> that perl core could do to make this sort of thing nicer ;)  
> > Exposing more of the internal mechanics of the perl runtime engine
> > via an API just makes it even harder to change or fix the engine in
> > future.  
> The question to ask is, What is Paul trying to achieve? Is it
> reasonable? If so, how can we allow him to accomplish this in a way
> that we could continue to support it.

I'm glad that came up - I was going to write a reply to the earlier
points by trying to make that motivation.

In brief, what I'm trying to add to Perl is the same syntax that quite
a number of other languages have all adopted, to make coöperatively-
scheduled coroutines work nicely for handling deferred parts of Future-
like asynchronous programming. Without wishing to go into lots of
detail (but the interested reader can watch my TPCiA talk [1]), each of
Python 3, EcmaScript 6 and C# 5 have all added basically the exact same
syntax shape for the concept. I know Rust is considering it, and I
expect other languages too. That three quite different languages have
all converged on the same syntax speaks volumes to me. I can foresee
this thing becoming as ubiquitous as such shapes as the `if` condition
and the `while` loop, and I'm very keen to see it brought to Perl too.

> Another question to ask is, should this part of core instead?

An exciting question.

By raw lines-of-code count, the vast majority (>90%) of the code in
Future/AsyncAwait.xs is the fairly boring low-level machinery involved
in suspending and later resuming a running pureperl sub, with the
myriad of complexities involved in hunting down those many stacks and
side-effects. The rest of the module is concerned with parsing the
`async/await` syntax out of the source code and implementing the two
new OP functions needed to back the semantics behind it. There's a
small amount of glue logic to talk to the Future instances involved,
and then invoke the suspend/resume logic.

I'm quite happy that the syntax parsing and new OP functions can sit in
a CPAN module, because on the surface they're using existing documented
APIs introduced back in 5.14 and seem quite stable now, in use by many
other CPAN modules than just mine. They're also very specific to the
exact async/await syntax and semantics.

In principle, I could see a lot of benefit to my code in core perl
providing some ability to suspend and resume a running sub. There might
be other use-cases (such as coroutine generator syntax) that could make
use of that. However in practice it's a little hard to imagine many
other cases besides my module that would really exercise it. The
interface into such would also be fairly fragile and special-cased to
this sort of situation.

I would definitely be interested in having some amount of core perl
support for the trickier internal fiddling that currently I have to do
in my XS code, but trying to work out where the boundary of
responsibility is, and how to make that a more generically useful
interface than simply being "these core functions exist in order to
make AsyncAwait work", is a harder question. It's one I want to start
looking at a little later when I have more of this implemented and
working well, so we can better see where the edges are. Already I've
found a few small "missing pieces" in the parser API for example, which
is unrelated to the context stack under discussion here, but still
relevant as a general principle.


[1] My TPCiA talk about Futures and the async/await syntax --

    the syntax part of the discussion begins at

Paul "LeoNerd" Evans      |  |

Thread Previous | Thread Next Perl Programming lists via nntp and http.
Comments to Ask Bjørn Hansen at | Group listing | About