async/await syntax - overview

((This is a mail in three parts - the first part (here) outlining the
  problem I'm attempting to solve. I'll follow this up with a second
  part giving the current attempt at a solution, and third
  call-to-action part that suggests what assistance I would find
  immediately useful.

  If you get bored reading these first two, just skip to the third.))

I'm intending to provide an async/await syntax for Perl, much in the
vein of ECMAScript, Python, C#, and so on. See for example:

  ECMAScript: https://tc39.github.io/ecmascript-asyncawait/
  Python:     https://www.python.org/dev/peps/pep-0492/
  C#:         https://msdn.microsoft.com/en-us/library/mt674882.aspx

That these three thematically-distinct languages have all converged on
basically the same syntax and semantics suggests to me this is a nice
design, and one I would like to replicate in Perl 5.

In essence, this is two keywords 'async' and 'await' that combine with
Futures to allow various forms of behaviour to be expressed more easily
than can be done without them.

To be clear: they don't in themselves add any new semantics or
behaviour, they simply allow typical asynchronous code flow to be
expressed in a far more natural and readable way, with less "machinery
noise" than can be achieved with other forms of notation.

((Readers not familiar with the concept of a future might suffice
  with the summary that:

    A future represents the result of an ongoing operation that may or
    may not have completed yet. A Future is in one of three states -
    pending (i.e. incomplete), successful with a result value, or failed
    with an exception. Once in either of the non-pending states it can
    change state no further.

  You might want to at least roughly skim-read
    https://metacpan.org/pod/Future
  on the subject of the ->done, ->fail, and ->get methods.

  Finally it is important to note that a Future can be observed; that
  a callback function can be placed on it that will be invoked when it
  changes state from pending to a final state.))

The async/await syntax comes in the form of two new keywords. 

The 'await' keyword is the simpler to explain initially, in vague
handwavy terms. 'await' operates on an expression of Future type,
yielding a value. If the Future instance value is ready (i.e. it is not
still pending) then this is equivalent to simply fetching the value out
of it using the ->get method; i.e. that

  my $val = await $f;

will fetch the result value of a successful future into $val and will
re-raise the exception contained in a failed future. I will for now
gloss over what happens if $f is still pending.

The 'async' keyword alters the behaviour of a function such that its
return value is always a Future. If the function would otherwise have
made a successful return, the modified function now returns a
successful Future instance with that result. If it would have raised an
exception, it now returns a failed Future instance with that exception.

  async sub { return 1 }    ===>  sub { return Future->done(1) }

  async sub { die "oops" }  ===>  sub { return Future->fail("oops") }

The power of these two keywords comes from their combination, because
they allow us to handle the final case of 'await'; when called on a
Future still pending.

When 'await' is invoked on a pending future, it causes the entire
containing function to be "suspended", returning a new pending future to
its own caller. At some later time when the inner future (the subject
of the 'await' expression) finally completes, the suspended function
is then resumed and takes the course of one of the cases already given
(fetching the result or re-raising the exception). The now-resumed
function continues its operation as normal, exactly as if it had never
suspended, until eventually it too returns a result or throws an
exception. At this point, the future that had earlier been returned to
the initial caller of this function can now be marked as complete with
the appropriate result.

It is a syntax error for an 'await' expression to be used unless its
*immediately* containing CODE block is marked 'async'. This provides a
good degree of localised safety - neither the caller of an 'async'
function, nor any functions called by 'await' expressions, need to care.
From the outside caller's perspective it simply invoked a function that
returned a Future instance, much like any other Future-returning
function does today. Similarly, any function invoked by the
subexpression to 'await' does not need to know; it simply has to return
a Future as normal. These mechanisms obviously can combine - an 'async'
function can clearly be 'await'ed on, but they don't have to be.

It is this element of lexical safety that I believe makes this
mechanism a good fit for Perl. It allows functions to be written using
this syntax that do not affect the operation of the code either side of
them.

As an extension some languages go further into starting to define uses
of 'await' as more than simply an expression; for example they might
suggest a certain amount of concurrency or parallelism by doing the
equivalent of

  await foreach (LIST) { BLOCK }

though for a first-attempt I'm considering that out of scope.

-----

Examples:

This program will print the numbers 1 to 5 in numerical order.

  my $f_inner = Future->new;

  async sub foo {
    say 2;
    await $f_inner;
    say 4;
  }

  say 1;
  my $f_outer = foo();
  say 3;
  $f_inner->done();
  say 5;

Rationale:

This syntax allows us to write a much simpler form of Future chaining
than is otherwise possible using only code blocks and ->then methods.
For example, a function currently written like

  sub foo {
    do_A()->then( sub {
      my ($result_A) = @_;
      do_B($result_A);
    })
  }

can now be written

  async sub foo {
    await do_B( await do_A() );
  }

An important upshot is that rather than starting new nested blocks each
time, we're able to reuse the -same- lexical scope. This allows things
like

  async sub bar {
    my $x = await do_X();

    ... do some things here

    my $y = await do_Y($x);

    ... other things here that can use $x and $y
  }

which would otherwise be a lot more convenient; typically resulting in
a leading

  my $y;

at the top of the function just so that all the nested CODE blocks
inside it can see.

It becomes yet more powerful when we start to look at code examples
where the 'await' keyword does not simply appear in the toplevel
mainline of the function, but instead nested within a conditional or
iteration; for example:

  async sub read_a_nonblank_line {
    while(1) {
      my $line = await read_a_line();
      length $line and return $line;
    }
  }

Which, when written without the async/await syntax using plain Futures
would typically require some helper function like Future::Utils::repeat.

-- 
Paul "LeoNerd" Evans

leonerd@leonerd.org.uk      |  https://metacpan.org/author/PEVANS
http://www.leonerd.org.uk/  |  https://www.tindie.com/stores/leonerd/

• async/await syntax - overview by Paul "LeoNerd" Evans
• Re: async/await syntax - overview by Josh Juran