develooper Front page | perl.perl5.porters | Postings from March 2022

criteria for ending an experiment

Thread Next
Neil Bowers
March 1, 2022 09:45
criteria for ending an experiment
Message ID:
Prompted by recent discussions on the list, at last week’s meeting the PSC talked about criteria for ending an experiment, and subsequently have iterated on our notes. Here’s our first full draft for discussion and further iteration.

perlpolicy says[1] that before we can drop "experimental" on a feature:

• it has to be experimental in 2 stable releases or more;
• it must be unchanged in behaviour for at least a full development cycle;
• it cannot have any design-changing bugs open against it.

We would like to temper that first point, so that simple language changes can be experimental for just one stable release, in the right circumstances. For example, when a simple function is added to builtin:: in the future, two stable releases as experimental might seem excessive.

For an experiment to be declared a success, we want to see that:

1. the feature has been fully exercised;
2. any bugs identified have been resolved;
3. performance isn't an issue.

"Fully exercised" means we'd like to see evidence of it being used:

• A good start is for it to be used in the software in perl5.git: in tests, Porting tools, etc.
• But preferably we'd like to see some "pull" from real-life Perl programmers, as we have had with signatures, for example.
• Usage in CPAN modules is good evidence, but unrealistic for most experiments, particularly given we want to start resolving them promptly, and most authors stay at least one or two releases behind the leading edge.
• There's a chicken-and-egg situation here: people don't want to use experimental features in production, and generally that's a good call. But non-production personal tools are a good place to play with experimental features.
• People writing tests and examples, to kick the tyres, should be a minimum. So if you want to lobby for something to come out of experiment, email p5p asking people to do this.
• If none of the above are true, and we drop the experimental tag, then we could end up shipping a buggy feature, which we're then committed to supporting.
• If no-one is using an experimental feature, and calls for tyre-kicking result in nothing, then we should consider dropping it – not all experiments have to succeed.

What can we do to make this happen?

• Encourage people to do pull requests to update the perl core to use experimental features. This is a source of drive-by contributions to Perl ("get your name in the AUTHORS file!")
• When we release a stable version of Perl with a new experiment, we should get someone to do a separate blog post on the experiment, (a) to tell people about it, and (b) to ask anyone interested to have a play with it and share their experiences either on p5p or comments on the blog post.
• When we're half-way through the experimental period, review how well it's been exercised, and consider:
• Email p5p asking people to play with it and share their experiences.
• Another public blog post.
• Improve our messaging around the "experimental" label: what it means, and when people might and might not consider using such features.



Thread Next Perl Programming lists via nntp and http.
Comments to Ask Bjørn Hansen at | Group listing | About