develooper Front page | perl.perl5.porters | Postings from May 2009

Defining stability and testing it

Thread Next
From:
David Golden
Date:
May 27, 2009 09:08
Subject:
Defining stability and testing it
Message ID:
5d4beb40905270908ha6030e1s96a2e0f0844f7c5e@mail.gmail.com
I've been thinking about the other thread that started with git
progress and devolved into release management, and one thing that
really jumped out at me is the constant refrain that what is "hard" in
preparing for a Perl release is knowing that a particular branch is
"stable".  (Depending on the pumpking, it sounded like compiling
perldelta is either "hard" or just "tedious")

So regardless of whether there is a shift towards more frequent
releases or more use of feature branches, it seems like we need to
attack the hard problem of knowing whether something is stable.  So
that means:

(1) We need to define some testable criteria for "stable"

(2) We need to be able to test arbitrary commit-points for stability

If we can do #1, then we do #2.  If we can automated most or all of #2
then we make the problem much less "hard".  If we can scale #2, then
we can test branches more frequently and/or test more branches at
once.  If we can report #2 in some centralized way, then we can make
stability blockers more obvious and attract attention towards fixing
them.

So for #1, what are some candidate criteria?

(a) Perl test suite passes on a number of platforms for a variety of
configuration options (ideally automated, but is at times just a
person saying "yes, it passes") -- what platforms and what config
options are enough to be considered 'stable'?

(b) Perl release checks pass (just on pumpking's platform?)

(c) Does not "break" an arbitrary list of CPAN modules (where "break"
means PASS becomes FAIL/NA/UNKNOWN since some prior stable point)  --
but on which platforms?  (And are there platform-specific lists?  E.g.
don't break Bundle::libwin32 on MSWin32)

What else?

I don't know the Perl guts well enough to productively hack there, but
I do know testing.  I've done (c) for Module::Build a couple times now
and have it as one of the use cases guiding for my work on CPAN
Testers 2.0.  So this is an area where I think I can contribute and I
suspect others in "the community" might be willing to work as well.

Maybe that doesn't address Nicholas' concern that there are less
people closing bugs, but if we can ease some of the "hard" problems,
maybe that will help more active porters be more productive with the
time they do have.

-- David

Thread Next


nntp.perl.org: Perl Programming lists via nntp and http.
Comments to Ask Bjørn Hansen at ask@perl.org | Group listing | About