develooper Front page | perl.perl5.porters | Postings from May 2009

Re: Defining stability and testing it

Thread Previous | Thread Next
From:
chromatic
Date:
May 27, 2009 15:10
Subject:
Re: Defining stability and testing it
Message ID:
200905271510.09182.chromatic@wgz.org
On Wednesday 27 May 2009 14:51:04 Nicholas Clark wrote:

I was under the assumption (based on something you said) that the most difficult 
and tedious part of the pumpking job was going through every commit to 
bleadperl and cherrypicking the right ones for maintperl.  If that's not the 
bottleneck to releasing more frequently, a lot of my other assumptions are 
wrong.

If the real bottleneck is "It takes a lot of time to figure out if this code is 
stable", that's much easier to address.

> > (a) Perl test suite passes on a number of platforms for a variety of
> > configuration options (ideally automated, but is at times just a
> > person saying "yes, it passes") -- what platforms and what config
> > options are enough to be considered 'stable'?
>
> I'm not certain. I think it's roughly
>
> "No tests fail on any platform we are interested in".

I think that's automatable.

> > (c) Does not "break" an arbitrary list of CPAN modules (where "break"
> > means PASS becomes FAIL/NA/UNKNOWN since some prior stable point)  --
> > but on which platforms?  (And are there platform-specific lists?  E.g.
> > don't break Bundle::libwin32 on MSWin32)
>
> Well, I wanted to know the cause of change from PASS of as many CPAN
> modules as could be automatically tested. (I think that Slaven was testing
> on FreeBSD, and Andreas on Linux. Not a great spread, but he who calls the
> piper...)

I think that's automatable too.

> 5: What caused the change?

Given sufficient smoke coverage, I think we can narrow this down.

> 6: "Can you fix it?"

That's not automatable, but it's not scut work like the other questions.

> I don't have a good list of which modules explicitly "must not break", but
> I'd like all the "classic" ones with long dependency chains to build from
> scratch, so things like Plagger, Catalyst, Jifty, Moose and RT spring to
> mind. I don't think that Inline::C is a dependency of any of the above, but
> I'd be unhappy if that broke.

I think that's automatable.

> For anything other than a .0 release - are we binary compatible with the
> previous release(s)?
>
> Roughly
>
> 1: Are all symbols exported by the previous release still exported, and
> still of the same type?
> 2: Are all public structures where size matters the same?
> 3: Do all other public structures start with the same members?

That's a little more difficult to automate, but I think it's doable.

> and, in practical terms:
>
> Do modules compiled with the previous version(s) still work with this
> version.
>
> We're not testing this well, if at all. I think parts of it can be
> automated, but it probably requires running CPAN module tests against an
> installed module, or faking up blib. What we'd need is roughly
>
> 1: Do all the candidate modules prerequisites pass? If no, stop here
> 2: Does the candidate module build and test cleanly against previous
> release? If no, stop here
> 3: Does the candidate module build and test cleanly against pending
> release? If no, this is interesting, but it's a general regression, not a
> binary compatibility issue
> 4: Does the candidate module *built with the previous release* pass its
> tests when run with the pending release?

> Help in scaling this up and automating it would be greatly appreciated.

Could we define a list of interesting platforms (or at least a sufficiently 
interesting subset of platforms we can run in VMs on a computing farm)?  I 
know PITA already solved some of these problems.  With $50,000 I think we 
could make progress in setting up a buildfarm and automating these tests to 
get meaningful (bisectable) results more frequently.

This might require doing something different than Test::Smoke (I'm partial to 
how Smolder works), but would once-a-day feedback on changes and their 
implications to platforms help?  Would once-a-week feedback on changes and 
their implications to CPAN help?

-- c

Thread Previous | Thread Next


nntp.perl.org: Perl Programming lists via nntp and http.
Comments to Ask Bjørn Hansen at ask@perl.org | Group listing | About