Front page | perl.perl5.porters |
Postings from May 2009
Re: Defining stability and testing it
Thread Previous
|
Thread Next
From:
David Golden
Date:
May 27, 2009 19:25
Subject:
Re: Defining stability and testing it
Message ID:
5d4beb40905271925t52fc34c8u687b3c8a69925bdb@mail.gmail.com
On Wed, May 27, 2009 at 5:51 PM, Nicholas Clark <nick@ccl4.org> wrote:
> That's not "all tests pass" as we may need to mark tests as todo for
> particular platforms, but we need to know which tests, and which platforms,
> and why. (I'm thinking for "it's a bug not yet fixed", which is much the same
> as "it's a new test", not, "it's an existing test, so this is a regression we
> want to ship")
I mean "all tests pass" in the sense of the harness reporting success,
which would ignore TODO failures.
> The level of knowledge drills down:
>
> 1: Which modules changed state?
> [Can certainly be automated]
>
> 2: What is the nature of the test failure?
> [ie what perl -Mblib t/what_goes_here.t do I want to run to see it?]
>
> 3: Which core change caused this?
> [distill the above to a script that can be fed to git bisect]
>
> 4: Which other modules fail in the same way?
> [hopefully as simple as "group newly failing modules by step 3]
>
> at which point it's possible to start to triage this cluster of failures.
> I'm not sure how automatable it is, but it's certainly distributable
> (and it was, to a degree, but Slaven and Andreas need not be the only ones
> doing it. Berlin 2, rest of world 0)
>
> And then the tougher parts
>
> 5: What caused the change?
>
> 6: "Can you fix it?"
The diagnostic might be automatable, but sounds more like the work to
resolve breakage after it's detected. Or is there something in all
that analysis that would lead you to make a determination of whether
the breakage can be ignored? (Aside from VMS, which I assume is often
the case.) So, yes, I can see how this helps you, but is it really
more than is necessary to get the green/red light?
> For anything other than a .0 release - are we binary compatible with the
> previous release(s)?
>
> Roughly
>
> 1: Are all symbols exported by the previous release still exported, and still
> of the same type?
> 2: Are all public structures where size matters the same?
> 3: Do all other public structures start with the same members?
OK. I'll assume that someone with decent C skills can code up some tests.
> 4: Does the candidate module *built with the previous release* pass its tests
> when run with the pending release?
Hmm. That's going to be a little trickier than what we can do easily
by adapting normal CPAN testing setups, but not impossible.
-- David
Thread Previous
|
Thread Next