Front page | perl.perl5.porters |
Postings from January 2017
Re: test randomization (Re: slowness of ext/XS-APItest/t/handy.t,utf8.t)
Thread Previous
|
Thread Next
From:
Craig A. Berry
Date:
January 26, 2017 02:57
Subject:
Re: test randomization (Re: slowness of ext/XS-APItest/t/handy.t,utf8.t)
Message ID:
CA+vYcVzuJxd-u+H4e9np9RT9_oTgrSqrEHDe=13GG2De4mv-MQ@mail.gmail.com
On Wed, Jan 25, 2017 at 4:33 AM, Dave Mitchell <davem@iabyn.com> wrote:
> On Tue, Jan 24, 2017 at 11:10:50AM -0600, Craig A. Berry wrote:
>> Setting expectations would help, but it's still a major sea change
>> that I don't think has been adequately discussed (sorry if it was and
>> I wasn't paying attention). Things like BBC reports, CPAN smokes, and
>> even basic bisecting depend on everything being the same except the
>> one thing you want to vary. Randomizing input data in tests takes
>> away people's choice about what gets varied.
>>
>> I've always thought the purpose of the test suite was to validate that
>> things known to be good are still good with a different
>> platform/toolchain/configuration/version, etc. There is nothing wrong
>> with exhaustively hunting down things whose goodness is not known, but
>> the core test suite, which is included with the release tarball to
>> certify the release, seems an odd place for that.
>
> I think random subset selection in the test suite should be a method of
> last resort. I believe we do it in one or two places already with Unicode
> stuff, since it would take far too long to test every permutation in that
> case.
>
> When we do this, I think ideally:
>
> 1) the test space should be logically divided into N subsets, and one of
> those subsets is randomly chosen for testing - i.e. there is only a single
> random number generated at the start of the test suite, and that selects
> which set of tests to run - so no doing a 1000-times loop and for each
> iteration choosing a random character and testing it.
> 2) if at least one test fails, then the random number chosen should
> be displayed on stderr (e.g. using diag()) so that it can be seen in smoke
> reports etc. It should also be reported to stdout always;
> 3) there should be a way to run a test script with a specified random number
> (e.g. via an environment variable);
> 4) N should be small enough (e.g. 10,20,30...) that all permutations are
> likely to have been tried by at least one smoker after a small number of
> days, so we don't get a sudden failure 6 months down the line.
> 5) there should be a method (e.g. via an env var) to make all test scripts
> in the test suite run all tests rather than a random subset.
On Wed, Jan 25, 2017 at 6:25 AM, Dave Mitchell <davem@iabyn.com> wrote:
>
> 6) this should be standardised and maybe implemented in test.pl or
> Test::More etc.
7) Some documentation of the above, not sure in which POD file; the
test suite does not seem to have its own.
But generally I like your plan.
> (Now Karl's going to point out how Unicode makes that nice simple scheme
> impractical... ;-)
And why utf8.t and handy.t can never be made fast and will only get
slower as the Unicode Consortium continues to add billions of
emoticons ;-) (ending that sentence with an emoticon). In any case,
sorry to hijack your thread.
Thread Previous
|
Thread Next