develooper Front page | perl.perl5.porters | Postings from August 2018

env vars for slow tests

Thread Next
From:
Dave Mitchell
Date:
August 10, 2018 08:28
Subject:
env vars for slow tests
Message ID:
20180810082844.GI2798@iabyn.com
Currently we seem to have a collection of ad-hoc environment variables for
skipping slow tests on slow machines and/or restricting the number or type
of test runs.

I'd like to rationalise these.

For example looking for strings of the form $ENV{PERL_....} under t/ and
in *,t test files, shows the following apparently timing-related vars
(along with their counts):

   2 PERL_DEBUG_FULL_TEST
        used by a few test files. Mentioned briefly in pod/perllocale.pod
        causes more tests to be run

   5 PERL_RUN_SLOW_TESTS
        used by Storable, lib/charnames.t and t/re/fold_grind.t
        but not documented outside those files

   1 PERL_SKIP_PSYCHO_TEST
        used by t/re/pat_psycho.t to skip the entire test file

   1 PERL_TEST_NUMCONVERTS
        used only by t/op/numconvert.t. Seems to affect how many tests it
        runs.

   2 PERL_TEST_TIME_OUT_FACTOR
        used by t/re/fold_grind.t and t/re/pat_psycho.t

Most of these vars are undocumented.  There's also

  14 PERL_TEST_MEMORY
    actually documented! Says how many Gb available for testing.
    If set, runs 7 test scripts under t/bigmem.
    Also used by Storable, 

I would like to eliminate most of these, and where possible:

1) auto-adapt - for example, in each problematic test script, time the
running of a first batch of tests, and based on that result, decide how
many more batches to run and/or skip.

2) If not possible to auto-adapt, have a *single* documented var which
indicates a 'slowness' factor for the host machine's CPU (c.f.
PERL_TEST_MEMORY to indicate how many Gb the host has for testing) and get
all test scripts to use that one var.

Then owners of smoke machines can just set that one var (and perhaps
PERL_TEST_MEMORY) and avoid the current random failures due to watchdog
timeouts.

I haven't (yet) made any firm suggestions, in case anyone already has some
ideas or WIP.

-- 
"Emacs isn't a bad OS once you get used to it.
It just lacks a decent editor."

Thread Next


nntp.perl.org: Perl Programming lists via nntp and http.
Comments to Ask Bjørn Hansen at ask@perl.org | Group listing | About