develooper Front page | | Postings from August 2000

Should we dump the perl5 t/ tests?

Thread Next
Brent LaVelle
August 31, 2000 23:07
Should we dump the perl5 t/ tests?
Message ID:
A long time ago on p5p I suggested dumping the current test scheme in
favor of a general purpose testing system where you could test things
programs like:
print "testing testing"
print " 1 2 3"

To verify you get a return code of 255 and something like
/^syntax error at $0 line [23], / from STDERR

Or run code like this:
#!/usr/local/bin/perl -w
my $hashref1={aa => "ay ay"};   #aa should be quoted
my $hashref2={0a => "zero ay"}; #0a should be quoted

Tests that can run this:
#!/usr/local/bin/perl -w

# sawtooth memory allocation with deallocation

use strict;
my $iterations = 100000;
my $i;

my $hash_ref = {
		'0a' => 'apattern'x10,
		'0b' => 'bpattern'x20,
		'0c' => 'cpattern'x30,
		'0d' => 'dpattern'x40

for ($i=1;$i<$iterations;$i++)
    delete $hash_ref->{($i-1).'a'};
    delete $hash_ref->{($i-1).'b'};
    delete $hash_ref->{($i-1).'c'};
    delete $hash_ref->{($i-1).'d'};

and see constant (bound) memory usage.

The observer linear memory usage from a similar example until the
program runs out of virtual memory:
#!/usr/local/bin/perl -w

# sawtooth memory allocation with deallocation

use strict;
my $iterations = 100000;
my $i;

my $hash_ref = {
		'0a' => 'apattern'x10,
		'0b' => 'bpattern'x20,
		'0c' => 'cpattern'x30,
		'0d' => 'dpattern'x40

for ($i=1;$i<$iterations;$i++)
    undef $hash_ref->{($i-1).'a'};
    undef $hash_ref->{($i-1).'b'};
    undef $hash_ref->{($i-1).'c'};
    undef $hash_ref->{($i-1).'d'};

The test system would also be platform aware so we can deal with
differing return codes and the absense of STDERR on various platforms.
Allowing Perl to test itself is like the fox guarding the hen house,
oddly enough it seems to work better than it sounds but I think this
should be re-thought for a number of reasons.

Not all tests finish when something goes wrong, a sane parent should be
  around to terminate the child and clean up resources.

Failing tests can grab too many system resources and needs a parent to
  look after it.

In testing in complex real world tests many dependent test cases are
  needed with dependent test cases that can be nested very deep.

Client server and race condition tests should have a way of running
  multiple processes an synchronizing the processes and verifying the
  results with respect to time.  For example, do A, do B, observe C,
  do D.

Understand known failures and regressions to previous known failures.
  This one is important as it takes what was a bunch of tests and now
  links the test system (repository now) with the bug tracking, version
  and release management systems.

The test system should be able to collect metrics.  What tests are good
  for what.

The metrics should be used to slice and dice the tests.  Based on
  platform (don't run fork tests on NT/VMS), 
  cost (if the test takes 5 hours to run on a high end Alpha
    don't run only for level <3 and if the test fill the $TMP
    partition only run for level 1,...)
  environment (if it connects to MySQL only test when MySQL is 
    available and this testing is desired)
  release (if 6.1 doesn't allow a identifier to start with the Ogham
    feather mark but 6.2 does then only run the test on 6.2 and higher)
  hell-of-it (user doesn't understand or care about signals, ignore
    signal tests)

Tests results should be saved for the "It works for me" discussions.

Tests should be able to run over the network, so from one machine I
  should be able to run tests on all the different type of platforms I
  have access to in my office without slinging a lot of script around.
  (Although I do enjoy doing it but we only need one of those wheels)

Blah blah blah it is getting late but I hope you get the picture (even
  if it seems Pictish is parts :)

When I proposed this before nobody seemed to get fired up, they where
worried about the expensive fork/exec that it added.  I'm worried about
50000 lines of Perl code that test the same 20% of the C (or whatever)
code that Perl is written in but take 100 times longer than the 500
lines needed to test that 20%.

Bottom line: when I tested compilers at Convex many years ago a
different program (that sort of sucks by todays standards) was employed
although you could write a Fortran test system in Fortran (hell the NIST
did, but they actually missed test case failures) but that was too
limiting.  When I tested medical software I could have rigged the
software to test itself but I again chose to write an independent
program to do this testing testing that cannot be done by the
program alone (like sig 9 the program and see if the database could be
hosed), the same at 7-11 although the invoicing system there would never
be able to test itself ;)

Below the bottom line:  the test system should be aware of the Perl
version but be and independent product that produces a set of tests for
Perl, t/ tests.  This test system would be a general purpose tool and
take Perl testing to a new level.

Thread Next Perl Programming lists via nntp and http.
Comments to Ask Bjørn Hansen at | Group listing | About