develooper Front page | perl.bootstrap | Postings from July 2000

RFC: Proper Addition Of New Tests

Michael G Schwern
July 26, 2000 13:52
RFC: Proper Addition Of New Tests
Message ID:
In One Sentence

It will be checked that new tests fail (they find the bug) before the fix is


Its is often forgotten that tests are code.  And code has bugs.  Very often
the test won't actually be testing what you think its testing.  So how do
you test a test?

A very simple way is to apply the test and make sure it fails, apply your
code patch, then re-run the test to make sure it passes.  This can all be
automated and should reduce the number of tests which don't work.


A simple patching wrapper can be written (or extend the existing wrapper
around patch that perl5 uses) which automates this process.

1)  Apply the test.
2)  Run the test.  Make sure it fails.
3)  Apply the code patch.
4)  Run the test again, make sure it passes.

Simple.  Stupid.  And there's no reason this can't be applied to Perl5.


It is necessary to enforce one patch, one test in order to make this
technique feasible.  So this RFC is linked to my other RFC about A Test For
Every Patch.

Things to do NOW

- Write the testing patching program and/or extend perl5's existing patch 
  wrapper, whichever is easier.  [Up for grabs]

- Begin using this technique on perl5 (independently, or with p5p's
  blessing) and gather statistics on how often tests fail to test what 
  they're testing.  [Up for grabs, too]


All blame for this RFC can be transfered to Michael G Schwern

Michael G Schwern
Just Another Stupid Consultant                      Perl6 Kwalitee Ashuranse
But why?  It's such a well designed cesspool of C++ code.  Why wouldn't
you want to hack mozilla?
                -- Ziggy Perl Programming lists via nntp and http.
Comments to Ask Bjørn Hansen at | Group listing | About