Front page | perl.bootstrap |
Postings from July 2000
In One Sentence
---------------
All patches to perl must have an associated testing patch.
Synopsis
--------
It has often be said that perl defines perl. I'd like to change that a bit
and say perl's regression tests should define perl. The state of the
existing tests is somewhat inadequate to the task. Part of the problem is
tests are written whenever someone feels like adding a test.
Instead, every new feature added/changed and bug fixed (ie. every code
patch) must have an associated patch to the testing suite. No test, no
acceptance. Period.
While this may seem draconian, it should rapidly raise the overall quality
of the patches by forcing authors to do more than dash off a little C code.
Problems
--------
If we have 5000 patches does that mean 5000 tests? Yes. Won't that take a
long time to run? Probably. Won't that make building Perl really annoying?
Probably. Solution? Break up the tests into mandatory and secondary. The
mandatory tests would be similar to what t/ is now and would be distributed
with Perl and run at every build. The complete set of secondary tests would
only need to be run periodically by the perl6 developers and not necessarily
with every build. "make more tests" or something. If the set of secondary
tests grows too large, it could be distributed seperately from the main
distribution.
Very few people write regression suites on a regular basis. As a result,
there will be many more people who know how to write code patches for Perl
than know how to write good tests for Perl. A testing tutorial will have to
be written to ease people along and get them comfortable with writing tests.
It is encouraged, but not necessary, that every patch come with a test. It
is simply necessary that the patch eventually aquire a test sometime before
its being accepted. As a result if someone is good at writing code, but not
tests its okay. Somebody will (hopefully) pick up the slack. Writing tests
also provides a "way in" for people who do not yet understand the perl
internals yet wish to help out with development.
Having a really cool code patch rejected because there's no test might
cheese people off. This policy, combined with the "Proper Addition Of New
Tests" RFC, provides a boolean, automated, impartial veto which hopefully
will minimize complaints.
*All* patches? Even documentation patches? Pretty much. There will
be exceptions, but even documentation can be tested. (See the
following "Embedded Tests" RFCs).
Corellaries
-----------
The regression tests should become a defacto standard of what defines perl.
Individual test suites can also test work-alike libraries, such as libraries
attempting to emulate perl's regexes.
This RFC will mesh in with the upcoming RFC about minimum patching
standards.
Things to do NOW
--------------
- Write a "how to test perl" tutorial/guide. [any takers?]
- Begin gnawing away at p5p for general acceptance of the one patch/one test
philosophy.
- Work on a simple script to automate test/code patches (or retrofit the
existing one).
Blame
-----
All blame for this RFC can be transfered to Michael G Schwern
<schwern@pobox.com>
--
Michael G Schwern http://www.pobox.com/~schwern/ schwern@pobox.com
Just Another Stupid Consultant Perl6 Kwalitee Ashuranse
Sometimes these hairstyles are exaggerated beyond the laws of physics
- Unknown narrator speaking about Anime
Thread Next