In One Sentence --------------- All patches to perl must have an associated testing patch. Synopsis -------- It has often be said that perl defines perl. I'd like to change that a bit and say perl's regression tests should define perl. The state of the existing tests is somewhat inadequate to the task. Part of the problem is tests are written whenever someone feels like adding a test. Instead, every new feature added/changed and bug fixed (ie. every code patch) must have an associated patch to the testing suite. No test, no acceptance. Period. While this may seem draconian, it should rapidly raise the overall quality of the patches by forcing authors to do more than dash off a little C code. Problems -------- If we have 5000 patches does that mean 5000 tests? Yes. Won't that take a long time to run? Probably. Won't that make building Perl really annoying? Probably. Solution? Break up the tests into mandatory and secondary. The mandatory tests would be similar to what t/ is now and would be distributed with Perl and run at every build. The complete set of secondary tests would only need to be run periodically by the perl6 developers and not necessarily with every build. "make more tests" or something. If the set of secondary tests grows too large, it could be distributed seperately from the main distribution. Very few people write regression suites on a regular basis. As a result, there will be many more people who know how to write code patches for Perl than know how to write good tests for Perl. A testing tutorial will have to be written to ease people along and get them comfortable with writing tests. It is encouraged, but not necessary, that every patch come with a test. It is simply necessary that the patch eventually aquire a test sometime before its being accepted. As a result if someone is good at writing code, but not tests its okay. Somebody will (hopefully) pick up the slack. Writing tests also provides a "way in" for people who do not yet understand the perl internals yet wish to help out with development. Having a really cool code patch rejected because there's no test might cheese people off. This policy, combined with the "Proper Addition Of New Tests" RFC, provides a boolean, automated, impartial veto which hopefully will minimize complaints. *All* patches? Even documentation patches? Pretty much. There will be exceptions, but even documentation can be tested. (See the following "Embedded Tests" RFCs). Corellaries ----------- The regression tests should become a defacto standard of what defines perl. Individual test suites can also test work-alike libraries, such as libraries attempting to emulate perl's regexes. This RFC will mesh in with the upcoming RFC about minimum patching standards. Things to do NOW -------------- - Write a "how to test perl" tutorial/guide. [any takers?] - Begin gnawing away at p5p for general acceptance of the one patch/one test philosophy. - Work on a simple script to automate test/code patches (or retrofit the existing one). Blame ----- All blame for this RFC can be transfered to Michael G Schwern <schwern@pobox.com> -- Michael G Schwern http://www.pobox.com/~schwern/ schwern@pobox.com Just Another Stupid Consultant Perl6 Kwalitee Ashuranse Sometimes these hairstyles are exaggerated beyond the laws of physics - Unknown narrator speaking about AnimeThread Next