Yet Another TODO Model for Test.pm

The ongoing search to find a good model for TODO tests in Pugs 
continues ...

To start with, the idea of some kind of "switch" to turn all failures 
into TODOs has been removed. It was a bad idea from the start and since 
it never actually was implemented, I have ditched it entirely.

Second, I would like to move away from the force_todo() usage, as it 
has a number of obvious problems. However, I cannot dismiss it's 
usefulness as a quick-fix during release preparation, so i will not yet 
remove it.

It was determined (on #perl6 and various emails) that what we really 
need in Pugs is a multi-layered TODO model. In Pugs we have several 
different kinds of TODO tests, along with the plain old TODO. Here is a 
list of some of the more common ones and examples of how the attributes 
might look;

- TODO

:todo, :todo(1)

Sometimes, it just doesn't fit anywhere else, so in that case it is 
just TODO.

- TODO bug

:todo<bug>, todo => 'bug', :todo('bug')

This is either an outright bug in Pugs, or it is a feature which 
/should/ be implemented, but for some reason (usually some kind of 
seemingly unrelated bug) it is not. This is a good "layer" for edge 
cases tests and other such ugly corners.

- TODO feature

:todo<feature>, todo => 'feature', :todo('feature')

These are tests written for perl6 features which are not yet 
implemented in Pugs. Objects is the perfect example of this.

- TODO parsefail

:todo<parsefail>, todo => 'parsefail', :todo('parsefail')

These are tests which fail to parse correctly. These are usually tested 
with eval_ok() or eval_is().

- TODO hardfail

:todo<hardfail>, todo => 'hardfail', :todo('hardfail')

These are tests which parse okay, but die with an uncatchable error. I 
will be experimenting with using the "is lazy" trait to catch and 
not-evaluate these (and therefore not kill pugs).

My goal was to support these different kinds of TODOs with Test.pm, and 
also make them visible through Test::Harness (and eventually in our 
smoke test). In exploring Test::Harness I noticed that TODO tests can 
also have a "reason", just like SKIP tests do (although Test::Harness 
seems to ignore them) and I have exploited this feature to accomplish 
my goal.

So currently (in the latest revision), Test.pm will support the 
following code:

ok(bad_function(), "... testing bad_function", :todo<bug>);

this will produce the following TAP output:

not ok 1 - ... testing bad_function # TODO bug
#   Failed (TODO bug) test (-e line 3, column 1-46)
#        Got: undef

I have also altered our local copy of Test::Harness (in inc/) so that 
if you were to run this with 'prove' or 'make test' you would see 
something like this:

t/my_test_file....ok
          1/5 TODO bug test
          2/5 TODO tests
          2/5 TODO feature tests
All tests successful.

This allows a simple top-level overview of not only the number of TODO 
tests, but of each specific TODO test "layer". Ideally we can support 
this level of TODO granularity in the test smoke interface as well.

Now all that is left is for people to start changing all the :todo 
tests to :todo<some_attribute>.

As always comments, question and suggestions are always welcome.

Thanks,

Stevan

• Yet Another TODO Model for Test.pm by Stevan Little
• Re: Yet Another TODO Model for Test.pm by Nathan Gray
• Re: Yet Another TODO Model for Test.pm by Stevan Little
• Re: Yet Another TODO Model for Test.pm by Nathan Gray