Front page | perl.perl5.porters |
Postings from May 2008
Re: TODO Tests
Thread Previous
|
Thread Next
From:
Bram
Date:
May 6, 2008 07:30
Subject:
Re: TODO Tests
Message ID:
20080506162918.e7b0y50pwwogoco4@horde.wizbit.be
Quoting Abigail <abigail@abigail.be>:
> On Mon, May 05, 2008 at 08:36:18PM +0200, Bram wrote:
>> While adding the todo tests for t/op/range.t I ran into some
>> limitations (IMHO).
>>
>> Currently the way to define a TODO test is:
>>
>> {
>> local $TODO = "test that foo() returns 1";
>> my $s1 = foo();
>> is($s1, "1");
>> }
>>
>>
>> foo() at the moment returns 3 instead of 1. Which is fine.
>
> I've read your suggestion a couple of times, and I think I'm lost at this
> step.
>
> If foo() returning 3 is fine, then why are we testing for it returning 1?
I see why you are confused... bad choice of words...
foo() returns something else then 1 (the value that is wanted).
Therefor a todo test gets added.
It is known that foo(), at the moment, returns 3.
If it starts returning something else then 3 then *something* changed
the behaviour - maybe/probably unintended.
>> Next month, as a side effect of a patch, foo() now returns 4.
>>
>> This is a step in the wrong direction but the tests suite won't pick
>> this up, since it still isn't equal to 1.
>>
>> Or an alternative: next month, as a side effect of a patch, foo()
>> now returns 2.
>>
>> The test suite won't pick this up either.
>>
>>
>> Proposal:
>>
>> Adding some syntax to support (among other things):
>>
>> todo_is($value, $value_wanted, $expected_value, $name_of_test);
>> (or perhaps: todo_is($value, $value_wanted, $name_of_test,
>> $expected_value);)
>>
>> Or following the example:
>> my $s1 = foo();
>> todo_is($s1, "1", "3", "test that foo() returns 1");
>>
>>
>> Now there are three options:
>>
>> a) $s1 contains 3, no special output is printed
>> b) $s1 contains 1, output the same as it is now: (basically)
>> unexpected pass for todo: "test that foo() returns 1"
>> c) $s1 contains 2 or 4, output: unexpected output for todo: "test that
>> foo() returns 1"
>>
>>
>> Any opinions on this? (Or better asked: are there reasons against this?)
>> (Or am I repeating an old idea?)
>
> I'm not getting it. Are you proposing we have regression tests that trigger
> a 'not ok' if it returns another wrong value?
I want it to trigger something but not a 'not ok' because it still is
a todo test (and therfor the result of the test suite should be pass
and not fail).
It should be marked as an unexpected result from a TODO test.
> If so, can't we just do:
>
> {
> my $s1 = foo ();
> #
> # Remove next two lines when bug gets fixed.
> #
> is ($1, 3, "Known wrong value");
> local $TODO = "It should really be 1";
> is ($1, 1, "Right");
> }
In theory yes, but that would produce a FAIL if foo() returns
something else then 3.
> But I'm still confused on what you are trying to catch.
What I want (when running cd t ../perl harness op/range.t ):
* foo() returns 1: (What we really want)
All tests successful.
Test Summary Report
-------------------
op/range.t (Wstat: 0 Tests: 135 Failed: 0)
TODO passed: 122
Files=1, Tests=135, 0 wallclock secs ( 0.05 usr 0.00 sys + 0.04
cusr 0.02 csys = 0.11 CPU)
Result: PASS
* foo() returns 3: (What it returned when the TODO test was added)
op/range......ok
All tests successful.
Files=1, Tests=135, 1 wallclock secs ( 0.07 usr 0.00 sys + 0.04
cusr 0.00 csys = 0.11 CPU)
Result: PASS
* foo() returns 4: (What is not expected)
All tests successful.
Test Summary Report
-------------------
op/range.t (Wstat: 0 Tests: 135 Failed: 0)
TODO unexpected output: 122 (output was 4 - expected 3)
Files=1, Tests=135, 0 wallclock secs ( 0.05 usr 0.00 sys + 0.04
cusr 0.02 csys = 0.11 CPU)
Result: PASS
Did this clear some things up?
(I will re-submit this idea (with the feedback integrated) to the
perl.qa and/or tap list.)
Kind regards,
Bram
Thread Previous
|
Thread Next