Front page | perl.perl5.porters |
Postings from December 2012
[perl #116242] t/op/not.t: Misspecified tests; add descriptions to tests lacking them
Thread Next
From:
James E Keenan
Date:
December 29, 2012 16:30
Subject:
[perl #116242] t/op/not.t: Misspecified tests; add descriptions to tests lacking them
Message ID:
rt-3.6.HEAD-17500-1356798584-1792.116242-75-0@perl.org
# New Ticket Created by James E Keenan
# Please include the string: [perl #116242]
# in the subject line of all future correspondence about this issue.
# <URL: https://rt.perl.org:443/rt3/Ticket/Display.html?id=116242 >
Since the St Louis Perl Hackathon in November, I and others have been
writing descriptions for tests in the Perl 5 core distribution's test
suite which currently lack them. I have encountered a case where adding
descriptions to two tests (a) in one case changes the outcome of the
test from PASS to FAIL; and (b) where in both cases the descriptions
fail to print -- as if they had never been written in the first place!
Consider tests 4 and 5 in t/op/not.t:
#####
# test not(..) and !
is(! 1, not 1);
is(! 0, not 0);
#####
Currently, none of the tests in t/op/not.t have descriptions. Hence,
the output of running these two tests is as opaque as that of any other
test in that file:
#####
$ ./perl t/op/not.t
1..16
ok 1
...
ok 4
ok 5
...
ok 16
#####
Now, suppose we add descriptions to all tests in this file. Here I
display only the first five:
#####
pass("logical negation of empty list") if not();
is(not(), 1, "logical negation of empty list in numeric comparison");
is(not(), not(0),
"logical negation of empty list compared with logical negation of
false value");
is(! 1, not 1, q{description 1});
is(! 0, not 0, q{description 0});
#####
The output I get is:
#####
$ ./perl t/op/xnot.t
1..5
ok 1 - logical negation of empty list
ok 2 - logical negation of empty list in numeric comparison
ok 3 - logical negation of empty list compared with logical negation of
false value
ok 4
not ok 5
# Failed test 5 - at t/op/xnot.t line 19
# got "1"
# expected ""
#####
Where did the descriptions for tests 4 and 5 go? And why is test 5 now
failing?
I posed this question on the St Louis and Toronto Perlmongers mailing
lists. Thanks to Nathan Nutter, Scott Smith, Liam R E Quin, Olaf
Alders, Uri Guttman, Tom Legrady, Fulko Hew for participating in the
discussion.
Adapting a diagnostic suggested by Fulko Hew, let's set aside the
"testing" aspect of the problem and define a function which prints out
its arguments:
#####
sub examine {print "arg: <$_>\n" for @_;}
#####
Let's first pass to this function the 3 arguments provided to is()
above, but without the 'not ' in the second argument:
#####
examine(! 1, 1, q{description 1});
examine(! 0, 0, q{description 0});
#####
The output:
#####
arg: <>
arg: <1>
arg: <description 1>
arg: <1>
arg: <0>
arg: <description 0>
#####
The high-precedence '!' operator binds to the number following it and
(apparently) immediately forces that argument to be evaluated. '! 1'
evaluates to Perl-false -- hence the empty string between the
angle-brackets in the first line of output. '! 0' evaluates to
Perl-true -- hence the '1' in the fourth line of output. As we would
intuitively expect, examine() reports 3 arguments in each call. And why
do we intuitively expect that? Because the commas inside the function
call are functioning as item separators within a list.
Now let's add the string 'not ' to the second argument in each call:
#####
print "Prepending 'not ' to second argument\n";
examine(! 1, not 1, q{description 1});
examine(! 0, not 0, q{description 0});
#####
This time, our overall output is:
#####
arg: <>
arg: <1>
arg: <description 1>
arg: <1>
arg: <0>
arg: <description 0>
Prepending 'not ' to second argument
arg: <>
arg: <>
arg: <1>
arg: <>
#####
Here we're seeing the apparently anomalous results we saw when we tried
to add a description to tests 4 and 5 in t/op/not.t. We note:
1. The high-precedence '!' operator is continuing to bind to the number
following it in each call, forcing immediate evaluation. The first and
third lines following "Prepending" correspond to the first and fourth
lines above "Prepending".
2. Each call, however, is only reporting two "arguments" -- and each
"argument" is being evaluated to an empty string (second and fourth
lines following "Prepending").
In the discussion on the Perlmonger mailing lists, Uri Guttman observed
that 'not' isn't just a lower precedence version of !. It is also a
named function with a prototype.
#####
./perl -le 'print prototype "CORE::not"'
$;
#####
Since 'not' as a function is prototyped to expect a scalar argument, the
comma to the right of each of 'not 1' and 'not 0' is no longer
functioning as an item separator in list context. Rather (to quote
'perlop'), "[i]n scalar context it evaluates its left argument, throws
that value away, then evaluates its right argument and returns that
value." The "right argument", in these cases, are q{description 1} and
q{description 0}. When evaluated by 'not' -- now serving as a unary
negation operator -- the expressions evaluate to Perl-false. Hence the
empty strings between the angle brackets in the second and fourth lines
after "Prepending".
If you adapt the diagnostic suggested by Fulko to use either
Test::More::is() or the is() from 't/test.pl', you should get similar
results.
This suggests that tests in 4 and 5 in t/op/not.t are not conducting a
precise test of what you would at first think they are testing. As
written, you would think they're testing that '! 1' and 'not 1' "do the
same thing" and that '! 0' and 'not 0' "do the same thing" as well. But
the fact that adding a description to the test affects the outcome of
the test when run in a testing context means that the tests are not
spelled sufficiently precisely.
Hence, I propose we apply the two patches attached. The first merely
puts parentheses around the second argument in each of tests 4 and 5.
The second adds descriptions to those two tests and to every other test
in the file as well.
Please review.
Thank you very much.
Jim Keenan
Thread Next
-
[perl #116242] t/op/not.t: Misspecified tests; add descriptions to tests lacking them
by James E Keenan