[perl #116242] t/op/not.t: Misspecified tests; add descriptions to tests lacking them

# New Ticket Created by  James E Keenan 
# Please include the string:  [perl #116242]
# in the subject line of all future correspondence about this issue. 
# <URL: https://rt.perl.org:443/rt3/Ticket/Display.html?id=116242 >


Since the St Louis Perl Hackathon in November, I and others have been 
writing descriptions for tests in the Perl 5 core distribution's test 
suite which currently lack them.  I have encountered a case where adding 
descriptions to two tests (a) in one case changes the outcome of the 
test from PASS to FAIL; and (b) where in both cases the descriptions 
fail to print -- as if they had never been written in the first place!

Consider tests 4 and 5 in t/op/not.t:

#####
# test not(..) and !
is(! 1, not 1);
is(! 0, not 0);
#####

Currently, none of the tests in t/op/not.t have descriptions.  Hence, 
the output of running these two tests is as opaque as that of any other 
test in that file:

#####
$ ./perl t/op/not.t
1..16
ok 1
...
ok 4
ok 5
...
ok 16
#####

Now, suppose we add descriptions to all tests in this file.  Here I 
display only the first five:

#####
pass("logical negation of empty list") if not();
is(not(), 1, "logical negation of empty list in numeric comparison");
is(not(), not(0),
     "logical negation of empty list compared with logical negation of 
false value");

is(! 1, not 1, q{description 1});
is(! 0, not 0, q{description 0});
#####

The output I get is:

#####
$ ./perl t/op/xnot.t
1..5
ok 1 - logical negation of empty list
ok 2 - logical negation of empty list in numeric comparison
ok 3 - logical negation of empty list compared with logical negation of 
false value
ok 4
not ok 5
# Failed test 5 - at t/op/xnot.t line 19
#      got "1"
# expected ""
#####

Where did the descriptions for tests 4 and 5 go?  And why is test 5 now 
failing?

I posed this question on the St Louis and Toronto Perlmongers mailing 
lists.  Thanks to Nathan Nutter, Scott Smith, Liam R E Quin, Olaf 
Alders, Uri Guttman, Tom Legrady, Fulko Hew for participating in the 
discussion.

Adapting a diagnostic suggested by Fulko Hew, let's set aside the 
"testing" aspect of the problem and define a function which prints out 
its arguments:

#####
sub examine {print "arg: <$_>\n" for @_;}
#####

Let's first pass to this function the 3 arguments provided to is() 
above, but without the 'not ' in the second argument:

#####
examine(! 1, 1, q{description 1});
examine(! 0, 0, q{description 0});
#####

The output:

#####
arg: <>
arg: <1>
arg: <description 1>
arg: <1>
arg: <0>
arg: <description 0>
#####

The high-precedence '!' operator binds to the number following it and 
(apparently) immediately forces that argument to be evaluated.  '! 1' 
evaluates to Perl-false -- hence the empty string between the 
angle-brackets in the first line of output.  '! 0' evaluates to 
Perl-true -- hence the '1' in the fourth line of output.  As we would 
intuitively expect, examine() reports 3 arguments in each call.  And why 
do we intuitively expect that?  Because the commas inside the function 
call are functioning as item separators within a list.

Now let's add the string 'not ' to the second argument in each call:

#####
print "Prepending 'not ' to second argument\n";
examine(! 1, not 1, q{description 1});
examine(! 0, not 0, q{description 0});
#####

This time, our overall output is:
#####
arg: <>
arg: <1>
arg: <description 1>
arg: <1>
arg: <0>
arg: <description 0>
Prepending 'not ' to second argument
arg: <>
arg: <>
arg: <1>
arg: <>
#####

Here we're seeing the apparently anomalous results we saw when we tried 
to add a description to tests 4 and 5 in t/op/not.t.  We note:

1. The high-precedence '!' operator is continuing to bind to the number 
following it in each call, forcing immediate evaluation.  The first and 
third lines following "Prepending" correspond to the first and fourth 
lines above "Prepending".

2. Each call, however, is only reporting two "arguments" -- and each 
"argument" is being evaluated to an empty string (second and fourth 
lines following "Prepending").

In the discussion on the Perlmonger mailing lists, Uri Guttman observed 
that 'not' isn't just a lower precedence version of !.  It is also a 
named function with a prototype.

#####
./perl -le 'print prototype "CORE::not"'
$;
#####

Since 'not' as a function is prototyped to expect a scalar argument, the 
comma to the right of each of 'not 1' and 'not 0' is no longer 
functioning as an item separator in list context.  Rather (to quote 
'perlop'), "[i]n scalar context it evaluates its left argument, throws 
that value away, then evaluates its right argument and returns that 
value."  The "right argument", in these cases, are q{description 1} and 
q{description 0}.  When evaluated by 'not' -- now serving as a unary 
negation operator -- the expressions evaluate to Perl-false.  Hence the 
empty strings between the angle brackets in the second and fourth lines 
after "Prepending".

If you adapt the diagnostic suggested by Fulko to use either 
Test::More::is() or the is() from 't/test.pl', you should get similar 
results.

This suggests that tests in 4 and 5 in t/op/not.t are not conducting a 
precise test of what you would at first think they are testing.  As 
written, you would think they're testing that '! 1' and 'not 1' "do the 
same thing" and that '! 0' and 'not 0' "do the same thing" as well.  But 
the fact that adding a description to the test affects the outcome of 
the test when run in a testing context means that the tests are not 
spelled sufficiently precisely.

Hence, I propose we apply the two patches attached.  The first merely 
puts parentheses around the second argument in each of tests 4 and 5. 
The second adds descriptions to those two tests and to every other test 
in the file as well.

Please review.

Thank you very much.
Jim Keenan

• [perl #116242] t/op/not.t: Misspecified tests; add descriptions to tests lacking them by James E Keenan
• Re: [perl #116242] t/op/not.t: Misspecified tests; add descriptionsto tests lacking them by Aristotle Pagaltzis