Re: Test::Buildder v. Test2 subtests

Actually at first glance a possible problem is this: You are using 2
functions that are not intended to be interchangeable.

Test::More::subtest() is a tool intended to run a subtest.
Test2::API::run_subtest() is a helper intended to write tools like
'subtest()'.

I would like to see the results when you use
Test2::Tools::Subtest::subtest_streamed() which is the actual equivalent to
Test::More::subtest().

-Chad

On Tue, Dec 13, 2016 at 8:09 AM, Chad Granum <exodist7@gmail.com> wrote:

> I will have to look into this when I have more time.
>
> On Tue, Dec 13, 2016 at 8:06 AM, Ricardo Signes <perl.qa@rjbs.manxome.org>
> wrote:
>
>> I've hit a nasty (to me) difference between Test2::API::run_subtest and
>> Test::Builder::subtest.  Shout out to Matthew Horsfall for helping
>> localize the
>> problem.  Here's my trivial reproducer:
>>
>>   #!perl
>>   use strict;
>>   use warnings;
>>   use Test::More;
>>   use Test2::API;
>>
>>   my $code = sub {
>>     if (fork) {
>>       wait;
>>       cmp_ok($?, '!=', 0, "subprocess died");
>>     } else {
>>       die "123";
>>     };
>>   };
>>
>>   my $call = $ARGV[0] ? \&Test::More::subtest : \&Test2::API::run_subtest;
>>   $call->(test => $code);
>>   done_testing;
>>
>> When running with Test::More...
>>
>>   ~$ perl foo 1
>>   # Subtest: test
>>   123 at foo line 12.
>>       ok 1 - subprocess died
>>       1..1
>>   ok 1 - test
>>   1..1
>>
>> The die in the subprocess causes the child process to exit nonzero and the
>> tests run normally.  Great!  This is how my tests work.  When I switch
>> them
>> to Test2::API::run_subtest, though, I get this:
>>
>>   ~$ perl foo 0
>>   # test
>>   not ok 1 - test
>>   # Failed test 'test'
>>   # at foo line 17.
>>   # Caught exception in subtest: 123 at foo line 12.
>>   1..1
>>       not ok 1 - subprocess died
>>       #   Failed test 'subprocess died'
>>       #   at foo line 10.
>>       #          got: 0
>>       #     expected: anything else
>>       1..1
>>   not ok 1 - test
>>   # Failed test 'test'
>>   # at foo line 17.
>>   1..1
>>   # Looks like you failed 1 test of 1.
>>
>> In the forked code, the die causes the subtest to end in failure *in the
>> fork*
>> meaning we get a repeat result for test 1: one from the child, one from
>> the
>> parent.  Also, the child is now exiting zero instead of nonzero, despite
>> the
>> fact that it threw an exception.
>>
>> It gets worse [for me]!  Above, I had a cmp_ok on $? that failed, so I had
>> duplicate *not ok* results for test 1.  If I replace that cmp_ok with a
>> pass, I
>> get duplicate *conflicting* results for test 1... and then the main
>> process
>> terminates zero.  This means that the test seems to pass if you look at
>> its $?,
>> but the harness sees it as a failure because its total output does include
>> failures (and non-increasing test numbers).
>>
>> Adding Test2::IPC does not help.  The issue is not synchronising test
>> counters,
>> or the like.  It's that I don't want the subprocess's exit to cause a
>> subtest
>> exit in the parent process.  I think that's right: the only process that
>> should
>> turn exceptions into subtest fails is the process that created the
>> subtest.
>>
>> a) am I wrong?
>> b) advice on how to procede?
>>
>> --
>> rjbs
>>
>
>

• Test::Buildder v. Test2 subtests by Ricardo Signes
• Re: Test::Buildder v. Test2 subtests by Chad Granum
• Re: Test::Buildder v. Test2 subtests by Chad Granum
• Re: Test::Buildder v. Test2 subtests by Ricardo Signes
• Re: Test::Buildder v. Test2 subtests by Chad Granum
• Re: Test::Buildder v. Test2 subtests by Ricardo Signes