develooper Front page | perl.qa | Postings from December 2016

Re: Test::Buildder v. Test2 subtests

Thread Previous | Thread Next
From:
Chad Granum
Date:
December 13, 2016 16:09
Subject:
Re: Test::Buildder v. Test2 subtests
Message ID:
CAJFr3ktWv4VvPN_E52s_2Zv-HtaB_juLN9rRYsbhRZPAukG3hg@mail.gmail.com
I will have to look into this when I have more time.

On Tue, Dec 13, 2016 at 8:06 AM, Ricardo Signes <perl.qa@rjbs.manxome.org>
wrote:

> I've hit a nasty (to me) difference between Test2::API::run_subtest and
> Test::Builder::subtest.  Shout out to Matthew Horsfall for helping
> localize the
> problem.  Here's my trivial reproducer:
>
>   #!perl
>   use strict;
>   use warnings;
>   use Test::More;
>   use Test2::API;
>
>   my $code = sub {
>     if (fork) {
>       wait;
>       cmp_ok($?, '!=', 0, "subprocess died");
>     } else {
>       die "123";
>     };
>   };
>
>   my $call = $ARGV[0] ? \&Test::More::subtest : \&Test2::API::run_subtest;
>   $call->(test => $code);
>   done_testing;
>
> When running with Test::More...
>
>   ~$ perl foo 1
>   # Subtest: test
>   123 at foo line 12.
>       ok 1 - subprocess died
>       1..1
>   ok 1 - test
>   1..1
>
> The die in the subprocess causes the child process to exit nonzero and the
> tests run normally.  Great!  This is how my tests work.  When I switch them
> to Test2::API::run_subtest, though, I get this:
>
>   ~$ perl foo 0
>   # test
>   not ok 1 - test
>   # Failed test 'test'
>   # at foo line 17.
>   # Caught exception in subtest: 123 at foo line 12.
>   1..1
>       not ok 1 - subprocess died
>       #   Failed test 'subprocess died'
>       #   at foo line 10.
>       #          got: 0
>       #     expected: anything else
>       1..1
>   not ok 1 - test
>   # Failed test 'test'
>   # at foo line 17.
>   1..1
>   # Looks like you failed 1 test of 1.
>
> In the forked code, the die causes the subtest to end in failure *in the
> fork*
> meaning we get a repeat result for test 1: one from the child, one from the
> parent.  Also, the child is now exiting zero instead of nonzero, despite
> the
> fact that it threw an exception.
>
> It gets worse [for me]!  Above, I had a cmp_ok on $? that failed, so I had
> duplicate *not ok* results for test 1.  If I replace that cmp_ok with a
> pass, I
> get duplicate *conflicting* results for test 1... and then the main process
> terminates zero.  This means that the test seems to pass if you look at
> its $?,
> but the harness sees it as a failure because its total output does include
> failures (and non-increasing test numbers).
>
> Adding Test2::IPC does not help.  The issue is not synchronising test
> counters,
> or the like.  It's that I don't want the subprocess's exit to cause a
> subtest
> exit in the parent process.  I think that's right: the only process that
> should
> turn exceptions into subtest fails is the process that created the subtest.
>
> a) am I wrong?
> b) advice on how to procede?
>
> --
> rjbs
>

Thread Previous | Thread Next


nntp.perl.org: Perl Programming lists via nntp and http.
Comments to Ask Bjørn Hansen at ask@perl.org | Group listing | About