Front page | perl.perl5.porters |
Postings from February 2020
Re: How to bisect for first commit at which a test passes?
Thread Previous
|
Thread Next
From:
James E Keenan
Date:
February 2, 2020 01:26
Subject:
Re: How to bisect for first commit at which a test passes?
Message ID:
20200202012636.3069.qmail@lists-nntp.develooper.com
On 2/1/20 7:29 PM, hv@crypt.org wrote:
> James E Keenan <jkeenan@pobox.com> wrote:
> :On 2/1/20 3:16 PM, hv@crypt.org wrote:
> :> I wrote:
> :> :James E Keenan <jkeenan@pobox.com> wrote:
> :> :[...]
> :> ::perl Porting/bisect.pl --start=v5.31.2 --end=v5.31.3 \
> :> ::-Dcc="g++7" \
> :> ::-Accflags="-Wl,-rpath=/usr/local/lib/gcc7" \
> :> ::-Aldflags="-Wl,-rpath=/usr/local/lib/gcc7" \
> :> ::--expect-fail --target t/op/sprintf2.t
> :> ::
> :> ::But that ends with this:
> :> ::
> :> ::t/op/sprintf2 ... ok
> :> ::All tests successful.
> :> ::Elapsed: 1 sec
> :> ::u=0.02 s=0.00 cu=0.24 cs=0.07 scripts=1 tests=1699
> :> ::HEAD is now at 45e0524192 5.31.2 today
> :> ::bad - zero exit from sh -c cd t && ./perl TEST op\/sprintf2\.t
> :> ::Runner returned 256, not 0 for start revision at
> :> :
> :> :That looks to me like it's saying the test passed at the start revision,
> :> :v5.31.2, unexpectedly.
> :>
> :> To expand on that: expect-fail is implemented by setting expect-pass to 0,
> :> and then handled in bisect-runner.pl report_and_exit(), where it is used
> :> to invert the return value seen by bisect.pl.
> :>
> :> So the test passed, the test run returned 0, so report_and_exit returned
> :> non-zero, and that's why it complains "Runner returned 256, not 0 for start
> :> revision".
> :
> :Hugo, thanks for spotting that.
> :
> [...]
> :So, what cleared up between v5.31.2 and v5.31.3 was the interaction
> :between the harness and the program. And that *might* explain the
> :bisection result.
> :
> :That's as far as I've gotten. Further ideas?
>
> So you should bisect on that invocation, which should be possible.
>
> I guess you'll need --target=test_prep; then I don't know if there's
> an easier way, but you can always invoke the harness direct from perl
> with -e like (untested):
> -e 'chdir("t"); exec qw{ ./perl harness op/sprintf.t }'
>
> Be sure about what you're observing though: check the exit code ($? in
> the shell, same as in perl) after those test runs to verify that the
> apparently passing one is actually exiting with 0. You are also not
> comparing like with like, because your harness invocation is from a
> different directory. So to confirm it's the harness that makes the
> difference, a fairer check would be something like:
>
> % ( cd t ; ./perl op/sprintf2.t >/dev/null ) ; echo $?
> 0
> % ( cd t ; ./perl harness op/sprintf2.t >/dev/null ) ; echo $?
> 139
> % # ^^ if it exited with SEGV, for example
>
> Hugo
>
Data for just the last part of your recommendations:
#####
[perl] $ ./perl -v | head -2 | tail -1
This is perl 5, version 31, subversion 2 (v5.31.2) built for
amd64-freebsd-thread-multi
[perl] $ ( cd t ; ./perl op/sprintf2.t >/dev/null ) ; echo $?
9
[perl] $ ( cd t ; ./perl harness op/sprintf2.t >/dev/null ) ; echo $?
9
[perl] $ cd ../perl2
[perl2] $ ./perl -v | head -2 | tail -1
This is perl 5, version 31, subversion 3 (v5.31.3) built for
amd64-freebsd-thread-multi
[perl2] $ ( cd t ; ./perl op/sprintf2.t >/dev/null ) ; echo $?
0
[perl2] $ ( cd t ; ./perl harness op/sprintf2.t >/dev/null ) ; echo $?
0
#####
Thank you very much.
Jim Keenan
Thread Previous
|
Thread Next