develooper Front page | perl.perl5.porters | Postings from February 2020

Re: How to bisect for first commit at which a test passes?

Thread Previous | Thread Next
February 2, 2020 00:46
Re: How to bisect for first commit at which a test passes?
Message ID:
James E Keenan <> wrote:
:On 2/1/20 3:16 PM, wrote:
:> I wrote:
:> :James E Keenan <> wrote:
:> :[...]
:> ::perl Porting/ --start=v5.31.2 --end=v5.31.3 \
:> ::-Dcc="g++7" \
:> ::-Accflags="-Wl,-rpath=/usr/local/lib/gcc7" \
:> ::-Aldflags="-Wl,-rpath=/usr/local/lib/gcc7" \
:> ::--expect-fail --target t/op/sprintf2.t
:> ::
:> ::But that ends with this:
:> ::
:> ::t/op/sprintf2 ... ok
:> ::All tests successful.
:> ::Elapsed: 1 sec
:> ::u=0.02  s=0.00  cu=0.24  cs=0.07  scripts=1  tests=1699
:> ::HEAD is now at 45e0524192 5.31.2 today
:> ::bad - zero exit from sh -c cd t && ./perl TEST op\/sprintf2\.t
:> ::Runner returned 256, not 0 for start revision at
:> :
:> :That looks to me like it's saying the test passed at the start revision,
:> :v5.31.2, unexpectedly.
:> To expand on that: expect-fail is implemented by setting expect-pass to 0,
:> and then handled in report_and_exit(), where it is used
:> to invert the return value seen by
:> So the test passed, the test run returned 0, so report_and_exit returned
:> non-zero, and that's why it complains "Runner returned 256, not 0 for start
:> revision".
:Hugo, thanks for spotting that.
:So, what cleared up between v5.31.2 and v5.31.3 was the interaction 
:between the harness and the program.  And that *might* explain the 
:bisection result.
:That's as far as I've gotten.  Further ideas?

So you should bisect on that invocation, which should be possible.

I guess you'll need --target=test_prep; then I don't know if there's
an easier way, but you can always invoke the harness direct from perl
with -e like (untested):
  -e 'chdir("t"); exec qw{ ./perl harness op/sprintf.t }'

Be sure about what you're observing though: check the exit code ($? in
the shell, same as in perl) after those test runs to verify that the
apparently passing one is actually exiting with 0. You are also not
comparing like with like, because your harness invocation is from a
different directory. So to confirm it's the harness that makes the
difference, a fairer check would be something like:

% ( cd t ; ./perl op/sprintf2.t >/dev/null ) ; echo $?
% ( cd t ; ./perl harness op/sprintf2.t >/dev/null ) ; echo $?
% # ^^ if it exited with SEGV, for example


Thread Previous | Thread Next Perl Programming lists via nntp and http.
Comments to Ask Bjørn Hansen at | Group listing | About