develooper Front page | perl.perl5.porters | Postings from February 2020

How to bisect for first commit at which a test passes?

Thread Next
From:
James E Keenan
Date:
February 1, 2020 17:18
Subject:
How to bisect for first commit at which a test passes?
Message ID:
20200201171755.19742.qmail@lists-nntp.develooper.com
I'm having trouble figuring out how to use Porting/bisect.pl to identify 
the first point at which, with given configuration options, a file in 
the test suite began to pass.

(At the present time, it may not be possible to do this at all.  But if 
it is, I'm not grokking it.)

The specific problem:

Almost two years ago, in what is now 
https://github.com/perl/perl5/issues/16441, I reported that 
t/op/sprintf2.t was FAILing when g++ version 7 was the C-compiler being 
used.  g++7 was apparently so new at that point, and our smoke-test 
reports were still so scanty, that we had not previously detected this 
problem.

The problem appears to have cleared up in the current dev cycle 
somewhere between v5.31.2 and v5.31.3.  I would like to identify the 
first commit at which this test file began to PASS.  Usually, of course, 
we are trying to identify the first commit at which a test began to 
FAIL.  Here, we're reversing what we're looking for.

On the basis of the documentation (perldoc Porting/bisect-runner.pl), I 
would expect that (on FreeBSD), something like this would work:

#####
perl Porting/bisect.pl --start=v5.31.2 --end=v5.31.3 \
-Dcc="g++7" \
-Accflags="-Wl,-rpath=/usr/local/lib/gcc7" \
-Aldflags="-Wl,-rpath=/usr/local/lib/gcc7" \
--expect-fail --target t/op/sprintf2.t
#####

But that ends with this:

#####
t/op/sprintf2 ... ok
All tests successful.
Elapsed: 1 sec
u=0.02  s=0.00  cu=0.24  cs=0.07  scripts=1  tests=1699
HEAD is now at 45e0524192 5.31.2 today
bad - zero exit from sh -c cd t && ./perl TEST op\/sprintf2\.t
Runner returned 256, not 0 for start revision at 
/home/jkeenan/gitwork/perl/Porting/bisect.pl line 240.
That took 240 seconds.
#####

Let's look at the section of bisect.pl cited.

#####
230 # Sanity check the first and last revisions:
231 system "git checkout $end" and die;
232 my $ret = system $^X, $runner, @ARGV;
233 die "Runner returned $ret for end revision" unless $ret;
234 die "Runner returned $ret for end revision, which is a skip"
235     if $ret == 125 * 256;
236
237 if (defined $start) {
238     system "git checkout $start" and die;
239     my $ret = system $^X, $runner, @ARGV;
240     die "Runner returned $ret, not 0 for start revision" if $ret;
241 } else {
242     # Try to find the earliest version for which the test works
243     my @tried;
...
#####

At this point $end is either HEAD or the commit specified as the 
argument to the '--end' option -- here, the v5.31.3 tag.  $runner is, 
typically, Porting/bisect-runner.pl followed by all command-line 
arguments, which bisect-runner.pl processes for options.  So the program 
first checks out the 'end' commit and runs a configure/build/test cycle. 
  At v5.31.3, t/op/sprintf2.t PASS, so at line 232 $ret is 0.  The 
program does not die at line 233 and advances to line 237.

We have provided 'v5.31.2' as the value for $start, so we check that out 
and run a cycle, here it appears that the system call returns '256'. 
That's a Perl-true value, so the program dies at line 240.

But in this case, we *already knew* that the test would fail at v5.31.2. 
  We needed the program to continue past the 'start' commit and 
determine the first *good* commit.  I was hoping that including the 
'--expect-fail' switch would cause this to happen, but it did not.

So, can anyone tell me whether:

(a) bisect.pl is currently capable of DWIMming here?

(b) if so, what is the correct command-line invocation?

Thank you very much.
Jim Keenan

Thread Next


nntp.perl.org: Perl Programming lists via nntp and http.
Comments to Ask Bjørn Hansen at ask@perl.org | Group listing | About