Front page | perl.perl5.porters |
Postings from February 2020
Re: How to bisect for first commit at which a test passes?
Thread Previous
|
Thread Next
From:
James E Keenan
Date:
February 1, 2020 22:19
Subject:
Re: How to bisect for first commit at which a test passes?
Message ID:
20200201221913.17618.qmail@lists-nntp.develooper.com
On 2/1/20 3:16 PM, hv@crypt.org wrote:
> I wrote:
> :James E Keenan <jkeenan@pobox.com> wrote:
> :[...]
> ::perl Porting/bisect.pl --start=v5.31.2 --end=v5.31.3 \
> ::-Dcc="g++7" \
> ::-Accflags="-Wl,-rpath=/usr/local/lib/gcc7" \
> ::-Aldflags="-Wl,-rpath=/usr/local/lib/gcc7" \
> ::--expect-fail --target t/op/sprintf2.t
> ::
> ::But that ends with this:
> ::
> ::t/op/sprintf2 ... ok
> ::All tests successful.
> ::Elapsed: 1 sec
> ::u=0.02 s=0.00 cu=0.24 cs=0.07 scripts=1 tests=1699
> ::HEAD is now at 45e0524192 5.31.2 today
> ::bad - zero exit from sh -c cd t && ./perl TEST op\/sprintf2\.t
> ::Runner returned 256, not 0 for start revision at
> :
> :That looks to me like it's saying the test passed at the start revision,
> :v5.31.2, unexpectedly.
>
> To expand on that: expect-fail is implemented by setting expect-pass to 0,
> and then handled in bisect-runner.pl report_and_exit(), where it is used
> to invert the return value seen by bisect.pl.
>
> So the test passed, the test run returned 0, so report_and_exit returned
> non-zero, and that's why it complains "Runner returned 256, not 0 for start
> revision".
>
> Hugo
>
Hugo, thanks for spotting that.
To better understand the problem, I built perl twice in the checkout
directory, the first time at tag v5.31.2, the second at v5.31.3. (Also,
to better simulate the problem in the bug report, I built with
'-Duseithreads', which I had neglected to include in the bisection calls
-- but I don't think that made a material difference here.)
Configuration in each:
#####
sh ./Configure -des -Dusedevel -Duseithreads \
-Dcc="g++7" \
-Accflags="-Wl,-rpath=/usr/local/lib/gcc7" \
-Aldflags="-Wl,-rpath=/usr/local/lib/gcc7" \
&& make test_prep
#####
v5.31.2
"Regular" run
#####
./perl -Ilib t/op/sprintf2.t
...
ok 1698 - special-case %.0f on -4503599627370503
ok 1699 - croak for very large numeric format results
1..1699
#####
No hint of a problem.
Run thru harness
#####
cd t;./perl harness op/sprintf2.t; cd -
...
ok 1699 - croak for very large numeric format results
Dubious, test returned 9 (wstat 2304, 0x900)
All 1699 subtests passed
(less 30 skipped subtests: 1669 okay)
Test Summary Report
-------------------
op/sprintf2.t (Wstat: 2304 Tests: 1699 Failed: 0)
Non-zero exit status: 9
Files=1, Tests=1699, 1 wallclock secs ( 0.15 usr 0.00 sys + 0.16 cusr
0.12 csys = 0.42 CPU)
Result: FAIL
/usr/home/jkeenan/gitwork/perl2
#####
So the problem has to do with the harness somehow!
v5.31.3
"Regular" run: no difference from above, no indication of error
Run thru harness:
#####
cd t;./perl harness op/sprintf2.t; cd -
op/sprintf2.t .. ok
All tests successful.
Files=1, Tests=1699, 0 wallclock secs ( 0.12 usr 0.00 sys + 0.18 cusr
0.09 csys = 0.39 CPU)
Result: PASS
/usr/home/jkeenan/gitwork/perl2
#####
So, what cleared up between v5.31.2 and v5.31.3 was the interaction
between the harness and the program. And that *might* explain the
bisection result.
That's as far as I've gotten. Further ideas?
Thank you very much.
Jim Keenan
Thread Previous
|
Thread Next