develooper Front page | perl.perl5.porters | Postings from February 2017

Re: Moving XS-APItest in harness testing

Thread Previous | Thread Next
James E Keenan
February 19, 2017 15:33
Re: Moving XS-APItest in harness testing
Message ID:
On 02/19/2017 12:20 AM, Karl Williamson wrote:
> Is there any reason not to move these tests away from the cpan, dist,
> ext tests? and into the tests for the core?
> I have a smoke-me that does this, if you'd like to see how it impacts
> your testing performance.

Here's one problem I just noticed.

In the FreeBSD-11.0 VM where I reported smoke-test results earlier today 
-- an environment in which $TEST_JOBS == 4 -- I timed 'make 
test_harness' at the following commits:

commit 3b1d752dd57309cbf5f5018ddf5ec58ce2fcb7a4
708.01 real       555.87 user       113.28 sys

khw-harness branch
705.78 real       539.69 user       119.21 sys

No significant improvement there.  But what is more troubling is that I 
got a failure in the khw-harness branch which I believe was induced by 
the re-ordering of the sequence of tests.

# Failed test 2 -    right time (waited 6 secs for 3-sec alarm) at 
op/alarm.t line 34
# Failed test 4 -    right time (waited 5 secs for 3-sec alarm) at 
op/alarm.t line 54

I then re-ran t/op/alarm.t by itself.  All tests PASSed.

 From that I infer that the failures I got were due to the fact the VM 
was busy at the point when t/op/alarm.t was run.  I've never observed 
this failure before, so I suspect it was due to memory-heavy tests such 
as those in XS-API now being run earlier in the harness and causing 
time-sensitive tests elsewhere to fail.


Thread Previous | Thread Next Perl Programming lists via nntp and http.
Comments to Ask Bjørn Hansen at | Group listing | About