Front page | perl.perl5.porters |
Postings from April 2018
Smoke tests inexplicably running more slowly
Thread Next
From:
James E Keenan
Date:
April 25, 2018 22:55
Subject:
Smoke tests inexplicably running more slowly
Message ID:
20180425225540.19858.qmail@lists-nntp.develooper.com
This message does not concern Perl 5 source code, but I couldn't think
of a more appropriate list for a code about smoke-testing the core
distribution.
Since October 2016 I have been running an instance of VMWare on my Linux
laptop, the VM holding, originally, FreeBSD-11.0. For a year-and-a-half
it was the source of my smoke test reports for this platform and OS
version -- at the time the only such reports. Most often I used a
Test-Smoke configuration (perlcurrent.cfg) which called for just two
configuration variants: one without -DDEBUGGING and one with. (This is
actually 4 separate build-configure-test cycles, since each variant is
run once with perlio land once with stdio.) Here are some report links
from earlier this year and their respective running times and dates of run:
Link HH:MM YYYY-MM-DD
https://perl5.test-smoke.org/report/62028 1:06 2018-01-24
https://perl5.test-smoke.org/report/62434 1:04 2018-02-05
https://perl5.test-smoke.org/report/63129 1:16 2018-02-25
https://perl5.test-smoke.org/report/64415 1:07 2018-03-30
Since January 2018 using the same VM, I have also been creating or
updating FreeBSD ports (e.g.,
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=225599).
Recently, however, I noticed a dramatic increase in the smoke-test's
running time for the exact same configuration:
https://perl5.test-smoke.org/report/64777 1:58 2018-04-10
https://perl5.test-smoke.org/report/65131 2:00 2018-04-19
https://perl5.test-smoke.org/report/65344 2:02 2018-04-24
These slower running times have been associated with an increased
likelihood of failure in what for me has always been the most
resource-sensitive test file: t/re/speed_thr.t. That file failed in
all three of the cases above, but before that point typically failed in
about one in fifteen smoke-test runs.
I explored possible explanations for this increase in running time. The
running time tends to increase over time because we add more tests and
the tests we've added in recent years tend to be long-running,
resource-intensive ones. But I have not observed a similar increase in
smoke-test running time on other smoke testing rigs, e.g., Carlos
Guevara's FreeBSD-11 runs.
There was, however, one significant change in the FreeBSD OS running in
the VM. In early April I ran 'freebsd-update' to update the entire OS
and packages from FreeBSD-11.0 to -11.1.
Why? Recall that I was using this environment to create FreeBSD ports.
In FreeBSD, the OS and the ports are released together -- which means
that they reach end-of-life together. 11.0 was out-of-date, such that
when I would run 'pkg' to add a package I would get a warning of
UNSUPPORTED SYSTEM. For a while I could work around this in my ports
development work by setting an environmental variable, but by early
April I could no longer do this. I consulted with more knowledgeable
BSD folks who bluntly informed me that it was time for 'freebsd-update'.
I didn't really expect this update to succeed within the context of a
VM, but to my surprise it did -- flawlessly and in just about ten
minutes. So I'm now running:
#####
$ uname -mrs
FreeBSD-11.1-RELEASE-p9 amd64
#####
But then I discovered that my Perl 5 smoke tests were taking nearly
twice as long to run as previously (see above). Previously, my
individual smoke-test runs tended to be 5 to 10 minutes quicker than
Carlos Guevara's for FreeBSD-11; now they're 15 minutes slower.
So the OS upgrade and the smoke-test slowdown are correlated, and it's
plausible that the former would have an effect on the latter. But why?
Has anyone encountered something like this before?
Thank you very much.
Jim Keenan
Thread Next
-
Smoke tests inexplicably running more slowly
by James E Keenan