Smoke tests inexplicably running more slowly

This message does not concern Perl 5 source code, but I couldn't think 
of a more appropriate list for a code about smoke-testing the core 
distribution.

Since October 2016 I have been running an instance of VMWare on my Linux 
laptop, the VM holding, originally, FreeBSD-11.0.  For a year-and-a-half 
it was the source of my smoke test reports for this platform and OS 
version -- at the time the only such reports.  Most often I used a 
Test-Smoke configuration (perlcurrent.cfg) which called for just two 
configuration variants: one without -DDEBUGGING and one with.  (This is 
actually 4 separate build-configure-test cycles, since each variant is 
run once with perlio land once with stdio.)  Here are some report links 
from earlier this year and their respective running times and dates of run:

Link                                      HH:MM YYYY-MM-DD
https://perl5.test-smoke.org/report/62028  1:06 2018-01-24
https://perl5.test-smoke.org/report/62434  1:04 2018-02-05
https://perl5.test-smoke.org/report/63129  1:16 2018-02-25
https://perl5.test-smoke.org/report/64415  1:07 2018-03-30

Since January 2018 using the same VM, I have also been creating or 
updating FreeBSD ports (e.g., 
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=225599).

Recently, however, I noticed a dramatic increase in the smoke-test's 
running time for the exact same configuration:

https://perl5.test-smoke.org/report/64777  1:58 2018-04-10
https://perl5.test-smoke.org/report/65131  2:00 2018-04-19
https://perl5.test-smoke.org/report/65344  2:02 2018-04-24

These slower running times have been associated with an increased 
likelihood of failure in what for me has always been the most 
resource-sensitive test file:  t/re/speed_thr.t.  That file failed in 
all three of the cases above, but before that point typically failed in 
about one in fifteen smoke-test runs.

I explored possible explanations for this increase in running time.  The 
running time tends to increase over time because we add more tests and 
the tests we've added in recent years tend to be long-running, 
resource-intensive ones.  But I have not observed a similar increase in 
smoke-test running time on other smoke testing rigs, e.g., Carlos 
Guevara's FreeBSD-11 runs.

There was, however, one significant change in the FreeBSD OS running in 
the VM.  In early April I ran 'freebsd-update' to update the entire OS 
and packages from FreeBSD-11.0 to -11.1.

Why?  Recall that I was using this environment to create FreeBSD ports. 
In FreeBSD, the OS and the ports are released together -- which means 
that they reach end-of-life together.  11.0 was out-of-date, such that 
when I would run 'pkg' to add a package I would get a warning of 
UNSUPPORTED SYSTEM.  For a while I could work around this in my ports 
development work by setting an environmental variable, but by early 
April I could no longer do this.  I consulted with more knowledgeable 
BSD folks who bluntly informed me that it was time for 'freebsd-update'.

I didn't really expect this update to succeed within the context of a 
VM, but to my surprise it did -- flawlessly and in just about ten 
minutes.  So I'm now running:

#####
$ uname -mrs
FreeBSD-11.1-RELEASE-p9 amd64
#####

But then I discovered that my Perl 5 smoke tests were taking nearly 
twice as long to run as previously (see above).  Previously, my 
individual smoke-test runs tended to be 5 to 10 minutes quicker than 
Carlos Guevara's for FreeBSD-11; now they're 15 minutes slower.

So the OS upgrade and the smoke-test slowdown are correlated, and it's 
plausible that the former would have an effect on the latter.  But why?

Has anyone encountered something like this before?

Thank you very much.
Jim Keenan

• Smoke tests inexplicably running more slowly by James E Keenan
• Re: Smoke tests inexplicably running more slowly by Tomasz Konojacki