[perl #127849] skip/todo timing tests on VMs

On Mon, 04 Sep 2017 01:29:40 -0700, davem wrote:
> On Sat, Sep 02, 2017 at 04:56:55PM -0700, James E Keenan via RT wrote:
> > On Thu, 07 Apr 2016 02:37:10 GMT, khw wrote:
> > > Standardize some mechanism so that tests that fail on smokers can
> > > be
> > > instead made TODO or skipped
> > > For instance, we could tell Time::Hi-Res that it was running on a
> > > loaded VM, so it wouldn't generate failures
> >
> > In my FreeBSD-10.3 VM, the last test found in t/re/speed.t is the one
> > that most frequently fails:
> >
> > #####
> > {
> >     # [perl #127855] Slowdown in m//g on COW strings of certain
> > lengths
> >     # this should take milliseconds, but took 10's of seconds.
> >     my $elapsed= -time;
> >     my $len= 4e6;
> >     my $zeros= 40000;
> >     my $str= ( "0" x $zeros ) . ( "1" x ( $len - $zeros ) );
> >     my $substr= substr( $str, 1 );
> >     1 while $substr=~m/0/g;
> >     $elapsed += time;
> >     ok( $elapsed <= 1, "should not COW on long string with substr and
> > m//g");
> > }
> > #####
> >
> > See, e.g., http://perl5.test-smoke.org/report/57470.
> >
> > All smoke test runs on this platform are built with a threaded perl.
> > The test seems to fail most frequently on DEBUGGING builds.  Whenever
> > I run the test file by itself, that is, outside the smoke-test run,
> > it always PASSes.
> >
> > How would you diagnose or address that?
> 
> Is it actually failing test 58, or is it that the watchdog timeout is
> triggering during test 58?
> 
> Try 'use Time::HiRes qw(time);' and print the value of $elapsed just
> before
> the ok test, e.g.
> 
> +++ b/t/re/speed.t
> @@ -146,6 +146,7 @@ PROG
>      {
>          # [perl #127855] Slowdown in m//g on COW strings of certain
> lengths
>          # this should take milliseconds, but took 10's of seconds.
> +        use Time::HiRes qw(time);
>          my $elapsed= -time;
>          my $len= 4e6;
>          my $zeros= 40000;
> @@ -153,6 +154,7 @@ PROG
>          my $substr= substr( $str, 1 );
>          1 while $substr=~m/0/g;
>          $elapsed += time;
> +        printf "e=%.6f\n", $elapsed;
>          ok( $elapsed <= 1, "should not COW on long string with substr
> and m//g");
>      }
> 
> Then run it several times and see what sort of values you get.
> On my system is see values of around 0.02s; if your system is taking
> nearer to 1s to run that test, then it could occasionally overlap two
> 1s clock ticks and fail the test. In that case it would be as simple
> as
> changing the '$elapsed <= 1' condition to '$elapsed <= 2' or maybe 3.

Was there any follow up to this?
-- 
Karl Williamson

---
via perlbug:  queue: perl5 status: open
https://rt.perl.org/Ticket/Display.html?id=127849

• [perl #127849] skip/todo timing tests on VMs by Karl Williamson via RT
• [perl #127849] skip/todo timing tests on VMs by Karl Williamson via RT