develooper Front page | perl.perl5.porters | Postings from February 2013

Re: Benchmark regression 2013-02

Thread Previous | Thread Next
From:
Nicholas Clark
Date:
February 26, 2013 12:01
Subject:
Re: Benchmark regression 2013-02
Message ID:
20130226120059.GM5653@plum.flirble.org
On Tue, Feb 26, 2013 at 12:28:06PM +0100, Steffen Schwigon wrote:
> Hi!
> 
> Short note form the benchmark front:
> 
> After weeks of consolidation function calls and method calls get slower
> again, for *non-threaded* (sic).
> 
>   http://speed.perlformance.net/timeline/#/?exe=36,35&base=14+80&ben=Fib&env=4&revs=50&equid=off
>   http://speed.perlformance.net/timeline/#/?exe=36,35&base=14+80&ben=FibOO&env=4&revs=50&equid=off

As far as I can make out, the jump on the graph is between commits
00a1356009c12c2c and c96939e471cb3942.

No commits in that range change any code even vaguely related to function
calls. I'm still suspicious that the numbers are also affected by code
alignment changes (due to unrelated function sizes changing), and that
these can be generating noise which is as large as changes caused by the
code. What compiler and linker flags are you using? I can't spot that on
the site.

My own experimentation revealed that even the *order* of linking of (the
same!) object files had repeated measurable differences on execution speed
of the same benchmark. And I have no idea why this was, and hence how to
eliminate it from what I'm measuring.

This seems to be a "real" problem (ie not just me), as this sort of thing
seems to be the motivation behind "Stablizer":

    Stablizer is a compiler pass andd runtime system that eliminates
    measurement bias, enabling rigorous performance evaluation using
    standard statistical techniques. Stabilizer uses LLVM to enable dynamic
    random placement of code, stack, heap, and globals for C and C++
    applications.

http://people.cs.umass.edu/~charlie/stabilizer.pdf
and http://github.com/ccurtsinger/stabilizer

I think that using something like that is going to be needed to avoid all
sorts of alignment noise that are currently plaguing our attempts at
understandable benchmark results.

Nicholas Clark

Thread Previous | Thread Next


nntp.perl.org: Perl Programming lists via nntp and http.
Comments to Ask Bjørn Hansen at ask@perl.org | Group listing | About