develooper Front page | perl.perl5.porters | Postings from September 2012

benchmarks in the style of "typical" programs

Thread Next
From:
Nicholas Clark
Date:
September 20, 2012 07:35
Subject:
benchmarks in the style of "typical" programs
Message ID:
20120920143541.GU9583@plum.flirble.org
Steffen Schwigon commented in another thread that it's very hard to find
benchmarks that are designed to behave like typical programs.

It's very easy to write a benchmark for a particular thing, and it's fairly
easy to get such a beast to show that a particular change will speed that
benchmark up.

But

Whilst such a benchmark will show you the behaviour of a specific feature
in isolation, they typically don't actually represent realistic code. They
usually don't throw enough data around or create enough objects to start
to stress the memory subsystems. And they don't do enough different things
to thrash any CPU instruction cache. So It's much harder to show whether a
particular change slows everything else down meaningfully enough to not be
worth it.

So

Does anyone have any code to use as benchmarks that behave like "typical"
Perl programs? This for the purpose of benchmarking the perl interpreter and
assessing the viability of potential optimisations, particularly
optimisations that involve trade offs that are going to make some cases
slower.


I'm pretty sure that any "functionality" test suite is not going to make
a good "load test" performance benchmark, because it's

a) trying to test corner cases, not common cases
b) trying to do this as efficiently as possible

and certainly not trying to do something that lasts for 20 to 30 seconds
for the sake of lasting for 20-30 seconds.


Needs to do realistic things on a big enough scale to stress a typical system.
Needs to avoid external library dependencies, or particular system specifics.
Needs to be constrained by CPU or RAM, not blocking on an I/O device.
Preferably needs to be doing something that Perl is typically used for.
Preferably needs to avoid being too Perl version specific.
Preferably needs to avoid being a maintenance headache itself.

Nicholas Clark

Thread Next


nntp.perl.org: Perl Programming lists via nntp and http.
Comments to Ask Bjørn Hansen at ask@perl.org | Group listing | About