Front page | perl.perl5.porters |
Postings from November 2011
On 2011.11.16 11:16 PM, H.Merijn Brand wrote:
>> * Make a realistic benchmark suite of both performance and memory [4]
>> * Set up a smoker to the benchmarks and report significant differences
>> and performance creeps to p5p, like with tests
>
> I know we're slow on this, but the new setup of Test::Smoke will store
> all core test run times in the database, so one can select runs for the
> same machine and compare them over time.
That's a good start.
It's hard to tease useful information out of that as the test time is
monolithic. It's difficult to know what caused a performance change... which
test got slower? Did perl get slower, or did the test change? How do you
usefully compare the performance of test runs between different versions of
Perl when the tests are constantly changing?
A benchmark suite has to be:
* Fine grained
* Repeatable
* Deterministic (ie. each set of runs produces the same result)
* Comparable between commits
* Applicable to real world performance situations
It should be able automatically answer the questions:
* What slowed down / sped up?
* When did it slow down / speed up?
It should provide the tools to answer:
* Why did it slow down?
Unfortunately the test suite can't tell us that.
--
<Schwern> What we learned was if you get confused, grab someone and swing
them around a few times
-- Life's lessons from square dancing