develooper Front page | | Postings from June 2008

The Problem with Non-Functional Metrics (was Re: CPAN Ratings and the problem of choice)

Thread Previous | Thread Next
June 30, 2008 11:21
The Problem with Non-Functional Metrics (was Re: CPAN Ratings and the problem of choice)
Message ID:
On Monday 30 June 2008 04:01:01 Salve J Nilsen wrote:

> If people are actually annoyed about getting in the "hall of shame", we
> shouldn't remove the hall, but instead give them useful info on how to get
> out of it. If authors add useless "workarounds" just to get on the top of
> the CPANTS game, we shouldn't remove the game, but instead find ways to make
> this tactic useless.

Let's try a thought experiment.  Suppose I manage a department of a thousand 
programmers -- clearly too many to evaluate individually on a quarterly 
basis, but still a fraction of the size of active CPAN contributors.

Suppose that I decide to measure the value of a programmer by the number of 
lines of codes he adds to the project.  What's going to happen to the 

Oh, but clearly that's a bad metric.  Let's add another metric: number of bugs 

In about a month, the problems with that metric should be obvious too.  Let's 
add another metric: all functions should have API-level documentation which 
fits in a specific template including inputs, outputs, pre-conditions, 
post-conditions, and revision histlry.

That helps the first metric, so it's a win, right?

At this point, quarterly reviews are coming soon, so let's add a final metric: 
everyone who suggests a new metric which gets adopted gets a bonus on his or 
her review.

Sure, the quality of the software has gone down, but at least now I can 
*measure* the... well okay, the only thing I can measure is the willingness 
of my thousand developers to mold their behavior to fit the metrics to keep 
their jobs.

After two review cycles, even I should realize that this isn't working.  (I 
like to think that I know a few things about software development 
management.)  Inspiration strikes -- instead of the threat of losing their 
jobs, I'll merely make a weekly list of the worst-performing employees, and 
publish it the local newspaper where everyone in the company can see it, 
without telling anyone that the list is there and without notifying anyone 
that his or her name is suddenly on the list.

If public humiliation doesn't encourage people to write better code, I'm not 
sure what kind of metric I can add that will.

That's because what you measure in software development determines the kind of 
behavior you're going to get.  You have to be very careful only to measure 
the kinds of things you want to promote.

(Please note that I'm not suggesting that CPANTS considers SLOC count a 
measure of Kwalitee.)

-- c

Thread Previous | Thread Next Perl Programming lists via nntp and http.
Comments to Ask Bjørn Hansen at | Group listing | About