develooper Front page | perl.perl5.porters | Postings from December 2010

why distributions are hard (Re: [perl #XXX] [PATCH] __SUB__)

Thread Previous | Thread Next
Nicholas Clark
December 14, 2010 12:26
why distributions are hard (Re: [perl #XXX] [PATCH] __SUB__)
Message ID:
On Tue, Dec 14, 2010 at 06:48:47PM +0100, demerphq wrote:
> On 14 December 2010 18:10, Nicholas Clark <> wrote:
> > On Mon, Dec 13, 2010 at 07:03:32PM +0100, demerphq wrote:
> >
> >> We have also seen that in many environments it is anywhere from nearly
> >> impossible to quite difficult to get non-standard modules installed at
> >> all, let alone ones that require compiling XS code. We have seen both
> >> of these problems with Scalar::Util. (I have personal experience of
> >> this one.)
> >
> > You're proposing a technical fix to a political problem.
> On one level yes, on another level no. Its a technical problem that it
> is quite difficult to make recursive closures. The existing technical
> solution for doing so is pretty sad. The proiposed solution is vaslty
> superior. So why should Perl essentially endorse the shit one and then
> say "but the best way you need to install extra modules"

I was meaning that "bundle a module" is a technical fix to a political

> On the other hand yes, we have devs in our community who work in
> environments that can be quite restrictive. Your position basically
> amounts to saying that you don't care about their opinion or needs. I
> dont think that is the way our community should work and I definitely
> dont think it is a productive attitude.

I care, but not so much that I'm prepared to advocate donating my time for
free to solving a problem the wrong way.

It's that simple.

If someone is being paid to work in these environments, and not contributing
anything back, why should they *expect* to externalise their costs onto me?

> > We can't include all of CPAN in the core just because other people wish
> > to externalize the costs of resolve their issues.
> This is a straw man argument. Nobody is arguing about including all of CPAN.

I remain of the firmly opinion that the following reasons for including CPAN
modules in core:

1: that someone wants it, but their local environment makes it hard to have
   anything other than core modules

2: that it's not "supported" on CPAN, but it is if it's in core

3: that if it's in core, it's the official way to do something, but if it's
   on CPAN, it can't be

are attempts to find a technical solution to a political problem, and I will
resist attempts to solve them the wrong way. Specifically

1: a) if it were to be added to the core perl, how is one going to persuade
      anyone to install a new version of *that* in place of what one has?
      Why is that going to be any easier (or faster) than solving the problem
      of locally installing module?
   b) If one doesn't solve (a), how many *years* until one can have it? Will it
      still matter?

2: What is different between the volunteers who contribute support on the perl
   core, and those who contribute support to their modules on CPAN?
   If it matters to a firm commercially, pay for commercial support. Either
   outsourced to someone competent (eg ActiveState) or, in house, by employing
   people with the relevant skills in house. It's all open source, so
   *nothing* stops any firm making and maintaining an internal fork.
   Either the value to the business is great enough that this is worth doing,
   or it's *not*.

3: Just do it.

There *are*, other, valid reasons (in my book) for including things.

> We are arguing about including critical XS code that cannot reasonably
> be simulated in Perl code and which serves a function which
> essentially cannot be implemented without XS.

I dispute "critical". Useful, yes. But Perl 5 is already a sufficiently
functional language that people have built half-billion dollar businesses from
it. The language is already way more than Turing complete. Most everything
proposed for addition is "nice to have", and there will never be complete
consensus on what is essential to add, and what is merely nice.

I'll add that whilst I've never felt need for this feature, it did seem
reasonable to add it (as ${^SUB}, not __SUB__), as part of the core language.
*Did* seem, until it was observed that there is still contention on *what* the right behaviour is.

So it's not clear to me that there is a right answer.

> This DOES NOT apply to XS code. And even with XS code its not like
> anyone is arguing that this applies to a lot code, but rather to a
> critical select few pieces that arguably should have been in perl from
> the beginning.

> > This is one of my hot buttons and I will argue against it until I am driven
> > away from Perl 5.
> I don't really like the threat here Nicholas, especially given it is
> in regard to a position that nobody has taken.
> If you really have a good argument for why this idea shouldn't move
> forward then post it, but this line about including all of cpan is
> just bogus. Bring it up when somebody actually makes that argument ok?

The threat is merely that I have a very strong opinion about what we should
add, and that I'm not going to shut up about it.

Basically, distributions are hard. People have, for about a decade, been
arguing that the core distribution should be an SDK, usually for one of the
3 reasons above. However, much talking, no action.

It's hard. No-one thinks it through. The problems, as I see it, with actually
*maintaining* a distribution, for the long term, are as follows:

So, if one wants to make a mega-distribution...

Q: What OSes will it to support?

A: Linux rules the world of hosting providers, and a chunk of the enterprise,
   so that's a no brainer. But how many variants of how many distributions?

   Likely Win32, as a chunk of the target audience have Win32 desktops, even
   if they are developing for a Linux deployment

   Probably OS X, as that's what a chunk of the opinion formers use

   But does one stop there? Or consider that you're targeting FreeBSD,
   Solaris, etc? One gets more reach, for little extra cost, but where is the
   sweet spot?

Q: Which modules to include?

A: Easy, all the current best practice modules. But, that leads on to...

Q: What is the policy on removals?

and that's where it starts to get hard...

So, which YAML module to include?

A0: None, as JSON is the future
A1a: YAML::Syck
A1c: YAML::Tiny

all have pros and cons

Now, consider the position 4 years ago. YAML was the future back then. So had
one started the project 4 years ago, one *would* have a YAML module in. So
what's the policy *now* - remove it, or keep it?

Today, one would (likely) pick DBI, DBD::SQLite and DBIx::Class.
But 8 years ago, it would be DBI, DBD::SQLite and Class::DBI.

So today, would the decision be to add DBIx::Class, and keep Class::DBI, or
drop the latter?

The "enterprise Perl" answer likely has to "it's still being used, so we have
to keep it". Hello ever increasing maintenance liability.

The "current best practice" answer is obviously "no, it's not current, so it
can go". Except that *that* defeats one of the goals of such a project, which
is to encourage OS vendors (and hosting providers, the gateway to recruiting
new blood), to ship a collection of modules that newbies can rely on to
*always* be there. Because if today's "Bundle::PBP" has DBIx::Class, but
that from 4 years ago as Class::DBI, and that's what 60% of hosting providers
still have, what's the point of this?

Q: What is the policy on dependent libraries?

Because as soon as one has YAML, the question is "which XML module?"
Again, no right answer. And does one bundle the relevant C library, or does one
expect the OS to provide that? If one wants modules to be always available, one
has to. (And if it's not always available, what's the point of bundling it?).
Which means one is now having to track and update on every CVE in (say) libxml.
And if one wants SSL support (likely considered "essential"), openssl.

Q: What is the policy on bug fixes to bundled modules?

Same problems as core perl, but with more bundled, more likely to hit it:

If someone reports a bug in a module you bundle, what should be done?
Fix it locally in the distribution? Report the bug (and fix) upstream, and
wait for it to come back?

If the former, how does one avoids upsetting module authors because it's now a
fork their module and uploaded to CPAN?

If the latter, and upstream is unresponsive, what does one say to the end
users? "Thanks for your patch, but months later we're still waiting on ..."

The more modules you have, the worse it gets.
The more OSes you want to support, the worse it is.

Hence why I'm (also) very wary of adding anything more to the perl core,
and fairly keen to get rid (to CPAN) of anything that is demonstrably
not used by >99.something% of CPAN.

Nicholas Clark

Thread Previous | Thread Next Perl Programming lists via nntp and http.
Comments to Ask Bjørn Hansen at | Group listing | About