Front page | perl.perl5.porters |
Postings from July 2012
Re: supporting untarring of extensions (was Re: Detecting duplicateextension directories)
Thread Previous
|
Thread Next
From:
Nicholas Clark
Date:
July 12, 2012 07:43
Subject:
Re: supporting untarring of extensions (was Re: Detecting duplicateextension directories)
Message ID:
20120712144341.GA9583@plum.flirble.org
On Wed, Jul 11, 2012 at 04:29:16PM -0500, Jesse Luehrs wrote:
> On Wed, Jul 11, 2012 at 09:49:18PM +0100, Nicholas Clark wrote:
> > On Wed, Jul 11, 2012 at 02:51:29PM -0500, Jesse Luehrs wrote:
> > > On Wed, Jul 11, 2012 at 08:23:09PM +0100, Nicholas Clark wrote:
> >
> > > > Other than Configure probes, what is causing pain for dual life modules
> > > > sitting in the existing build process? We must fix that, I agree.
> > > > But I'm not aware of the specifics of the problems, which makes it hard
> > > > to help.
> > > >
> > > > (This stuff is hard to get working at all, as it has to work on 3 disjoint
> > > > families of platforms, which make the differences between Linux and AIX seem
> > > > tiny.)
> > >
> > > The problems I'm referring to have to do with the way we currently ship
> > > dual-life modules. For instance: the version.pm that ships with core
> > > doesn't include the code to monkey-patch UNIVERSAL::VERSION, because it
> > > knows that since it's part of core, UNIVERSAL::VERSION already works the
> > > way it wants it to. The version on CPAN *does* include this code, even
> > > though it claims to be the same version number. This sort of thing is
> > > not uncommon among the modules that we ship with core.
> >
> > You're never going to fix version.pm with this, because parts of it are
> > hard coded into util.c. Sorry.
> >
> > (But if it *is* possible to kick parts of util.c out into XS code, then
> > it may well be possible to fix how we handle version.pm without changing the
> > build system at all)
> >
> > Which other modules are pain?
>
> Sure, version.pm has core bits, but it is dual life. It installs from
> CPAN just fine, so there can't be any reason why the code in
> lib/version.pm has to be any different from the code that's shipped as
> version.pm on CPAN. This is the only point I'm trying to make here.
Ah OK. Right, I don't know the specifics of why it differs.
I think that the only way forward is for someone to actually start looking
at the specifics of it, and trying to work out *how* to reduce the
differences to zero.
And
a) this doesn't have to be me
b) I think actually that this *shouldn't* be me.
Really.
Because one of the problems seems to be that I "know" everything.
(This is not true. I don't)
But the illusion of this can come about partly because
i) I *do* know *more* about quite a few obscure things than many other
people
ii) Where I don't, I dig and find out
And I think that the only way that knowledge is going to be *spread* is if
other people bravely step up and attempt to solve problems, get stuck,
ask questions, and learn things.
Note, it doesn't specifically *have* to be any other individual either.
But if *noone* does it, it will never happen.
Also, I don't have a solution to make this happen. Money (alone) doesn't
seem to be the solution, and the tasks that need doing are not shiny or
sexy.
> My point is less that this *can't* work, and more that this isn't
> currently how we do things. We don't just drop an unpacked tarball
> directly into cpan/Whatever, we unpack it, remove a bunch of files,
> customize a bunch of other files, shift other things around to various
> parts of the source tree, and things like that (all of the things
> represented by EXCLUDED, CUSTOMIZED, and MAP in Maintainers.pl). Whether
> it's a packed or unpacked tarball is pretty irrelevant; the thing that
> would make things easier is for the contents to actually be the same.
OK. Which I think means that the way to go is to chip away at each and every
entry in Maintainers.pl that is EXCLUDED, CUSTOMIZED or MAP. Work out *why*
it currently is, and work out how to avoid it.
Specifically, advice from Klortho:
#11949 What happens when you try?
One or more "you"s need to try
I looked at Maintainers.PL
Things that seemed to crop up more than once in EXCLUDED were
1) bundled modules in inc/ or t/lib
2) tests. I infer that the core would skip for some reason
3) examples
4) Makefile.PL and other build stuff the core doesn't need, or causes problems
In the case of (2) and (4), the right solution might be to add skips for the
tests (and liase with upstream to get them in), and to figure out how to
refactor the Makefile.PL to behave suitably for the core, or how to get the
core to behave suitably for the Makefile.PL
For (3), if examples are never installed, then shipping them in the core is
(sort of) deadweight
For (1), shipping something that duplicates what is in the core is not just
deadweight, but potentially counterproductive. (eg which of the 5 files
named More.pm in the core distribution is the one I edit to locally hack
Test::More?)
But I think it's really going to take a lot of piece by piece work chipping
away at these on a case by case, and starting to see the patterns.
In some cases, EXCLUDED only happened because the structure in Maintainers.PL
reflects how things *are*, and when we moved files out from lib into the
current structure (remember, things were *much worse* before) we moved the
files that were in git, but didn't go an (re)add in files from the distros
that weren't previously in lib/. So it's possible that the right solution is
simply to add the files, and delete the EXCLUDED entry. Again, it's
"#11949 What happens when you try?"
However, I think that of the 3, EXCLUDED is the least worrying.
MAP is potentially more tricky. Some of these came about because of
restrictions on where things had to be that are now resolved, but weren't
2 years ago.
For example, thanks to rafl, the build system can now see Pod files that
aren't in pod/. Likewise, the various Pod conversion scripts can live in
their own distros in dist/ and cpan/ whilst previously they had to be in
pod/ too
One that *isn't* done yet is everything in utils/ that's for extensions.
Really, they should go "back" into ext/ cpan/ and dist/ and let
ExtUtils::MakeMaker deal with them. But do to that *right* we need to test
that it correctly does not just the pl->bat conversion for Win32 (pretty sure
that it does) but the pl->com conversion for VMS (not so sure about that)
Because just moving them and then finding out $n months later that something
was broken on VMS isn't the way to go.
But again, it's working through them systematically and bumping them off,
one by one, or few by few.
I think for a chunk of these, the exceptions were put in because it was a
triaged solution to getting things moved out of lib/. We simply had too
much to do, and making it pretty and right "right now" would have killed
the whole attempt. It's "just" that cleaning things up is never a fun job,
and there are always 5 other non-fun jobs also wanting attention.
> The packed vs. unpacked distinction isn't relevant, see above. There
> would be bootstrapping issues here, it's true - it's possible that there
> would have to be a distinction between dual life toolchain modules and
> other dual life modules, or that toolchain modules would need to have
> two separate ways to be built. I'm not sure how good of an idea this is,
> I'm just throwing out ideas to see what the potential problems here
> might be.
OK. Bootstrapping is a gnarly problem. I believe that we have a lot of that
working. It may be ugly - well, it *is* ugly, but it works.
What we also have is a very effective parallel build and parallel testing
system, which *massively* drops the total time taken to build perl
(hence why Jenkins can do a clean build and report back in about 5 minutes)
It *is* capable of expressing build dependencies between modules in ext/
etc, but there are actually very few.
It's also capable of generating a single coherent summary of the tests run,
and what passed and what failed.
The CPAN shell is nowhere near on either of these, but to be fair, it
doesn't try to be, because it's out solving different hard problems.
It's also not *obvious* why various things turn out not to work.
(eg it took me about 2 hours to try various things to get to the point of
deciding that version::vpp isn't good enough for miniperl)
So I'm not convinced that pure brainstorming is the right way to go here.
Prototyping feels necessary, because a lot of this stuff isn't obvious.
>
> > > Additionally, this would allow external packagers to have a much easier
> > > time building a stripped-down version of perl for whatever reason (as
> > > Fedora and whoever else seem interested in doing). All you would (well,
> > > should) have to do would be to remove the appropriate tarballs from the
> > > right place, and they would be gone. No need to fiddle with the build
> > > system at all.
> >
> > Yes. I'd love this to be possible. And in the original message I suggested
> > a route that should permit this.
> >
> > But right now, what you're describing would *also* seem to be possible using
> > an external driver script that
> >
> > a) configures, builds and installs perl
> > b) fires up CPAN.pm
> > c) gets it to build things
> >
> > (and doesn't *need* to be in the core at all)
> >
> > Which means that we seem to be discussing two possibly disjoint problems
> > here
> >
> > 1) that the existing handling of dual life modules has a reputation for being
> > painful
> >
> > 2) that we would also like to be able to handle building arbitrary
> > collections of extra tarballs in *an* integrated process
>
> and 3) we would like to easily be able to remove dists from the set that
> typically ships with perl
Right now we have no way to disable *building* non-XS modules found in ext/
etc. (other than editing config.sh)
This is probably the first thing to work on.
Once that's available, and people can easily test what happens when various
extensions are skipped, then from there what remains is probably only a SMOP
to edit Porting/makerel so that it can make tarballs without various of the
dists.
>
> > So part of my questioning now would be "what does the core do wrong that
> > stops an external driver script being useful?" Because it's going to be
> > the same things that need solving whether such a driver is internal or
> > external.
> >
> > I'm sorry if I'm sounding like an inquisition. I appreciate that there *is*
> > pain, but I'm really failing to grasp any specific details of where that
> > pain is coming from.
>
> The external driver script would work fine for (2) (and I imagine that
> this is similar to how things like Strawberry Perl work currently), but
> is irrelevant for (1), and wouldn't really help with (3), unless I'm
> missing something?
Probably would if it edits config.sh and then re-runs ./Configure -S
> > There is potentially a very simple solution to that - a third perl binary
> > between miniperl and perl. A sort of mesoperl (hybrid), which *does* have
> > DynaLoader (so can run XS code) but also has the miniperl buildcustomize
> > logic (so has a way to fix @INC to be absolute paths into the build
> > directory)
>
> Sure, this would be a totally valid alternative (and would likely be
> easier too). I don't think that the lack of XS at this stage hurts
> anything currently, but if we want to be able to drop arbitrary external
> tarballs into somewhere to build them too, the fewer restrictions those
> external things have, the better.
Yes. Tony Cook had an experiment, and (for example) right now you can't
use ExtUtils::CBuilder with miniperl because it wants to load IO, and
Can't load module IO, dynamic loading not available in this perl.
So, sure, one has to continue not to build *IO* with ExtUtils::CBuilder
(and DynaLoader), and one would need to encode a dependency to ensure that
a parallel build completes lib/auto/IO/IO.$dlext before starting to use
anything that has a "build_requires" on IO, but the infrastructure to do
all of that stuff exists already.
It's mostly a case of knowing what the bits needed are, where they are
(or finding them), and trying stuff out until it works.
Nicholas Clark
Thread Previous
|
Thread Next