develooper Front page | perl.perl5.porters | Postings from July 2012

Re: supporting untarring of extensions (was Re: Detecting duplicateextension directories)

Thread Previous | Thread Next
Jesse Luehrs
July 12, 2012 08:12
Re: supporting untarring of extensions (was Re: Detecting duplicateextension directories)
Message ID:
On Thu, Jul 12, 2012 at 03:43:41PM +0100, Nicholas Clark wrote:
> On Wed, Jul 11, 2012 at 04:29:16PM -0500, Jesse Luehrs wrote:
> > Sure, has core bits, but it is dual life. It installs from
> > CPAN just fine, so there can't be any reason why the code in
> > lib/ has to be any different from the code that's shipped as
> > on CPAN. This is the only point I'm trying to make here.
> Ah OK. Right, I don't know the specifics of why it differs.
> I think that the only way forward is for someone to actually start looking
> at the specifics of it, and trying to work out *how* to reduce the
> differences to zero.
> And
> a) this doesn't have to be me
> b) I think actually that this *shouldn't* be me.
>    Really.
>    Because one of the problems seems to be that I "know" everything.
>    (This is not true. I don't)
>    But the illusion of this can come about partly because
>    i)  I *do* know *more* about quite a few obscure things than many other 
>        people
>    ii) Where I don't, I dig and find out
>    And I think that the only way that knowledge is going to be *spread* is if
>    other people bravely step up and attempt to solve problems, get stuck,
>    ask questions, and learn things.
> Note, it doesn't specifically *have* to be any other individual either.
> But if *noone* does it, it will never happen.
> Also, I don't have a solution to make this happen. Money (alone) doesn't
> seem to be the solution, and the tasks that need doing are not shiny or
> sexy.

Yes, none of this was meant to imply that you should be doing any of
this(: You are just one of the few people who actually understands the
build system well enough to know what needs to be done without breaking
things, so I thought hammering out some of those details here would be a
good place to start.

> > My point is less that this *can't* work, and more that this isn't
> > currently how we do things. We don't just drop an unpacked tarball
> > directly into cpan/Whatever, we unpack it, remove a bunch of files,
> > customize a bunch of other files, shift other things around to various
> > parts of the source tree, and things like that (all of the things
> > represented by EXCLUDED, CUSTOMIZED, and MAP in Whether
> > it's a packed or unpacked tarball is pretty irrelevant; the thing that
> > would make things easier is for the contents to actually be the same.
> OK. Which I think means that the way to go is to chip away at each and every
> entry in that is EXCLUDED, CUSTOMIZED or MAP. Work out *why*
> it currently is, and work out how to avoid it.
> Specifically, advice from Klortho:
>     #11949 What happens when you try?
> One or more "you"s need to try
> I looked at Maintainers.PL
> Things that seemed to crop up more than once in EXCLUDED were
> 1) bundled modules in inc/ or t/lib
> 2) tests. I infer that the core would skip for some reason
> 3) examples
> 4) Makefile.PL and other build stuff the core doesn't need, or causes problems
> In the case of (2) and (4), the right solution might be to add skips for the
> tests (and liase with upstream to get them in), and to figure out how to
> refactor the Makefile.PL to behave suitably for the core, or how to get the
> core to behave suitably for the Makefile.PL
> For (3), if examples are never installed, then shipping them in the core is
> (sort of) deadweight
> For (1), shipping something that duplicates what is in the core is not just
> deadweight, but potentially counterproductive. (eg which of the 5 files
> named in the core distribution is the one I edit to locally hack
> Test::More?)
> But I think it's really going to take a lot of piece by piece work chipping
> away at these on a case by case, and starting to see the patterns.
> In some cases, EXCLUDED only happened because the structure in Maintainers.PL
> reflects how things *are*, and when we moved files out from lib into the
> current structure (remember, things were *much worse* before) we moved the
> files that were in git, but didn't go an (re)add in files from the distros
> that weren't previously in lib/.  So it's possible that the right solution is
> simply to add the files, and delete the EXCLUDED entry. Again, it's
> "#11949 What happens when you try?"
> However, I think that of the 3, EXCLUDED is the least worrying.
> MAP is potentially more tricky. Some of these came about because of
> restrictions on where things had to be that are now resolved, but weren't
> 2 years ago.
> For example, thanks to rafl, the build system can now see Pod files that
> aren't in pod/. Likewise, the various Pod conversion scripts can live in
> their own distros in dist/ and cpan/ whilst previously they had to be in
> pod/ too
> One that *isn't* done yet is everything in utils/ that's for extensions.
> Really, they should go "back" into ext/ cpan/ and dist/ and let
> ExtUtils::MakeMaker deal with them. But do to that *right* we need to test
> that it correctly does not just the pl->bat conversion for Win32 (pretty sure
> that it does) but the pl->com conversion for VMS (not so sure about that)
> Because just moving them and then finding out $n months later that something
> was broken on VMS isn't the way to go.
> But again, it's working through them systematically and bumping them off,
> one by one, or few by few.
> I think for a chunk of these, the exceptions were put in because it was a
> triaged solution to getting things moved out of lib/. We simply had too
> much to do, and making it pretty and right "right now" would have killed
> the whole attempt. It's "just" that cleaning things up is never a fun job,
> and there are always 5 other non-fun jobs also wanting attention.

Yes, this is exactly the sort of information I was looking for. It's
hard to tell sometimes whether certain parts of the way that perl does
things are actually important from some reason, or if it's just left
over from historical things that nobody has gotten around to actually
cleaning up yet.

Knowing the difference between "this was designed this way because we
thought about it and it's the best idea we could come up with" and "this
is like this because it was worse in the past, and nobody has yet had
the time or motivation to finish cleaning it up" is useful, and also
important to know when moving forward (so that we don't end up
perpetuating the parts that are less than useful).

A lot of work that gets done on the core these days (especially by
people new to core development) involves a lot of cargo culting because
it's not clear whether the way things are done are actually important or
not, but at least the way things are done do actually work. Reducing the
amount that that needs to happen, even if things don't get cleaned up
immediately, is still useful.

> > The packed vs. unpacked distinction isn't relevant, see above. There
> > would be bootstrapping issues here, it's true - it's possible that there
> > would have to be a distinction between dual life toolchain modules and
> > other dual life modules, or that toolchain modules would need to have
> > two separate ways to be built. I'm not sure how good of an idea this is,
> > I'm just throwing out ideas to see what the potential problems here
> > might be.
> OK. Bootstrapping is a gnarly problem. I believe that we have a lot of that
> working. It may be ugly - well, it *is* ugly, but it works.
> What we also have is a very effective parallel build and parallel testing
> system, which *massively* drops the total time taken to build perl
> (hence why Jenkins can do a clean build and report back in about 5 minutes)
> It *is* capable of expressing build dependencies between modules in ext/
> etc, but there are actually very few.
> It's also capable of generating a single coherent summary of the tests run,
> and what passed and what failed.
> The CPAN shell is nowhere near on either of these, but to be fair, it
> doesn't try to be, because it's out solving different hard problems.
> It's also not *obvious* why various things turn out not to work.
> (eg it took me about 2 hours to try various things to get to the point of
> deciding that version::vpp isn't good enough for miniperl)
> So I'm not convinced that pure brainstorming is the right way to go here.
> Prototyping feels necessary, because a lot of this stuff isn't obvious.

Fair enough. I also have no desire to do anything that will cause issues
for parallel builds - it's one of the main things that makes working on
the core somewhat tolerable.

> > and 3) we would like to easily be able to remove dists from the set that
> > typically ships with perl
> Right now we have no way to disable *building* non-XS modules found in ext/
> etc. (other than editing
> This is probably the first thing to work on.
> Once that's available, and people can easily test what happens when various
> extensions are skipped, then from there what remains is probably only a SMOP
> to edit Porting/makerel so that it can make tarballs without various of the
> dists.
> > The external driver script would work fine for (2) (and I imagine that
> > this is similar to how things like Strawberry Perl work currently), but
> > is irrelevant for (1), and wouldn't really help with (3), unless I'm
> > missing something?
> Probably would if it edits and then re-runs ./Configure -S
> > Sure, this would be a totally valid alternative (and would likely be
> > easier too). I don't think that the lack of XS at this stage hurts
> > anything currently, but if we want to be able to drop arbitrary external
> > tarballs into somewhere to build them too, the fewer restrictions those
> > external things have, the better.
> Yes. Tony Cook had an experiment, and (for example) right now you can't
> use ExtUtils::CBuilder with miniperl because it wants to load IO, and
>     Can't load module IO, dynamic loading not available in this perl.
> So, sure, one has to continue not to build *IO* with ExtUtils::CBuilder
> (and DynaLoader), and one would need to encode a dependency to ensure that
> a parallel build completes lib/auto/IO/IO.$dlext before starting to use
> anything that has a "build_requires" on IO, but the infrastructure to do
> all of that stuff exists already.
> It's mostly a case of knowing what the bits needed are, where they are
> (or finding them), and trying stuff out until it works.

Thanks, this has all been very helpful.


Thread Previous | Thread Next Perl Programming lists via nntp and http.
Comments to Ask Bjørn Hansen at | Group listing | About