develooper Front page | perl.perl5.porters | Postings from July 2012

Re: supporting untarring of extensions (was Re: Detecting duplicateextension directories)

Thread Previous | Thread Next
From:
Jesse Luehrs
Date:
July 11, 2012 12:11
Subject:
Re: supporting untarring of extensions (was Re: Detecting duplicateextension directories)
Message ID:
20120711191101.GW30375@tozt.net
On Fri, Jan 27, 2012 at 07:39:39PM +0000, Nicholas Clark wrote:
> I *think* I can see the genesis of a general solution
> to how to bootstrap arbitrary extensions:

I would really love to see this happen.

> Backstory:
> 
> The build on *nix has never depended on having Perl installed.
> I assume because it originates from the days before Perl was commonly
> installed.
> 
> But it has this interesting and I think useful side effect - bootstrapping
> to a new platform.
> 
> Our cross compiling is about as functional as a chocolate teapot.
> 
> *But* (I belive) all our build tools cross compile nicely. (sh, sed, awk,
> grep, make, cc). Hence one *can* bootstrap Perl 5 onto a new platform, albeit
> in a rather round about way, by first bootstrapping a native toolchain.
> 
> So we aren't actually stuffed. I think it's important to keep this lack of
> dependency on perl 5 to build perl 5, at least as long as we are unable to
> cross compile.
> 
> 
> Building arbitrary extensions from tarballs:
> 
> This is somewhat a de-lux feature. A value add.
> 
> I see no reason why *it* can't rely on having a perl already installed
> in order to work.
> 
> Moreover, the core ships a very nice new toolchain, so that installed perl
> doesn't actually need to have any modules installed.
> 
> So, I think it might be possible take advantage of all the current build
> system as follows:
> 
> 1) require that the tarballs are in a new directory, analogous to cpan/
>    For want of a better name, and to keep BooK happy for now:
> 
> 19:20 <+meta> Nicholas: crunch
> 
>   crunch/
> 
> 2) in Configure, if it can detect an installed perl that is new enough to
>    run the toolchain (5.8.1+ currently?), it then calls out to a bootstrap
>    script written in Perl. (Much nicer than shell. And portable)
> 
> 3) that is run with an -I options sufficient to use all the new, shipped
>    toolchain modules (much like the list in write_buildcustomize.pl)
> 
> 4) it uses whatever means CPAN can currently use to unpack tarballs.
>    I think that they *could* be into crunch/ but it's probably clearer to the
>    end user to use a new name:
> 
> 19:22 <+meta> Nicholas: clank
> 
> 5) everything is sufficiently set up now that the rest of the core build
>    system will pick up the new modules. This should work for simple things

So by "arbitrary extensions" here, are you referring to things that
already exist as dual-life in the core, or entirely new modules to add
to a custom distribution? (or both?) My personal interest is in getting
as many dual-life modules as possible extracted out of the main build
process, because keeping things in sync with their CPAN versions is
quite a pain, due to the amount of changes that are necessary to get the
distribution to build in-core.

I think I can see how this would be possible - keep a very minimal
(doesn't use perl) build process available for building just perl and
the toolchain modules, to allow for bootstrapping. Then, just build and
install that "perl-minimal", and that will allow you to run the full
build process, which builds everything with perl via CPAN tarballs. Is
this what you are envisioning?

> Refinements:
> 
> 6) I believe that which tests are run is determined by MANIFEST, not by
>    scanning the filesystem. I think that this means that the extraction script
>    should be writing out a new file containing pathnames of tests, and
>    t/harness adapted to read from it too
> 
> 7) dependency analysis can be done by the extraction script and whatever CPAN
>    uses. requires and test_requires are fairly easy. build_requires is a bit
>    more tricky
> 
>    a) I think that such modules need to be added to the list used by
>        write_buildcustomize.pl. I suspect that the answer is to write out a
>        file and teach it to read it, to avoid editing it in place
>    b) I suspect that the modules need to be build ahead of their dependents.
>       There is this section in Makefile.SH:
> 
> : Prepare dependency lists for Makefile.
> dynamic_list=' '
> dynamic_ext_re="lib/auto/re/re.$dlext"
> extra_dep=''
> for f in $dynamic_ext; do
>     : the dependency named here will never exist
>       base=`echo "$f" | sed 's/.*\///'`
>     this_target="lib/auto/$f/$base.$dlext"  
>     dynamic_list="$dynamic_list $this_target"
> 
>     : Parallel makes reveal that we have some interdependencies
>     case $f in
> 	Math/BigInt/FastCalc|Devel/NYTProf) extra_dep="$extra_dep
> $this_target: $list_util_dep" ;;
> 	Unicode/Normalize) extra_dep="$extra_dep
> $this_target: uni.data" ;;
>     esac
> done
> 
>       Again, I suspect that the right solution is to write out a file and
>       teach Makefile.SH to suck it into the Makefile it writes, to avoid
>       editing anything in place.
> 
> 8) Everything is set up to assume nested Makefiles. I have this suspicion
>    that to cater for modules that require Module::Build to build, the
>    initial path of least resistance is to adapt make_ext.pl to write out a
>    Makefile that has targets that call do ../../miniperl Build $target
> 
> 9) The core by necessity builds extensions with miniperl not perl
>    If extensions really need perl (ie dynamic loading) this might get fun.
>    It's either going to be a lot of PERLLIB games (how make_ext.pl used to
>    work) which risks busting the environment variable length limit
>    *or* compile a third perl binary, a sort of hybrid between miniperl
>    (so that it uses lib/buildcustomize.pl to set @INC) and perl
>    (to give it DynaLoader.o and hence dynamic module loading)
> 
>    In which case, it makes sense to have the "extras" unpacked into clank/ -
>    we
>    i)  don't do anything in clank/ until pretty much all of test_prep is made
>    ii) we use hybridperl rather than miniperl (or perl) to build them
> 
> 
> 
> The above, I think, should get us
> 
> *) general purpose tarball extraction
> *) parallel make
> *) parallel testing
> 
> for very little modification of the existing build bootstrapping.
> 
> Which, if I'm right, is a win.
> 
> Nicholas Clark
> 
> PS No, I'm not proposing to do this. It's not my itch. But I'm hoping that
>    it's a reasonable plan for scratching for those who have the itch.

-doy

Thread Previous | Thread Next


nntp.perl.org: Perl Programming lists via nntp and http.
Comments to Ask Bjørn Hansen at ask@perl.org | Group listing | About