Front page | perl.perl5.porters |
Postings from January 2012
supporting untarring of extensions (was Re: Detecting duplicate extension directories)
Thread Previous
|
Thread Next
From:
Nicholas Clark
Date:
January 27, 2012 11:39
Subject:
supporting untarring of extensions (was Re: Detecting duplicate extension directories)
Message ID:
20120127193938.GZ18528@plum.flirble.org
Useful bit at end.
On Fri, Jan 27, 2012 at 04:38:29PM +0000, Nicholas Clark wrote:
> On Fri, Jan 27, 2012 at 11:18:35AM -0500, David Golden wrote:
> In this case, I think it's impossible (in the general case) to be able to
> automatically decide the right thing to do.
>
> Given that
>
> a: Configure assumes that any directory under ext/ dist/ or cpan/ is an
> extension. The structure in them is variable enough that it's not really
> possible to work out which is "live", and which is build detritus
> b: The approach Configure takes (still) supports placing any unpacked
> CPAN distribution below ext/ (etc) and trying to build it. We don't
> parse MANIFEST to find the list of extensions. I think that it's useful
> to continue being able to build anything put there
> (I've used it in the past to build Devel::NYTProf to profile blead's
> build and test scripts. Particularly to get parallel testing working)
> c: We can't know if any file is actually something the user cares about
>
> Given (a) and (b), the only solution as far as Configure is concerned is
> to remove the "wrong" directory, to leave only the "right" directory.
> The problem is that we can't rm -rf dist/$wrong because it might contain
> user files. eg given this:
>
> $ ls */Hash-Util-FieldHash/
> dist/Hash-Util-FieldHash/:
> Valuable.pm
>
> ext/Hash-Util-FieldHash/:
> Changes FieldHash.o Makefile.PL pm_to_blib
> FieldHash.bs FieldHash.xs blib t
> FieldHash.c Makefile lib
>
> we can't know whether it's safe to delete dist/Hash-Util-FieldHash/Valuable.pm
> so as to be able to delete dist/Hash-Util-FieldHash/
>
> but if we don't delete dist/Hash-Util-FieldHash/ Configure will list
> it as both a non-XS and an XS extension. In turn Makefile.SH will create a
> Makefile that tries to build it as a non-XS extension. In turn, that fails
> without any obvious diagnostic as to *why*. The error will probably be about
> not finding ExtUtils::ParseXS or somesuch, but the underlying cause is that
> the XS part of the toolchain isn't bootstrapped yet.
>
> That's why I think think that it's not possible to solve the general problem
> of automatically cleaning up for this scenario.
Actually, in the general case, I don't think it's possible *even* if we
"cheat" by consulting MANIFEST. If it's an extension that the user has
provided, eg:
$ ls */Foo
dist/Foo:
Foo.pm
ext/Foo:
Foo.pm Foo.xs
we can't know which is the correct directory to build.
But. Thinking about this, and why I have this gut feeling that it's important
*not* to restrict ourselves to extensions listed in MANIFEST - ie to retain
the current probing, I *think* I can see the genesis of a general solution
to how to bootstrap arbitrary extensions:
Backstory:
The build on *nix has never depended on having Perl installed.
I assume because it originates from the days before Perl was commonly
installed.
But it has this interesting and I think useful side effect - bootstrapping
to a new platform.
Our cross compiling is about as functional as a chocolate teapot.
*But* (I belive) all our build tools cross compile nicely. (sh, sed, awk,
grep, make, cc). Hence one *can* bootstrap Perl 5 onto a new platform, albeit
in a rather round about way, by first bootstrapping a native toolchain.
So we aren't actually stuffed. I think it's important to keep this lack of
dependency on perl 5 to build perl 5, at least as long as we are unable to
cross compile.
Building arbitrary extensions from tarballs:
This is somewhat a de-lux feature. A value add.
I see no reason why *it* can't rely on having a perl already installed
in order to work.
Moreover, the core ships a very nice new toolchain, so that installed perl
doesn't actually need to have any modules installed.
So, I think it might be possible take advantage of all the current build
system as follows:
1) require that the tarballs are in a new directory, analogous to cpan/
For want of a better name, and to keep BooK happy for now:
19:20 <+meta> Nicholas: crunch
crunch/
2) in Configure, if it can detect an installed perl that is new enough to
run the toolchain (5.8.1+ currently?), it then calls out to a bootstrap
script written in Perl. (Much nicer than shell. And portable)
3) that is run with an -I options sufficient to use all the new, shipped
toolchain modules (much like the list in write_buildcustomize.pl)
4) it uses whatever means CPAN can currently use to unpack tarballs.
I think that they *could* be into crunch/ but it's probably clearer to the
end user to use a new name:
19:22 <+meta> Nicholas: clank
5) everything is sufficiently set up now that the rest of the core build
system will pick up the new modules. This should work for simple things
Refinements:
6) I believe that which tests are run is determined by MANIFEST, not by
scanning the filesystem. I think that this means that the extraction script
should be writing out a new file containing pathnames of tests, and
t/harness adapted to read from it too
7) dependency analysis can be done by the extraction script and whatever CPAN
uses. requires and test_requires are fairly easy. build_requires is a bit
more tricky
a) I think that such modules need to be added to the list used by
write_buildcustomize.pl. I suspect that the answer is to write out a
file and teach it to read it, to avoid editing it in place
b) I suspect that the modules need to be build ahead of their dependents.
There is this section in Makefile.SH:
: Prepare dependency lists for Makefile.
dynamic_list=' '
dynamic_ext_re="lib/auto/re/re.$dlext"
extra_dep=''
for f in $dynamic_ext; do
: the dependency named here will never exist
base=`echo "$f" | sed 's/.*\///'`
this_target="lib/auto/$f/$base.$dlext"
dynamic_list="$dynamic_list $this_target"
: Parallel makes reveal that we have some interdependencies
case $f in
Math/BigInt/FastCalc|Devel/NYTProf) extra_dep="$extra_dep
$this_target: $list_util_dep" ;;
Unicode/Normalize) extra_dep="$extra_dep
$this_target: uni.data" ;;
esac
done
Again, I suspect that the right solution is to write out a file and
teach Makefile.SH to suck it into the Makefile it writes, to avoid
editing anything in place.
8) Everything is set up to assume nested Makefiles. I have this suspicion
that to cater for modules that require Module::Build to build, the
initial path of least resistance is to adapt make_ext.pl to write out a
Makefile that has targets that call do ../../miniperl Build $target
9) The core by necessity builds extensions with miniperl not perl
If extensions really need perl (ie dynamic loading) this might get fun.
It's either going to be a lot of PERLLIB games (how make_ext.pl used to
work) which risks busting the environment variable length limit
*or* compile a third perl binary, a sort of hybrid between miniperl
(so that it uses lib/buildcustomize.pl to set @INC) and perl
(to give it DynaLoader.o and hence dynamic module loading)
In which case, it makes sense to have the "extras" unpacked into clank/ -
we
i) don't do anything in clank/ until pretty much all of test_prep is made
ii) we use hybridperl rather than miniperl (or perl) to build them
The above, I think, should get us
*) general purpose tarball extraction
*) parallel make
*) parallel testing
for very little modification of the existing build bootstrapping.
Which, if I'm right, is a win.
Nicholas Clark
PS No, I'm not proposing to do this. It's not my itch. But I'm hoping that
it's a reasonable plan for scratching for those who have the itch.
Thread Previous
|
Thread Next