develooper Front page | perl.perl5.porters | Postings from January 2012

supporting untarring of extensions (was Re: Detecting duplicate extension directories)

Thread Previous | Thread Next
From:
Nicholas Clark
Date:
January 27, 2012 11:39
Subject:
supporting untarring of extensions (was Re: Detecting duplicate extension directories)
Message ID:
20120127193938.GZ18528@plum.flirble.org
Useful bit at end.

On Fri, Jan 27, 2012 at 04:38:29PM +0000, Nicholas Clark wrote:
> On Fri, Jan 27, 2012 at 11:18:35AM -0500, David Golden wrote:

> In this case, I think it's impossible (in the general case) to be able to
> automatically decide the right thing to do.
> 
> Given that
> 
> a: Configure assumes that any directory under ext/ dist/ or cpan/ is an
>    extension. The structure in them is variable enough that it's not really
>    possible to work out which is "live", and which is build detritus
> b: The approach Configure takes (still) supports placing any unpacked
>    CPAN distribution below ext/ (etc) and trying to build it. We don't
>    parse MANIFEST to find the list of extensions. I think that it's useful
>    to continue being able to build anything put there
>    (I've used it in the past to build Devel::NYTProf to profile blead's
>     build and test scripts. Particularly to get parallel testing working)
> c: We can't know if any file is actually something the user cares about
>    
> Given (a) and (b), the only solution as far as Configure is concerned is
> to remove the "wrong" directory, to leave only the "right" directory.
> The problem is that we can't rm -rf dist/$wrong because it might contain
> user files. eg given this:
> 
> $ ls */Hash-Util-FieldHash/
> dist/Hash-Util-FieldHash/:
> Valuable.pm
> 
> ext/Hash-Util-FieldHash/:
> Changes         FieldHash.o     Makefile.PL     pm_to_blib
> FieldHash.bs    FieldHash.xs    blib            t
> FieldHash.c     Makefile        lib
> 
> we can't know whether it's safe to delete dist/Hash-Util-FieldHash/Valuable.pm
> so as to be able to delete dist/Hash-Util-FieldHash/
> 
> but if we don't delete dist/Hash-Util-FieldHash/ Configure will list
> it as both a non-XS and an XS extension. In turn Makefile.SH will create a
> Makefile that tries to build it as a non-XS extension. In turn, that fails
> without any obvious diagnostic as to *why*. The error will probably be about
> not finding ExtUtils::ParseXS or somesuch, but the underlying cause is that
> the XS part of the toolchain isn't bootstrapped yet.
> 
> That's why I think think that it's not possible to solve the general problem
> of automatically cleaning up for this scenario.
 
Actually, in the general case, I don't think it's possible *even* if we
"cheat" by consulting MANIFEST. If it's an extension that the user has
provided, eg:

$ ls */Foo
dist/Foo:
Foo.pm

ext/Foo:
Foo.pm  Foo.xs


we can't know which is the correct directory to build.


But. Thinking about this, and why I have this gut feeling that it's important
*not* to restrict ourselves to extensions listed in MANIFEST - ie to retain
the current probing, I *think* I can see the genesis of a general solution
to how to bootstrap arbitrary extensions:


Backstory:

The build on *nix has never depended on having Perl installed.
I assume because it originates from the days before Perl was commonly
installed.

But it has this interesting and I think useful side effect - bootstrapping
to a new platform.

Our cross compiling is about as functional as a chocolate teapot.

*But* (I belive) all our build tools cross compile nicely. (sh, sed, awk,
grep, make, cc). Hence one *can* bootstrap Perl 5 onto a new platform, albeit
in a rather round about way, by first bootstrapping a native toolchain.

So we aren't actually stuffed. I think it's important to keep this lack of
dependency on perl 5 to build perl 5, at least as long as we are unable to
cross compile.


Building arbitrary extensions from tarballs:

This is somewhat a de-lux feature. A value add.

I see no reason why *it* can't rely on having a perl already installed
in order to work.

Moreover, the core ships a very nice new toolchain, so that installed perl
doesn't actually need to have any modules installed.

So, I think it might be possible take advantage of all the current build
system as follows:

1) require that the tarballs are in a new directory, analogous to cpan/
   For want of a better name, and to keep BooK happy for now:

19:20 <+meta> Nicholas: crunch

  crunch/

2) in Configure, if it can detect an installed perl that is new enough to
   run the toolchain (5.8.1+ currently?), it then calls out to a bootstrap
   script written in Perl. (Much nicer than shell. And portable)

3) that is run with an -I options sufficient to use all the new, shipped
   toolchain modules (much like the list in write_buildcustomize.pl)

4) it uses whatever means CPAN can currently use to unpack tarballs.
   I think that they *could* be into crunch/ but it's probably clearer to the
   end user to use a new name:

19:22 <+meta> Nicholas: clank

5) everything is sufficiently set up now that the rest of the core build
   system will pick up the new modules. This should work for simple things


Refinements:

6) I believe that which tests are run is determined by MANIFEST, not by
   scanning the filesystem. I think that this means that the extraction script
   should be writing out a new file containing pathnames of tests, and
   t/harness adapted to read from it too

7) dependency analysis can be done by the extraction script and whatever CPAN
   uses. requires and test_requires are fairly easy. build_requires is a bit
   more tricky

   a) I think that such modules need to be added to the list used by
       write_buildcustomize.pl. I suspect that the answer is to write out a
       file and teach it to read it, to avoid editing it in place
   b) I suspect that the modules need to be build ahead of their dependents.
      There is this section in Makefile.SH:

: Prepare dependency lists for Makefile.
dynamic_list=' '
dynamic_ext_re="lib/auto/re/re.$dlext"
extra_dep=''
for f in $dynamic_ext; do
    : the dependency named here will never exist
      base=`echo "$f" | sed 's/.*\///'`
    this_target="lib/auto/$f/$base.$dlext"  
    dynamic_list="$dynamic_list $this_target"

    : Parallel makes reveal that we have some interdependencies
    case $f in
	Math/BigInt/FastCalc|Devel/NYTProf) extra_dep="$extra_dep
$this_target: $list_util_dep" ;;
	Unicode/Normalize) extra_dep="$extra_dep
$this_target: uni.data" ;;
    esac
done

      Again, I suspect that the right solution is to write out a file and
      teach Makefile.SH to suck it into the Makefile it writes, to avoid
      editing anything in place.

8) Everything is set up to assume nested Makefiles. I have this suspicion
   that to cater for modules that require Module::Build to build, the
   initial path of least resistance is to adapt make_ext.pl to write out a
   Makefile that has targets that call do ../../miniperl Build $target

9) The core by necessity builds extensions with miniperl not perl
   If extensions really need perl (ie dynamic loading) this might get fun.
   It's either going to be a lot of PERLLIB games (how make_ext.pl used to
   work) which risks busting the environment variable length limit
   *or* compile a third perl binary, a sort of hybrid between miniperl
   (so that it uses lib/buildcustomize.pl to set @INC) and perl
   (to give it DynaLoader.o and hence dynamic module loading)

   In which case, it makes sense to have the "extras" unpacked into clank/ -
   we
   i)  don't do anything in clank/ until pretty much all of test_prep is made
   ii) we use hybridperl rather than miniperl (or perl) to build them



The above, I think, should get us

*) general purpose tarball extraction
*) parallel make
*) parallel testing

for very little modification of the existing build bootstrapping.

Which, if I'm right, is a win.

Nicholas Clark

PS No, I'm not proposing to do this. It's not my itch. But I'm hoping that
   it's a reasonable plan for scratching for those who have the itch.

Thread Previous | Thread Next


nntp.perl.org: Perl Programming lists via nntp and http.
Comments to Ask Bjørn Hansen at ask@perl.org | Group listing | About