develooper Front page | perl.perl5.porters | Postings from August 2009

The plan for ext/ and dual-life modules

Thread Next
Nicholas Clark
August 28, 2009 13:55
The plan for ext/ and dual-life modules
Message ID:
Currently we have a lot of dual life modules in the perl core. They're all
"installed" into lib/, with their tests next to them, or in a directory near
them, or somewhere completely different, and if they have test fixtures those
are somewhere else, and ...

Yes, it's a maze of special cases. And nothing is laid out the same way as
its CPAN distribution. Even for the modules in ext/, which can at least be
laid out the same way as their distribution tarball from CPAN, the tests are
still run with a current directory of t/, so their fixtures aren't where they
thought they were. Or at least, that's how it was yesterday.

So my plan is as follows:

0: Change it so that each module in ext/ runs its tests in its directory.

   (Schwern made a branch that did this earlier in the year, but Dave asked us
    to hold off merging it until 5.10.1 was released. Well, 5.10.1 has now been
    released, and I've merged as much as possible, but decided to take a
    slightly different approach to Schwern's way. Thanks Schwern, for doing
    most of this work for me)

   As of last night, this is in.

   It's a bit rough, and there are currently exceptions in place for specific
   directories in ext/, some to disable setting $ENV{PERL_CORE}, and one to
   still run ext/File-Glob/t/*.t from t, as those tests expect to see t/TEST
   and various other files, but IT WORKS.
   (If anyone would like to help, figuring out how to change them to run from
    ext/File-Glob would be useful)

   This has at least 3 benefits

a: the expected one - the layout of tests and fixtures is the same as when
   the test runs from the unpacked CPAN distribution. So almost no special-
   case conditional code on $ENV{PERL_CORE}
b: different modules run tests in different directories - there are fewer
   test race conditions where tests for module P write files in t/, and
   tests for module Q (usually IO), glob their current directory, and fail
   because it changed underneath them
c: an unexpected one - as I've changed the test invocation to use PERL5LIB
   to force @INC to be that of the installed perl, it's looking like most of
   the time we don't need $ENV{PERL_CORE} boilerplate in tests for dual life
   modules. (Unless they do fancy things like run under taint or spawn more

1: Clean up the existing tests to ext, feeding changes back upstream to their

2: Migrate as many dual modules as possible from lib/ to ext/, giving them the
   exact same layout as their CPAN distributions.

   (As a module gets moved from lib/$somewhere to ext/$name the directory its
    tests are run from changes from t to ext/$name, so there may be some
    fixing needed, but as this is all 1 module at a time, it's manageable)

   This also means fixing the MAP field in Porting/ to reflect
   the new, untangled layout. (Hopefully removing the MAP field completely)

   Go for the easy ones first. I think that most everything is "easy", except
   the modules needed to build things in ext/ - ie ExtUtils::MakeMaker, and
   anything it calls or depends on. So lib/ likely can't easily
   move to somewhere in ext/constant. I'm not worried - there are dozens which

   Feed all the fixes back upstream

3: I think, but I've not checked the code, let alone tried it, that it might be
   possible to have more than 1 "ext" directory. In which case, we can split
   ext into "ext" and "dual". All the core-only modules can stay in ext/
   All the dual life modules can go in dual/

   That would makes it incredibly easy for anyone to see which code we
   "control", and which code and documentation we get from upstream, and so
   should send patches and bug reports upstream, instead of attempting to fix
   in place.

Well, that's my plan. If people think it's a good idea, and want to help,
that would be great. Pick anything to help with, but in particular

1: please try to convert the File::Glob tests to run from ext/File-Glob
2: pick any dual life module currently in lib, get the CPAN tarball
   (named in Porting/Maintainers.PL), and try to move it to ext/

Nicholas Clark

Thread Next Perl Programming lists via nntp and http.
Comments to Ask Bjørn Hansen at | Group listing | About