Re: @INC search order.

Front page | perl.perl5.porters | Postings from February 2009
Thread Previous | Thread Next
From:
Nicholas Clark
Date:
February 15, 2009 10:17
Subject:
Re: @INC search order.
Message ID:
20090215181701.GA80373@plum.flirble.org
Because this is somewhat old (and I've dug it up from the archive
http://www.xray.mpe.mpg.de/mailing-lists/perl5-porters/2002-04/msg02380.html
) I'll quote a lot of the context first:

On Mon, Apr 29, 2002 at 03:41:47PM -0400, Andy Dougherty wrote:
> As part of a package of NetBSD-specific patches, Johnny Lam
> offered the following thoughtful patch
> 
> >     (4)	We changed the default module/library search path to have
> > 	site_perl come before the standard directories.  In other words,
> > 	the previous search path on an i386 was:
> > 
> > 	    /usr/pkg/lib/perl5/5.6.1/i386-netbsd
> > 	    /usr/pkg/lib/perl5/5.6.1
> > 	    /usr/pkg/lib/perl5/site_perl/5.6.1/i386-netbsd
> > 	    /usr/pkg/lib/perl5/site_perl/5.6.1
> > 	    /usr/pkg/lib/perl5/site_perl
> > 
> > 	but we changed it to:
> > 
> > 	    /usr/pkg/lib/perl5/site_perl/5.6.1/i386-netbsd
> > 	    /usr/pkg/lib/perl5/site_perl/5.6.1
> > 	    /usr/pkg/lib/perl5/site_perl
> > 	    /usr/pkg/lib/perl5/5.6.1/i386-netbsd
> > 	    /usr/pkg/lib/perl5/5.6.1
> > 
> > The rationale for (4) is that when we install a module that is newer
> > than one in the standard library, the new module goes into the
> > site_perl directory as it's an add-on module.  However, we can't use
> > the newer module without modifying either the scripts or the Perl
> > environment to find the newer module explicitly because of the order
> > of the library search path: the site_perl directories come after the
> > standard directories.  The solution on a locally-compiled perl is to
> > directly replace the module in the standard library with the newer
> > module.  However, this isn't really on option when installing via a
> > package system as the older module files are owned by the perl
> > package.  By placing the the site_perl directories before the
> > standard directories, newer modules that we install via pkgsrc are
> > simply found before the older ones in the standard library.
> 
> This is indeed a real problem, and the only other easy solution I see is
> for the package distributor to re-distribute the entire perl package
> whenever any single module in it is updated.  Some distributions, such as
> Debian, reduce the size of the updated packages by splitting the perl
> package into parts and then only updating the parts that need it, but that
> introduces other compilcations of its own and may not be appropriate for
> NetBSD.
> 
> However, I think the proposed patch (which I haven't attached) isn't quite
> right either.  To see why, it's necessary to think both forward and
> backward to imagine several co-existing versions all on the same system.
> 
> I apologize for this long message, but I've never found a clear way to
> discuss this briefly.
> 
> First, the natural question is (as extracted from Porting/pumpkin.pod)
> 
>     =head2 Why isn't there a directory to override Perl's library?
> 
>     Mainly because no one's gotten around to making one.  Note that
>     "making one"  involves changing perl.c, Configure, config_h.SH (and
>     associated files, see above), and I<documenting> it all in the
>     INSTALL file.
> 
> The only thing I have to add to that is that you'd also have to add an
> appropriate MakeMaker target INSTALLDIRS=override.
> 
> The main reason why the site-specific libraries are loaded last is
> that such an ordering used to be necessary to support using the same
> extension across multiple binary-compatible perl versions.  For a
> detailed discussion, see the INSTALL file in perl5.005_03.  Look for
> the paragraph that ends with
> 
>     (This is also why perl searches the site-specific libraries last.)
> 
> Starting in perl5.6.1, perl was enhanced to search backwards through
> compatible versions.  This removed some of the earlier constraints and
> indeed does make the search ordering of libraries less critical.
> However, since perl has searched site-specific libraries last for the
> past 8 years, changing it would require careful thought and explicit
> documentation.
> 
> The main issue that arises with any sort of override directory is that
> you need to remember to remove it when a new version of the standard
> library renders it obsolete.  The proposed ordering doesn't do that.
> This is best illustrated with an example, taken from 5.7.3's INSTALL
> file:

> One possible solution is to change the search order to be more
> version-oriented than prefix-oriented.  That is consult sitelib,
> vendorlib, and privlib in that order for each version, as appropriate,
> something like the following:
> 	
> 	# CURRENT VERSION
> 	# sitelib
> 	/usr/local/lib/perl5/site_perl/5.6.1/$archname
> 	/usr/local/lib/perl5/site_perl/5.6.1
> 	# vendorlib
> 	/usr/local/lib/perl5/vendor_perl/5.6.1/$archname
> 	/usr/local/lib/perl5/vendor_perl/5.6.1
> 	# privlib
> 	/usr/local/lib/perl5/5.6.1/$archname
> 	/usr/local/lib/perl5/5.6.1
> 
> 	# OLDER VERSIONS
> 	# sitelib
> 	/usr/local/lib/perl5/site_perl/5.6.0/$archname
> 	/usr/local/lib/perl5/site_perl/5.6.0
> 	# vendorlib
> 	/usr/local/lib/perl5/vendor_perl/5.6.0/$archname
> 	/usr/local/lib/perl5/vendor_perl/5.6.0
> 	# Don't search privlib for older versions
> 
> 	# No vendorlib in 5.005
> 	/usr/local/lib/perl5/site_perl/5.005/$archname
> 	/usr/local/lib/perl5/site_perl/5.005
> 
> 	# 5.004 and earlier
> 	/usr/local/lib/perl5/site_perl/
> 
> With this scheme, a user-installed CGI.pm in 5.6.0 *would* be
> overridden by a perl-5.6.1 version, for perl-5.6.1.  If for some
> reason you still wanted your local override of CGI.pm, you'd have to
> reinstall with 5.6.1 also, but I don't think that's onerous.
> 
> Hmm.  This scheme might make more sense.  It would require re-writing parts
> of perl.c:incpush() and lib/lib.pm (as well as documenting it in
> INSTALL).  I'm not volunteering to do it myself, though.  I'm unlikely
> to have time to do much of anything before the end of May.

Yes, it does require that...

> I'm also not sure whether it's right or not.  The existing scheme has
> been in place a long time.  Some users might be relying on it working
> the way it currently does.  'Twould need very careful thought.

Right-ho. With the kitchen sink of every @INC modifying mechanism I know of:

config.sh has:
otherlibdirs='/Users/nick/Sandpit/511v/other0:/Users/nick/Sandpit/511v/other1';
inc_version_list_init='"pie/good", "pie", "beer/foamy", "beer", 0'

[the latter is as if 5.11.0 is binary compatible with a pre-installed
 version "pie" and version "beer". 5.8.9 has
inc_version_list='5.8.8/darwin-2level 5.8.8 5.8.7/darwin-2level 5.8.7 5.8.6/darwin-2level 5.8.6';
]

Set the undocumented APPLLIB_EXP as a symbol for the C compiler:

-DAPPLLIB_EXP=\"/Users/nick/Sandpit/511v/appllib\"

Set in PERL5LIB the environment:

PERL5LIB=/Users/nick/Sandpit/511v/perl5lib0:/Users/nick/Sandpit/511v/perl5lib1

Add an -I option to the #! line (which is now put at the start of @INC)

$ cat dump_inc.pl
#!perl -I/Users/nick/Sandpit/511v/hashbangI

print "$_\n" foreach @INC;
__END__

and then run with a command line -I option.

In the output below, note that ARCHLIB_EXP and PRIVLIB_EXP are the only two
locations that the core installs to.

Before I changed things, this order, annotated with the source of elements

# -I on the #! line
/Users/nick/Sandpit/511v/hashbangI/5.11.0/darwin-thread-multi-64int-2level
/Users/nick/Sandpit/511v/hashbangI/5.11.0
/Users/nick/Sandpit/511v/hashbangI/darwin-thread-multi-64int-2level
/Users/nick/Sandpit/511v/hashbangI/beer/foamy
/Users/nick/Sandpit/511v/hashbangI/beer
/Users/nick/Sandpit/511v/hashbangI
# -I on the command line
/Users/nick/Sandpit/511v/cliI/5.11.0/darwin-thread-multi-64int-2level
/Users/nick/Sandpit/511v/cliI/5.11.0
/Users/nick/Sandpit/511v/cliI/darwin-thread-multi-64int-2level
/Users/nick/Sandpit/511v/cliI/pie/good
/Users/nick/Sandpit/511v/cliI/pie
/Users/nick/Sandpit/511v/cliI
# PERL5LIB from the environment, split on :
/Users/nick/Sandpit/511v/perl5lib0/5.11.0/darwin-thread-multi-64int-2level
/Users/nick/Sandpit/511v/perl5lib0/5.11.0
/Users/nick/Sandpit/511v/perl5lib0/darwin-thread-multi-64int-2level
/Users/nick/Sandpit/511v/perl5lib0/pie/good
/Users/nick/Sandpit/511v/perl5lib0/pie
/Users/nick/Sandpit/511v/perl5lib0
/Users/nick/Sandpit/511v/perl5lib1/5.11.0/darwin-thread-multi-64int-2level
/Users/nick/Sandpit/511v/perl5lib1/5.11.0
/Users/nick/Sandpit/511v/perl5lib1/darwin-thread-multi-64int-2level
/Users/nick/Sandpit/511v/perl5lib1/beer/foamy
/Users/nick/Sandpit/511v/perl5lib1/beer
/Users/nick/Sandpit/511v/perl5lib1
# The undocumented APPLLIB_EXP
/Users/nick/Sandpit/511v/appllib/5.11.0/darwin-thread-multi-64int-2level
/Users/nick/Sandpit/511v/appllib/5.11.0
/Users/nick/Sandpit/511v/appllib/darwin-thread-multi-64int-2level
/Users/nick/Sandpit/511v/appllib/beer/foamy
/Users/nick/Sandpit/511v/appllib/beer
/Users/nick/Sandpit/511v/appllib
# ARCHLIB_EXP (core installs here)
/Users/nick/Sandpit/511v/lib/perl5/5.11.0/darwin-thread-multi-64int-2level
# PRIVLIB_EXP (core installs here)
/Users/nick/Sandpit/511v/lib/perl5/5.11.0
# SITEARCH_EXP (CPAN defaults to install here)
/Users/nick/Sandpit/511v/lib/perl5/site_perl/5.11.0/darwin-thread-multi-64int-2level
# SITELIB_EXP (CPAN defaults to install here)
/Users/nick/Sandpit/511v/lib/perl5/site_perl/5.11.0
# SITELIB_STEM
/Users/nick/Sandpit/511v/lib/perl5/site_perl/pie/good
/Users/nick/Sandpit/511v/lib/perl5/site_perl/pie
/Users/nick/Sandpit/511v/lib/perl5/site_perl
# VENDORARCH_EXP
/Users/nick/Sandpit/511v/lib/perl5/vendor_perl/5.11.0/darwin-thread-multi-64int-2level
# VENDORLIB_EXP
/Users/nick/Sandpit/511v/lib/perl5/vendor_perl/5.11.0
# VENDORLIB_STEM
/Users/nick/Sandpit/511v/lib/perl5/vendor_perl/beer/foamy
/Users/nick/Sandpit/511v/lib/perl5/vendor_perl/beer
/Users/nick/Sandpit/511v/lib/perl5/vendor_perl
# PERL_OTHERLIBDIRS, split on :
/Users/nick/Sandpit/511v/other0/5.11.0/darwin-thread-multi-64int-2level
/Users/nick/Sandpit/511v/other0/5.11.0
/Users/nick/Sandpit/511v/other0/darwin-thread-multi-64int-2level
/Users/nick/Sandpit/511v/other0/beer/foamy
/Users/nick/Sandpit/511v/other0/beer
/Users/nick/Sandpit/511v/other0
/Users/nick/Sandpit/511v/other1/5.11.0/darwin-thread-multi-64int-2level
/Users/nick/Sandpit/511v/other1/5.11.0
/Users/nick/Sandpit/511v/other1/darwin-thread-multi-64int-2level
/Users/nick/Sandpit/511v/other1/pie/good
/Users/nick/Sandpit/511v/other1/pie
/Users/nick/Sandpit/511v/other1
# Last but not least, the current directory
.



Currently, with commit b9ba2fadb18b54e35e5de54f945111a56cbcb249

# -I on the #! line
/Users/nick/Sandpit/511v/hashbangI/5.11.0/darwin-thread-multi-64int-2level
/Users/nick/Sandpit/511v/hashbangI/5.11.0
/Users/nick/Sandpit/511v/hashbangI/darwin-thread-multi-64int-2level
/Users/nick/Sandpit/511v/hashbangI/beer/foamy
/Users/nick/Sandpit/511v/hashbangI/beer
/Users/nick/Sandpit/511v/hashbangI
# -I on the command line
/Users/nick/Sandpit/511v/cliI/5.11.0/darwin-thread-multi-64int-2level
/Users/nick/Sandpit/511v/cliI/5.11.0
/Users/nick/Sandpit/511v/cliI/darwin-thread-multi-64int-2level
/Users/nick/Sandpit/511v/cliI/pie/good
/Users/nick/Sandpit/511v/cliI/pie
/Users/nick/Sandpit/511v/cliI
# "The current version", from PERL5LIB from the environment, split on :
/Users/nick/Sandpit/511v/perl5lib0/5.11.0/darwin-thread-multi-64int-2level
/Users/nick/Sandpit/511v/perl5lib0/5.11.0
/Users/nick/Sandpit/511v/perl5lib0/darwin-thread-multi-64int-2level
/Users/nick/Sandpit/511v/perl5lib0
/Users/nick/Sandpit/511v/perl5lib1/5.11.0/darwin-thread-multi-64int-2level
/Users/nick/Sandpit/511v/perl5lib1/5.11.0
/Users/nick/Sandpit/511v/perl5lib1/darwin-thread-multi-64int-2level
/Users/nick/Sandpit/511v/perl5lib1
/Users/nick/Sandpit/511v/appllib/5.11.0/darwin-thread-multi-64int-2level
# "The current version", from the undocumented APPLLIB_EXP
/Users/nick/Sandpit/511v/appllib/5.11.0
/Users/nick/Sandpit/511v/appllib/darwin-thread-multi-64int-2level
/Users/nick/Sandpit/511v/appllib
# SITEARCH_EXP (CPAN defaults to install here)
/Users/nick/Sandpit/511v/lib/perl5/site_perl/5.11.0/darwin-thread-multi-64int-2level
# SITELIB_EXP (CPAN defaults to install here)
/Users/nick/Sandpit/511v/lib/perl5/site_perl/5.11.0
# "The current version", from SITELIB_STEM
/Users/nick/Sandpit/511v/lib/perl5/site_perl
# VENDORARCH_EXP
/Users/nick/Sandpit/511v/lib/perl5/vendor_perl/5.11.0/darwin-thread-multi-64int-2level
# VENDORLIB_EXP
/Users/nick/Sandpit/511v/lib/perl5/vendor_perl/5.11.0
# "The current version", from VENDORLIB_STEM
/Users/nick/Sandpit/511v/lib/perl5/vendor_perl
# ARCHLIB_EXP (core installs here)
/Users/nick/Sandpit/511v/lib/perl5/5.11.0/darwin-thread-multi-64int-2level
# PRIVLIB_EXP (core installs here)
/Users/nick/Sandpit/511v/lib/perl5/5.11.0
# "The current version", from PERL_OTHERLIBDIRS, split on :
/Users/nick/Sandpit/511v/other0/5.11.0/darwin-thread-multi-64int-2level
/Users/nick/Sandpit/511v/other0/5.11.0
/Users/nick/Sandpit/511v/other0/darwin-thread-multi-64int-2level
/Users/nick/Sandpit/511v/other0
/Users/nick/Sandpit/511v/other1/5.11.0/darwin-thread-multi-64int-2level
/Users/nick/Sandpit/511v/other1/5.11.0
/Users/nick/Sandpit/511v/other1/darwin-thread-multi-64int-2level
/Users/nick/Sandpit/511v/other1
# Older versions, from PERL5LIB from the environment, split on :
/Users/nick/Sandpit/511v/perl5lib0/pie/good
/Users/nick/Sandpit/511v/perl5lib0/pie
/Users/nick/Sandpit/511v/perl5lib1/beer/foamy
/Users/nick/Sandpit/511v/perl5lib1/beer
# Older versions, from the undocumented APPLLIB_EXP
/Users/nick/Sandpit/511v/appllib/beer/foamy
/Users/nick/Sandpit/511v/appllib/beer
# Older versions, from SITELIB_STEM
/Users/nick/Sandpit/511v/lib/perl5/site_perl/pie/good
/Users/nick/Sandpit/511v/lib/perl5/site_perl/pie
# Older versions, from VENDORLIB_STEM
/Users/nick/Sandpit/511v/lib/perl5/vendor_perl/beer/foamy
/Users/nick/Sandpit/511v/lib/perl5/vendor_perl/beer
/Users/nick/Sandpit/511v/lib/perl5/5.11.0
# Older versions, from PERL_OTHERLIBDIRS, split on :
/Users/nick/Sandpit/511v/other0/beer/foamy
/Users/nick/Sandpit/511v/other0/beer
/Users/nick/Sandpit/511v/other1/pie/good
/Users/nick/Sandpit/511v/other1/pie
# Last but not least, the current directory
.


This is *roughly* Andy's proposal. Specific discrepancies are

1: Andy has these two later, as they are assumed to be "old" (pre 5.005):

/Users/nick/Sandpit/511v/lib/perl5/site_perl
/Users/nick/Sandpit/511v/lib/perl5/vendor_perl

   Should they be later in @INC? If so, where?

2: Andy's layout didn't take into account the fact that after

	/usr/local/lib/perl5/site_perl/5.6.1/$archname
	/usr/local/lib/perl5/site_perl/5.6.1

  perl searches for 

	/usr/local/lib/perl5/site_perl/$archname    # surprise bonus!


  That directory doesn't have a version number in it? Does it matter that it's
  early? Or should it be later than all directories with version numbers in?
  [remember that for this example, "pie" and "beer" are version numbers, and
   that for version "pie", CPAN would install modules to 
   /Users/nick/Sandpit/511v/lib/perl5/site_perl/pie/good
   /Users/nick/Sandpit/511v/lib/perl5/site_perl/pie
  ]



So the layout meets the design goal of the (default) CPAN install location
being earlier in @INC that the core's install location, and that upgrading
perl results in its core modules being earlier in @INC than modules installed
from CPAN for the earlier perl.

Feedback requested.

Nicholas Clark
Thread Previous | Thread Next