develooper Front page | perl.perl5.porters | Postings from October 2018

Re: [perl #133587] toolchain possibilities for avoiding "binariesmismatched" error

Thread Previous | Thread Next
From:
frederik
Date:
October 18, 2018 18:58
Subject:
Re: [perl #133587] toolchain possibilities for avoiding "binariesmismatched" error
Message ID:
20181018185832.i4gytcqp4qllgk4u@ofb.net
Thank you Graham for figuring out the reason for the behavior I'm
experiencing, and why it didn't match Dave's test-case.

I guess a summary of this thread is that most Perl developers either
(1) don't use Perl on systems where they don't have root access, or
(2) compile their own Perl and install it somewhere in ~/ in such
cases.

That this would be necessary seems odd to me, for example my shell
configuration is portable across multiple non-privileged acounts
without having to recompile my shell for different architectures and
distributions.

Of course, it is not necessary if I don't use 'cpan'/'cpanm', and if I
stick to installing (pure-Perl) modules by hand.

My question would be if there is a way to use 'cpan'/'cpanm' for the
case I am interested in, e.g. either (a) installing only pure-Perl
modules, with dependencies, in my home directory, or (b) installing
binary modules in such a way that they only get used when they are
compatible with the architecture and Perl verion. The goal is to have
a "portable home directory"...

Would adding support for versioned paths to ExtUtils::MakeMaker and
Module::Build accomplish (b)?

I am confused by something Dave Mitchell said:

> > How are users notified of the fact that system-level modules are being
> > reinstalled locally [...]
>
> Users will only get this behaviour if they request it.

That suggests that I configured 'cpan' at some time to do this but I
don't remember having done so.

It would be a big help if I could tell 'cpan' to avoid reinstalling
modules which already exist in PERL5LIB. Would this create problems
with users experiencing false bugs that are actually due to module
version mismatches? Relatedly, when CPAN modules depend on each other,
do they depend on a specific version number (which could be checked by
"module managers" like 'cpanm', when deciding to rely on an existing
system-level module installation)?

Thanks,

Frederick

On Tue, Oct 16, 2018 at 04:15:50AM -0700, Graham Knop via RT wrote:
> On Mon, Oct 15, 2018 at 1:39 PM Dave Mitchell <davem@iabyn.com> wrote:
> >
> > On Sat, Oct 13, 2018 at 12:15:09PM -0700,  (via RT) wrote:
> > > 1. In using 'cpanm' and local::lib, I thought I was doing the
> > > "standard thing", for example cpanm mentions local::lib under the
> > > documentation for the "-l, --local-lib" flag. But other people seem to
> > > be installing modules in "version-specific directories", which I don't
> > > even know how to do. It doesn't seem to be the default, since I just
> > > ran 'cpan' in a new user account and after hitting 'enter' at every
> > > prompt and typing 'install List::BinarySearch', I found that the XS
> > > module was installed here:
> > >
> > > ~/perl5/lib/perl5/x86_64-linux-thread-multi/auto/List/BinarySearch/XS/XS.so
> > >
> > > I assume this location is not version-specific, since it has no
> > > version number, just the architecture. How does someone (like Dave?)
> > > install a module like List::BinarySearch without encountering the
> > > above scenario, which he describes as "relatively rare"? (I don't
> > > think my distribution has List::BinarySearch pre-packaged, but let's
> > > assume that it doesn't)
> >
> > A perl configured and built using default options *will* use
> > version-specific paths. Here's a simple test I did just now:
> >
> > $ tar xfz ~/perl5/tarballs/perl-5.28.0.tar.gz
> > $ cd perl-5.28.0/
> > $ PATH=/bin:/usr/bin sh Configure -des -Dprefix=/home/davem/tmp/foo/perl  && TEST_JOBS=16  HARNESS_OPTIONS=j16  make  -j 16 test_harness && make  -j 16 install
> > $ cd ..; $ rm -rf perl-5.28.0
> > $ cd /home/davem/tmp/foo
> > $ perl/bin/cpan
> > cpan[1]> install List::BinarySearch
> > Would you like to install List::BinarySearch::XS? [y]
> > $ find . |grep BinarySearch
> > ./perl/lib/site_perl/5.28.0/x86_64-linux/auto/List/BinarySearch
> > ./perl/lib/site_perl/5.28.0/x86_64-linux/auto/List/BinarySearch/XS
> > ./perl/lib/site_perl/5.28.0/x86_64-linux/auto/List/BinarySearch/XS/XS.so
> > ./perl/lib/site_perl/5.28.0/x86_64-linux/auto/List/BinarySearch/XS/.packlist
> > ./perl/lib/site_perl/5.28.0/x86_64-linux/auto/List/BinarySearch/.packlist
> > ./perl/lib/site_perl/5.28.0/x86_64-linux/List/BinarySearch
> > ./perl/lib/site_perl/5.28.0/x86_64-linux/List/BinarySearch/XS.pm
> > ./perl/lib/site_perl/5.28.0/List/BinarySearch.pm
> > ./perl/lib/site_perl/5.28.0/List/BinarySearch
> > ./perl/lib/site_perl/5.28.0/List/BinarySearch/PP.pm
> > ./perl/man/man3/List::BinarySearch.3
> > ./perl/man/man3/List::BinarySearch::XS.3
> > ./perl/man/man3/List::BinarySearch::PP.3
> >
> > Please look carefully at the above, and bear it very keenly in mind for
> > any further discussion. If that doesn't convince you that perl does in
> > indeed use version-specific paths *by default*, then you'll have to be
> > very clear and specific about what your doubts are.
> >
> > > 2. I think people were taken aback when I mentioned that I have common
> > > binary modules like Compress::Raw::Zlib installed in $HOME. I later
> > > learned this is something that 'cpan' and 'cpanm' do automatically
> >
> > No, by default cpan etc will try to install under the site_perl directory
> > of wherever perl was installed, unless you tell it otherwise, e.g. with
> > options at perl build time, or by using an external tool like local::lib
> > to set some environment variables which tells cpan to install somewhere
> > else.
> 
> This is true if you have permissions to install in site_perl.  But if
> you try to install modules using system perl without running as root,
> you probably won't have access to do that.
> 
> In that case, both CPAN.pm and cpanm will default to using local::lib
> and installing into ~/perl5, using ExtUtils::MakeMaker's INSTALL_BASE
> option and Module::Build's --install_base option.  Those options don't
> use versioned paths.
> 
> >
> > > How are users notified of the fact that system-level modules are being
> > > reinstalled locally, turned into "dual-life modules" as it were?
> >
> > Users will only get this behaviour if they request it.
> >
> > The term 'dual-life' is usually used (by us at least) to refer to
> > something different - where a module is both bundled with perl and
> > available (possibly in a newer version) on CPAN. It doesn't (very much)
> > relate to where the module gets installed).
> >
> > > How are other people (like Andy Dougherty?) avoiding 'cpan' install
> > > actions that pull in binary dependencies like Compress::Raw::Zlib? I'm
> > > guessing he didn't just "hit enter" at one of the prompts, like I did?
> >
> > Again, this just doesn't normally occur with a default installation.
> >
> > > Perhaps I'm taking the wrong approach, maybe I should not be using the
> > > 'cpan' tool in my use case - namely "unprivileged user with simple
> > > utility scripts that have a few external dependencies". However, I'm
> > > guessing it is a pretty common case, and I realized that the problem
> > > I'm running into with the confusing "binaries mismatched" error
> > > message is something that I could completely avoid if I could tell
> > > 'cpan' to refrain from installing what Dan called "dual-life modules",
> > > in other words to stick with installing only modules that are not
> > > already installed system-wide (or elsewhere in PERL5LIB).
> >
> > No, what you really want to do is, if installing under your home
> > directory, install version-sensitive files under version-specific paths.
> > This appears to be a infelicity in the current local::lib tool. I don't
> > know a lot about local::lib, and I'm not a "toolchain guy", so I don't
> > know whether there's a good reason why local::lib does things the way it
> > does or whether it can be altered / extended to include versions in paths.
> 
> local::lib is using the options provided by ExtUtils::MakeMaker and
> Module::Build.  I've worked in the past on using additional options to
> force installing into versioned directories, but didn't end up
> finishing the work.  I had found other options for myself, and nobody
> else pushed for fixing the issue until now, so I hadn't gotten back to
> it.
> 
> >
> > > Further, when there is an option to install a binary version of a
> > > module, it would be nice to be able to tell 'cpan'/'cpanm' that I am
> > > only interested in the pure-Perl implementation, or at least for it to
> > > install the binary component in such a way that Perl can automatically
> > > fall back on the pure-Perl version of the module, when I find myself
> > > using my scripts on a machine with a different architecture or Perl
> > > version or whatever else would make it impossible to employ the binary
> > > module. In rare cases where I want to depend on a binary module, I
> > > would probably do it by installing the distribution package for that
> > > module, since there is a big overlap between modules which have XS
> > > code and modules which are available in distribution-specific
> > > packages. Basically, a big feature of Perl is backwards compatibility,
> > > and it seems like I should be able to assemble a collection of
> > > modules, using automatic tools like 'cpan', in such a way that they
> > > continue to run correctly on a wide variety of systems.
> >
> > I think such functionality would have to be done by the distribution
> > itself on a case-by-case basis.
> >
> > > It seems to me that there are a few bugs or potential areas for
> > > improvement here. Is anyone else concerned about these things, what is
> > > the status, and am I missing something?
> >
> > My thoughts:
> >
> > * The situation is not ideal, but
> > * fixes are not easy: this whole area is complex (and not one I
> >   understand well)
> >
> > --
> > Technology is dominated by two types of people: those who understand what
> > they do not manage, and those who manage what they do not understand.
> 

Thread Previous | Thread Next


nntp.perl.org: Perl Programming lists via nntp and http.
Comments to Ask Bjørn Hansen at ask@perl.org | Group listing | About