develooper Front page | perl.perl5.porters | Postings from October 2018

[perl #133587] toolchain possibilities for avoiding "binariesmismatched" error

Thread Next
From:
perlbug-followup
Date:
October 13, 2018 19:15
Subject:
[perl #133587] toolchain possibilities for avoiding "binariesmismatched" error
Message ID:
rt-4.0.24-11700-1539458109-1143.133587-75-0@perl.org
# New Ticket Created by   
# Please include the string:  [perl #133587]
# in the subject line of all future correspondence about this issue. 
# <URL: https://rt.perl.org/Ticket/Display.html?id=133587 >


This is a bug report for perl from frederik@ofb.net,
generated with the help of perlbug 1.41 running under perl 5.28.0.

-----------------------------------------------------------------
[Please describe your issue here]

This is a follow-up to an earlier bug report, which complains about
the fact that users see an uninformative error message "loadable
library and perl binaries are mismatched", after a Linux distribution
upgrade, if they have installed Perl modules locally e.g. using the
'cpan' tool.

https://rt.perl.org/Public/Bug/Display.html?id=133440

Thanks to everyone who responded to the original. I guess we're
waiting for key developers to give us some guidance on improving the
error message and documentation patches I sent, but looking back over
these emails, there are a few things that I'm still confused about and
I thought maybe we could make some progress if I tried to resolve
those deficits in my understanding.

These mostly have to do with the toolchain, 'cpan'/'cpanm', and
probably ExtUtils::MakeMaker and other modules, so I'm submitting a
new bug report which I hope can focus on the toolchain aspect of this
problem. I've never tried to contribute a module to CPAN so I am
unfamiliar with how to submit bugs against toolchain components. If
this bug should be submitted elsewhere or against certain modules,
then please let me know. I'm pasting some of the original bug thread
for background below, and then I follow that with a few comments.

----------------------------------------------------------------

Date: Fri, 10 Aug 2018 13:18:56 -0700
From: Andy Dougherty via RT <perlbug-followup@perl.org>

I'm not sure why you have Compress::Raw::Zlib installed locally, since
it has been bundled with perl since version 5.9.4.  [...]

Date: Tue, 14 Aug 2018 20:27:23 -0400
From: Dan Book <grinnz@gmail.com>

These are good ideas. The fundamental issue though is that modules you
installed for one major version of Perl must always be installed again for
another, and users don't tend to expect this whether they use local::lib or
not, it will always be a problem if you install modules without using your
package manager in a Perl managed by your package manager.

Date: Sat, 18 Aug 2018 23:27:46 -0700
From: Dan Book via RT <perlbug-followup@perl.org>

> Thanks for your reply. While trying to downgrade my system (debugging
> another issue) I discovered a possible reason why I have problematic
> binary modules in my home directory which are also in Perl core
> (causing cpan to break when I upgrade Perl). The reason appears to be
> that the "recompile" cpan command reinstalls modules which I had
> removed earlier using 'cpanm'.
> [...]

These are dual-life modules; it is expected and often desired to install
them a second time in site or vendor libs to upgrade them from the core
module version. Since Perl 5.12, site and vendor libs take precedence
in @INC for that reason. local::lib of course takes precedence over all
standard lib directories. [...]

Date: Tue, 21 Aug 2018 16:43:58 -0700
From: frederik@ofb.net

> These are dual-life modules; [...]

You're saying that these modules are purposefully installed by 'cpan'
even when compatible versions already exist in 'core'?

I guess not everyone expects this, because when I first submitted this
bug report I got a response from Andy Dougherty saying "I'm not sure
why you have Compress::Raw::Zlib installed locally, since it has been
bundled with perl since version 5.9.4." And I have in my shell history
that I uninstalled it, while now it is back in ~/.local. Although when
I removed it again and re-ran 'cpan -r', it didn't reappear, so I'm
not entirely sure what's happening.

Date: Tue, 21 Aug 2018 17:00:08 -0700
From: Dan Book via RT <perlbug-followup@perl.org>

That is correct. His response seemed to account for this possibility but
was asking why you had installed a newer version in a local lib without the
use of version specific directories, which could be considered a deficiency
in the toolchain. Regardless, upgrades to dual-life modules are installed
in sitelib or local libs very commonly, this is the reason they are
available from CPAN. I can't comment on how CPAN.pm's recompile function
interacts with dual-life modules or local libs.

Date: Tue, 18 Sep 2018 10:43:27 +0100
From: Dave Mitchell <davem@iabyn.com>

[...]
The particular scenario you encountered seems to me to be relatively rare.
It required the following combination of circumstances:

1) You were using the OS's perl installation.
2) You didn't use the OS's packing system to add extra CPAN modules:
   for example on Fedora to install the Time::ParseDate module,
   I would install the perl-Time-ParseDate RPM provided by Fedora.
3) You didn't use cpan or similar in a way that installs the extra packages
   as part of the perl installation which the cpan tool is a part of.
4) You used the third-party local::lib tool install the extra CPAN modules
   under your home directory, which doesn't install them in
   version-specific paths.

----------------

My comments:

1. In using 'cpanm' and local::lib, I thought I was doing the
"standard thing", for example cpanm mentions local::lib under the
documentation for the "-l, --local-lib" flag. But other people seem to
be installing modules in "version-specific directories", which I don't
even know how to do. It doesn't seem to be the default, since I just
ran 'cpan' in a new user account and after hitting 'enter' at every
prompt and typing 'install List::BinarySearch', I found that the XS
module was installed here:

~/perl5/lib/perl5/x86_64-linux-thread-multi/auto/List/BinarySearch/XS/XS.so

I assume this location is not version-specific, since it has no
version number, just the architecture. How does someone (like Dave?)
install a module like List::BinarySearch without encountering the
above scenario, which he describes as "relatively rare"? (I don't
think my distribution has List::BinarySearch pre-packaged, but let's
assume that it doesn't)

2. I think people were taken aback when I mentioned that I have common
binary modules like Compress::Raw::Zlib installed in $HOME. I later
learned this is something that 'cpan' and 'cpanm' do automatically
(see "dual-life modules" above). (Dan Book said "His response seemed
to account for this possibility" but I'm not sure what was meant by
that...)

How are users notified of the fact that system-level modules are being
reinstalled locally, turned into "dual-life modules" as it were? Is
this in the documentation? I see the term "dual-life" appearing in the
cpanm manual page, but it is not defined, neither are neighboring
terms like "shadow files" defined:

       --uninst-shadows
           Uninstalls the shadow files of the distribution that you're
           installing. This eliminates the confusion if you're trying to
           install core (dual-life) modules from CPAN against perl 5.10 or
           older, or modules that used to be XS-based but switched to pure
           perl at some version.

How are other people (like Andy Dougherty?) avoiding 'cpan' install
actions that pull in binary dependencies like Compress::Raw::Zlib? I'm
guessing he didn't just "hit enter" at one of the prompts, like I did?

3. My own reason for wanting things to be installed in my home
directory is so that I can use some custom scripts easily on systems
where I don't have administrative access, just by copying everything
over with my shell config. My scripts are pretty simple, I'm not
trying to deploy a website or anything so I'm not sure that including
a separate build of Perl is warranted (although I see that libperl is
only 3.3M...).

Perhaps I'm taking the wrong approach, maybe I should not be using the
'cpan' tool in my use case - namely "unprivileged user with simple
utility scripts that have a few external dependencies". However, I'm
guessing it is a pretty common case, and I realized that the problem
I'm running into with the confusing "binaries mismatched" error
message is something that I could completely avoid if I could tell
'cpan' to refrain from installing what Dan called "dual-life modules",
in other words to stick with installing only modules that are not
already installed system-wide (or elsewhere in PERL5LIB).

Further, when there is an option to install a binary version of a
module, it would be nice to be able to tell 'cpan'/'cpanm' that I am
only interested in the pure-Perl implementation, or at least for it to
install the binary component in such a way that Perl can automatically
fall back on the pure-Perl version of the module, when I find myself
using my scripts on a machine with a different architecture or Perl
version or whatever else would make it impossible to employ the binary
module. In rare cases where I want to depend on a binary module, I
would probably do it by installing the distribution package for that
module, since there is a big overlap between modules which have XS
code and modules which are available in distribution-specific
packages. Basically, a big feature of Perl is backwards compatibility,
and it seems like I should be able to assemble a collection of
modules, using automatic tools like 'cpan', in such a way that they
continue to run correctly on a wide variety of systems.


It seems to me that there are a few bugs or potential areas for
improvement here. Is anyone else concerned about these things, what is
the status, and am I missing something?

Thank you,

Frederick

[Please do not change anything below this line]
-----------------------------------------------------------------
---
Flags:
    category=install
    severity=medium
---
Site configuration information for perl 5.28.0:

Configured by builduser at Wed Aug  1 10:43:08 CEST 2018.

Summary of my perl5 (revision 5 version 28 subversion 0) configuration:
   
  Platform:
    osname=linux
    osvers=4.17.11-arch1
    archname=x86_64-linux-thread-multi
    uname='linux flo-64s 4.17.11-arch1 #1 smp preempt sun jul 29 10:11:16 utc 2018 x86_64 gnulinux '
    config_args='-des -Dusethreads -Duseshrplib -Doptimize=-march=x86-64 -mtune=generic -O2 -pipe -fstack-protector-strong -fno-plt -Dprefix=/usr -Dvendorprefix=/usr -Dprivlib=/usr/share/perl5/core_perl -Darchlib=/usr/lib/perl5/5.28/core_perl -Dsitelib=/usr/share/perl5/site_perl -Dsitearch=/usr/lib/perl5/5.28/site_perl -Dvendorlib=/usr/share/perl5/vendor_perl -Dvendorarch=/usr/lib/perl5/5.28/vendor_perl -Dscriptdir=/usr/bin/core_perl -Dsitescript=/usr/bin/site_perl -Dvendorscript=/usr/bin/vendor_perl -Dinc_version_list=none -Dman1ext=1perl -Dman3ext=3perl -Dcccdlflags='-fPIC' -Dlddlflags=-shared -Wl,-O1,--sort-common,--as-needed,-z,relro,-z,now -Dldflags=-Wl,-O1,--sort-common,--as-needed,-z,relro,-z,now'
    hint=recommended
    useposix=true
    d_sigaction=define
    useithreads=define
    usemultiplicity=define
    use64bitint=define
    use64bitall=define
    uselongdouble=undef
    usemymalloc=n
    default_inc_excludes_dot=define
    bincompat5005=undef
  Compiler:
    cc='cc'
    ccflags ='-D_REENTRANT -D_GNU_SOURCE -fwrapv -fno-strict-aliasing -pipe -fstack-protector-strong -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64'
    optimize='-march=x86-64 -mtune=generic -O2 -pipe -fstack-protector-strong -fno-plt'
    cppflags='-D_REENTRANT -D_GNU_SOURCE -fwrapv -fno-strict-aliasing -pipe -fstack-protector-strong -I/usr/local/include'
    ccversion=''
    gccversion='8.1.1 20180531'
    gccosandvers=''
    intsize=4
    longsize=8
    ptrsize=8
    doublesize=8
    byteorder=12345678
    doublekind=3
    d_longlong=define
    longlongsize=8
    d_longdbl=define
    longdblsize=16
    longdblkind=3
    ivtype='long'
    ivsize=8
    nvtype='double'
    nvsize=8
    Off_t='off_t'
    lseeksize=8
    alignbytes=8
    prototype=define
  Linker and Libraries:
    ld='cc'
    ldflags ='-Wl,-O1,--sort-common,--as-needed,-z,relro,-z,now -fstack-protector-strong -L/usr/local/lib'
    libpth=/usr/local/lib /usr/lib/gcc/x86_64-pc-linux-gnu/8.1.1/include-fixed /usr/lib /lib/../lib /usr/lib/../lib /lib /lib64 /usr/lib64
    libs=-lpthread -lgdbm -ldb -ldl -lm -lcrypt -lutil -lc -lgdbm_compat
    perllibs=-lpthread -ldl -lm -lcrypt -lutil -lc
    libc=libc-2.27.so
    so=so
    useshrplib=true
    libperl=libperl.so
    gnulibc_version='2.27'
  Dynamic Linking:
    dlsrc=dl_dlopen.xs
    dlext=so
    d_dlsymun=undef
    ccdlflags='-Wl,-E -Wl,-rpath,/usr/lib/perl5/5.28/core_perl/CORE'
    cccdlflags='-fPIC'
    lddlflags='-shared -Wl,-O1,--sort-common,--as-needed,-z,relro,-z,now -L/usr/local/lib -fstack-protector-strong'


---
@INC for perl 5.28.0:
    /home/frederik/scripts-misc/perl
    /home/frederik/.local/lib/perl5/5.28.0/x86_64-linux-thread-multi
    /home/frederik/.local/lib/perl5/5.28.0
    /home/frederik/.local/lib/perl5/x86_64-linux-thread-multi
    /home/frederik/.local/lib/perl5
    /usr/lib/perl5/5.28/site_perl
    /usr/share/perl5/site_perl
    /usr/lib/perl5/5.28/vendor_perl
    /usr/share/perl5/vendor_perl
    /usr/lib/perl5/5.28/core_perl
    /usr/share/perl5/core_perl

---
Environment for perl 5.28.0:
    HOME=/home/frederik
    LANG=en_US.UTF-8
    LANGUAGE (unset)
    LD_LIBRARY_PATH=/home/frederik/.local/arch/x86_64/lib:/home/frederik/.local/lib:/usr/local/lib
    LOGDIR (unset)
    PATH=/home/frederik/.local/bin:/home/frederik/projects/mailproc:/home/frederik/scripts-misc:/home/frederik/.local/arch/x86_64/bin:/usr/bin/core_perl:/usr/bin/vendor_perl:/usr/bin/site_perl:/usr/local/bin:/usr/local/sbin:/usr/bin
    PERL5LIB=/home/frederik/scripts-misc/perl:/home/frederik/.local/lib/perl5:
    PERL_BADLANG (unset)
    PERL_LOCAL_LIB_ROOT=/home/frederik/.local/:/home/frederik/.local/:/home/frederik/.local/
    PERL_MB_OPT=--install_base "/home/frederik/.local/"
    PERL_MM_OPT=INSTALL_BASE=/home/frederik/.local/
    SHELL=/bin/zsh


Thread Next


nntp.perl.org: Perl Programming lists via nntp and http.
Comments to Ask Bjørn Hansen at ask@perl.org | Group listing | About