Front page | perl.perl5.porters |
Postings from October 2015
eliminate bootstrap files on platforms that do not use@DynaLoader::dl_resolve_using?
Thread Next
From:
bulk88
Date:
October 30, 2015 11:00
Subject:
eliminate bootstrap files on platforms that do not use@DynaLoader::dl_resolve_using?
Message ID:
20151030110026.15180.qmail@lists-nntp.develooper.com
This is an in depth followup to
http://www.nntp.perl.org/group/perl.perl5.porters/2015/10/msg232207.html
which got no answers.
Today I tried to make and use a .bs file on Win32, to load certain
shared libraries into perl.exe, before I load an XS DLL that depends on
that shared library, and that XS DLL WILL NOT AUTOMATICALLY find the
dependent DLL for Win32-ish reasons (I wont explain to keep things
short). It was a disaster. I struggle to understand how anyone can use
.bs files.
I did a "dmake "BSLOADLIBS=C:\Program Files\Expat 2.1.0\B
in\libexpat.dll"" on a XS module. This is the .bs file that was generated.
----------------------------------------------------
# Expat DynaLoader bootstrap file for MSWin32 architecture.
# Do not edit this file, changes will be lost.
# This file was automatically generated by the
# Mkbootstrap routine in ExtUtils::Mkbootstrap (v7.10).
@DynaLoader::dl_resolve_using = qw(C:\Program Files\Expat
2.1.0\Bin\libexpat.dll);
1;
------------------------------------------------------
qw? oh really?
It gets worse
-------------------------------------------------------
# --- MakeMaker dynamic_bs section:
BOOTSTRAP = $(BASEEXT).bs
# As Mkbootstrap might not write a file (if none is required)
# we use touch to prevent make continually trying to remake it.
# The DynaLoader only reads a non-empty file.
$(BOOTSTRAP) : $(FIRST_MAKEFILE) $(BOOTDEP)
$(INST_ARCHAUTODIR)$(DFSEP).exists
$(NOECHO) $(ECHO) "Running Mkbootstrap for $(NAME) ($(BSLOADLIBS))"
$(NOECHO) $(PERLRUN) \
"-MExtUtils::Mkbootstrap" \
-e "Mkbootstrap('$(BASEEXT)','$(BSLOADLIBS)');"
$(NOECHO) $(TOUCH) "$@"
$(CHMOD) $(PERM_RW) "$@"
-------------------------------------------------------
$(BASEEXT) is "Bar", as in "Foo::Bar", so the .bs file was built in the
root src tree, next to the .o file and next to the .xs file, not in
blib, next to the dll/shared lib (.so/.dll). So DynaLoader/XSLoader
never found the file during a "dmake test" or a "perl -Mblib t/test.t",
unless I install the module, the .bs file won't be found and wont run,
so you can't pass a "make test" if you actually MUST run a .bs file, and
MUST load additional shlibs before loading the XS shlib. Therefore, I
say nobody is using .bs files.
Next, only 2 blead platforms use @DynaLoader::dl_resolve_using freemint
and HPUX in ext\DynaLoader. While a .bs file is arbitrary perl code, it
is very complicated and very "not intended" to allow you to make a
custom .bs file with arbitrary code.
The only thing I can find with CPAN grep (my tries
http://grep.cpan.me/?q=\w_BS\W&page=1
http://grep.cpan.me/?q=file%3A.[bB][sS]+...
http://grep.cpan.me/?q=file%3A_[bB][sS]+... ) that uses *_BS files, is
the EUMM test
https://metacpan.org/source/RJBS/perl-5.22.0/cpan/ExtUtils-MakeMaker/t/Mkbootstrap.t#L54
for *_BS files.
I decided to do the "very complicated", and I tried making a *_BS file
myself that emits a .bs file, this works, but only by accident.
-------------------------------------------------------
open(FH, '>', 'Expat.bs');
print FH <<'END';
use DynaLoader ();
my $file = 'C:\Program Files\Expat 2.1.0\Bin\libexpat.dll';
my $module = 'Expat';
my $libref = DynaLoader::dl_load_file($file, 0) or
die("Can't load '$file' for module $module: ".DynaLoader::dl_error());
push(@DynaLoader::dl_librefs,$libref); # record loaded object
END
-------------------------------------------------------
and the only reason it works is @all at
https://metacpan.org/source/RJBS/perl-5.22.0/cpan/ExtUtils-MakeMaker/lib/ExtUtils/Mkbootstrap.pm#L48
is empty on Win32 by default, which means it is not intended for the
*_BS file to itself generate a .bs file, see also
https://metacpan.org/pod/release/RJBS/perl-5.22.0/cpan/ExtUtils-MakeMaker/lib/ExtUtils/Mkbootstrap.pm
but officially you can only ADD code to the generated .bs (the $bscode
var), not specify the entire contents of the file. Remember Win32
DynaLoader never uses @DynaLoader::dl_resolve_using.
To really use .bs files sanely, you need to overwrite the EUMM section
to have arbitrary code as far as I can tell instead playing games with
Mkboostrap.pm, but is exactly one module on cpan (
http://grep.cpan.me/?q=[^r][^+]dynamic_bs )
https://metacpan.org/source/XAN/Archive-Tar-Builder-2.0000/mk/MY.pm#L40
that modifies dynamic_bs.
Now about @DynaLoader::dl_resolve_using. All the
@DynaLoader::dl_resolve_using platforms leak the libraries they load
with @DynaLoader::dl_resolve_using.
Here is HPUX
---------------------------------------
void
dl_load_file(filename, flags=0)
char * filename
int flags
PREINIT:
shl_t obj = NULL;
int i, max, bind_type;
dMY_CXT;
CODE:
DLDEBUG(1,PerlIO_printf(Perl_debug_log, "dl_load_file(%s,%x):\n",
filename,flags));
if (flags & 0x01)
Perl_warn(aTHX_ "Can't make loaded symbols global on this platform
while loading %s",filename);
if (dl_nonlazy) {
bind_type = BIND_IMMEDIATE|BIND_VERBOSE;
} else {
bind_type = BIND_DEFERRED;
/* For certain libraries, like DCE, deferred binding often causes run
* time problems. Adding BIND_NONFATAL to BIND_IMMEDIATE still
allows
* unresolved references in situations like this. */
/* bind_type = BIND_IMMEDIATE|BIND_NONFATAL; */
}
/* BIND_NOSTART removed from bind_type because it causes the
shared library's */
/* initialisers not to be run. This causes problems with all of
the static objects */
/* in the library. */
#ifdef DEBUGGING
if (dl_debug)
bind_type |= BIND_VERBOSE;
#endif /* DEBUGGING */
max = AvFILL(dl_resolve_using);
for (i = 0; i <= max; i++) {
char *sym = SvPVX(*av_fetch(dl_resolve_using, i, 0));
DLDEBUG(1,PerlIO_printf(Perl_debug_log, "dl_load_file(%s)
(dependent)\n", sym));
obj = shl_load(sym, bind_type, 0L);
if (obj == NULL) {
goto end;
}
^^^^^^^^^^^^^^obj IS LEAKED^^^^^^^^^^^^^^
}
DLDEBUG(1,PerlIO_printf(Perl_debug_log, "dl_load_file(%s): ",
filename));
obj = shl_load(filename, bind_type, 0L);
DLDEBUG(2,PerlIO_printf(Perl_debug_log, " libref=%p\n", (void*)obj));
end:
ST(0) = sv_newmortal() ;
if (obj == NULL)
SaveError(aTHX_ "%s",Strerror(errno));
else
sv_setiv( ST(0), PTR2IV(obj) );
---------------------------------------------------------
HPUX DynaLoader has dl_unload_file(), but never saves the shared lib
object pointers loaded with @DynaLoader::dl_resolve_using, they are just
leaked (they could be stored in @DynaLoader::dl_librefs so they are
unloaded at END time from dl_unload_all_files(), but IDK if
@DynaLoader::dl_librefs and @DynaLoader::dl_modules becoming
unsynchronized is a bad idea/forbidden).
Now onto FREEMINT.
-----------------------------------------------------------
void
dl_load_file(filename, flags=0)
char * filename
int flags
PREINIT:
int dlderr,x,max;
GV *gv;
dMY_CXT;
CODE:
DLDEBUG(1,PerlIO_printf(Perl_debug_log, "dl_load_file(%s,%x):\n",
filename,flags));
if (flags & 0x01)
Perl_croak(aTHX_ "Can't make loaded symbols global on this platform
while loading %s",filename);
max = AvFILL(dl_require_symbols);
for (x = 0; x <= max; x++) {
char *sym = SvPVX(*av_fetch(dl_require_symbols, x, 0));
DLDEBUG(1,PerlIO_printf(Perl_debug_log, "dld_create_ref(%s)\n", sym));
if (dlderr = dld_create_reference(sym)) {
SaveError(aTHX_ "dld_create_reference(%s): %s", sym,
dld_strerror(dlderr));
goto haverror;
}
}
DLDEBUG(1,PerlIO_printf(Perl_debug_log, "dld_link(%s)\n", filename));
if (dlderr = dld_link(filename)) {
SaveError(aTHX_ "dld_link(%s): %s", filename, dld_strerror(dlderr));
goto haverror;
}
DLDEBUG(1,PerlIO_printf(Perl_debug_log, "dld_link(libm.a)\n"));
if (dlderr = dld_link("/usr/lib/libm.a")) {
SaveError(aTHX_ "dld_link(libm.a): %s", dld_strerror(dlderr));
goto haverror;
}
DLDEBUG(1,PerlIO_printf(Perl_debug_log, "dld_link(libc.a)\n"));
if (dlderr = dld_link("/usr/lib/libc.a")) {
SaveError(aTHX_ "dld_link(libc.a): %s", dld_strerror(dlderr));
goto haverror;
}
max = AvFILL(dl_resolve_using);
for (x = 0; x <= max; x++) {
char *sym = SvPVX(*av_fetch(dl_resolve_using, x, 0));
DLDEBUG(1,PerlIO_printf(Perl_debug_log, "dld_link(%s)\n", sym));
if (dlderr = dld_link(sym)) {
SaveError(aTHX_ "dld_link(%s): %s", sym, dld_strerror(dlderr));
goto haverror;
}
^^^^^^^^^LEAK dld_link added a refcount notch on that shlib, char * sym
was not saved to later pass to dld_unlink_by_file^^^^^^^^
}
DLDEBUG(2,PerlIO_printf(Perl_debug_log, "libref=%s\n", filename));
haverror:
ST(0) = sv_newmortal() ;
if (dlderr == 0)
sv_setiv(ST(0), PTR2IV(filename));
^^^^^^^^^^^^^^^^^^^^^^^^^^this code put a pointer to a lexical SVPV's PV
buffer, in an IV, and how is this number/freed pointer supposed to be
usable after the XSRETURN(1)???? It should really be "ST(0) =
filenameSV;" or "ST(0) = sv_2mortal(newSVsv(filenameSV));, or most
efficiently, nothing at all, since incoming ST(0) is outgoing ST(0) ;-)
^^^^^^^^^^^^^^^^^^^^^^^^^^
XSRETURN(1);
-----------------------------------------------------------
FREEMINT doesn't implement an dl_unload_file() XSUB (the OS allows it
though, "dld_unlink_by_file") so it is impossible to unload a shlib on
it with perl.
So should bootstrap files be eliminated, both in generation or attempted
generation by EUMM, and reading by XSLoader/DynaLoader on all platforms
(remember "make test" can never read the .bs file, so nobody is using
this feature)?
Or eliminate generation and reading on every platform except HPUX and
FREEMINT?
are .bs files bs? (pun intended)
I managed to actually use a .bs file for its supposed purpose on Win32,
but there are a huge amount of hurdles and bugs in the way, to use it
seriously, I am thinking it is dead code.
The Win32 specific stuff I left out at the start is how DLL deps are
resolved if they aren't system/OS DLLs in C:/Windows. 90% of people fix
it, by adding the dir with the dll to env var PATH, either with
$ENV{PATH} inside perl in the .pm before loading the XS DLL, or forever
in Win32's .rc file equivalent at login time. 10% of people will call
Win32::LoadLibrary('C:\absolute\path\to.dll');. The dll filename dep
list in a dll is always "to.dll", if a "to.dll", with any path, is
already loaded in the process, that "to.dll" instance will be used to
resolve symbols for "to.dll". If there are 2 "to.dll" instances in a
process (one had to be loaded by absolute path), most recent one to be
loaded wins (you can still use the older to.dll for resolving symbols at
runtime, with their handles, explicitly, but isn't the "automatic
dynamic linking" system, this is the dlsym analog). Perl's CPAN
version.pm had a bunch of drama, since perl had a XS "version.dll" for
CPAN version::, but there is a Microsoft OS DLL called "version.dll" for
reading metadata from DLL/EXE files. Win32.pm 's XS DLL links to
"version.dll", so if you install CPAN version::, either CPAN version::
will fail in DynaLoader, or Win32:: will fail in DynaLoader, which ever
was 2nd. CPAN version:: renamed its XS library to "vxs" to fix the problem.
Thread Next
-
eliminate bootstrap files on platforms that do not use@DynaLoader::dl_resolve_using?
by bulk88