eliminate bootstrap files on platforms that do not use@DynaLoader::dl_resolve_using?

This is an in depth followup to 
http://www.nntp.perl.org/group/perl.perl5.porters/2015/10/msg232207.html 
which got no answers.

Today I tried to make and use a .bs file on Win32, to load certain 
shared libraries into perl.exe, before I load an XS DLL that depends on 
that shared library, and that XS DLL WILL NOT AUTOMATICALLY find the 
dependent DLL for Win32-ish reasons (I wont explain to keep things 
short). It was a disaster. I struggle to understand how anyone can use 
.bs files.

I did a "dmake "BSLOADLIBS=C:\Program Files\Expat 2.1.0\B
in\libexpat.dll"" on a XS module. This is the .bs file that was generated.

----------------------------------------------------
# Expat DynaLoader bootstrap file for MSWin32 architecture.
# Do not edit this file, changes will be lost.
# This file was automatically generated by the
# Mkbootstrap routine in ExtUtils::Mkbootstrap (v7.10).
@DynaLoader::dl_resolve_using =   qw(C:\Program Files\Expat 
2.1.0\Bin\libexpat.dll);

1;
------------------------------------------------------

qw? oh really?

It gets worse

-------------------------------------------------------
# --- MakeMaker dynamic_bs section:
BOOTSTRAP = $(BASEEXT).bs

# As Mkbootstrap might not write a file (if none is required)
# we use touch to prevent make continually trying to remake it.
# The DynaLoader only reads a non-empty file.
$(BOOTSTRAP) : $(FIRST_MAKEFILE) $(BOOTDEP) 
$(INST_ARCHAUTODIR)$(DFSEP).exists
	$(NOECHO) $(ECHO) "Running Mkbootstrap for $(NAME) ($(BSLOADLIBS))"
	$(NOECHO) $(PERLRUN) \
		"-MExtUtils::Mkbootstrap" \
		-e "Mkbootstrap('$(BASEEXT)','$(BSLOADLIBS)');"
	$(NOECHO) $(TOUCH) "$@"
	$(CHMOD) $(PERM_RW) "$@"
-------------------------------------------------------

$(BASEEXT) is "Bar", as in "Foo::Bar", so the .bs file was built in the 
root src tree, next to the .o file and next to the .xs file, not in 
blib, next to the dll/shared lib (.so/.dll). So DynaLoader/XSLoader 
never found the file during a "dmake test" or a "perl -Mblib t/test.t", 
unless I install the module, the .bs file won't be found and wont run, 
so you can't pass a "make test" if you actually MUST run a .bs file, and 
MUST load additional shlibs before loading the XS shlib. Therefore, I 
say nobody is using .bs files.

Next, only 2 blead platforms use @DynaLoader::dl_resolve_using freemint 
and HPUX in ext\DynaLoader. While a .bs file is arbitrary perl code, it 
is very complicated and very "not intended" to allow you to make a 
custom .bs file with arbitrary code.

The only thing I can find with CPAN grep (my tries 
http://grep.cpan.me/?q=\w_BS\W&page=1 
http://grep.cpan.me/?q=file%3A.[bB][sS]+... 
http://grep.cpan.me/?q=file%3A_[bB][sS]+... ) that uses *_BS files, is 
the EUMM test 
https://metacpan.org/source/RJBS/perl-5.22.0/cpan/ExtUtils-MakeMaker/t/Mkbootstrap.t#L54 
for *_BS files.

I decided to do the "very complicated", and I tried making a *_BS file 
myself that emits a .bs file, this works, but only by accident.
-------------------------------------------------------
open(FH, '>', 'Expat.bs');
print FH <<'END';
use DynaLoader ();
my $file = 'C:\Program Files\Expat 2.1.0\Bin\libexpat.dll';
my $module = 'Expat';
my $libref = DynaLoader::dl_load_file($file, 0) or
     die("Can't load '$file' for module $module: ".DynaLoader::dl_error());
push(@DynaLoader::dl_librefs,$libref);  # record loaded object

END
-------------------------------------------------------

and the only reason it works is @all at 
https://metacpan.org/source/RJBS/perl-5.22.0/cpan/ExtUtils-MakeMaker/lib/ExtUtils/Mkbootstrap.pm#L48 
is empty on Win32 by default, which means it is not intended for the 
*_BS file to itself generate a .bs file, see also 
https://metacpan.org/pod/release/RJBS/perl-5.22.0/cpan/ExtUtils-MakeMaker/lib/ExtUtils/Mkbootstrap.pm 
but officially you can only ADD code to the generated .bs (the $bscode 
var), not specify the entire contents of the file. Remember Win32 
DynaLoader never uses @DynaLoader::dl_resolve_using.

To really use .bs files sanely, you need to overwrite the EUMM section 
to have arbitrary code as far as I can tell instead playing games with 
Mkboostrap.pm, but is exactly one module on cpan ( 
http://grep.cpan.me/?q=[^r][^+]dynamic_bs ) 
https://metacpan.org/source/XAN/Archive-Tar-Builder-2.0000/mk/MY.pm#L40 
that modifies dynamic_bs.

Now about @DynaLoader::dl_resolve_using. All the 
@DynaLoader::dl_resolve_using platforms leak the libraries they load 
with @DynaLoader::dl_resolve_using.

Here is HPUX
---------------------------------------
void
dl_load_file(filename, flags=0)
     char *	filename
     int		flags
     PREINIT:
     shl_t obj = NULL;
     int	i, max, bind_type;
     dMY_CXT;
     CODE:
     DLDEBUG(1,PerlIO_printf(Perl_debug_log, "dl_load_file(%s,%x):\n", 
filename,flags));
     if (flags & 0x01)
	Perl_warn(aTHX_ "Can't make loaded symbols global on this platform 
while loading %s",filename);
     if (dl_nonlazy) {
       bind_type = BIND_IMMEDIATE|BIND_VERBOSE;
     } else {
       bind_type = BIND_DEFERRED;
       /* For certain libraries, like DCE, deferred binding often causes run
        * time problems.  Adding BIND_NONFATAL to BIND_IMMEDIATE still 
allows
        * unresolved references in situations like this.  */
       /* bind_type = BIND_IMMEDIATE|BIND_NONFATAL; */
     }
     /* BIND_NOSTART removed from bind_type because it causes the 
shared library's	*/
     /* initialisers not to be run.  This causes problems with all of 
the static objects */
     /* in the library.	   */
#ifdef DEBUGGING
     if (dl_debug)
	bind_type |= BIND_VERBOSE;
#endif /* DEBUGGING */

     max = AvFILL(dl_resolve_using);
     for (i = 0; i <= max; i++) {
	char *sym = SvPVX(*av_fetch(dl_resolve_using, i, 0));
	DLDEBUG(1,PerlIO_printf(Perl_debug_log, "dl_load_file(%s) 
(dependent)\n", sym));
	obj = shl_load(sym, bind_type, 0L);
	if (obj == NULL) {
	    goto end;
	}
^^^^^^^^^^^^^^obj IS LEAKED^^^^^^^^^^^^^^
     }
     DLDEBUG(1,PerlIO_printf(Perl_debug_log, "dl_load_file(%s): ", 
filename));
     obj = shl_load(filename, bind_type, 0L);

     DLDEBUG(2,PerlIO_printf(Perl_debug_log, " libref=%p\n", (void*)obj));
end:
     ST(0) = sv_newmortal() ;
     if (obj == NULL)
         SaveError(aTHX_ "%s",Strerror(errno));
     else
         sv_setiv( ST(0), PTR2IV(obj) );
---------------------------------------------------------

HPUX DynaLoader has dl_unload_file(), but never saves the shared lib 
object pointers loaded with @DynaLoader::dl_resolve_using, they are just 
leaked (they could be stored in @DynaLoader::dl_librefs so they are 
unloaded at END time from dl_unload_all_files(), but IDK if 
@DynaLoader::dl_librefs and @DynaLoader::dl_modules becoming 
unsynchronized is a bad idea/forbidden).


Now onto FREEMINT.

-----------------------------------------------------------
void
dl_load_file(filename, flags=0)
     char *	filename
     int		flags
     PREINIT:
     int dlderr,x,max;
     GV *gv;
     dMY_CXT;
     CODE:
     DLDEBUG(1,PerlIO_printf(Perl_debug_log, "dl_load_file(%s,%x):\n", 
filename,flags));
     if (flags & 0x01)
	Perl_croak(aTHX_ "Can't make loaded symbols global on this platform 
while loading %s",filename);
     max = AvFILL(dl_require_symbols);
     for (x = 0; x <= max; x++) {
	char *sym = SvPVX(*av_fetch(dl_require_symbols, x, 0));
	DLDEBUG(1,PerlIO_printf(Perl_debug_log, "dld_create_ref(%s)\n", sym));
	if (dlderr = dld_create_reference(sym)) {
	    SaveError(aTHX_ "dld_create_reference(%s): %s", sym,
		      dld_strerror(dlderr));
	    goto haverror;
	}
     }

     DLDEBUG(1,PerlIO_printf(Perl_debug_log, "dld_link(%s)\n", filename));
     if (dlderr = dld_link(filename)) {
	SaveError(aTHX_ "dld_link(%s): %s", filename, dld_strerror(dlderr));
	goto haverror;
     }

     DLDEBUG(1,PerlIO_printf(Perl_debug_log, "dld_link(libm.a)\n"));
     if (dlderr = dld_link("/usr/lib/libm.a")) {
	SaveError(aTHX_ "dld_link(libm.a): %s", dld_strerror(dlderr));
	goto haverror;
     }

     DLDEBUG(1,PerlIO_printf(Perl_debug_log, "dld_link(libc.a)\n"));
     if (dlderr = dld_link("/usr/lib/libc.a")) {
	SaveError(aTHX_ "dld_link(libc.a): %s", dld_strerror(dlderr));
	goto haverror;
     }

     max = AvFILL(dl_resolve_using);
     for (x = 0; x <= max; x++) {
	char *sym = SvPVX(*av_fetch(dl_resolve_using, x, 0));
	DLDEBUG(1,PerlIO_printf(Perl_debug_log, "dld_link(%s)\n", sym));
	if (dlderr = dld_link(sym)) {
	    SaveError(aTHX_ "dld_link(%s): %s", sym, dld_strerror(dlderr));
	    goto haverror;
	}
^^^^^^^^^LEAK dld_link added a refcount notch on that shlib, char * sym 
was not saved to later pass to dld_unlink_by_file^^^^^^^^
     }
     DLDEBUG(2,PerlIO_printf(Perl_debug_log, "libref=%s\n", filename));
haverror:
     ST(0) = sv_newmortal() ;
     if (dlderr == 0)
	sv_setiv(ST(0), PTR2IV(filename));
^^^^^^^^^^^^^^^^^^^^^^^^^^this code put a pointer to a lexical SVPV's PV 
buffer, in an IV, and how is this number/freed pointer supposed to be 
usable after the XSRETURN(1)???? It should really be "ST(0) = 
filenameSV;" or "ST(0) = sv_2mortal(newSVsv(filenameSV));, or most 
efficiently, nothing at all, since incoming ST(0) is outgoing ST(0) ;-) 
^^^^^^^^^^^^^^^^^^^^^^^^^^
     XSRETURN(1);
-----------------------------------------------------------
FREEMINT doesn't implement an dl_unload_file() XSUB (the OS allows it 
though, "dld_unlink_by_file") so it is impossible to unload a shlib on 
it with perl.

So should bootstrap files be eliminated, both in generation or attempted 
generation by EUMM, and reading by XSLoader/DynaLoader on all platforms 
(remember "make test" can never read the .bs file, so nobody is using 
this feature)?

Or eliminate generation and reading on every platform except HPUX and 
FREEMINT?

are .bs files bs? (pun intended)

I managed to actually use a .bs file for its supposed purpose on Win32, 
but there are a huge amount of hurdles and bugs in the way, to use it 
seriously, I am thinking it is dead code.

The Win32 specific stuff I left out at the start is how DLL deps are 
resolved if they aren't system/OS DLLs in C:/Windows. 90% of people fix 
it, by adding the dir with the dll to env var PATH, either with 
$ENV{PATH} inside perl in the .pm before loading the XS DLL, or forever 
in Win32's .rc file equivalent at login time. 10% of people will call 
Win32::LoadLibrary('C:\absolute\path\to.dll');. The dll filename dep 
list in a dll is always "to.dll", if a "to.dll", with any path, is 
already loaded in the process, that "to.dll" instance will be used to 
resolve symbols for "to.dll". If there are 2 "to.dll" instances in a 
process (one had to be loaded by absolute path), most recent one to be 
loaded wins (you can still use the older to.dll for resolving symbols at 
runtime, with their handles, explicitly, but isn't the "automatic 
dynamic linking" system, this is the dlsym analog). Perl's CPAN 
version.pm had a bunch of drama, since perl had a XS "version.dll" for 
CPAN version::, but there is a Microsoft OS DLL called "version.dll" for 
reading metadata from DLL/EXE files. Win32.pm 's XS DLL links to 
"version.dll", so if you install CPAN version::, either CPAN version:: 
will fail in DynaLoader, or Win32:: will fail in DynaLoader, which ever 
was 2nd. CPAN version:: renamed its XS library to "vxs" to fix the problem.

• eliminate bootstrap files on platforms that do not use@DynaLoader::dl_resolve_using? by bulk88
• Re: eliminate bootstrap files on platforms that do not use@DynaLoader::dl_resolve_using? by Leon Timmermans
• Re: eliminate bootstrap files on platforms that do notuse@DynaLoader::dl_resolve_using? by bulk88
• Re: eliminate bootstrap files on platforms that donotuse@DynaLoader::dl_resolve_using? by bulk88
• Re: eliminate bootstrap files on platforms that do not use@DynaLoader::dl_resolve_using? by Ivan Pozdeev