Front page | perl.perl5.porters |
Postings from April 2018
did static perl/makeaperl on Win32 ever work in history?
Thread Next
From:
bulk88
Date:
April 9, 2018 18:07
Subject:
did static perl/makeaperl on Win32 ever work in history?
Message ID:
20180409180245.30729.qmail@lists-nntp.develooper.com
Recently because of https://rt.perl.org/Ticket/Display.html?id=132992 it
shows someone doing win32 static builds. I've never paid attention to a
static win32 perl before until now. But in my testing, there are still
many things broken with it.
2 styles of perl are build, a stub perl.exe 9KB and a massive 6.2MB
perl527.dll (aka libperl) that has an xs_init() that has 55 ::bootstrap
NewXS registration calls. This perl527.dll has a normal export table,
and uneventfully can compile, and load XS DLLs into the process, just
like regular (active/strawberry/default) dynamic Win32 perl. I haven't
experimented if its possible to upgrade/replace the static linked day 1
XS modules inside perl527.dll with XS DLL modules compiled off CPAN.
And a "perl-static.exe" 5.5MB, that also has an xs_init() that has 55
::bootstrap NewXS registration calls. But it doesn't have an export
table (bug or intentional?). BUTTTTT it will try to load an XS DLL into
the process if there is one in auto dir in @INC and you call
Dynaloader/XSLoader. This results in a handshake error/SEGV or "can't
find perl527.dll" from the OS, since the XS DLL is compiled to import
perl symbols from "perl527.dll" not from "perl-static.exe". Is
Dynaloader trying to load XS DLLs into perl-static.exe a bug?
I also tried with an XS module to do "make test_static", which generates
a Makefile.aperl and a new perlmain.c and tries to build a new
perl-static.exe. This was horribly broken on Win32. The code in
MM_Unix.pm:makeaperl is Win32-unaware.
Example, hard coded "nm" in MM_Unix.pm:xs_static_lib_is_xs
https://perl5.git.perl.org/perl.git/blob/908f2cb56527d29c9176e478fa7eee0d02a7c77e:/cpan/ExtUtils-MakeMaker/lib/ExtUtils/MM_Unix.pm#l2748
hard coding "cat" which is unix only
https://perl5.git.perl.org/perl.git/blob/908f2cb56527d29c9176e478fa7eee0d02a7c77e:/cpan/ExtUtils-MakeMaker/lib/ExtUtils/MM_Unix.pm#l2607
https://perl5.git.perl.org/perl.git/blob/908f2cb56527d29c9176e478fa7eee0d02a7c77e:/cpan/ExtUtils-MakeMaker/lib/ExtUtils/MM_Unix.pm#l2614
`` for shell execution, not win32 compliant
https://perl5.git.perl.org/perl.git/blob/908f2cb56527d29c9176e478fa7eee0d02a7c77e:/cpan/ExtUtils-MakeMaker/lib/ExtUtils/MM_Unix.pm#l2614
compile a .c and link in 1 shot, g++ specific, not MSVC compliant
https://perl5.git.perl.org/perl.git/blob/908f2cb56527d29c9176e478fa7eee0d02a7c77e:/cpan/ExtUtils-MakeMaker/lib/ExtUtils/MM_Unix.pm#l2614
Another problem is, the new perlmain.c that is generated by
ExtUtils::Miniperl::writemain() doesn't have the correct list of Xsub
bootstrap registration calls.
"C:\perl527s\bin\perl-static.exe" "-Iblib\arch" "-Iblib\lib"
"-Ic:\perl527s\lib" "-Ic:\perl527s\lib" "-MExtUtils::Miniperl" -e
"writemain(grep(s#.*/auto/##s, @ARGV), q(DynaLoader))"
"blib\arch\auto\Cpanel\JSON\XS\XS.lib"
----------------------------------------
/* Register any extra external extensions */
EXTERN_C void boot_DynaLoader (pTHX_ CV* cv);
static void
xs_init(pTHX)
{
static const char file[] = __FILE__;
dXSUB_SYS;
PERL_UNUSED_CONTEXT;
/* DynaLoader is a special case */
newXS("DynaLoader::boot_DynaLoader", boot_DynaLoader, file);
}
---------------------------------------
With a little bit of cmd line massaging, specifically flipping the dir
seps, I got something slightly sane
C:\sources\Cpanel-JSON-XS>"C:\perl527s\bin\perl-static.exe"
"-Iblib\arch" "-Iblib\lib" "-Ic:\perl527s\lib" "-Ic:\perl527s\lib"
"-MExtUtils::Miniperl" -e"writemain(grep(s#.*/auto/##s, @ARGV),
q(DynaLoader))" "blib/arch/auto/Cpanel/JSON/XS/XS.lib"
---------------------------------------
/* Register any extra external extensions */
EXTERN_C void boot_Cpanel__JSON__XS (pTHX_ CV* cv);
EXTERN_C void boot_DynaLoader (pTHX_ CV* cv);
static void
xs_init(pTHX)
{
static const char file[] = __FILE__;
dXSUB_SYS;
PERL_UNUSED_CONTEXT;
newXS("Cpanel::JSON::XS::bootstrap", boot_Cpanel__JSON__XS, file);
/* DynaLoader is a special case */
newXS("DynaLoader::boot_DynaLoader", boot_DynaLoader, file);
}
--------------------------------------
But that is still broken, where are the 55 other XS libraries in static
perl? the new static perl binary will have no XS libs inside it except
DynaLoader and the 1 afterwards XS library I am trying to compile.
Anyways, I then manually fix the Makefile.aperl to link a new static
perl.exe. Changing all the references libperl.lib to perl527s.lib.
removing "`cat $(INST_ARCHAUTODIR)/extralibs.all`" since cat doesnt work
and extralibs.all is empty. Change "/Foperl" to "/out:perl". etc. etc.
---------------------------------------
C:\sources\Cpanel-JSON-XS>gmake -f Makefile.aperl test_static
cl -c -nologo -GF -W3 -O1 -MD -Zi -DNDEBUG -GL -DWIN32 -D_CONSOLE
-DNO_STRICT
-DPERL_TEXTMODE_SCRIPTS -DPERL_IMPLICIT_CONTEXT -DPERL_IMPLICIT_SYS
-DWIN32_NO_R
EGISTRY -DUSE_PERLIO -D_USE_32BIT_TIME_T -DPERLDLL -O1 -MD -Zi -DNDEBUG
-GL -D
VERSION=\"3.0103\" -DXS_VERSION=\"3.0103\" "-IC:\perl527s\lib\CORE" XS.c
XS.c
XS.xs(2632) : warning C4018: '>' : signed/unsigned mismatch
XS.xs(2653) : warning C4102: 'modechange' : unreferenced label
"C:\perl527s\bin\perl-static.exe" -MExtUtils::Command -e rm_f --
"blib\arch\auto
\Cpanel\JSON\XS\XS.lib"
lib -ltcg -out:blib\arch\auto\Cpanel\JSON\XS\XS.lib XS.obj
Microsoft (R) Library Manager Version 7.10.6030
Copyright (C) Microsoft Corporation. All rights reserved.
"C:\perl527s\bin\perl-static.exe" -MExtUtils::Command -e chmod -- 755
blib\arch\
auto\Cpanel\JSON\XS\XS.lib
Skip blib\lib\Cpanel\JSON\XS.pm (unchanged)
Skip blib\lib\Cpanel\JSON\XS\Boolean.pm (unchanged)
"C:\perl527s\bin\perl-static.exe" -MExtUtils::Command -e cp --
bin/cpanel_json_x
s blib\script\cpanel_json_xs
pl2bat.bat blib\script\cpanel_json_xs
Writing perlmain.c
"C:\perl527s\bin\perl-static.exe" -MExtUtils::Command -e mv --
perlmain.ct perlm
ain.c
\|/\|/\|/\|/\|/\|/\|/\|/\|/\|/\|/
perlmain.c -> perlmain.o
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
cl -c -nologo -GF -W3 -O1 -MD -Zi -DNDEBUG -GL -DWIN32 -D_CONSOLE
-DNO_STRICT
-DPERL_TEXTMODE_SCRIPTS -DPERL_IMPLICIT_CONTEXT -DPERL_IMPLICIT_SYS
-DWIN32_NO_R
EGISTRY -DUSE_PERLIO -D_USE_32BIT_TIME_T -DPERLDLL -O1 -MD -Zi -DNDEBUG
-GL -D
VERSION=\"3.0103\" -DXS_VERSION=\"3.0103\" "-IC:\perl527s\lib\CORE"
perlmain.
c
perlmain.c
\|/\|/\|/\|/\|/\|/\|/\|/\|/\|/\|/
link new perl.exe
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
link -nologo -nodefaultlib -debug -opt:ref,icf -ltcg
-libpath:"c:\perl527s\lib\C
ORE" -machine:x86 XS.obj ./perlmain.obj /out:perl.exe
"blib\arch\auto\Cpanel\J
SON\XS\XS.lib" "c:\perl527s\lib/CORE/perl527s.lib" oldnames.lib
kernel32.lib us
er32.lib gdi32.lib winspool.lib comdlg32.lib advapi32.lib shell32.lib
ole32.lib
oleaut32.lib netapi32.lib uuid.lib ws2_32.lib mpr.lib winmm.lib
version.lib odbc
32.lib odbccp32.lib comctl32.lib msvcrt.lib
Creating library perl.lib and object perl.exp
Generating code
Finished generating code
LINK : warning LNK4020: a type record in
'c:\perl521\srcnew\cpan\encode\cn\vc70.
pdb' is corrupted; some symbols and types may not be accessible from the
debugge
r
LINK : warning LNK4020: a type record in
'c:\perl521\srcnew\cpan\encode\cn\vc70.
pdb' is corrupted; some symbols and types may not be accessible from the
debugge
r
To install the new 'perl' binary, call
gmake -f Makefile.aperl inst_perl MAP_TARGET=perl
gmake -f Makefile.aperl map_clean
\|/\|/\|/\|/\|/\|/\|/\|/\|/\|/\|/
EXECUTING new perl.exe
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
"C:\sources\Cpanel-JSON-XS\perl" "-Iblib\arch" "-Iblib\lib"
"-Ic:\perl527s\lib"
"-Ic:\perl527s\lib" "-MExtUtils::Command::MM" "-MTest::Harness" "-e"
"undef *Tes
t::Harness::Switches; test_harness(0, 'blib\lib', 'blib\arch')" t/*.t
Can't locate loadable object for module Cwd in @INC (@INC contains:
blib\arch bl
ib\lib c:\perl527s\lib c:\perl527s\lib) at
c:\perl527s\lib/File/Spec/Win32.pm li
ne 5.
Compilation failed in require at c:\perl527s\lib/File/Spec/Win32.pm line 5.
BEGIN failed--compilation aborted at c:\perl527s\lib/File/Spec/Win32.pm
line 5.
Compilation failed in require at c:\perl527s\lib/File/Spec.pm line 21.
Compilation failed in require at c:\perl527s\lib/TAP/Harness.pm line 7.
BEGIN failed--compilation aborted at c:\perl527s\lib/TAP/Harness.pm line 7.
Compilation failed in require at c:\perl527s\lib/Test/Harness.pm line 11.
BEGIN failed--compilation aborted at c:\perl527s\lib/Test/Harness.pm
line 11.
Compilation failed in require.
BEGIN failed--compilation aborted.
Makefile.aperl:1135: recipe for target 'test_static' failed
gmake: *** [test_static] Error 2
C:\sources\Cpanel-JSON-XS>
--------------------------------
So Cwd's XS code is missing from the new perl.exe because the new
perlmain.c didn't have it.
Another problem is, every single "makeaperl" binary SEGVs on exit.
"Unhandled exception at 0x0040110d in perl.exe: 0xC0000005: Access
violation reading location 0x00e26954."
crash callstack
-------------------------------
perl.exe!main(int argc=2, char * * argv=0x00e22bb8, char * *
env=0x00e23140) Line 166 + 0x5 C
perl.exe!mainCRTStartup() Line 398 + 0xe C
kernel32.dll!_BaseProcessStart@4() + 0x23
-------------------------------
Cut of perlmain.c with crash location
------------------------------
#ifdef NO_ENV_ARRAY_IN_MAIN
extern char **environ;
int
main(int argc, char **argv)
#else
int
main(int argc, char **argv, char **env)
#endif
{
int exitstatus, i;
#ifdef PERL_GLOBAL_STRUCT
struct perl_vars *my_vars = init_global_struct();
# ifdef PERL_GLOBAL_STRUCT_PRIVATE
int veto;
my_plvarsp = my_vars;
# endif
#endif /* PERL_GLOBAL_STRUCT */
#ifndef NO_ENV_ARRAY_IN_MAIN
PERL_UNUSED_ARG(env);
#endif
#ifndef PERL_USE_SAFE_PUTENV
PL_use_safe_putenv = FALSE;
#endif /* PERL_USE_SAFE_PUTENV */
/* if user wants control of gprof profiling off by default */
/* noop unless Configure is given -Accflags=-DPERL_GPROF_CONTROL */
PERL_GPROF_MONCONTROL(0);
#ifdef NO_ENV_ARRAY_IN_MAIN
PERL_SYS_INIT3(&argc,&argv,&environ);
#else
PERL_SYS_INIT3(&argc,&argv,&env);
#endif
#if defined(USE_ITHREADS)
/* XXX Ideally, this should really be happening in perl_alloc() or
* perl_construct() to keep libperl.a transparently fork()-safe.
* It is currently done here only because Apache/mod_perl have
* problems due to lack of a call to cancel pthread_atfork()
* handlers when shared objects that contain the handlers may
* be dlclose()d. This forces applications that embed perl to
* call PTHREAD_ATFORK() explicitly, but if and only if it hasn't
* been called at least once before in the current process.
* --GSAR 2001-07-20 */
PTHREAD_ATFORK(Perl_atfork_lock,
Perl_atfork_unlock,
Perl_atfork_unlock);
#endif
PERL_SYS_FPU_INIT;
if (!PL_do_undump) {
my_perl = perl_alloc();
if (!my_perl)
exit(1);
perl_construct(my_perl);
PL_perl_destruct_level = 0;
}
PL_exit_flags |= PERL_EXIT_DESTRUCT_END;
if (!perl_parse(my_perl, xs_init, argc, argv, (char **)NULL))
perl_run(my_perl);
#ifndef PERL_MICRO
/* Unregister our signal handler before destroying my_perl */
for (i = 1; PL_sig_name[i]; i++) {
if (rsignal_state(PL_sig_num[i]) == (Sighandler_t) PL_csighandlerp) {
rsignal(PL_sig_num[i], (Sighandler_t) SIG_DFL);
}
}
#endif
exitstatus = perl_destruct(my_perl);
perl_free(my_perl);
#if defined(USE_ENVIRON_ARRAY) && defined(PERL_TRACK_MEMPOOL) &&
!defined(NO_ENV_ARRAY_IN_MAIN)
/*
* The old environment may have been freed by perl_free()
* when PERL_TRACK_MEMPOOL is defined, but without having
* been restored by perl_destruct() before (this is only
* done if destruct_level > 0).
*
* It is important to have a valid environment for atexit()
* routines that are eventually called.
*/
environ = env;
#endif
PERL_SYS_TERM();
#ifdef PERL_GLOBAL_STRUCT
# ifdef PERL_GLOBAL_STRUCT_PRIVATE
veto = my_plvarsp->Gveto_cleanup;
# endif
free_global_struct(my_vars);
# ifdef PERL_GLOBAL_STRUCT_PRIVATE
if (!veto)
my_plvarsp = NULL;
/* Remember, functions registered with atexit() can run after this
point,
and may access "global" variables, and hence end up calling
Perl_GetVarsPrivate() */
#endif
#endif /* PERL_GLOBAL_STRUCT */
exit(exitstatus);<<<<<<<CRASH
}
------------------------------------
Aseembly of crash site
------------------------------------
perl_free(my_perl);
004010F6 8B 0D B0 42 97 00 mov ecx,dword ptr [__fmode+4 (9742B0h)]
004010FC 8B F0 mov esi,eax
004010FE E8 95 11 00 00 call perl_free (402298h)
#if defined(USE_ENVIRON_ARRAY) && defined(PERL_TRACK_MEMPOOL) &&
!defined(NO_ENV_ARRAY_IN_MAIN)
/*
* The old environment may have been freed by perl_free()
* when PERL_TRACK_MEMPOOL is defined, but without having
* been restored by perl_destruct() before (this is only
* done if destruct_level > 0).
*
* It is important to have a valid environment for atexit()
* routines that are eventually called.
*/
environ = env;
#endif
PERL_SYS_TERM();
00401103 E8 EB 00 00 00 call Perl_sys_term (4011F3h)
#ifdef PERL_GLOBAL_STRUCT
# ifdef PERL_GLOBAL_STRUCT_PRIVATE
veto = my_plvarsp->Gveto_cleanup;
# endif
free_global_struct(my_vars);
# ifdef PERL_GLOBAL_STRUCT_PRIVATE
if (!veto)
my_plvarsp = NULL;
/* Remember, functions registered with atexit() can run after this
point,
and may access "global" variables, and hence end up calling
Perl_GetVarsPrivate() */
#endif
#endif /* PERL_GLOBAL_STRUCT */
exit(exitstatus);
00401108 E8 B6 F2 03 00 call Perl_get_context (4403C3h)
0040110D 8B B8 40 09 00 00 mov edi,dword ptr
[eax+940h]<<<<<<<<<<<<<<<<<<<<<<<<CRASH
00401113 56 push esi
00401114 E8 AA F2 03 00 call Perl_get_context (4403C3h)
00401119 FF B0 40 09 00 00 push dword ptr [eax+940h]
0040111F FF 57 08 call dword ptr [edi+8]
00401122 59 pop ecx
00401123 59 pop ecx
00401124 5F pop edi
00401125 5E pop esi
}
00401126 33 C0 xor eax,eax
00401128 5B pop ebx
00401129 C9 leave
0040112A C3 ret
/* Register any extra external extensions */
EXTERN_C void boot_DynaLoader (pTHX_ CV* cv);
static void
xs_init(pTHX)
-------------------------------------
Well I know what happened, exit() was hooked by threaded win32 perl.
-----------------------------
# define exit PerlProc_exit
-----------------------------
from XSUB.h
PerlProc_exit requires a my_perl. We just called perl_free(my_perl) and
Perl_sys_term(). my_perl struct has been freed. Perl_get_context() will
return uninit mem or NULL. I think its actually a bug that
Perl_get_context() returned 0x00E26014 instead of NULL after a perl_free
but its programmer error ANYWAYS to use my_perl interp instances from
wrong threads or after being freed. So probably no point of making
perl_free() do a PERL_SET_THX(NULL) to match perl_alloc doing a
PERL_SET_THX(my_perl) binding a interp to an OS thread.
The reason why dynamic full perl.exe and large static lib perl.exe work
is, they never call exit(). They use supplied win32/runperl.c not
ExtUtils::Miniperl. And runperl.c simply returns an exit code from perl
controled main() back to libc. It doesn't call exit().
So back to question, why didn't ExtUtils::Miniperl::writemain put in all
the 55 core XS libs into the new xs_init?
ExtUtils::Miniperl::writemain takes the .lib list on command line from
the Makefile.aperl file's MAP_STATICDEP target which is generated by
MM_Unix.pm:_find_static_libs(), which scans the file sys for .lib files
in /auto, but Win32 static perl uses the big perl527s.lib file and
doesn't copy from the source tree the smaller per-module .lib files. If
I copy .lib files manually from the source tree to
C:/installedstaticperl/lib/auto and do a big rewrite of
xs_static_lib_is_xs(), the modules appear inside perlmain.c correctly.
So why does perl527s.lib exist? IDK. If I was writing this code from
scratch, I'd say $Config{static_ext} must be read out by
_find_static_libs() and added every time, inaddition to the .lib in
/auto search done now, since thats gonna be the contents of whats baked
into perl527s.lib.
Vadim asked something similar in
https://www.nntp.perl.org/group/perl.perl5.porters/2018/03/msg249982.html
https://rt.perl.org/Ticket/Display.html?id=133070#txn-1544876
but I dont think he ever tried a "makeaperl" target.
Thread Next
-
did static perl/makeaperl on Win32 ever work in history?
by bulk88