develooper Front page | perl.perl5.porters | Postings from September 2012

[perl #114704] win32: rebase and bind the core and all XS module dlls during build process

Thread Next
From:
bulk 88
Date:
September 1, 2012 19:59
Subject:
[perl #114704] win32: rebase and bind the core and all XS module dlls during build process
Message ID:
rt-3.6.HEAD-11172-1346554762-1459.114704-75-0@perl.org
# New Ticket Created by  bulk 88 
# Please include the string:  [perl #114704]
# in the subject line of all future correspondence about this issue. 
# <URL: https://rt.perl.org:443/rt3/Ticket/Display.html?id=114704 >





This is a bug report for perl from bulk88@hotmail.com,
generated with the help of perlbug 1.39 running under perl 5.17.4.


-----------------------------------------------------------------
[Please describe your issue here]

TLDR, Win32 Perl needs to rebase and bind all its DLLs and EXE files during the build process and save alot of memory on Win32 and speed up process start up time very easily.

In http://perl5.git.perl.org/perl.git/commitdiff/729a02f2394e6a50fe8352c9471391b53e54ac40 a change was done to the VC makefile in win32/Makefile. This change was to avoid DLL relocation on windows to speed up Perl starting up. While the perl core dll (perl5**.dll) does not get relocated, all XS modules (which are DLLs in Win32 Perl) sit at the default 0x10000000. Perl 5.17.4 comes with the following DLLs on Win32 out of the box. The list was hand pruned to remove /site's DLLs.
_______________________________________________________
C:\perl517>dir /s /b *.dll
C:\perl517\bin\perl517.dll
C:\perl517\lib\auto\arybase\arybase.dll
C:\perl517\lib\auto\attributes\attributes.dll
C:\perl517\lib\auto\B\B.dll
C:\perl517\lib\auto\Compress\Raw\Bzip2\Bzip2.dll
C:\perl517\lib\auto\Compress\Raw\Zlib\Zlib.dll
C:\perl517\lib\auto\Cwd\Cwd.dll
C:\perl517\lib\auto\Data\Dumper\Dumper.dll
C:\perl517\lib\auto\Devel\Peek\Peek.dll
C:\perl517\lib\auto\Devel\PPPort\PPPort.dll
C:\perl517\lib\auto\Digest\MD5\MD5.dll
C:\perl517\lib\auto\Digest\SHA\SHA.dll
C:\perl517\lib\auto\Encode\Encode.dll
C:\perl517\lib\auto\Encode\Byte\Byte.dll
C:\perl517\lib\auto\Encode\CN\CN.dll
C:\perl517\lib\auto\Encode\EBCDIC\EBCDIC.dll
C:\perl517\lib\auto\Encode\JP\JP.dll
C:\perl517\lib\auto\Encode\KR\KR.dll
C:\perl517\lib\auto\Encode\Symbol\Symbol.dll
C:\perl517\lib\auto\Encode\TW\TW.dll
C:\perl517\lib\auto\Encode\Unicode\Unicode.dll
C:\perl517\lib\auto\Fcntl\Fcntl.dll
C:\perl517\lib\auto\File\Glob\Glob.dll
C:\perl517\lib\auto\Filter\Util\Call\Call.dll
C:\perl517\lib\auto\Hash\Util\Util.dll
C:\perl517\lib\auto\Hash\Util\FieldHash\FieldHash.dll
C:\perl517\lib\auto\IO\IO.dll
C:\perl517\lib\auto\List\Util\Util.dll
C:\perl517\lib\auto\Math\BigInt\FastCalc\FastCalc.dll
C:\perl517\lib\auto\MIME\Base64\Base64.dll
C:\perl517\lib\auto\mro\mro.dll
C:\perl517\lib\auto\Opcode\Opcode.dll
C:\perl517\lib\auto\PerlIO\encoding\encoding.dll
C:\perl517\lib\auto\PerlIO\mmap\mmap.dll
C:\perl517\lib\auto\PerlIO\scalar\scalar.dll
C:\perl517\lib\auto\PerlIO\via\via.dll
C:\perl517\lib\auto\POSIX\POSIX.dll
C:\perl517\lib\auto\re\re.dll
C:\perl517\lib\auto\SDBM_File\SDBM_File.dll
C:\perl517\lib\auto\Socket\Socket.dll
C:\perl517\lib\auto\Storable\Storable.dll
C:\perl517\lib\auto\Sys\Hostname\Hostname.dll
C:\perl517\lib\auto\Text\Soundex\Soundex.dll
C:\perl517\lib\auto\threads\threads.dll
C:\perl517\lib\auto\threads\shared\shared.dll
C:\perl517\lib\auto\Tie\Hash\NamedCapture\NamedCapture.dll
C:\perl517\lib\auto\Time\HiRes\HiRes.dll
C:\perl517\lib\auto\Time\Piece\Piece.dll
C:\perl517\lib\auto\Unicode\Collate\Collate.dll
C:\perl517\lib\auto\Unicode\Normalize\Normalize.dll
C:\perl517\lib\auto\Win32\Win32.dll
C:\perl517\lib\auto\Win32API\File\File.dll
________________________________________________
So after the 1st XS module is loaded, all further DLL modules must be relocated (google the term), since they all share the same address. I belive the read only PE sections can not be COWed/shared between different win32 processes then (atleast on Win XP, NT 6 might be different because of ASLR changes). Of course read write PE sections can never be shared between perl processes. "Rebase" ing (rebasing) the perl.exe might also be useful so in the million to one chance a XS module links with a DLL that links with a EXE (an EXE that exports functions), that 2nd EXE image won't have to be relocated. I think by default with Visual C, EXEs do not contain relocation data and are therefore unrelocatable, but 3rd party creating an exe that exports and is inteded to be linked with in a different process will include the reloc info. Rebasing is a very large way to save process specific memory on Windows. The savings are all the read only data in the PE file (machine code/C const data) 
 that was relocated, per PE file, per Perl process.

A 2nd way smaller way of decreasing the private memory usage of Perl images is to "bind" the EXEs and DLLs. That mean that the import table is prefilled with the function pointers of the functions to import, along with a CRC and timestamp of the exporting DLL, if the CRC and timestamp of the exporting DLL in the importer DLL match the exporting DLL on disk, no fixups are required. This reduces the private memory by 4KB (1 page) per EXE/DLL. This is highly Windows OS release and MS security fix specific. One strange thing about Visual C linker is, it merges the ".idata" section, which itself is marked as read/write in .lib files, into the ".rdata" section of the PE file. The ".idata" section contains the function pointer import table. This means, that unless a DLL is binded, 1 page of C const "read only" data is always unsharable since the DLL loaded will mark the page read write to do fixup the function pointers, then mark it back to read only, but now its unsharable with oth
 er Perl processes. This can be confirmed with VMMap on WinXP. Using the -merge option to VC linker to force the .idata section into .data (read write initialized data) where is rightfully belongs results in linker error LNK1272.

An example of the bind command, note it is missing the winsxs specific directory of comctl32.dll V6 so perl5**.dll is not fully binadable I guess.

bind� -p "C:\WINDOWS\system32;C:\perl517\bin"� -u "C:\perl517\bin\perl.exe" "C:\perl517\bin\perl517.dll" "C:\perl517\lib\auto\arybase\arybase.dll" ......

An example of the rebase command, 

rebase -b 0x70000000 -d� -R "." -G "C:/Documents and Settings/Owner/Desktop/p517dlls_to_rebase.txt"

p517dlls_to_rebase.txt is newline separated dll names, like the list above.

Both bind and rebase are MS command line tools included with Visual C/Platform SDK. I dont know if any of this ticket applies to Mingw Perl or not. I've heard rumors rebase has been discontinued for Windows 8's SDK, probably due to ASLR caching/interprocess sharing of relocated DLLs.

A rebasing should be done atleast build time. Perhaps a bat file or a perl script can be written and installed to /bin so if a user has rebase and bind they can rebase all the DLLs in /lib and /site/lib to be neatly in order. The bind is most important since it is end user OS patch level specific. Of course the user would have to have bind and rebase in their PATH already and have a copy of paid Visual Studio or free Platform SDK. I'm not sure if there are any FOSS equivelents of rebase and bind, and whether they would be Perl license compatible. A fancier idea more complicated more unrealistic idea idea is to have ExtUtils::MakeMaker or Config.pm or IDK what keep a record on disk of the last used base address, and lower it or rebase the XS DLL to be just below the last known lowest base address on the last EUMM compiled module, each time a XS DLL is built, this way /site/lib XS modules are always rebased to not ever conflict and a manual tool or perl script does not have to 
 be run on demand by the user to keep their /site/lib rebased and binded.

Looking at ActivePerl 5.12.3 32 bit, none of the /lib XS DLLs are rebased, all of them are 0x10000000. I'm surprised nobody including AS ever thought of this before.

These 2 ideas presented here can reduce accusations that Perl is a memory hog/slow startup/etc on Windows. I'm not sure how much impact on "Mem Usage" column in Task Manager these ideas will have, but the results will be measurable in Process Explorer and VMMap.

I dont have the time to do anything about this idea right now or in the near future, so I'm creating this as a todo ticket for me in the future, or some other Win32 Perl dev to look at one day.



[Please do not change anything below this line]
-----------------------------------------------------------------
---
Flags:
��� category=core
��� severity=wishlist
---
Site configuration information for perl 5.17.4:

Configured by Owner at Fri Aug 24 11:02:59 2012.

Summary of my perl5 (revision 5 version 17 subversion 4 patch blead 2012-08-19.11:53:47 0374b0a2a50c6e91951723a4d9ee1d7f534b03eb v5.17.2-394-g0374b0a) configuration:
� Snapshot of: 0374b0a2a50c6e91951723a4d9ee1d7f534b03eb
� Platform:
��� osname=MSWin32, osvers=5.1, archname=MSWin32-x86-multi-thread
��� uname=''
��� config_args='undef'
��� hint=recommended, useposix=true, d_sigaction=undef
��� useithreads=define, usemultiplicity=define
��� useperlio=define, d_sfio=undef, uselargefiles=define, usesocks=undef
��� use64bitint=undef, use64bitall=undef, uselongdouble=undef
��� usemymalloc=n, bincompat5005=undef
� Compiler:
��� cc='cl', ccflags ='-nologo -GF -W3 -Od -MD -Zi -DDEBUGGING -DWIN32 -D_CONSOLE -DNO_STRICT -DPERL_POISON� -DPERL_TEXTMODE_SCRIPTS -DPERL_IMPLICIT_CONTEXT -DPERL_IMPLICIT_SYS -DUSE_PERLIO -D_USE_32BIT_TIME_T',
��� optimize='-Od -MD -Zi -DDEBUGGING',
��� cppflags='-DWIN32'
��� ccversion='13.10.6030', gccversion='', gccosandvers=''
��� intsize=4, longsize=4, ptrsize=4, doublesize=8, byteorder=1234
��� d_longlong=undef, longlongsize=8, d_longdbl=define, longdblsize=8
��� ivtype='long', ivsize=4, nvtype='double', nvsize=8, Off_t='__int64', lseeksize=8
��� alignbytes=8, prototype=define
� Linker and Libraries:
��� ld='link', ldflags ='-nologo -nodefaultlib -debug� -libpath:"c:\perl517\lib\CORE"� -machine:x86'
��� libpth="C:\Program Files\Microsoft Visual Studio .NET 2003\VC7\lib"
��� libs=oldnames.lib kernel32.lib user32.lib gdi32.lib winspool.lib� comdlg32.lib advapi32.lib shell32.lib ole32.lib oleaut32.lib� netapi32.lib uuid.lib ws2_32.lib mpr.lib winmm.lib� version.lib odbc32.lib odbccp32.lib comctl32.lib msvcrt.lib
��� perllibs=oldnames.lib kernel32.lib user32.lib gdi32.lib winspool.lib� comdlg32.lib advapi32.lib shell32.lib ole32.lib oleaut32.lib� netapi32.lib uuid.lib ws2_32.lib mpr.lib winmm.lib� version.lib odbc32.lib odbccp32.lib comctl32.lib msvcrt.lib
��� libc=msvcrt.lib, so=dll, useshrplib=true, libperl=perl517.lib
��� gnulibc_version=''
� Dynamic Linking:
��� dlsrc=dl_win32.xs, dlext=dll, d_dlsymun=undef, ccdlflags=' '
��� cccdlflags=' ', lddlflags='-dll -nologo -nodefaultlib -debug� -libpath:"c:\perl517\lib\CORE"� -machine:x86'

Locally applied patches:
��� 

---
@INC for perl 5.17.4:
��� C:/perl517/site/lib
��� C:/perl517/lib
��� .

---
Environment for perl 5.17.4:
��� HOME (unset)
��� LANG (unset)
��� LANGUAGE (unset)
��� LD_LIBRARY_PATH (unset)
��� LOGDIR (unset)
��� PATH=C:\perl517\bin;C:\Program Files\Microsoft Visual Studio .NET 2003\Common7\IDE;C:\Program Files\Microsoft Visual Studio .NET 2003\VC7\BIN;C:\Program Files\Microsoft Visual Studio .NET 2003\Common7\Tools;C:\Program Files\Microsoft Visual Studio .NET 2003\Common7\Tools\bin\prerelease;C:\WINDOWS\system32;C:\WINDOWS;C:\WINDOWS\system32\wbem;
��� PERL_BADLANG (unset)
��� SHELL (unset)

 		 	   		  


Thread Next


nntp.perl.org: Perl Programming lists via nntp and http.
Comments to Ask Bjørn Hansen at ask@perl.org | Group listing | About