Front page | perl.perl5.porters |
Postings from May 2015
[perl #125296] CvFILE on threaded, and gv_fetchfile on all perls, burn mad memory
Thread Previous
From:
bulk88
Date:
May 30, 2015 23:29
Subject:
[perl #125296] CvFILE on threaded, and gv_fetchfile on all perls, burn mad memory
Message ID:
rt-4.0.18-931-1433028550-1747.125296-75-0@perl.org
# New Ticket Created by bulk88
# Please include the string: [perl #125296]
# in the subject line of all future correspondence about this issue.
# <URL: https://rt.perl.org/Ticket/Display.html?id=125296 >
This is a bug report for perl from bulk88@hotmail.com,
generated with the help of perlbug 1.40 running under perl 5.22.0.
-----------------------------------------------------------------
[Please describe your issue here]
commit a636914a235cd8650ed6efcb07e302c0332fa638 "Re: CvFILE corruption
under ithreads" from 5.7.1 from 2001 drastically increased memory usage
of threaded perl.
Using "perl -e"system pause; require Test::More; system 'pause'"" as a
benchmark, and procdump, strings @8 chars, diff, then manually removing
sensitive information (user32/gdi32 shared memory with passwords from
long gone GUI apps) produced the 2 logs (attached later in this ticket).
You can see that each PP sub creates a unique copy of the absolute file
path.
-----------------------------------------------------------------
05/28/2015 05:21 PM 11,819,415 perl.exe_nothd_af.dmp
05/28/2015 05:21 PM 10,832,263 perl.exe_nothd_b4.dmp
05/28/2015 05:35 PM 12,069,271 perl.exe_thd_af.dmp
05/28/2015 05:34 PM 10,910,087 perl.exe_thd_b4.dmp
-----------------------------------------------------------------
unthreaded took 11,819,415-10,832,263=987,152 bytes of memory.
threaded took 12,069,271-10,910,087=1,159,184 bytes of memory
1,159,184-987,152=172,032 bytes of memory.
So threaded takes 172KB or 17% more memory to load the same Perl module.
Some more processing of the diff shows 1503 instances of new "+C:/"
paths, it comes to 63988 bytes of extra file paths, (attached later).
This file excludes alignment roundup+libc malloc headers+perl malloc
headers. 63988-(1503*1)<CRLF to NULL byte>+(1503*8)<Win32 32 bit malloc
header>+(1503*12)<Perl's vmem.h on 32 bits>+(1503*4)<average alighnment
roundup to 8 on win32-32>. Adding all of that comes to 98557 bytes of
file paths as a rough estimate.
Reverting ("#define CvFILE_set_from_cop(sv, cop) (CvFILE(sv) =
CopFILE(cop), CvDYNFILE_off(sv))") commit
a636914a235cd8650ed6efcb07e302c0332fa638 "Re: CvFILE corruption under
ithreads" causes causes mem corruption (use after free) and some test
fails (I am using PERL_POISON+Win32 Debugging Heap (gflags) on the whole
smoke). Here is the test fails. More tests fail individually when I run
them with Dr Memory.
-----------------------------------------------------------------
Test Summary Report
-------------------
op/stash.t (Wstat:
0 Tests:
51 Failed: 1)
Failed test: 35
uni/stash.t (Wstat:
0 Tests:
49 Failed: 1)
Failed test: 36
../ext/Devel-Peek/t/Peek.t (Wstat:
768 Test
s: 114 Failed: 3)
Failed tests: 25, 27, 51
Non-zero exit status: 3
Files=2385, Tests=697642, 5307 wallclock secs (257.47 usr + 7.23 sys =
264.70 C
PU)
Result: FAIL
b88dmake: Error code 181, while making 'test'
-----------------------------------------------------------------
Dr Memory on op/stash.t shows failures like
-----------------------------------------------------------------
Error #1: UNADDRESSABLE ACCESS: reading 0x1161539c-0x1161539d 1 byte(s)
# 0 replace_strlen
[d:\drmemory_package\drmemory\replace.c:375]
# 1 perl522.dll!Perl_newSVpv [c:\perl521\srcnewb4opt\sv.c:9215]
# 2 B.dll!XS_B__IV_IVX [c:\perl521\srcnewb4opt\ext\b\b.xs:1676]
# 3 perl522.dll!Perl_pp_entersub [c:\perl521\srcnewb4opt\pp_hot.c:3268]
# 4 perl522.dll!Perl_runops_debug [c:\perl521\srcnewb4opt\dump.c:2234]
# 5 perl522.dll!S_run_body
[c:\perl521\srcnewb4opt\win32\perl.c:2451]
# 6 perl522.dll!perl_run
[c:\perl521\srcnewb4opt\win32\perl.c:2374]
# 7 perl522.dll!RunPerl
[c:\perl521\srcnewb4opt\win32\perllib.c:258]
# 8 main
[c:\perl521\srcnewb4opt\win32\perlmain.c:39]
-----------------------------------------------------------------
CvFILE is either a pointer to malloc freed sentinel pattern, or some
other perl src code file in the .t process and therefore the tests fail.
CvFILE always comes from CopFILE(), but CopFILE() is very different on
threads and unthreaded because of the "no SV*s in OPs due to shared
OP/OP COW" limitation.
threaded
-----------------------------------------------------------------
# define CopFILE(c) ((c)->cop_file)
-----------------------------------------------------------------
unthreaded
-----------------------------------------------------------------
# define CopFILE(c) (CopFILEGV(c) \
? GvNAME(CopFILEGV(c))+2 : NULL)
-----------------------------------------------------------------
a section of the COP struct
-----------------------------------------------------------------
#ifdef USE_ITHREADS
PADOFFSET cop_stashoff; /* offset into PL_stashpad, for the
package the line was compiled in */
char * cop_file; /* file name the following line # is from */
#else
HV * cop_stash; /* package line was compiled in */
GV * cop_filegv; /* file the following line # is from */
#endif
-----------------------------------------------------------------
The current design of CvFILE on threads has to change. I am currrently
experimenting with only selectivly using "savepv(CopFILE(cop)),
CvDYNFILE_on(sv))". It seems to me (and Im not a optree expert) 99% of
the time, the filename is available in the 1st op of the sub, which is a
pp_nextstate. Predeclared subs have CV*s, but CvSTART and CvROOT are
null, so in Perl_newSTUB a savepv() is required. The PL_curcop filename
is freed on savestack with
-----------------------------------------------------------------
Note: # 0 replace_free
[d:\drmemory_package\common\alloc_replace.c:2380]
Note: # 1 perl522.dll!VMem::Free
[c:\perl521\srcnewb4opt\win32\vmem.h:213]
Note: # 2 perl522.dll!CPerlHost::FreeShared
[c:\perl521\srcnewb4opt\win32\perlhost.h:101]
Note: # 3 perl522.dll!PerlMemSharedFree
[c:\perl521\srcnewb4opt\win32\perlhost.h:360]
Note: # 4 perl522.dll!Perl_leave_scope
[c:\perl521\srcnewb4opt\scope.c:852]
Note: # 5 perl522.dll!Perl_pop_scope
[c:\perl521\srcnewb4opt\scope.c:104]
Note: # 6 perl522.dll!Perl_pp_leaveeval
[c:\perl521\srcnewb4opt\pp_ctl.c:4368]
Note: # 7 perl522.dll!Perl_runops_debug
[c:\perl521\srcnewb4opt\dump.c:2234]
Note: # 8 perl522.dll!S_run_body
[c:\perl521\srcnewb4opt\win32\perl.c:2451]
Note: # 9 perl522.dll!perl_run
[c:\perl521\srcnewb4opt\win32\perl.c:2374]
Note: #10 perl522.dll!RunPerl
[c:\perl521\srcnewb4opt\win32\perllib.c:258]
Note: #11 main
[c:\perl521\srcnewb4opt\win32\perlmain.c:39]
-----------------------------------------------------------------
callstack.
Another idea proposed in
http://www.nntp.perl.org/group/perl.perl5.porters/2001/05/msg36781.html
was to eliminate CvFILE entirely since its value is not used by PP code.
There will be breakage on CPAN to remove CvFILE http://grep. cpan.
me/?q=CvFILE but all the CvFILE uses are either to print debugging info
to a human, or setting CvFILE to maintain perlapi compatibility since
core does that. This would save a ptr slot in the CV body plus the
malloc. Here are the usages that look most, "interesting" to me.
https://metacpan.org/source/MLEHMANN/Devel-FindRef-1.44/FindRef.xs#L204
https://metacpan.org/source/NWCLARK/Devel-Arena-0.23/Arena.xs#L453
https://metacpan.org/source/RGARCIA/Sub-Identify-0.10/Identify.xs#L49
Another idea is, change CvFILE slot in the CV body to be HEK * owned or
GV * (gv_fetchfile's GV*) un/owned by the CV * instead of the current
unowned static char * for XS, or the malloced char * for PP.
That also brings up, why does gv_fetchfile exist to create typeglobs
like this hand trimmed of non "_<" entries list? What use do these GVs have?
-----------------------------------------------------------------
C:\Documents and Settings\Owner>perl -e"$,=\"\n\", print keys%main::"
_<perl.c
_<.\win32.c
_<..\universal.c
_<-e
_<perllib.c
_<..\perlio.c
_<..\mro_core.c
_<Win32CORE.c
C:\Documents and Settings\Owner>
-----------------------------------------------------------------
Or
-----------------------------------------------------------------
C:\Documents and Settings\Owner>perl -e"require Test::Harness;
$,=\"\n\", print
keys%main::"
_<C:/perl521/lib/strict.pm
_<C:/perl521/lib/auto/Time/HiRes/HiRes.dll
_<C:/perl521/lib/auto/Fcntl/Fcntl.dll
_<..\universal.c
_<C:/perl521/lib/XSLoader.pm
_<C:/perl521/lib/auto/IO/IO.dll
_<Cwd.c
_<-e
_<C:/perl521/lib/base.pm
_<C:/perl521/lib/auto/POSIX/POSIX.dll
_<Win32CORE.c
_<POSIX.xs
_<perllib.c
_<C:/perl521/lib/warnings.pm
_<C:/perl521/lib/Carp.pm
_<C:/perl521/lib/auto/Cwd/Cwd.dll
_<C:/perl521/site/lib/TAP/Parser/IteratorFactory.pm
_<..\mro_core.c
_<.\win32.c
_<DynaLoader.c
_<HiRes.c
_<perl.c
_<POSIX.c
_<Fcntl.c
_<IO.c
_<..\perlio.c
C:\Documents and Settings\Owner>
-----------------------------------------------------------------
Even worse, these GVs have a SV * inside, with YET another unique copy
of the filename (see later attached pic, gp_file_hek is the 1st copy,
the SV * in the GV* is another copy, different pointers).
-----------------------------------------------------------------
C:\Documents and Settings\Owner>perl -e"require Test::Harness; require
Devel::Pe
ek; Devel::Peek::Dump(*{'_<C:/perl521/lib/strict.pm'})"
SV = PVGV(0x90032c) at 0x9119f4
REFCNT = 1
FLAGS = (MULTI)
NAME = "_<C:/perl521/lib/strict.pm"
NAMELEN = 26
GvSTASH = 0x3680f4 "main"
FLAGS = 0x2
GP = 0x914f7c
SV = 0x911a04
REFCNT = 1
IO = 0x0
FORM = 0x0
AV = 0x0
HV = 0x0
CV = 0x0
CVGEN = 0x0
GPFLAGS = 0x0 ()
LINE = 14
FILE = "C:/perl521/lib/strict.pm"
EGV = 0x9119f4 "_<C:/perl521/lib/strict.pm"
C:\Documents and Settings\Owner>perl -e"require Test::Harness; require
Devel::Pe
ek; Devel::Peek::Dump(*{'_<C:/perl521/lib/strict.pm'}{SCALAR})"
SV = IV(0x3681f8) at 0x3681fc
REFCNT = 1
FLAGS = (TEMP,ROK)
RV = 0x911d3c
SV = PV(0x36906c) at 0x911d3c
REFCNT = 2
FLAGS = (POK,pPOK)
PV = 0x91527c "C:/perl521/lib/strict.pm"\0
CUR = 24
LEN = 26
C:\Documents and Settings\Owner>
-----------------------------------------------------------------
While threaded perl wastes memory much worse than unthreaded. Unthreaded
still has 3 copies of the file path for every .pm file in memory (see
later attached). I guess I'll have to newSVpvn_share that, or something
else, since I do have a HEK* already from the GV, which is an easy fix.
Now note, I have not said there are any memory leaks, it is just that
threaded Perl burns memory like an oil tycoon setting pallets of money
on fire. All this code (file path storage) could only have been designed
by a monkey house and has to change, to something else. Exactly what, is
why I filed this ticket.
[Please do not change anything below this line]
-----------------------------------------------------------------
---
Flags:
category=core
severity=high
---
Site configuration information for perl 5.22.0:
Configured by Owner at Thu May 28 17:26:18 2015.
Summary of my perl5 (revision 5 version 22 subversion 0) configuration:
Commit id: db8192ecf78d7f18a7fdbcc047f3dad0577bb287
Platform:
osname=MSWin32, osvers=5.1, archname=MSWin32-x86-multi-thread
uname=''
config_args='undef'
hint=recommended, useposix=true, d_sigaction=undef
useithreads=define, usemultiplicity=define
use64bitint=undef, use64bitall=undef, uselongdouble=undef
usemymalloc=n, bincompat5005=undef
Compiler:
cc='cl', ccflags ='-nologo -GF -W3 -O1 -MD -Zi -DNDEBUG -GL -DWIN32
-D_CONSOLE -DNO_STRICT -DPERL_TEXTMODE_SCRIPTS -DPERL_IMPLICIT_CONTEXT
-DPERL_IMPLICIT_SYS -D_USE_32BIT_TIME_T',
optimize='-O1 -MD -Zi -DNDEBUG -GL',
cppflags='-DWIN32'
ccversion='13.10.6030', gccversion='', gccosandvers=''
intsize=4, longsize=4, ptrsize=4, doublesize=8, byteorder=1234,
doublekind=3
d_longlong=undef, longlongsize=8, d_longdbl=define, longdblsize=8,
longdblkind=0
ivtype='long', ivsize=4, nvtype='double', nvsize=8, Off_t='__int64',
lseeksize=8
alignbytes=8, prototype=define
Linker and Libraries:
ld='link', ldflags ='-nologo -nodefaultlib -debug -opt:ref,icf -ltcg
-libpath:"c:\perl\lib\CORE" -machine:x86'
libpth=\lib
libs=oldnames.lib kernel32.lib user32.lib gdi32.lib winspool.lib
comdlg32.lib advapi32.lib shell32.lib ole32.lib oleaut32.lib
netapi32.lib uuid.lib ws2_32.lib mpr.lib winmm.lib version.lib
odbc32.lib odbccp32.lib comctl32.lib msvcrt.lib
perllibs=oldnames.lib kernel32.lib user32.lib gdi32.lib winspool.lib
comdlg32.lib advapi32.lib shell32.lib ole32.lib oleaut32.lib
netapi32.lib uuid.lib ws2_32.lib mpr.lib winmm.lib version.lib
odbc32.lib odbccp32.lib comctl32.lib msvcrt.lib
libc=msvcrt.lib, so=dll, useshrplib=true, libperl=perl522.lib
gnulibc_version=''
Dynamic Linking:
dlsrc=dl_win32.xs, dlext=dll, d_dlsymun=undef, ccdlflags=' '
cccdlflags=' ', lddlflags='-dll -nologo -nodefaultlib -debug
-opt:ref,icf -ltcg -libpath:"c:\perl\lib\CORE" -machine:x86'
Locally applied patches:
RC2
---
@INC for perl 5.22.0:
C:/perl521/srcnewb4opt/lib
.
---
Environment for perl 5.22.0:
HOME (unset)
LANG (unset)
LANGUAGE (unset)
LD_LIBRARY_PATH (unset)
LOGDIR (unset)
PATH=C:\sperl\c\bin;C:\perl521\bin;C:\WINDOWS\system32;C:\Program
Files\Microsoft Visual Studio .NET 2003\Vc7\bin;C:\Program
Files\Microsoft Visual Studio .NET
2003\Common7\IDE;C:\WINDOWS;C:\Program Files\Git\cmd;C:\Program
Files\Microsoft Visual Studio .NET 2003\Common7\Tools\bin;C:\perl\bin;
PERL_BADLANG (unset)
PERL_JSON_BACKEND=Cpanel::JSON::XS
PERL_YAML_BACKEND=YAML::XS
SHELL (unset)
Thread Previous