develooper Front page | perl.perl5.porters | Postings from May 2015

[perl #125296] CvFILE on threaded, and gv_fetchfile on all perls, burn mad memory

Thread Previous
From:
bulk88
Date:
May 30, 2015 23:29
Subject:
[perl #125296] CvFILE on threaded, and gv_fetchfile on all perls, burn mad memory
Message ID:
rt-4.0.18-931-1433028550-1747.125296-75-0@perl.org
# New Ticket Created by  bulk88 
# Please include the string:  [perl #125296]
# in the subject line of all future correspondence about this issue. 
# <URL: https://rt.perl.org/Ticket/Display.html?id=125296 >


This is a bug report for perl from bulk88@hotmail.com,
generated with the help of perlbug 1.40 running under perl 5.22.0.


-----------------------------------------------------------------
[Please describe your issue here]

commit a636914a235cd8650ed6efcb07e302c0332fa638 "Re: CvFILE corruption 
under ithreads" from 5.7.1 from 2001 drastically increased memory usage 
of threaded perl.

Using "perl -e"system pause; require Test::More; system 'pause'"" as a 
benchmark, and procdump, strings @8 chars, diff, then manually removing 
sensitive information (user32/gdi32 shared memory with passwords from 
long gone GUI apps) produced the 2 logs (attached later in this ticket). 
You can see that each PP sub creates a unique copy of the absolute file 
path.
-----------------------------------------------------------------
05/28/2015  05:21 PM        11,819,415 perl.exe_nothd_af.dmp
05/28/2015  05:21 PM        10,832,263 perl.exe_nothd_b4.dmp
05/28/2015  05:35 PM        12,069,271 perl.exe_thd_af.dmp
05/28/2015  05:34 PM        10,910,087 perl.exe_thd_b4.dmp
-----------------------------------------------------------------

unthreaded took 11,819,415-10,832,263=987,152 bytes of memory.
threaded took 12,069,271-10,910,087=1,159,184 bytes of memory

1,159,184-987,152=172,032 bytes of memory.

So threaded takes 172KB or 17% more memory to load the same Perl module. 
Some more processing of the diff shows 1503 instances of new "+C:/" 
paths, it comes to 63988 bytes of extra file paths, (attached later). 
This file excludes alignment roundup+libc malloc headers+perl malloc 
headers. 63988-(1503*1)<CRLF to NULL byte>+(1503*8)<Win32 32 bit malloc 
header>+(1503*12)<Perl's vmem.h on 32 bits>+(1503*4)<average alighnment 
roundup to 8 on win32-32>. Adding all of that comes to 98557 bytes of 
file paths as a rough estimate.

Reverting ("#define CvFILE_set_from_cop(sv, cop) (CvFILE(sv) = 
CopFILE(cop), CvDYNFILE_off(sv))") commit 
a636914a235cd8650ed6efcb07e302c0332fa638 "Re: CvFILE corruption under 
ithreads" causes causes mem corruption (use after free) and some test 
fails (I am using PERL_POISON+Win32 Debugging Heap (gflags) on the whole 
smoke). Here is the test fails. More tests fail individually when I run 
them with Dr Memory.

-----------------------------------------------------------------
Test Summary Report
-------------------
op/stash.t                                                      (Wstat: 
0 Tests:
 51 Failed: 1)
  Failed test:  35
uni/stash.t                                                     (Wstat: 
0 Tests:
 49 Failed: 1)
  Failed test:  36
../ext/Devel-Peek/t/Peek.t                                      (Wstat: 
768 Test
s: 114 Failed: 3)
  Failed tests:  25, 27, 51
  Non-zero exit status: 3
Files=2385, Tests=697642, 5307 wallclock secs (257.47 usr +  7.23 sys = 
264.70 C
PU)
Result: FAIL
b88dmake:  Error code 181, while making 'test'
-----------------------------------------------------------------

Dr Memory on op/stash.t shows failures like
-----------------------------------------------------------------
Error #1: UNADDRESSABLE ACCESS: reading 0x1161539c-0x1161539d 1 byte(s)
# 0 replace_strlen                  
[d:\drmemory_package\drmemory\replace.c:375]
# 1 perl522.dll!Perl_newSVpv        [c:\perl521\srcnewb4opt\sv.c:9215]
# 2 B.dll!XS_B__IV_IVX              [c:\perl521\srcnewb4opt\ext\b\b.xs:1676]
# 3 perl522.dll!Perl_pp_entersub    [c:\perl521\srcnewb4opt\pp_hot.c:3268]
# 4 perl522.dll!Perl_runops_debug   [c:\perl521\srcnewb4opt\dump.c:2234]
# 5 perl522.dll!S_run_body          
[c:\perl521\srcnewb4opt\win32\perl.c:2451]
# 6 perl522.dll!perl_run            
[c:\perl521\srcnewb4opt\win32\perl.c:2374]
# 7 perl522.dll!RunPerl             
[c:\perl521\srcnewb4opt\win32\perllib.c:258]
# 8 main                            
[c:\perl521\srcnewb4opt\win32\perlmain.c:39]
-----------------------------------------------------------------

CvFILE is either a pointer to malloc freed sentinel pattern, or some 
other perl src code file in the .t process and therefore the tests fail.

CvFILE always comes from CopFILE(), but CopFILE() is very different on 
threads and unthreaded because of the "no SV*s in OPs due to shared 
OP/OP COW" limitation.

threaded
-----------------------------------------------------------------
#  define CopFILE(c)        ((c)->cop_file)
-----------------------------------------------------------------
unthreaded
-----------------------------------------------------------------
#  define CopFILE(c)        (CopFILEGV(c) \
                    ? GvNAME(CopFILEGV(c))+2 : NULL)
-----------------------------------------------------------------
a section of the COP struct
-----------------------------------------------------------------
#ifdef USE_ITHREADS
    PADOFFSET    cop_stashoff;    /* offset into PL_stashpad, for the
                   package the line was compiled in */
    char *    cop_file;    /* file name the following line # is from */
#else
    HV *    cop_stash;    /* package line was compiled in */
    GV *    cop_filegv;    /* file the following line # is from */
#endif
-----------------------------------------------------------------
The current design of CvFILE on threads has to change. I am currrently 
experimenting with only selectivly using "savepv(CopFILE(cop)), 
CvDYNFILE_on(sv))". It seems to me (and Im not a optree expert) 99% of 
the time, the filename is available in the 1st op of the sub, which is a 
pp_nextstate. Predeclared subs have CV*s, but CvSTART and CvROOT are 
null, so in Perl_newSTUB a savepv() is required. The PL_curcop filename 
is freed on savestack with
-----------------------------------------------------------------
Note: # 0 replace_free                        
[d:\drmemory_package\common\alloc_replace.c:2380]
Note: # 1 perl522.dll!VMem::Free              
[c:\perl521\srcnewb4opt\win32\vmem.h:213]
Note: # 2 perl522.dll!CPerlHost::FreeShared   
[c:\perl521\srcnewb4opt\win32\perlhost.h:101]
Note: # 3 perl522.dll!PerlMemSharedFree       
[c:\perl521\srcnewb4opt\win32\perlhost.h:360]
Note: # 4 perl522.dll!Perl_leave_scope        
[c:\perl521\srcnewb4opt\scope.c:852]
Note: # 5 perl522.dll!Perl_pop_scope          
[c:\perl521\srcnewb4opt\scope.c:104]
Note: # 6 perl522.dll!Perl_pp_leaveeval       
[c:\perl521\srcnewb4opt\pp_ctl.c:4368]
Note: # 7 perl522.dll!Perl_runops_debug       
[c:\perl521\srcnewb4opt\dump.c:2234]
Note: # 8 perl522.dll!S_run_body              
[c:\perl521\srcnewb4opt\win32\perl.c:2451]
Note: # 9 perl522.dll!perl_run                
[c:\perl521\srcnewb4opt\win32\perl.c:2374]
Note: #10 perl522.dll!RunPerl                 
[c:\perl521\srcnewb4opt\win32\perllib.c:258]
Note: #11 main                                
[c:\perl521\srcnewb4opt\win32\perlmain.c:39]
-----------------------------------------------------------------
callstack.

Another idea proposed in 
http://www.nntp.perl.org/group/perl.perl5.porters/2001/05/msg36781.html 
was to eliminate CvFILE entirely since its value is not used by PP code. 
There will be breakage on CPAN to remove CvFILE http://grep. cpan. 
me/?q=CvFILE but all the CvFILE uses are either to print debugging info 
to a human, or setting CvFILE to maintain perlapi compatibility since 
core does that. This would save a ptr slot in the CV body plus the 
malloc. Here are the usages that look most, "interesting" to me.

https://metacpan.org/source/MLEHMANN/Devel-FindRef-1.44/FindRef.xs#L204
https://metacpan.org/source/NWCLARK/Devel-Arena-0.23/Arena.xs#L453
https://metacpan.org/source/RGARCIA/Sub-Identify-0.10/Identify.xs#L49

Another idea is, change CvFILE slot in the CV body to be HEK * owned or 
GV * (gv_fetchfile's GV*) un/owned by the CV * instead of the current 
unowned static char * for XS, or the malloced char * for PP.

That also brings up, why does gv_fetchfile exist to create typeglobs 
like this hand trimmed of non "_<" entries list? What use do these GVs have?
-----------------------------------------------------------------
C:\Documents and Settings\Owner>perl -e"$,=\"\n\", print keys%main::"
_<perl.c
_<.\win32.c
_<..\universal.c
_<-e
_<perllib.c
_<..\perlio.c
_<..\mro_core.c
_<Win32CORE.c
C:\Documents and Settings\Owner>
-----------------------------------------------------------------

Or
-----------------------------------------------------------------
C:\Documents and Settings\Owner>perl -e"require Test::Harness; 
$,=\"\n\", print
keys%main::"
_<C:/perl521/lib/strict.pm
_<C:/perl521/lib/auto/Time/HiRes/HiRes.dll
_<C:/perl521/lib/auto/Fcntl/Fcntl.dll
_<..\universal.c
_<C:/perl521/lib/XSLoader.pm
_<C:/perl521/lib/auto/IO/IO.dll
_<Cwd.c
_<-e
_<C:/perl521/lib/base.pm
_<C:/perl521/lib/auto/POSIX/POSIX.dll
_<Win32CORE.c
_<POSIX.xs
_<perllib.c
_<C:/perl521/lib/warnings.pm
_<C:/perl521/lib/Carp.pm
_<C:/perl521/lib/auto/Cwd/Cwd.dll
_<C:/perl521/site/lib/TAP/Parser/IteratorFactory.pm
_<..\mro_core.c
_<.\win32.c
_<DynaLoader.c
_<HiRes.c
_<perl.c
_<POSIX.c
_<Fcntl.c
_<IO.c
_<..\perlio.c
C:\Documents and Settings\Owner>
-----------------------------------------------------------------

Even worse, these GVs have a SV * inside, with YET another unique copy 
of the filename (see later attached pic, gp_file_hek is the 1st copy, 
the SV * in the GV* is another copy, different pointers).
-----------------------------------------------------------------
C:\Documents and Settings\Owner>perl -e"require Test::Harness; require 
Devel::Pe
ek; Devel::Peek::Dump(*{'_<C:/perl521/lib/strict.pm'})"
SV = PVGV(0x90032c) at 0x9119f4
  REFCNT = 1
  FLAGS = (MULTI)
  NAME = "_<C:/perl521/lib/strict.pm"
  NAMELEN = 26
  GvSTASH = 0x3680f4    "main"
  FLAGS = 0x2
  GP = 0x914f7c
    SV = 0x911a04
    REFCNT = 1
    IO = 0x0
    FORM = 0x0
    AV = 0x0
    HV = 0x0
    CV = 0x0
    CVGEN = 0x0
    GPFLAGS = 0x0 ()
    LINE = 14
    FILE = "C:/perl521/lib/strict.pm"
    EGV = 0x9119f4      "_<C:/perl521/lib/strict.pm"

C:\Documents and Settings\Owner>perl -e"require Test::Harness; require 
Devel::Pe
ek; Devel::Peek::Dump(*{'_<C:/perl521/lib/strict.pm'}{SCALAR})"
SV = IV(0x3681f8) at 0x3681fc
  REFCNT = 1
  FLAGS = (TEMP,ROK)
  RV = 0x911d3c
  SV = PV(0x36906c) at 0x911d3c
    REFCNT = 2
    FLAGS = (POK,pPOK)
    PV = 0x91527c "C:/perl521/lib/strict.pm"\0
    CUR = 24
    LEN = 26

C:\Documents and Settings\Owner>
-----------------------------------------------------------------
While threaded perl wastes memory much worse than unthreaded. Unthreaded 
still has 3 copies of the file path for every .pm file in memory (see 
later attached). I guess I'll have to newSVpvn_share that, or something 
else, since I do have a HEK* already from the GV, which is an easy fix.

Now note, I have not said there are any memory leaks, it is just that 
threaded Perl burns memory like an oil tycoon setting pallets of money 
on fire. All this code (file path storage) could only have been designed 
by a monkey house and has to change, to something else. Exactly what, is 
why I filed this ticket.

[Please do not change anything below this line]
-----------------------------------------------------------------
---
Flags:
    category=core
    severity=high
---
Site configuration information for perl 5.22.0:

Configured by Owner at Thu May 28 17:26:18 2015.

Summary of my perl5 (revision 5 version 22 subversion 0) configuration:
  Commit id: db8192ecf78d7f18a7fdbcc047f3dad0577bb287
  Platform:
    osname=MSWin32, osvers=5.1, archname=MSWin32-x86-multi-thread
    uname=''
    config_args='undef'
    hint=recommended, useposix=true, d_sigaction=undef
    useithreads=define, usemultiplicity=define
    use64bitint=undef, use64bitall=undef, uselongdouble=undef
    usemymalloc=n, bincompat5005=undef
  Compiler:
    cc='cl', ccflags ='-nologo -GF -W3 -O1 -MD -Zi -DNDEBUG -GL -DWIN32 
-D_CONSOLE -DNO_STRICT  -DPERL_TEXTMODE_SCRIPTS -DPERL_IMPLICIT_CONTEXT 
-DPERL_IMPLICIT_SYS -D_USE_32BIT_TIME_T',
    optimize='-O1 -MD -Zi -DNDEBUG -GL',
    cppflags='-DWIN32'
    ccversion='13.10.6030', gccversion='', gccosandvers=''
    intsize=4, longsize=4, ptrsize=4, doublesize=8, byteorder=1234, 
doublekind=3
    d_longlong=undef, longlongsize=8, d_longdbl=define, longdblsize=8, 
longdblkind=0
    ivtype='long', ivsize=4, nvtype='double', nvsize=8, Off_t='__int64', 
lseeksize=8
    alignbytes=8, prototype=define
  Linker and Libraries:
    ld='link', ldflags ='-nologo -nodefaultlib -debug -opt:ref,icf -ltcg 
        -libpath:"c:\perl\lib\CORE"         -machine:x86'
    libpth=\lib
    libs=oldnames.lib kernel32.lib user32.lib gdi32.lib winspool.lib 
comdlg32.lib advapi32.lib shell32.lib ole32.lib oleaut32.lib 
netapi32.lib uuid.lib ws2_32.lib mpr.lib winmm.lib version.lib 
odbc32.lib odbccp32.lib comctl32.lib msvcrt.lib
    perllibs=oldnames.lib kernel32.lib user32.lib gdi32.lib winspool.lib 
comdlg32.lib advapi32.lib shell32.lib ole32.lib oleaut32.lib 
netapi32.lib uuid.lib ws2_32.lib mpr.lib winmm.lib version.lib 
odbc32.lib odbccp32.lib comctl32.lib msvcrt.lib
    libc=msvcrt.lib, so=dll, useshrplib=true, libperl=perl522.lib
    gnulibc_version=''
  Dynamic Linking:
    dlsrc=dl_win32.xs, dlext=dll, d_dlsymun=undef, ccdlflags=' '
    cccdlflags=' ', lddlflags='-dll -nologo -nodefaultlib -debug 
-opt:ref,icf -ltcg         -libpath:"c:\perl\lib\CORE"         -machine:x86'

Locally applied patches:
    RC2

---
@INC for perl 5.22.0:
    C:/perl521/srcnewb4opt/lib
    .

---
Environment for perl 5.22.0:
    HOME (unset)
    LANG (unset)
    LANGUAGE (unset)
    LD_LIBRARY_PATH (unset)
    LOGDIR (unset)
    PATH=C:\sperl\c\bin;C:\perl521\bin;C:\WINDOWS\system32;C:\Program 
Files\Microsoft Visual Studio .NET 2003\Vc7\bin;C:\Program 
Files\Microsoft Visual Studio .NET 
2003\Common7\IDE;C:\WINDOWS;C:\Program Files\Git\cmd;C:\Program 
Files\Microsoft Visual Studio .NET 2003\Common7\Tools\bin;C:\perl\bin;
    PERL_BADLANG (unset)
    PERL_JSON_BACKEND=Cpanel::JSON::XS
    PERL_YAML_BACKEND=YAML::XS
    SHELL (unset)


Thread Previous


nntp.perl.org: Perl Programming lists via nntp and http.
Comments to Ask Bjørn Hansen at ask@perl.org | Group listing | About