[perl #57564] Refcounting bug with reused implicit lexicals as nested loop iterators

# New Ticket Created by  fish-perlbug@uc.org 
# Please include the string:  [perl #57564]
# in the subject line of all future correspondence about this issue. 
# <URL: http://rt.perl.org/rt3/Ticket/Display.html?id=57564 >



This is a bug report for perl from fish-perlbug@uc.org,
generated with the help of perlbug 1.35 running under perl 5.10.0.

Re-using the same implicitly lexical iterator variable in nested loops
decrements recount on wrong aliased target.

Terrible, terrible coding style, I know, but still, a bug.

Self-contained example:

--start--

my @list1 = ( qw{ foo } );
my @list2 = ( qw{ bar } );
my $i;

for $i ( @list1 ) {
    for $i ( @list2 ) {
    }
}

--cut--

Result:
Attempt to free unreferenced scalar: SV 0x1233ad8, Perl interpreter: 0x1220240.

This happens consistently on perl 5.10 but not on 5.8.* nor does it appear to
happen in blead.

It is specific to the nesting of the foreach-style loops with implicit lexical
loop variables with the same name.  What appears to be happening here is as
follows:

The entry of the first loop aliases $i to the first SV of the first list.  This
increases the refcount of that item.  Similarly the second loop entry aliases
the inner $i to the first SV of the second list.

However, on exit of the inner list, the refcount of the first item of the first
list is decremented, rather than the correct one.  Thus, the "foo" SV is
decremented twice and freed, but the "bar" is not decremented and thus leaks:

--start--

use Devel::Peek;

my @list1 = ( qw{ foo } );
my @list2 = ( qw{ foo } );
my $i;

for $i ( @list1 ) {
    for $i ( @list2 ) {
    }
}

print Dump \@list1;
print Dump \@list2;

--cut--

SV = RV(0x1233938) at 0x1233928
  REFCNT = 1
  FLAGS = (TEMP,ROK)
  RV = 0x125f650
  SV = PVAV(0x1235300) at 0x125f650
    REFCNT = 2
    FLAGS = (PADMY)
    ARRAY = 0x1270490
    FILL = 0
    MAX = 3
    ARYLEN = 0x0
    FLAGS = (REAL)
    Elt No. 0
    SV = PV(0x1231040) at 0x1233ad8
      REFCNT = 1                        # refcount is 1, but SVPV has simply been reused
      FLAGS = (POK,pPOK)
      PV = 0x12e8040 "(0x1233938) at 0x1233928\n  REFCNT = 1\n  FLAGS = (TEMP,ROK)"\0
      CUR = 58
      LEN = 64
SV = RV(0x1233938) at 0x1233928
  REFCNT = 1
  FLAGS = (TEMP,ROK)
  RV = 0x125f590
  SV = PVAV(0x1235318) at 0x125f590
    REFCNT = 2
    FLAGS = (PADMY)
    ARRAY = 0x123a9b0
    FILL = 0
    MAX = 3
    ARYLEN = 0x0
    FLAGS = (REAL)
    Elt No. 0
    SV = PV(0x12310a8) at 0x1233ce8
      REFCNT = 2                    # ref count too high
      FLAGS = (POK,pPOK)
      PV = 0x12327f0 "foo"\0
      CUR = 3
      LEN = 4

Comparing that with bleadperl:

SV = IV(0x1423928) at 0x1423928
  REFCNT = 1
  FLAGS = (TEMP,ROK)
  RV = 0x144f608
  SV = PVAV(0x14252e8) at 0x144f608
    REFCNT = 2
    FLAGS = (PADMY)
    ARRAY = 0x142a9a0
    FILL = 0
    MAX = 3
    ARYLEN = 0x0
    FLAGS = (REAL)
    Elt No. 0
    SV = PV(0x1421048) at 0x1423ad8
      REFCNT = 1
      FLAGS = (POK,pPOK)
      PV = 0x142202c "foo"\0
      CUR = 3
      LEN = 4
SV = IV(0x1423928) at 0x1423928
  REFCNT = 1
  FLAGS = (TEMP,ROK)
  RV = 0x144f590
  SV = PVAV(0x1425300) at 0x144f590
    REFCNT = 2
    FLAGS = (PADMY)
    ARRAY = 0x142a830
    FILL = 0
    MAX = 3
    ARYLEN = 0x0
    FLAGS = (REAL)
    Elt No. 0
    SV = PV(0x1421090) at 0x1423c88
      REFCNT = 1
      FLAGS = (POK,pPOK)
      PV = 0x14226d4 "foo"\0
      CUR = 3
      LEN = 4

Dumping \$i after the exit of the inner loop, the refcount of the SV is 2,
compared with 3 if I explicitly lexicalize the inner loop variable.

If the $i is a package global in the declaration, this issue does not manifest.
And similarly, if the lexicals have different names, or explicit mys, there is
no issue.

Not sure if this is a known issue - it doesn't seem to happen in bleadperl, but
I don't see a resolved bug that looks like this, and I suspect the issue is
somewhere in cop.h, which is very different in blead than in 5.10.  

The effect of this seems to vary a fair bit.  In the case I was seeing this in
a production system, the outer loop array was simply being emptied without
error, leading to surprizing action at a distance.

Not cygwin specific, seen on linux as well.

---
Flags:
    category=core
    severity=medium
---
This perlbug was built using Perl 5.10.0 - Tue Mar 18 00:26:31 EDT 2008
It is being executed now by  Perl 5.10.0 - Mon Jun 30 16:03:19 GMT 2008.

Site configuration information for perl 5.10.0:

Configured by rurban at Mon Jun 30 16:03:19 GMT 2008.

Summary of my perl5 (revision 5 version 10 subversion 0 patch 34065) configuration:
  Platform:
    osname=cygwin, osvers=1.5.25(0.15642), archname=cygwin-thread-multi-64int
    uname='cygwin_nt-5.1 reini 1.5.25(0.15642) 2008-06-12 19:34 i686 cygwin '
    config_args='-de -Dmksymlinks -Dusethreads -Dmad=y -Dusedevel'
    hint=recommended, useposix=true, d_sigaction=define
    useithreads=define, usemultiplicity=define
    useperlio=define, d_sfio=undef, uselargefiles=define, usesocks=undef
    use64bitint=define, use64bitall=undef, uselongdouble=undef
    usemymalloc=y, bincompat5005=undef
  Compiler:
    cc='gcc', ccflags ='-DPERL_USE_SAFE_PUTENV -U__STRICT_ANSI__ -fno-strict-aliasing -pipe -I/usr/local/include',
    optimize='-O3',
    cppflags='-DPERL_USE_SAFE_PUTENV -U__STRICT_ANSI__ -fno-strict-aliasing -pipe -I/usr/local/include'
    ccversion='', gccversion='3.4.4 (cygming special, gdc 0.12, using dmd 0.125)', gccosandvers=''
    intsize=4, longsize=4, ptrsize=4, doublesize=8, byteorder=12345678
    d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=12
    ivtype='long long', ivsize=8, nvtype='double', nvsize=8, Off_t='off_t', lseeksize=8
    alignbytes=8, prototype=define
  Linker and Libraries:
    ld='g++', ldflags =' -Wl,--enable-auto-import -Wl,--export-all-symbols -Wl,--stack,8388608 -Wl,--enable-auto-image-base -L/usr/local/lib'
    libpth=/usr/local/lib /usr/lib /lib
    libs=-lgdbm -ldb -ldl -lcrypt -lgdbm_compat
    perllibs=-ldl -lcrypt
    libc=/usr/lib/libc.a, so=dll, useshrplib=true, libperl=libperl.a
    gnulibc_version=''
  Dynamic Linking:
    dlsrc=dl_dlopen.xs, dlext=dll, d_dlsymun=undef, ccdlflags=' '
    cccdlflags=' ', lddlflags=' --shared  -Wl,--enable-auto-import -Wl,--export-all-symbols -Wl,--stack,8388608 -Wl,--enable-auto-image-base -L/usr/local/lib'

Locally applied patches:
    

---
@INC for perl 5.10.0:
    /usr/lib/perl5/5.10/i686-cygwin
    /usr/lib/perl5/5.10
    /usr/lib/perl5/site_perl/5.10/i686-cygwin
    /usr/lib/perl5/site_perl/5.10
    /usr/lib/perl5/vendor_perl/5.10/i686-cygwin
    /usr/lib/perl5/vendor_perl/5.10
    /usr/lib/perl5/vendor_perl/5.10
    /usr/lib/perl5/site_perl/5.8
    /usr/lib/perl5/vendor_perl/5.8
    .

---
Environment for perl 5.10.0:
    HOME=/home/fishy
    LANG (unset)
    LANGUAGE (unset)
    LD_LIBRARY_PATH (unset)
    LOGDIR (unset)
    PATH=/usr/local/bin:/usr/bin:/bin:/usr/X11R6/bin:/cygdrive/c/WINDOWS/system32:/cygdrive/c/WINDOWS:/cygdrive/c/WINDOWS/System32/Wbem:/cygdrive/c/Program Files/putty:/cygdrive/c/Program Files/Common Files/Adobe/AGL:/usr/bin:/usr/local/bin:/cygdrive/c/Program Files/Perforce:/cygdrive/c/PROGRA~1/ATT/Graphviz/bin:/cygdrive/c/Program Files/ActiveState Perl Dev Kit 7.0/bin:/cygdrive/c/Program Files/ActiveState Komodo IDE 4.2/:/usr/bin:/home/fishy/bin
    PERL_BADLANG (unset)
    SHELL (unset)

• [perl #57564] Refcounting bug with reused implicit lexicals as nested loop iterators by perlbug-followup
• Re: [perl #57564] Refcounting bug with reused implicit lexicals asnested loop iterators by Bram
• [PATCH] Tests for [perl #57564] and [perl #24524] Refcounting bug by Bram
• Re: [PATCH] Tests for [perl #57564] and [perl #24524] Refcountingbug by H.Merijn Brand