develooper Front page | perl.perl5.porters | Postings from August 2001

[ID 20010807.013] Garbage collection and/or memory leak problemin perl5.

Thread Previous
From:
Steve Weigand
Date:
August 7, 2001 08:58
Subject:
[ID 20010807.013] Garbage collection and/or memory leak problemin perl5.
Message ID:
200108071557.KAA02455@hammer.centtech.com

This is a bug report for perl from weigand@centtech.com,
generated with the help of perlbug 1.28 running under perl v5.6.0.


-----------------------------------------------------------------
[Please enter your report here]

I've discovered a bug in some Perl 5 builds,  including the latest
one it looks like.  The problem has to do with the way Perl does
garbage collection.  Here's an example program to be run on Linux,
Solaris,  or other unices:

   #!/usr/local/bin/perl5

   %T = ();
   system("vmstat 1 2");
   $T{"one"}{"two"} = "1" x 200000000;
   system("vmstat 1 2");
   %T = ();
   system("vmstat 1 2");
   $T{"one"}{"two"} = "1" x 200000000;
   system("vmstat 1 2");
   %T = ();
   system("vmstat 1 2");
   $T{"one"}{"two"} = "1" x 200000000;
   system("vmstat 1 2");
   %T = ();
   system("vmstat 1 2");
   $T{"one"}{"two"} = "1" x 200000000;
   system("vmstat 1 2");
   %T = ();
   system("vmstat 1 2");
   $T{"one"}{"two"} = "1" x 200000000;
   system("vmstat 1 2");

So what I'm doing above is I'm creating an associative array %T,
then I'm doing the unix command "vmstat" which shows you how much
memory is free at the system level.  Then I'm filling up the
associative array with 200MB of characters,  basically just gulping
up a large amount of memory.  Next,  I display "vmstat" again.  The
vmstat output should reflect the fact that I've just grabbed up
200MB of memory,  and it does.  Next,  I supposedly free up the
memory by doing "%T = ();".  Next,  I do vmstat again,  and it
displays that I'm still using 200MB.  But this is perfectly normal
and expected,  since I don't get that back until the program
exits.  But what isn't normal or expected is what happens next...

Next, I grab another 200MB and store it in my associative array.
When I now do vmstat,  it should *not* show any additional memory
usage,  since it should just use the same memory address space
of the original gulp of 200MB that I just discarded.  Instead,
vmstat shows I just grabbed up a lot more memory.  So instead of
just 200MB being used at this point,  it's actually using 400MB.
It's like it didn't free up the associative array's memory like I
asked it to do.

As the script continues,  it's supposed to free up the memory we just
grabbed and then grab another gulp of 200MB.  But at no time in the
program should I be using any more than 200MB of memory.  So,  why is
it that when I run this script using some versions of perl5 do I see
the memory being drained as it goes through the script,  eventually
running out of memory?

It's as if the garbage collection algorithm is not being run.

Seems like a HUGE bug to me!!!

I've tried this script on Linux, Solaris 2.6, and Solaris 2.7.  It
seems independent of OS type. I've tried it on Perl versions 5.6.0,
5.005_03, and 5.00401.  All those versions show the same bug.  But
when I use version 5.003,  it doesn't show the bug and behaves as
I would expect (correctly).  So Perl version 5.003 is the only version
I've seen which doesn't have this bug.

Furthermore,  I've used "delete" and "undef" to try to free up the
memory instead of just using "%T = ();",  and both result in the same
problem.  It's as if Perl is not freeing the memory up.

Here's the output of my script running on Perl 5.6.0.  It actually
shows that I run out of memory.  Remember that in solaris,  vmstat's
first output line is junk.  So only look at the second output line:

 procs     memory            page            disk          faults      cpu
 r b w   swap  free  re  mf pi po fr de sr f0 s0 -- --   in   sy   cs us sy id
 0 0 0   1280   736   0  31  5  3  4  0  0  0  1  0  0  320  291   95  6  1 93
 0 0 11 5697480 1794264 0 4  0  0  0  0  0  0  0  0  0  302   94   72  0  0 100
 procs     memory            page            disk          faults      cpu
 r b w   swap  free  re  mf pi po fr de sr f0 s0 -- --   in   sy   cs us sy id
 0 0 0   1296   736   0  31  5  3  4  0  0  0  1  0  0  320  291   95  6  1 93
 0 0 11 4648296 817072 0  4  0  0  0  0  0  0 17  0  0  338  118  103  0  4 96
 procs     memory            page            disk          faults      cpu
 r b w   swap  free  re  mf pi po fr de sr f0 s0 -- --   in   sy   cs us sy id
 0 0 0   1296   736   0  31  5  3  4  0  0  0  1  0  0  320  291   95  6  1 93
 0 0 11 4648296 817072 0  4  0  0  0  0  0  0  0  0  0  304  172   85  0  0 100
 procs     memory            page            disk          faults      cpu
 r b w   swap  free  re  mf pi po fr de sr f0 s0 -- --   in   sy   cs us sy id
 0 0 0   1304   744   0  31  5  3  4  0  0  0  1  0  0  320  291   95  6  1 93
 0 0 11 4124008 328776 0  5  8  0  0  0  0  0 52  0  0  462  118  164  0  2 98
 procs     memory            page            disk          faults      cpu
 r b w   swap  free  re  mf pi po fr de sr f0 s0 -- --   in   sy   cs us sy id
 0 0 0   1304   744   0  31  5  3  4  0  0  0  1  0  0  320  291   95  6  1 93
 0 0 11 4124008 328776 0  4  0  0  0  0  0  0  0  0  0  300   83   56  0  0 100
 procs     memory            page            disk          faults      cpu
 r b w   swap  free  re  mf pi po fr de sr f0 s0 -- --   in   sy   cs us sy id
 0 0 0   1320   744   0  31  5  4  4  0  0  0  1  0  0  320  291   95  6  1 93
 0 0 11 3604768 31408 0   5  0 256 264 0 33 0 20  0  0  344   83  102  0  5 94
 procs     memory            page            disk          faults      cpu
 r b w   swap  free  re  mf pi po fr de sr f0 s0 -- --   in   sy   cs us sy id
 0 0 0   1320   744   0  31  5  4  4  0  0  0  1  0  0  320  291   95  6  1 93
 0 0 11 3604752 31656 0   4  0  0  0  0  0  0  0  0  0  300  118   66  0  0 100
 procs     memory            page            disk          faults      cpu
 r b w   swap  free  re  mf pi po fr de sr f0 s0 -- --   in   sy   cs us sy id
 0 0 0   1384   744   0  31  5  4  4  0  0  0  1  0  0  320  291   95  6  1 93
 0 0 11 3080008 64040 0  19 120 0  0  0  0  0 15  0  0  345  116   98  0  2 98
 procs     memory            page            disk          faults      cpu
 r b w   swap  free  re  mf pi po fr de sr f0 s0 -- --   in   sy   cs us sy id
 0 0 0   1384   744   0  31  5  4  4  0  0  0  1  0  0  320  291   95  6  1 93
 0 0 11 3079992 64024 0   4  0  0  0  0  0  0  0  0  0  301  130   68  0  0 100
Out of memory during "large" request for 536875008 bytes at ./testthis3.pl line 30.

Notice how it was grabbing continually more and more memory from the
system.  Bad !!

Here's the run on Perl 5.003:

 procs     memory            page            disk          faults      cpu
 r b w   swap  free  re  mf pi po fr de sr f0 s0 -- --   in   sy   cs us sy id
 0 0 0   3200  1304   0  31  5  4  4  0  0  0  1  0  0  320  291   95  6  1 93
 0 0 28 5707752 1821120 0 4  0  0  0  0  0  0  0  0  0  301   83   55  0  0 100
 procs     memory            page            disk          faults      cpu
 r b w   swap  free  re  mf pi po fr de sr f0 s0 -- --   in   sy   cs us sy id
 0 0 0   3208  1312   0  31  5  4  4  0  0  0  1  0  0  320  291   95  6  1 93
 0 0 28 4659168 844544 0  5  8  0  0  0  0  0  1  0  0  307  125   95  0  2 97
 procs     memory            page            disk          faults      cpu
 r b w   swap  free  re  mf pi po fr de sr f0 s0 -- --   in   sy   cs us sy id
 0 0 0   3208  1312   0  31  5  4  4  0  0  0  1  0  0  320  291   95  6  1 93
 0 0 28 4659168 844544 0  4  0  0  0  0  0  0  0  0  0  300   83   52  0  0 100
 procs     memory            page            disk          faults      cpu
 r b w   swap  free  re  mf pi po fr de sr f0 s0 -- --   in   sy   cs us sy id
 0 0 0   3216  1312   0  31  5  4  4  0  0  0  1  0  0  320  291   95  6  1 93
 0 0 28 4659168 844544 0  4  0  0  0  0  0  0  0  0  0  300  118   65  0  0 100
 procs     memory            page            disk          faults      cpu
 r b w   swap  free  re  mf pi po fr de sr f0 s0 -- --   in   sy   cs us sy id
 0 0 0   3216  1312   0  31  5  4  4  0  0  0  1  0  0  320  291   95  6  1 93
 0 0 28 4659168 844544 0  4  0  0  0  0  0  0  0  0  0  300  147   67  0  0 100
   ..etc.

As you can see,  in this version of Perl (5.003),  it behaves correctly.
Vmstat shows that the memory is gulped up once during the beginning of
the program's run,  but no additional memory is gulped up as the program
continues.  And the program completes without an out-of-memory error.
No swap-thrashing occurs,  either,  like it did in the previous example.

Has anyone seen this before?  Is there anything I can do to fix this?
Is there a compile-time option I can use?  Is there a more up to date
version of Perl5 I can use?

I've tried programs like this,  too:

  LOCAL_BLOCK: {
    my %T = ();
    system("vmstat 1 2");
    $T{"one"}{"two"} = "1" x 20000000;
    system("vmstat 1 2");
  }
  LOCAL_BLOCK: {
    my %T = ();
    system("vmstat 1 2");
    $T{"one"}{"two"} = "1" x 20000000;
    system("vmstat 1 2");
  }
  LOCAL_BLOCK: {
    my %T = ();
    system("vmstat 1 2");
    $T{"one"}{"two"} = "1" x 20000000;
    system("vmstat 1 2");
  }
  LOCAL_BLOCK: {
    my %T = ();
    system("vmstat 1 2");
    $T{"one"}{"two"} = "1" x 20000000;
    system("vmstat 1 2");
  }

    ... etc.

The above script will also result in the same out-of-memory error.
This is really unexpected,  since the %T variable goes out of scope
after the curly-braces I would expect.

The only time when I've been able to get Perl to free up unused
memory is by encapsulating it in a subroutine:

   sub TestThis {
     my %T = ();
     system("vmstat 1 2");
     $T{"one"}{"two"} = "1" x 500000000;
     system("vmstat 1 2");
     %T = ();
   }
   
   for ($counter = 0; $counter <= 5; $counter++) {
     &TestThis;
   }

In the above example,  this script will work just fine.  It frees up
the memory used by %T after the subroutine scope is terminated.

So what this means is that no garbage collection or free'ing of memory
occurs for any global variable.  If you use delete() or undef() or
just set your variable to some other value in a global scope,  then
that memory is going to be tied up indefinitely and will never be
freed.

This looks like a rather large bug.  I work in the VLSI industry.  We
use Perl scripts for just about everything,  and our database sizes
are huge.  Something like this is enough to make us start switching
to other languages,  because Perl will result in excessive run times
due to swap-thrashing or will just crash altogether due to out of
memory problems.

Thanks,
  - Steve Weigand



[Please do not change anything below this line]
-----------------------------------------------------------------
---
Flags:
    category=core
    severity=high
---
Site configuration information for perl v5.6.0:

Configured by mjb at Tue Apr  4 12:37:13 CDT 2000.

Summary of my perl5 (revision 5.0 version 6 subversion 0) configuration:
  Platform:
    osname=solaris, osvers=2.6, archname=sun4-solaris
    uname='sunos sting 5.6 generic_105181-15 sun4u sparc sunw,ultra-2 '
    config_args=''
    hint=recommended, useposix=true, d_sigaction=define
    usethreads=undef use5005threads=undef useithreads=undef usemultiplicity=undef
    useperlio=undef d_sfio=undef uselargefiles=define 
    use64bitint=undef use64bitall=undef uselongdouble=undef usesocks=undef
  Compiler:
    cc='gcc', optimize='-O', gccversion=2.8.1
    cppflags='-I/usr/local/include'
    ccflags ='-I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64'
    stdchar='unsigned char', d_stdstdio=define, usevfork=false
    intsize=4, longsize=4, ptrsize=4, doublesize=8
    d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=16
    ivtype='long', ivsize=4, nvtype='double', nvsize=8, Off_t='off_t', lseeksize=8
    alignbytes=8, usemymalloc=y, prototype=define
  Linker and Libraries:
    ld='gcc', ldflags =' -L/usr/local/lib '
    libpth=/usr/local/lib /lib /usr/lib /usr/ccs/lib
    libs=-lsocket -lnsl -ldl -lm -lc -lcrypt -lsec
    libc=/lib/libc.so, so=so, useshrplib=false, libperl=libperl.a
  Dynamic Linking:
    dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef, ccdlflags=' '
    cccdlflags='-fPIC', lddlflags='-G -L/usr/local/lib'

Locally applied patches:
    

---
@INC for perl v5.6.0:
    /usr/local/lib/perl5/5.6.0/sun4-solaris
    /usr/local/lib/perl5/5.6.0
    /usr/local/lib/perl5/site_perl/5.6.0/sun4-solaris
    /usr/local/lib/perl5/site_perl/5.6.0
    /usr/local/lib/perl5/site_perl
    .

---
Environment for perl v5.6.0:
    HOME=/home/users3/weigand
    LANG=C
    LANGUAGE (unset)
    LD_LIBRARY_PATH=/usr/openwin/lib:/usr/X11/lib:/usr/lib/X11:/usr/dt/lib:/Server/csi/dynacell/releases/latest/ld_libs:/Server/csi/dynacore/releases/latest/ld_libs:/Server/cadmos/Q200-0221-arc/lib/sparc-solaris
    LOGDIR (unset)
    PATH=/home/users3/weigand/bin.private:/Server/csi/dynacore/releases/latest:/Server/csi/dynacell/releases/latest:.:/home/users3/weigand/bin:/usr/local/bin:/bin:/usr/bin:/usr/ucb:/usr/sbin:/usr/openwin/bin:/usr/bin/X11:/etc:/usr/etc:/usr/dt/bin:/vlsi/centaur/c5x/bin:/vlsi/cad2/bin:/vlsi/bin:/Server/epc/5.5p3/bin:/Server/csi/dynacore/releases/latest/java_lib:/Server/csi/dynacore/releases/latest/jre1.2.2/bin:/Server/pearl/sol/latest/bin:/vlsi/centaur/pearl/bin:/Server/cadmos/Q200-0221-arc/bin/sparc-solaris:/Server/veri/verilog/tools/verilog/bin:/Server/view/6.1:/vlsi/centaur/c5x/schematic/bin:/Server/bin
    PERL_BADLANG (unset)
    SHELL=/usr/local/bin/tcsh


Thread Previous


nntp.perl.org: Perl Programming lists via nntp and http.
Comments to Ask Bjørn Hansen at ask@perl.org | Group listing | About