develooper Front page | perl.perl5.porters | Postings from September 2010

[perl #77854] Memory leak using threads

Thread Next
From:
Jon Combe
Date:
September 16, 2010 00:16
Subject:
[perl #77854] Memory leak using threads
Message ID:
rt-3.6.HEAD-5116-1284479539-1142.77854-75-0@perl.org
# New Ticket Created by  Jon Combe 
# Please include the string:  [perl #77854]
# in the subject line of all future correspondence about this issue. 
# <URL: http://rt.perl.org/rt3/Ticket/Display.html?id=77854 >


Cc: jon.combe@telindus.co.uk
Subject: Memory leak using threads
Message-Id: <5.12.2_7636_1284476987@ubuntu-test>
Reply-To: jon.combe@telindus.co.uk
To: perlbug@perl.org


This is a bug report for perl from jon.combe@telindus.co.uk,
generated with the help of perlbug 1.39 running under perl 5.12.2.


-----------------------------------------------------------------
[Please describe your issue here]
I have a Perl script using threads and have found that when it has been running for a period of time the memory usage gets very high (around 1GB). I have managed to find the problem seems to be thread related. To help isolate the problem I created the simple Perl script below. This simply spawns a fixed number of threads and each thread simply increments and then decrements a thread counter. At the end, the code checks the thread count is zero (it should be) and then counts down from 1 to 30. This gives me 30 seconds to capture the final memory usage prior to the script ending. I have observed the following memory usage when changing the number of threads in the first "for" loop in terms of memory usage when running the script:-

Thread Count            VIRT MEM        RES MEM
50000                   106m            93m
100000                  195m            181m
150000                  287m            275m

Note that this output is from a version of Perl I compiled myself. I've also tried with the version of Perl that comes packaged with CentOS 5.4 (5.8.8) and that comes with Ubuntu 10.2 (Perl 5.10.1) with similar results. I've also tried removing the shared variable ($thread_count) and making the subroutine entirely empty with the same results. I suspect therefore it's a memory leak in creating threads.

#!/usr/bin/perl -w

use strict;
use threads;
use threads::shared;

my $thread_count : shared;

$thread_count = 0;
for ( my $i = 0 ; $i < 150000 ; $i++ )
{
        my $thread = threads->create('thread_function');
        $thread -> join();
        if ( $i % 1000 == 0 )
        {
                printf ( "%010i\n" , $i );
        }
}

print "DONE\n";
print "$thread_count\n";

while ( $thread_count > 0 )
{
        print "zzzz\n";
        sleep(1);
}
for ( my $i = 0 ; $i < 30 ; $i++ )
{
        print "Count down " . (30-$i) . "\n";
        sleep(1);
}

sub thread_function
{
        $thread_count++;
        $thread_count--;
}


[Please do not change anything below this line]
-----------------------------------------------------------------
---
Flags:
    category=core
    severity=high
---
Site configuration information for perl 5.12.2:

Configured by jcombe at Tue Sep 14 15:33:59 BST 2010.

Summary of my perl5 (revision 5 version 12 subversion 2) configuration:

  Platform:
    osname=linux, osvers=2.6.35-19-generic-pae, archname=i686-linux-thread-multi
    uname='linux ubuntu-test 2.6.35-19-generic-pae #28-ubuntu smp sun aug 29 08:17:04 utc 2010 i686 gnulinux '
    config_args=''
    hint=recommended, useposix=true, d_sigaction=define
    useithreads=define, usemultiplicity=define
    useperlio=define, d_sfio=undef, uselargefiles=define, usesocks=undef
    use64bitint=undef, use64bitall=undef, uselongdouble=undef
    usemymalloc=n, bincompat5005=undef
  Compiler:
    cc='cc', ccflags ='-D_REENTRANT -D_GNU_SOURCE -fno-strict-aliasing -pipe -fstack-protector -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64',
    optimize='-O2',
    cppflags='-D_REENTRANT -D_GNU_SOURCE -fno-strict-aliasing -pipe -fstack-protector -I/usr/local/include'
    ccversion='', gccversion='4.4.5', gccosandvers=''
    intsize=4, longsize=4, ptrsize=4, doublesize=8, byteorder=1234
    d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=12
    ivtype='long', ivsize=4, nvtype='double', nvsize=8, Off_t='off_t', lseeksize=8
    alignbytes=4, prototype=define
  Linker and Libraries:
    ld='cc', ldflags =' -fstack-protector -L/usr/local/lib'
    libpth=/usr/local/lib /lib /usr/lib
    libs=-lnsl -ldl -lm -lcrypt -lutil -lpthread -lc
    perllibs=-lnsl -ldl -lm -lcrypt -lutil -lpthread -lc
    libc=/lib/libc-2.12.1.so, so=so, useshrplib=false, libperl=libperl.a
    gnulibc_version='2.12.1'
  Dynamic Linking:
    dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef, ccdlflags='-Wl,-E'
    cccdlflags='-fPIC', lddlflags='-shared -O2 -L/usr/local/lib -fstack-protector'

Locally applied patches:


---
@INC for perl 5.12.2:
    /usr/local/lib/perl5/site_perl/5.12.2/i686-linux-thread-multi
    /usr/local/lib/perl5/site_perl/5.12.2
    /usr/lib/perl5/5.12.2/i686-linux-thread-multi
    /usr/lib/perl5/5.12.2
    .

---
Environment for perl 5.12.2:
    HOME=/home/jcombe
    LANG=en_GB.UTF-8
    LANGUAGE (unset)
    LD_LIBRARY_PATH (unset)
    LOGDIR (unset)
    PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games
    PERL_BADLANG (unset)
    SHELL=/bin/bash

This email is private and may be confidential and is for the intended recipient only. If misdirected, please notify us by telephone and confirm that it has been deleted from your system and any copies destroyed. If you are not the intended recipient you are strictly prohibited from using, printing, copying, distributing or disseminating this email or any information contained in it. We use reasonable endeavours to virus scan all emails leaving the Company but no warranty is given that this email and any attachments are virus free. You should undertake your own virus checking. The right to monitor email communications through our network is reserved by us.

Telindus Limited is a company registered in England and Wales under number 02020395. The registered office is Centurion, Riverside Way, Watchmoor Park, Blackwater Valley Road, Camberley, Surrey, GU15 3YA.


Thread Next


nntp.perl.org: Perl Programming lists via nntp and http.
Comments to Ask Bjørn Hansen at ask@perl.org | Group listing | About