develooper Front page | perl.perl5.porters | Postings from February 2018

Re: [perl #132892] Possibly memory leak when matching strings usingpre-compiled regexes stored in hash key (perl >= v5.26)

Thread Previous | Thread Next
From:
Shlomi Fish
Date:
February 22, 2018 09:04
Subject:
Re: [perl #132892] Possibly memory leak when matching strings usingpre-compiled regexes stored in hash key (perl >= v5.26)
Message ID:
20180222094445.6f287186@telaviv1.shlomifish.org
On Wed, 21 Feb 2018 07:49:20 -0800
(via RT) <perlbug-followup@perl.org> wrote:

> # New Ticket Created by   
> # Please include the string:  [perl #132892]
> # in the subject line of all future correspondence about this issue. 
> # <URL: https://rt.perl.org/Ticket/Display.html?id=132892 >
> 
>

Hi all!

I reworked the example code to remove some potential problems, and verified
that htop sees its RAM consumption increasing:

#!/usr/bin/perl

use strict;
use warnings;

# my $getmem = "ps -o vsz= -o rss= -p $$",

print "== Test result on $^O perl-$^V\n";

my %h = (
  qr{abcd} => [ 'ABCD', 2],
  qr{efgh} => [ 'EFGH', 1],
  qr{ijkl} => [ 'IJKL', 1],
  qr{mnop} => [ 'MNOP', 1],
  qr{qrst} => [ 'QRST', 1],
  qr{uvwx} => [ 'UVWX', 1],
);

my @list = qw/ abcd abcd efgh ijkl mnop qrst uvwx /;

while (1) {
  # system($getmem);
  for my $r (keys %h) {
    my $c = 0;
    my ($name, $expected) = @{$h{$r}};
    for my $e ( @list ) {
      $c++ if $e =~ $r;
    }

    print ">>> Count mismatch for $name\n" if ( $c != $expected );
  }
}


 
> Hello guys,
> 
> I have a long running perl script which allows the user configure the
> program behavior with a hash that got pre-compiled regexps
> as keys and the corresponding callback code as the value like below.
> Recently I have upgraded to perl v5.26 and the memory
> usage grows since then, so I have fall back to perl v5.24.3 as a temporary
> solution. What I observed before fallback is that the
> memory grows when I try to iterates the hash keys and matches them against
> some text.
> 
> %action = (
> 
>   qr/^regex here/ => sub { print "match" },
> 
> );
> 
> 
> I have written a small test for this problem, you can find it in the
> attached perlbug.tar.xz. 20180221-mem-leak-poc.pl is the prof of
> concept code. It will output the platform and perl version as the first
> line, and make use of the ps command to retrieve the vsz and
> rss of the perl interpreter before each iteration. The test result of
> different platform are stored in the rslt/ sub-directory for your reference.
> According to the results, with perl <v5.26, the size in memory remain
> unchanged; While the memory usage keeps increasing when the
> script is run with perl >=v5.26. I have limited the number of iteration,
> the memory usage grows further if you increases the number of iteration.
> 
> |-- 20180221-mem-leak-poc.pl
> `-- rslt
>     |-- centos6-p5.10.1.log
>     |-- centos6-p5.24.3.log
>     |-- centos6-p5.26.1.log
>     |-- centos6-p5.27.8.log
>     |-- dfbsd-p5.24.3.log
>     |-- dfbsd-p5.26.1.log
>     |-- freebsd-p5.24.3.log
>     `-- freebsd-p5.26.1.log
> 
> According to my empirical result, this problem occurs when using perl newer
> than v5.26, the first line of perl -v of the affected perl
> interpreters below.
> 
> > This is perl 5, version 26, subversion 1 (v5.26.1) built for amd64-freebsd
> > This is perl 5, version 27, subversion 8 (v5.27.8) built for x86_64-linux
> > This is perl 5, version 26, subversion 1 (v5.26.1) built for x86_64-linux  
> 
> This is perl 5, version 26, subversion 1 (v5.26.1) built for
> > x86_64-dragonfly  
> 
> 
> I have traced the problem a little bit and I *guess* the problem is in the
> function S_concat_pat of regcomp.c and the new SVs
> copied out are immortal, so every time my program iterate through the keys
> of hash to match against things, it creates more
> copies of the regexes. I can try to dig deeper when I am free to build the
> debug versions of perl, but that would be great if this
> case can be taken care by the professionals here. Thanks a lot.
> ​
> 
> ​
> Best regards,
> Baggio

Thread Previous | Thread Next


nntp.perl.org: Perl Programming lists via nntp and http.
Comments to Ask Bjørn Hansen at ask@perl.org | Group listing | About