Front page | perl.perl5.porters |
Postings from March 2008
Hi, porters.
I'm by no means fluent in the perl internals, but I *think*
Text::CSV_XS->getline is leaking.
Here's the script I used to check the memory being consumed.
Basically the idea is to create a relatively large chunk of TSV string,
and open it in memory, and pass it to getline() to see what it does.
use strict;
use blib;
use Text::CSV_XS 0.36;
print "My PID: $$\n";
my $line = join("\t", ('a'..'z')) . "\n";
my $csv = Text::CSV_XS->new({ sep_char => "\t" });
my $iter = 10000;
my $data = $line x $iter;
my $count = 0;
while(1) {
open(my $io, '<', \$data);
while(my $row = $csv->getline($io) ) {
}
$count += $iter;
print "Did $count iterations\n";
sleep(2);
}
When I run this code, I see memory increase of about a 400K ~ 600K per
iteration of the outer while loop on my MacBook. This was run with
Text::CSV_XS 0.36.
From glancing at the source code, I think it's a simple matter of
mortalizing one AV:
--- Text-CSV_XS-0.36/CSV_XS.xs 2008-03-06 00:26:41.000000000 +0900
+++ Text-CSV_XS-0.36.patched/CSV_XS.xs 2008-03-11 18:19:00.000000000 +0900
@@ -1083,6 +1083,7 @@
CSV_XS_SELF;
av = newAV ();
avf = newAV ();
+ sv_2mortal((SV *) avf);
ST (0) = xsParse (hv, av, avf, io, 1)
? sv_2mortal (newRV_noinc ((SV *)av))
: &PL_sv_undef;
I don't know exactly what avf is doing, but getline() is the only code
that's creating the said AV within it's call stack. Everything else is
receiving avf from somewhere outside... and it's not being returned or
anything. This seemes like a candidate to be mortalized.
Running the same test code with the patch applied, I see no significant
increase in memory usage, which seemed to indicate that this is the culprit.
Crossing my fingers that I'm not way off.
--d
Thread Next