Debugging Memory Usage in Perl

Hi!

I recently did a talk at YAPC about Scalability, Reliability, and
Performance at ZipRecruiter.  One of the talking points was the
difficulty of tracking down the cause of memory usage in Perl.  James
Keenan reached out to me and asked if I could send this feedback to
P5P in the hopes that the situation could be improved.

It is definitely possible that I (and my coworkers and the people in
the community that I reached out to) missed some tools or techniques,
so if that's the case feel free to let me know and move on.

The problem that we sometimes run into is that a process is using a
lot of memory and we want to see why.  We have already ruled out a
leak and instead believe that this is just a really high baseline for
code (and presumably some caches getting pre-warmed or something.)

So as far as I know the best tools out there are Devel::SizeMe
(obliquely referenced in the talk) and Devel::MAT.  Both are tools for
examining the current state of memory usage.  If I want to find out
*which* code allocated all of this memory, as far as I know I am out
of luck.  I could theoretically dump from MAT or SizeMe periodically
and use that to bisect, but that's more work than I am interested in
doing right now.

If I had my druthers we would have two hooks in the Perl compiler:

 * One that would get called when allocating from an already created
   arena

 * One that would get called when allocating memory from the operating
   system

Given these we could log a stacktrace, make a flame graph, whatever.
I suspect that alone would give me what I needed, though I could see
it being really expensive.

Anyway, just an idea; even if it never gets implemented Perl will
likely continue to be the primary programming language in use at
ZipRecruiter.

Thanks!

P.S. please keep me on Cc or To as I am not subscribed to the list.

-- 
fREW Schmidt
https://blog.afoolishmanifesto.com

• Debugging Memory Usage in Perl by fREW Schmidt
• Re: Debugging Memory Usage in Perl by Sergey Aleynikov