Re: Debugging Memory Usage in Perl

Hi,

2017-08-20 22:25 GMT+03:00 fREW Schmidt <frioux@gmail.com>:
> It is definitely possible that I (and my coworkers and the people in
> the community that I reached out to) missed some tools or techniques,
> so if that's the case feel free to let me know and move on.

There're two additional methods available.

First, on a regular perl build, you can walk Perl arenas (places Perl
interpreter allocates SV's from) with Devel::Gladiator and create a
top of most-created objects. That's kind-of like what Devel::MAT can
do, but can be applied to a live process and shows data split up by
class for blessed references. If your code is object-oriented enough,
you can track down what's allocated a lot and look more closely only
at those allocations.

When that's not enough (either you have a lot of plain SVs allocated
or you have a lot of objects of the same type allocated from too many
places) you can build a perl with '-A
ccflags=-DDEBUG_LEAKING_SCALARS'. That'll add some leak/double-free
debugging output (if you have some SV level leaks), but more
importantly, to provide this info, it also records an exact allocation
line and file for each individual SV. Here's how it looks like:

% ./perl -Ilib -e 'use Devel::Peek; Dump []'
ALLOCATED at -e:1 by anonlist (parent 0x0); serial 1721
SV = IV(0x7fc7a9004eb0) at 0x7fc7a9004ec0
  REFCNT = 1
  FLAGS = (TEMP,ROK)
  RV = 0x7fc7a9004b08
  ALLOCATED at -e:1 by anonlist (parent 0x0); serial 1720
  SV = PVAV(0x7fc7a9006270) at 0x7fc7a9004b08
    REFCNT = 1
    FLAGS = ()
    ARRAY = 0x0
    FILL = -1
    MAX = -1
    FLAGS = (REAL)

So you can break further down allocations of a single SV type. You can
find added fields' definition in sv.h. Note that this mode carries
some caveats - there're some use-after-free bugs when dumping
double-freed SV's (if you have ones).

Best regards,
Sergey Aleynikov


2017-08-20 22:25 GMT+03:00 fREW Schmidt <frioux@gmail.com>:
> Hi!
>
> I recently did a talk at YAPC about Scalability, Reliability, and
> Performance at ZipRecruiter.  One of the talking points was the
> difficulty of tracking down the cause of memory usage in Perl.  James
> Keenan reached out to me and asked if I could send this feedback to
> P5P in the hopes that the situation could be improved.
>
> It is definitely possible that I (and my coworkers and the people in
> the community that I reached out to) missed some tools or techniques,
> so if that's the case feel free to let me know and move on.
>
> The problem that we sometimes run into is that a process is using a
> lot of memory and we want to see why.  We have already ruled out a
> leak and instead believe that this is just a really high baseline for
> code (and presumably some caches getting pre-warmed or something.)
>
> So as far as I know the best tools out there are Devel::SizeMe
> (obliquely referenced in the talk) and Devel::MAT.  Both are tools for
> examining the current state of memory usage.  If I want to find out
> *which* code allocated all of this memory, as far as I know I am out
> of luck.  I could theoretically dump from MAT or SizeMe periodically
> and use that to bisect, but that's more work than I am interested in
> doing right now.
>
> If I had my druthers we would have two hooks in the Perl compiler:
>
>  * One that would get called when allocating from an already created
>    arena
>
>  * One that would get called when allocating memory from the operating
>    system
>
> Given these we could log a stacktrace, make a flame graph, whatever.
> I suspect that alone would give me what I needed, though I could see
> it being really expensive.
>
> Anyway, just an idea; even if it never gets implemented Perl will
> likely continue to be the primary programming language in use at
> ZipRecruiter.
>
> Thanks!
>
> P.S. please keep me on Cc or To as I am not subscribed to the list.
>
> --
> fREW Schmidt
> https://blog.afoolishmanifesto.com

• Debugging Memory Usage in Perl by fREW Schmidt
• Re: Debugging Memory Usage in Perl by Sergey Aleynikov