[perl #121974] [PATCH] Module::CoreList::TieHashDelta stop massive recursion

On Wed May 28 05:45:02 2014, bulk88 wrote:
> Ran into a problem with Module::CoreList::TieHashDelta when working on
> https://rt.perl.org/Ticket/Display.html?id=121923 "RFC/WIP replace
> Perl stack with self allocating/expanding (C-style) stack on Win32" I
> ran into a series of tests that ate up all process VM space and caused
> an out of memory exit. They are were related to Module::CoreList. The
> problem is, the "exists" magic, worst case, can recurse, "X perl
> releases ever" deep. Each tied call out frame swaps perl stacks with "
> PUSHSTACKi(PERLSI_MAGIC);", do this enough times on "self allocating
> stack Perl" and its OOM. I included a C callstack showing the problem.
> The callstack ends at about 28 tie frames deep since at that point it
> OOMs. Its probably deeper. Removing the magic/tie recursion fixed all
> the OOMing tests. Even if self allocating stack Perl never comes to
> be, this patch will make Module::CoreList faster, less memory, etc.
> 
> While debugging Module::CoreList::TieHashDelta and Module::CoreList,
> it looked to me like it does N^2 ($releases_ever_of_perl *
> $releases_ever_of_perl) searching in first_release_by_date and
> first_release_raw by the number of EXISTS calls. I am not interesting
> in optimizing Module::CoreList, but if someone else is interested, I
> suggest they step into EXISTS and make sure the searching algo is ok.

Thanks, applied as 42484eedeb4abcee1d893010daa67c1a42ec6424.

I did a simple test to see how deep the recursion went, with C<< corelist CGI >> the maximum depth was 64, if your stack replacement can't handle that, it's not suitable for core.

It does look like the process could use some caching - EXISTS was called with duplicate parameters several times for my CGI test.

Tony


---
via perlbug:  queue: perl5 status: open
https://rt.perl.org/Ticket/Display.html?id=121974

• [perl #121974] [PATCH] Module::CoreList::TieHashDelta stop massive recursion by Tony Cook via RT