Re: memory problem of "eval" in perl 5.10?

On Thu, Nov 26, 2009 at 08:54:48PM +0100, christian4perl@lists.muthpartners.de wrote:
> Hello,
> 
> We have a problem with the eval of the "new" perl 5.10.
> 
> Before, a brief background. We have run several applications which were 
> developed with design patterns, especially the state machine. In 
> conjunction with a data flow logic, analogous to the Unix filter. Perl’s 
> internal state of the state machine will be dumped at the end of every 
> "step" (Data:: Dumper), and stored in flat files or database CLOB 
> columns. Later, these dumps are reread, evaluated and then the work on 
> the data will continue.
> 
> Now it seems, that the "eval" from perl 5.10 are partially being very 
> rude with the available memory resources. It gives a mismatch of 1:1000; 
> every byte to be evaluated, needs 1Kbyte main memory. That means, the 
> data which could be processed with perl 5.8.8, with perl 5.10.0 creates 
> an "out of memory" (just as all makable 5.9.*). For reproducing tests, 
> the script eval.pl attached. The offender (eval) is on line 65.
> 
> The function memory () and expandSI () are to determine the memory 
> consumption, as it sees the operating system (it works under linux 
> only). So that eval.pl has something to do, we also attached the support 
> script get.pl. It loads eight outer space images from the Internet and 
> creates files with the changed Data::Dumper HTTP::Response instances 
> (lines 72-75 of get.pl).
> 
> We presume that the evaluation has a caching mechanism. Because a 
> functional evaluation repeats often; no new memory is required. Even two 
> different evaluations are possible repeatedly, therefore there is enough 
> space for everyone alone.
> 
> However, most of our application steps evaluate smaller structures. It 
> never takes a long time. The computer will interactively be tough and 
> will relatively quickly get the application from linux as an "out of 
> memory".
> 
> Can you help us?
> 
> (a) Is there an error in the procedure? Although the applications run 
> under perl 5.6 and 5.8 for many of years.
> 
> (b) Does it require different "parameters" of the modules or compiling 
> it?
> 
> (c) How can we find a solution to our problem.


Short answer: when evaling the string which was read in from the file that
was created using Data::Dumper, avoid using using a variable called '$obj';
i.e.  in  these lines:

  my	$obj ;
  ...
  $obj = <$fh> ;
  ...
  $obj = eval $obj ;
  ...
  return $obj ;

rename the variable, eg

  my	$data ;
  ...
  $data = <$fh> ;
  ...
  {
    my $obj;
    $data = eval $data ;
  }
  ...
  return $data ;

Long answer:

The file created by Data::Dumper looks roughly like:

    $obj = {
	    content => "multi-megabyte string containing 1000s of ' chars",
	    foo      => $obj->{bar},
    }


You read the file into $obj then eval it. The "$obj->{bar}" bit makes the
string be treated as a symbolic reference, so it is treated as a variable
name where "'" is the old perl-4 equivalent of the package separator "::",
so it tries to create a new variable that is nested thousands deep
package-wise, and uses all available memory to create all the packages.

Other answer:

Don't use Data::Dumper for dumping large binary data; you're probably
better off using Storable.

p5pers:

I'm still a bit confused by the fact that the OP's code worked in 5.8.x
but not 5.10.x; all my attempts to reduce the code soon got 5.8.x eating
memory too. Also, the memory usage seems proportional to the *square* of
the number of "'" in the variable name, e.g. in the following code:

    my $quotes = "'" x 4000;
    $quotes->{foo};
    system "ps -flyp $$";

double it to 8000 and memory usage quadruples.

Also, I'm not clear why Data::Dumper produces output like

    $obj = { foo  => $obj->{bar} }

since evalling this doesn't regenerate the circular reference.

-- 
Overhead, without any fuss, the stars were going out.
    -- Arthur C Clarke

• memory problem of "eval" in perl 5.10? by christian4perl
• Re: memory problem of "eval" in perl 5.10? by Dave Mitchell
• Re[2]: memory problem of "eval" in perl 5.10? Solved (with one new question ;-)) by christian4perl
• Re: Re[2]: memory problem of "eval" in perl 5.10? Solved (with one new question ;-)) by Eric Brine