develooper Front page | perl.perl5.porters | Postings from January 2019

[perl #128227] vfork should be used for spawning external processes

From:
slaven@rezic.de via RT
Date:
January 29, 2019 10:18
Subject:
[perl #128227] vfork should be used for spawning external processes
Message ID:
rt-4.0.24-18305-1548757130-991.128227-15-0@perl.org
Dana Mon, 28 Jan 2019 02:55:53 -0800, Mark@Overmeer.net reče:
> * Eric Wong (e@80x24.org) [190128 09:23]:
> > I hit ENOMEM when attempting to spawn one small subprocess from
> > a giant Perl process.  I had enough RAM for the small subprocess
> > (but not enough for a full fork of the giant Perl process).
> 
> Well, reading fork(2) tells me
> 
>    ENOMEM   fork()  failed to allocate the necessary kernel structures
>             because memory is tight.
> 
> It's not directly about the normal RAM used by the giant process, but
> about the RAM used for kernel administration, which may have restrictions.
> 
> You may be able to find an error or warning in dmesg about which resource
> is really the limitation the kernel bounces into.  That might show that
> your processor is too small for the task (number of page tables, DMA
> range, ...)

I doubt it's about kernel memory. Just tried a experiment on a recent Linux system (debian/stretch) with 32GB RAM + 1 GB Swap (at the moment used memory by other processes: < 1GB). The following script just allocates memory and then tries a simple system() call:

#!/usr/bin/perl

use strict;

my $alloc_gb = shift || 4;

my $mb = " " x 1024**2;
my $buf;
for (1..$alloc_gb*1024) {
    $buf .= $mb;
}

system 'echo', 'system() was successful';
print "Return values of system() call: \$?=$?, \$!=$!\n";

__END__

Running script allocating 16GB of RAM:

    $ perl /tmp/m.pl 16
    Return values of system() call: $?=0, $!=Cannot allocate memory

No logging happens in any of the files in /var/log at this time.

Running the script using with 15GB is successful.

Now another script using spawn() provided by POSIX::RT::Spawn (which unfortunately does not build anymore for perl >= 5.28.0):

#!/usr/bin/perl

use strict;
use POSIX::RT::Spawn;

my $alloc_gb = shift || 4;

my $mb = " " x 1024**2;
my $buf;
for (1..$alloc_gb*1024) {
    $buf .= $mb;
}

my $pid = spawn 'echo', 'spawn() was successful'
    or die "failed to spawn: $!";
waitpid $pid, 0;
die "command failed with status: ", $?>>8 if $?;

__END__

No problem running this script with 28GB.

Regards,
    Slaven


---
via perlbug:  queue: perl5 status: open
https://rt.perl.org/Ticket/Display.html?id=128227



nntp.perl.org: Perl Programming lists via nntp and http.
Comments to Ask Bjørn Hansen at ask@perl.org | Group listing | About