develooper Front page | perl.perl5.porters | Postings from December 2019

[PATCH] use posix_spawn [was: [perl #128227] vfork should be usedfor spawning external processes]

Thread Next
Eric Wong
December 7, 2019 06:11
[PATCH] use posix_spawn [was: [perl #128227] vfork should be usedfor spawning external processes]
Message ID:
Leon Timmermans via RT <> wrote:
> It may actually be sane to implement system on top of posix_spawn though
> (which can use vfork internally). That at least can be reasoned about, and
> seems to generally do what we want (I've never actually used it, there may
> be pitfalls that aren't obvious yet).
> Not sure either option is really worth the development effort though, any
> program where fork is a serious bottleneck is likely to be a mess anyway
> unless you're writing a shell or make.

I'm getting rid of a GNU make dependency somewhere and
relying on Perl5 for portability.

This is my first experience hacking with in the Perl5
interpreter and also my first time using posix_spawn /
posix_spawnp functions.  I'm still learning my way around
perlguts and such...

For old *nix hackers, the posix_spawn* functions are part of the
newer breed of functions like posix_{memalign,fadvise,madvise}
and pthread_* functions which return error numbers rather than
setting `errno' like traditional functions.

If posix_spawn uses vfork/CLONE_VFORK (as it does by default
with glibc 2.24+), this would allow Perl to implement
`backtick`, system(), and pipe opens on machines without an MMU.

It also increases performance for normal systems with small
scripts, but the effect is more pronounced on large scripts.
Numbers using the attached script on a laptop
(glibc 2.28 on x86-64 Linux):

default small process:

	spawn - ~0.9s
	blead - ~1.1s

bigger process (setting PAD_NR=50000000 in env):

	spawn - ~0.9s
	blead - ~4.1s

And 50MB isn't particularly big these days on amd64.  I think
FreeBSD should see an improvement, too (will confirm next week
when power is restored to my dev VM).

The only incompatibility which I had to workaround was to
implement the /bin/sh fallback on ENOEXEC.  execvp(3) falls back
to using /bin/sh on ENOEXEC, but posix_spawn(3) does not always
(glibc did up until 2.15).  And the workaround I put in for that
has a minor, (but IMHO insignificant) TOCTOU race (see comments
in code), though.

I'd still like to clean this up a bit, but I figure I'd post
this now in case something bad happens to me in bad weather.  I
won't be online much for a few days, but if I survive and return
next week, I'll see about some TODO items and any comments on
the code you guys might have.


- integrate with build system, metaconfig (right now it's
  only enabled for FreeBSD and glibc.  I haven't figured out
  metaconfig, yet (or can somebody else take care of that?)

- maybe reduce code duplication between Perl_do_aexec3,
  do_posix_spawn_popen, and do_spawn_posix...

- it would be good to eventually get the fork+execve paths
  modeled after posix_spawn

- system/popen could eventually support options like chdir,
  arbitrary redirects, open/close...  That's a long-term thing,

Fwiw, I want to get setrlimit supported in glibc posix_spawn,
and hopefuly into POSIX:
So I think I might work on that for glibc, next, and at some point
I'd like it in Perl, too.

test_harness passes.  Patches attached, and also available via
git HTTPS or Tor .onion

(note: my domain is likely going away in a few years
 since I won't pay .org extortionists; not sure what I'll do
 when they move on to extorting .net after that, too...)

The following changes since commit d8b6bcdacb63bae3a33687b2b1b1bac052800769:

  Update IO-Compress to CPAN version 2.092 (2019-12-06 14:44:28 +0000)

are available in the Git repository at: spawn-v1
  http://ou63pmih66umazou.onion/perl.git spawn-v1

for you to fetch changes up to 935b42974ebe131d956c9bd32f961cf95bf53101:

  use posix_spawn for pp_system (2019-12-07 05:23:34 +0000)

Eric Wong (2):
      use posix_spawn for popen_list and popen
      use posix_spawn for pp_system

 pp_sys.c | 280 ++++++++++++++++++++++++++++++++++-
 util.c   | 510 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++-
 2 files changed, 786 insertions(+), 4 deletions(-)

P.S. don't bother trying to convince me to use proprietary platforms
     or JavaScript.  Everybody's tried, everybody's failed :P

Thread Next Perl Programming lists via nntp and http.
Comments to Ask Bjørn Hansen at | Group listing | About