Front page | perl.perl5.porters |
Postings from March 2014
[perl #121490] Lock ordering issue; deadlock in malloc()/Perl_atfork_lock()
Thread Previous
|
Thread Next
From:
Philipp Rumpf
Date:
March 23, 2014 22:36
Subject:
[perl #121490] Lock ordering issue; deadlock in malloc()/Perl_atfork_lock()
Message ID:
rt-4.0.18-5004-1395507201-721.121490-75-0@perl.org
# New Ticket Created by Philipp Rumpf
# Please include the string: [perl #121490]
# in the subject line of all future correspondence about this issue.
# <URL: https://rt.perl.org/Ticket/Display.html?id=121490 >
This is a bug report for perl from prumpf@gmail.com,
generated with the help of perlbug 1.40 running under perl 5.19.11.
-----------------------------------------------------------------
Hi! I've run into a deadlock situation with the current git versions
of perl (5.19.11) and glibc (2.19), on x86_64-pc-linux-gnu with
ithreads and MY_MALLOC, though I've run into it with other setups
(recent Debian versions of Perl and glibc, no MY_MALLOC) as well. I
believe I've been able to track down the issue and come up with a
workaround, although I've not yet found the time to come up with a small
reproducible test case. Please feel free to ask me for one if it's
absolutely required, though, or ask for other information, and I'll do my
best.
In summary, the problem is inconsistent lock ordering between Perl's
PL_malloc_mutex and glibc's malloc/arena.c's list_lock. The situation
arises when one thread tries to fork() at the same time that another
thread calls malloc().
Perl runs pthread_atfork before the first malloc() makes glibc install
its atfork handlers, so fork() calls ptmalloc_lock_all() first, then
Perl_atfork_lock(). That means locking glibc's list_lock first, then
PL_malloc_mutex. (pthread_atfork() has LIFO semantics)
However, Perl's malloc implementation locks PL_malloc_mutex first,
then (sometimes) runs out of memory and calls the real malloc(), which
tries to lock list_lock. We thus have a race condition and a deadlock,
which I've seen in practice.
I believe this is fundamentally a glibc bug: its implementation of
pthread_atfork() behaves erratically depending on whether malloc() is
first called before or after pthread_atfork(). However, since the
broken versions of glibc are out there and multiplying, we should also
work around the issue in Perl itself.
The workaround should be as easy as including an extra
PerlMem_free(PerlMem_malloc(1024)) call before calling PTHREAD_ATFORK,
but gcc has started "optimizing" such (otherwise) useless calls. I've
found a deliberately duplicate call to perl_alloc() works, but that's
both a one-time memory leak and horribly ugly, and most likely breaks
whatever code uses PL_do_undump.
Nevertheless, I'll include it here, because most of the work was
probably in tracking down the bug, and fixing it should be easier,
even if I cannot presently think of a good fix.
diff --git a/ext/ExtUtils-Miniperl/lib/ExtUtils/Miniperl.pm
b/ext/ExtUtils-Miniperl/lib/ExtUtils/Miniperl.pm
index 730c565..a8092bf 100644
--- a/ext/ExtUtils-Miniperl/lib/ExtUtils/Miniperl.pm
+++ b/ext/ExtUtils-Miniperl/lib/ExtUtils/Miniperl.pm
@@ -129,6 +129,19 @@ main(int argc, char **argv, char **env)
* call PTHREAD_ATFORK() explicitly, but if and only if it hasn't
* been called at least once before in the current process.
* --GSAR 2001-07-20 */
+ /* There's a nasty race condition with the current versions of Perl and
+ * glibc: the call to PTHREAD_ATFORK in Perl's main() might be reached
+ * before the first malloc happens, in which
+ * case fork() locks malloc/arena.c's list_lock first, then tries to
lock
+ * PL_malloc_lock; another thread might have locked PL_malloc_lock
first,
+ * then tries to lock list_lock, resulting in a deadlock.
+ *
+ * A proper fix would be in glibc, ensuring that ptmalloc_init() is
called
+ * earlier, but a workaround is to make a malloc call ourselves. */
+ /* This leaks memory, but works. */
+ (void)perl_alloc();
+ /* This doesn't leak memory, but is optimized away by gcc */
+ PerlMem_free(PerlMem_malloc(1024));
PTHREAD_ATFORK(Perl_atfork_lock,
Perl_atfork_unlock,
Perl_atfork_unlock);
[Please do not change anything below this line]
-----------------------------------------------------------------
---
Flags:
category=core
severity=medium
---
Site configuration information for perl 5.19.11:
Configured by pip at Sat Mar 22 10:40:51 UTC 2014.
Summary of my perl5 (revision 5 version 19 subversion 11) configuration:
Derived from: b51c3e77dbb7e510319342a73163b3fbb59baf5a
Platform:
osname=linux, osvers=3.12-1-amd64, archname=x86_64-linux-thread-multi
uname='linux philadelphia 3.12-1-amd64 #1 smp debian 3.12.8-1
(2014-01-19) x86_64 gnulinux '
config_args='-er'
hint=previous, useposix=true, d_sigaction=define
useithreads=define, usemultiplicity=define
use64bitint=define, use64bitall=define, uselongdouble=undef
usemymalloc=y, bincompat5005=undef
Compiler:
cc='cc', ccflags ='-D_REENTRANT -D_GNU_SOURCE -fno-strict-aliasing
-pipe -fstack-protector -I/usr/local/include -D_LARGEFILE_SOURCE
-D_FILE_OFFSET_BITS=64',
optimize='-O2 -g',
cppflags='-D_REENTRANT -D_GNU_SOURCE -fno-strict-aliasing -pipe
-fstack-protector -I/usr/local/include -D_REENTRANT -D_GNU_SOURCE
-fno-strict-aliasing -pipe -fstack-protector -I/usr/local/include
-D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -D_REENTRANT -D_GNU_SOURCE
-fno-strict-aliasing -pipe -fstack-protector -I/usr/local/include
-D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -DPERL_POISON -D_REENTRANT
-D_GNU_SOURCE -fno-strict-aliasing -pipe -fstack-protector
-I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64
-D_REENTRANT -D_GNU_SOURCE -fno-strict-aliasing -pipe -fstack-protector
-I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64
-D_REENTRANT -D_GNU_SOURCE -fno-strict-aliasing -pipe -fstack-protector
-I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64
-D_REENTRANT -D_GNU_SOURCE -fno-strict-aliasing -pipe -fstack-protector
-I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64
-D_REENTRANT -D_GNU_SOURCE -fno-strict-aliasing -pipe -fstack-protector
-I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64'
ccversion='', gccversion='4.8.2', gccosandvers=''
intsize=4, longsize=8, ptrsize=8, doublesize=8, byteorder=12345678
d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=16
ivtype='long', ivsize=8, nvtype='double', nvsize=8, Off_t='off_t',
lseeksize=8
alignbytes=8, prototype=define
Linker and Libraries:
ld='cc', ldflags =' -fstack-protector -L/usr/local/lib'
libpth=/usr/local/lib /usr/lib/gcc/x86_64-linux-gnu/4.8/include-fixed
/usr/include/x86_64-linux-gnu /usr/lib /lib/x86_64-linux-gnu /lib/../lib
/usr/lib/x86_64-linux-gnu /usr/lib/../lib /lib /usr/local/lib
/usr/lib/gcc/x86_64-linux-gnu/4.8/include-fixed
/usr/include/x86_64-linux-gnu /usr/lib
libs=-lnsl -ldl -lm -lcrypt -lutil -lpthread -lc
perllibs=-lnsl -ldl -lm -lcrypt -lutil -lpthread -lc
libc=libc-2.17.so, so=so, useshrplib=false, libperl=libperl.a
gnulibc_version='2.18'
Dynamic Linking:
dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef, ccdlflags='-Wl,-E'
cccdlflags='-fPIC', lddlflags='-shared -O2 -L/usr/local/lib
-fstack-protector'
Locally applied patches:
uncommitted-changes
---
@INC for perl 5.19.11:
/usr/local/lib/perl5/site_perl/5.19.10/x86_64-linux-thread-multi
/usr/local/lib/perl5/site_perl/5.19.10
/usr/local/lib/perl5/5.19.10/x86_64-linux-thread-multi
/usr/local/lib/perl5/5.19.10
.
---
Environment for perl 5.19.11:
HOME=/home/pip
LANG=en_US.UTF-8
LANGUAGE (unset)
LC_ALL=en_US.utf8
LC_CTYPE=
LD_LIBRARY_PATH (unset)
LOGDIR (unset)
PATH=/bin:/usr/bin:/sbin:/usr/sbin:/usr/local/bin:/usr/local/sbin
PERL_BADLANG (unset)
SHELL=/bin/zsh-beta
Thread Previous
|
Thread Next