Front page | perl.perl5.porters |
Postings from August 2013
[perl #119445] performance bug: perl Thread::Queue is 20x slower than Unix pipe
Thread Previous
|
Thread Next
From:
johnh @ isi . edu
Date:
August 26, 2013 20:28
Subject:
[perl #119445] performance bug: perl Thread::Queue is 20x slower than Unix pipe
Message ID:
rt-3.6.HEAD-1873-1377304081-1315.119445-75-0@perl.org
# New Ticket Created by johnh@isi.edu
# Please include the string: [perl #119445]
# in the subject line of all future correspondence about this issue.
# <URL: https://rt.perl.org:443/rt3/Ticket/Display.html?id=119445 >
This is a bug report for perl from johnh@isi.edu,
generated with the help of perlbug 1.39 running under perl 5.16.3.
-----------------------------------------------------------------
Why is Thread::Queue *so* slow?
I understand it has to do locking and be careful about data
structures, but it seems like it is about 20x slower than opening up a
Unix pipe, printing to that, reading it back and parsing the result.
Thread::Queue is correct, but I suggest that 20x slower is a performance bug.
One would think that IPC through memory would be at least as fast as a
pipe through the kernel, and ideally it should be faster.
Here's timing of a test program that sends 500k integers between two threads,
using Thread::Queue or pipe(2).
$ ./thread_ipc_perf.pl -m queue
benchmark took 14 wallclock secs (14.71 usr + 2.51 sys = 17.22 CPU) @ 0.06/s (n=1)
$ ./thread_ipc_perf.pl -m pipe
benchmark took 0 wallclock secs ( 0.59 usr + 0.00 sys = 0.59 CPU) @ 1.69/s (n=1)
Here's a larger run (1M integers) with the same kind of results.
$ ./thread_ipc_perf.pl -N 1000000 -m queue
benchmark took 30 wallclock secs (32.69 usr + 6.06 sys = 38.75 CPU) @ 0.03/s (n=1)
$ ./thread_ipc_perf.pl -N 1000000 -m pipe
benchmark took 1 wallclock secs ( 1.23 usr + 0.00 sys = 1.23 CPU) @ 0.81/s (n=1)
Source code for the above simple benchmark is at
http://www.isi.edu/~johnh/SOFTWARE/FSDB/thread_ipc_perf.pl.txt
We can quibble over the exact multiplier (maybe it's only 15x slower),
but it's *really* slow.
Any suggestions? I get similar results if I simplify Thread::Queue to
bare minimum code.
To speculate, I'm thinking the cost is in making all IPC data shared.
It would be great if one could have data that is sent over
Thread::Queue that is copied, not shared.
Thanks for any suggestions,
-John Heidemann
[Please do not change anything below this line]
-----------------------------------------------------------------
---
Flags:
category=library
severity=medium
module=Thread::Queue
---
Site configuration information for perl 5.16.3:
Configured by Red Hat, Inc. at Tue Jun 18 09:17:09 UTC 2013.
Summary of my perl5 (revision 5 version 16 subversion 3) configuration:
Platform:
osname=linux, osvers=2.6.32-358.2.1.el6.x86_64, archname=x86_64-linux-thread-multi
uname='linux buildvm-05.phx2.fedoraproject.org 2.6.32-358.2.1.el6.x86_64 #1 smp wed feb 20 12:17:37 est 2013 x86_64 x86_64 x86_64 gnulinux '
config_args='-des -Doptimize=-O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic -Dccdlflags=-Wl,--enable-new-dtags -Dlddlflags=-shared -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic -Wl,-z,relro -DDEBUGGING=-g -Dversion=5.16.3 -Dmyhostname=localhost -Dperladmin=root@localhost -Dcc=gcc -Dcf_by=Red Hat, Inc. -Dprefix=/usr -Dvendorprefix=/usr -Dsiteprefix=/usr/local -Dsitelib=/usr/local/share/perl5 -Dsitearch=/usr/local/lib64/perl5 -Dprivlib=/usr/share/perl5 -Dvendorlib=/usr/share/perl5/vendor_perl -Darchlib=/usr/lib64/perl5 -Dvendorarch=/usr/lib64/perl5/vendor_perl -Darchname=x86_64-linux-thread-multi -Dlibpth=/usr/local/lib64 /lib64 /usr/lib64 -Duseshrplib -Dusethreads -Duseithreads -Dusedtrace=/usr/bin/dtrace -Duselargefiles -Dd_semctl_semun -Di_db -Ui_ndbm -Di_gdbm -Di_shadow -Di_sysl!
og -Dman3
ext=3pm -Duseperlio -Dinstallusrbinperl=n -Ubincompat5005 -Uversiononly -Dpager=/usr/bin/less -isr -Dd_gethostent_r_proto -Ud_endhostent_r_proto -Ud_sethostent_r_proto -Ud_endprotoent_r_proto -Ud_setprotoent_r_proto -Ud_endservent_r_proto -Ud_setservent_r_proto -Dscriptdir=/usr/bin -Dusesitecustomize'
hint=recommended, useposix=true, d_sigaction=define
useithreads=define, usemultiplicity=define
useperlio=define, d_sfio=undef, uselargefiles=define, usesocks=undef
use64bitint=define, use64bitall=define, uselongdouble=undef
usemymalloc=n, bincompat5005=undef
Compiler:
cc='gcc', ccflags ='-D_REENTRANT -D_GNU_SOURCE -fno-strict-aliasing -pipe -fstack-protector -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64',
optimize='-O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic',
cppflags='-D_REENTRANT -D_GNU_SOURCE -fno-strict-aliasing -pipe -fstack-protector -I/usr/local/include'
ccversion='', gccversion='4.8.1 20130603 (Red Hat 4.8.1-1)', gccosandvers=''
intsize=4, longsize=8, ptrsize=8, doublesize=8, byteorder=12345678
d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=16
ivtype='long', ivsize=8, nvtype='double', nvsize=8, Off_t='off_t', lseeksize=8
alignbytes=8, prototype=define
Linker and Libraries:
ld='gcc', ldflags =' -fstack-protector'
libpth=/usr/local/lib64 /lib64 /usr/lib64
libs=-lresolv -lnsl -lgdbm -ldb -ldl -lm -lcrypt -lutil -lpthread -lc -lgdbvm_compat
perllibs=-lresolv -lnsl -ldl -lm -lcrypt -lutil -lpthread -lc
libc=, so=so, useshrplib=true, libperl=libperl.so
gnulibc_version='2.17'
Dynamic Linking:
dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef, ccdlflags='-Wl,--enable-new-dtags -Wl,-rpath,/usr/lib64/perl5/CORE'
cccdlflags='-fPIC', lddlflags='-shared -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic -Wl,-z,relro '
Locally applied patches:
---
@INC for perl 5.16.3:
/usr/local/lib64/perl5
/usr/local/share/perl5
/usr/lib64/perl5/vendor_perl
/usr/share/perl5/vendor_perl
/usr/lib64/perl5
/usr/share/perl5
.
---
Environment for perl 5.16.3:
HOME=/home/johnh
LANG=en_US.UTF-8
LANGUAGE (unset)
LD_LIBRARY_PATH=/usr/local/lib
LOGDIR (unset)
PATH=/bin:/usr/bin:/usr/local/sbin:/etc:/sbin:/usr/sbin
PERL_BADLANG (unset)
SHELL=/bin/bash
Thread Previous
|
Thread Next