develooper Front page | perl.perl5.porters | Postings from August 2013

[perl #119445] performance bug: perl Thread::Queue is 20x slower than Unix pipe

Thread Previous | Thread Next
johnh @ isi . edu
August 26, 2013 20:28
[perl #119445] performance bug: perl Thread::Queue is 20x slower than Unix pipe
Message ID:
# New Ticket Created by 
# Please include the string:  [perl #119445]
# in the subject line of all future correspondence about this issue. 
# <URL: >

This is a bug report for perl from,
generated with the help of perlbug 1.39 running under perl 5.16.3.


Why is Thread::Queue *so* slow?

I understand it has to do locking and be careful about data
structures, but it seems like it is about 20x slower than opening up a
Unix pipe, printing to that, reading it back and parsing the result.

Thread::Queue is correct, but I suggest that 20x slower is a performance bug.

One would think that IPC through memory would be at least as fast as a
pipe through the kernel, and ideally it should be faster.

Here's timing of a test program that sends 500k integers between two threads,
using Thread::Queue or pipe(2).

$ ./ -m queue
benchmark took 14 wallclock secs (14.71 usr +  2.51 sys = 17.22 CPU) @  0.06/s (n=1)

$ ./ -m pipe
benchmark took  0 wallclock secs ( 0.59 usr +  0.00 sys =  0.59 CPU) @  1.69/s (n=1)

Here's a larger run (1M integers) with the same kind of results.

$ ./ -N 1000000 -m queue
benchmark took 30 wallclock secs (32.69 usr +  6.06 sys = 38.75 CPU) @  0.03/s (n=1)

$ ./ -N 1000000 -m pipe
benchmark took  1 wallclock secs ( 1.23 usr +  0.00 sys =  1.23 CPU) @  0.81/s (n=1)

Source code for the above simple benchmark is at

We can quibble over the exact multiplier (maybe it's only 15x slower),
but it's *really* slow.

Any suggestions?  I get similar results if I simplify Thread::Queue to
bare minimum code.  

To speculate, I'm thinking the cost is in making all IPC data shared.
It would be great if one could have data that is sent over
Thread::Queue that is copied, not shared.

Thanks for any suggestions,
   -John Heidemann

[Please do not change anything below this line]
Site configuration information for perl 5.16.3:

Configured by Red Hat, Inc. at Tue Jun 18 09:17:09 UTC 2013.

Summary of my perl5 (revision 5 version 16 subversion 3) configuration:
    osname=linux, osvers=2.6.32-358.2.1.el6.x86_64, archname=x86_64-linux-thread-multi
    uname='linux 2.6.32-358.2.1.el6.x86_64 #1 smp wed feb 20 12:17:37 est 2013 x86_64 x86_64 x86_64 gnulinux '
    config_args='-des -Doptimize=-O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -grecord-gcc-switches  -m64 -mtune=generic -Dccdlflags=-Wl,--enable-new-dtags -Dlddlflags=-shared -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -grecord-gcc-switches  -m64 -mtune=generic -Wl,-z,relro  -DDEBUGGING=-g -Dversion=5.16.3 -Dmyhostname=localhost -Dperladmin=root@localhost -Dcc=gcc -Dcf_by=Red Hat, Inc. -Dprefix=/usr -Dvendorprefix=/usr -Dsiteprefix=/usr/local -Dsitelib=/usr/local/share/perl5 -Dsitearch=/usr/local/lib64/perl5 -Dprivlib=/usr/share/perl5 -Dvendorlib=/usr/share/perl5/vendor_perl -Darchlib=/usr/lib64/perl5 -Dvendorarch=/usr/lib64/perl5/vendor_perl -Darchname=x86_64-linux-thread-multi -Dlibpth=/usr/local/lib64 /lib64 /usr/lib64 -Duseshrplib -Dusethreads -Duseithreads -Dusedtrace=/usr/bin/dtrace -Duselargefiles -Dd_semctl_semun -Di_db -Ui_ndbm -Di_gdbm -Di_shadow -Di_sysl!
 og -Dman3
 ext=3pm -Duseperlio -Dinstallusrbinperl=n -Ubincompat5005 -Uversiononly -Dpager=/usr/bin/less -isr -Dd_gethostent_r_proto -Ud_endhostent_r_proto -Ud_sethostent_r_proto -Ud_endprotoent_r_proto -Ud_setprotoent_r_proto -Ud_endservent_r_proto -Ud_setservent_r_proto -Dscriptdir=/usr/bin -Dusesitecustomize'
    hint=recommended, useposix=true, d_sigaction=define
    useithreads=define, usemultiplicity=define
    useperlio=define, d_sfio=undef, uselargefiles=define, usesocks=undef
    use64bitint=define, use64bitall=define, uselongdouble=undef
    usemymalloc=n, bincompat5005=undef
    cc='gcc', ccflags ='-D_REENTRANT -D_GNU_SOURCE -fno-strict-aliasing -pipe -fstack-protector -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64',
    optimize='-O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic',
    cppflags='-D_REENTRANT -D_GNU_SOURCE -fno-strict-aliasing -pipe -fstack-protector -I/usr/local/include'
    ccversion='', gccversion='4.8.1 20130603 (Red Hat 4.8.1-1)', gccosandvers=''
    intsize=4, longsize=8, ptrsize=8, doublesize=8, byteorder=12345678
    d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=16
    ivtype='long', ivsize=8, nvtype='double', nvsize=8, Off_t='off_t', lseeksize=8
    alignbytes=8, prototype=define
  Linker and Libraries:
    ld='gcc', ldflags =' -fstack-protector'
    libpth=/usr/local/lib64 /lib64 /usr/lib64
    libs=-lresolv -lnsl -lgdbm -ldb -ldl -lm -lcrypt -lutil -lpthread -lc -lgdbvm_compat
    perllibs=-lresolv -lnsl -ldl -lm -lcrypt -lutil -lpthread -lc
    libc=, so=so, useshrplib=true,
  Dynamic Linking:
    dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef, ccdlflags='-Wl,--enable-new-dtags -Wl,-rpath,/usr/lib64/perl5/CORE'
    cccdlflags='-fPIC', lddlflags='-shared -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic -Wl,-z,relro '

Locally applied patches:

@INC for perl 5.16.3:

Environment for perl 5.16.3:
    LANGUAGE (unset)
    LOGDIR (unset)
    PERL_BADLANG (unset)

Thread Previous | Thread Next Perl Programming lists via nntp and http.
Comments to Ask Bjørn Hansen at | Group listing | About