develooper Front page | perl.perl5.porters | Postings from August 2013

Re: [perl #119445] performance bug: perl Thread::Queue is 20x slowerthan Unix pipe

Thread Previous | Thread Next
Leon Timmermans
August 27, 2013 10:12
Re: [perl #119445] performance bug: perl Thread::Queue is 20x slowerthan Unix pipe
Message ID:
On Mon, Aug 26, 2013 at 5:58 PM, John Heidemann <> wrote:

>  I understand that Thread::Queue and perl threads allow shared data, and
> that
> that's much more than a pipe.
> My concern is that Thread::Queue also *forces* shared data, even when
> it's not rqeuired.  If that sharing comes with a 20x performance hit,
> that should be clear.
> From perlthrtut, the "Pipeline" model
>        The pipeline model divides up a task into a series of steps, and
> passes
>        the results of one step on to the thread processing the next.  Each
>        thread does one thing to each piece of data and passes the results
> to
>        the next thread in line.
> For the pipeline model, one does not need repeated sharing, just a
> one-time hand-off.  Each queue is FIFO with data touched by only one
> thread at a time.  That's exactly what my particular applications needs
> to do.
> But one does not *want* sharing (for the pipeline model) there if it's a
> 20x performance hit.
> If the statement is that queues should require shared data and the
> corresponding performance hit, that's a design choice one could make.
> Then I'd suggest the bug becomes: perlthrtut should say "don't use
> Thread::Queue for the pipeline model if you expect high performance,
> roll your own IPC".

Actually I did write a queue implementation for threads::lite that should
be a lot faster for simple data structures, but I never released it as a
separate module that could be used with

> Alternatively, I'd love some mechanism to share data between threads
> that allows a one-time handoff (not repeated sharing) with pipe-like
> performance.  One would *think* that shared memory should be able to be
> faster than round-tripping through a pipe (with perl parsing and kernel
> IO).  It seems like a shame that perl is forcing full-on sharing since
> it's slow and not required (in this case).

I don't think that would be faster than a queue, given perl's memory model
(memory has to be owned by a thread, shared memory has to be be handled
manually) a copy or two is necessary anyway.


Thread Previous | Thread Next Perl Programming lists via nntp and http.
Comments to Ask Bjørn Hansen at | Group listing | About