develooper Front page | perl.perl5.porters | Postings from August 2013

Re: [perl #119445] performance bug: perl Thread::Queue is 20x slowerthan Unix pipe

Thread Previous | Thread Next
Nicholas Clark
August 30, 2013 13:31
Re: [perl #119445] performance bug: perl Thread::Queue is 20x slowerthan Unix pipe
Message ID:
On Tue, Aug 27, 2013 at 05:15:09PM -0700, John Heidemann wrote:
> On Tue, 27 Aug 2013 11:18:57 +0100, Nicholas Clark wrote: 

> >Do you have a suggestion for where we should document this, such that you
> >would have read it had it been there? (Even better if you can suggest a
> >suitable change)
> A proposed patch to perlthrtut is attached at the end of this message.


> It's too bad there's no way to move data between two threads without
> making the data shared (other than the move).  A one-time copy from
> thread A to B.  C-only programs have done this for ages (see for
> example, "The Duality of Memory and Communication in the
> Implementation of a Multiprocessor Operating
> System" by Young et al, ACM SOSP 1987).

Agree that's it's frustrating.

That paper seems to predate Perl 1 by about 5 weeks, but I don't think that
the complexity trade off to facilitate concurrency became a concern of
mainstream development until some point after Perl 5 shipped in 1994.
By which time, of course, it's too late to add it in from the start.
(And the Perl 5 codebase is a rewrite of Perl 4, which traces history all the
way back to Perl 1, so really it needed to be in by December 1987 to be

I feel that it's the same fundamental problem as attempting to retrofit
Unicode support. Bolting it on later will never work completely - it has to
be in the design from the start.

> ----------------------------------------------------------------------
> --- perlthrtut.pod-	2013-08-27 08:47:16.347167972 -0700
> +++ perlthrtut.pod	2013-08-27 08:53:26.159772710 -0700
> @@ -465,6 +465,13 @@
>  data inconsistency and race conditions. Note that Perl will protect its
>  internals from your race conditions, but it won't protect you from you.
> +=head2 Thread Pitfalls: Performance
> +
> +Shared data is and locking expensive, slowing down access.
> +As of perl 5.18, one should expect sharing data between threads
> +with tools such as L<Thread::Queue> to be about 15-20x slower
> +than copying the data through L<pipe(2)>.
> +
>  =head1 Synchronization and control
>  Perl provides a number of mechanisms to coordinate the interactions

On Wed, Aug 28, 2013 at 11:30:58PM -0400, Ronald J Kimball wrote:

> I think this sentence got a bit mixed up.

I think also that it should mention your insight about what's not obvious
about performance - lack of handoff. I don't think that the performance has
changed much historically, and I foresee a way to change it in the future,
so I think that having a version number in there isn't that useful.
So this instead?

    Shared data and locking are expensive, slowing down access.
    Perl 5 has no way of passing ownership of data between threads, so all
    thread operations involve data becoming shared. One should expect sharing
    data between threads with tools such as L<Thread::Queue> to be about
    15-20x slower than copying the data through L<pipe(2)>.

If in the future someone does radically improve thread performance, then I'd
expect them to revisit the documentation to update it with new figures
(and publicise their success).

Nicholas Clark

Thread Previous | Thread Next Perl Programming lists via nntp and http.
Comments to Ask Bjørn Hansen at | Group listing | About