develooper Front page | perl.perl5.porters | Postings from August 2013

Re: [perl #119445] performance bug: perl Thread::Queue is 20x slowerthan Unix pipe

Thread Previous | Thread Next
From:
Nicholas Clark
Date:
August 27, 2013 10:19
Subject:
Re: [perl #119445] performance bug: perl Thread::Queue is 20x slowerthan Unix pipe
Message ID:
20130827101857.GE66035@plum.flirble.org
On Mon, Aug 26, 2013 at 08:58:14AM -0700, John Heidemann wrote:

> My concern is that Thread::Queue also *forces* shared data, even when
> it's not rqeuired.  If that sharing comes with a 20x performance hit,
> that should be clear.

Yes, I agree that that's a valid concern, and we could document that better.

As someone rather too close to the code, it's not easy to pull back far
enough to work out where someone reading the documentation for the first
time would have expected to have found such a warning.

Do you have a suggestion for where we should document this, such that you
would have read it had it been there? (Even better if you can suggest a
suitable change)

> Alternatively, I'd love some mechanism to share data between threads
> that allows a one-time handoff (not repeated sharing) with pipe-like
> performance.  One would *think* that shared memory should be able to be
> faster than round-tripping through a pipe (with perl parsing and kernel
> IO).  It seems like a shame that perl is forcing full-on sharing since
> it's slow and not required (in this case).

Agree, I'd love this too. It would permit a lot of effective higher level
concurrency designs to work*. But sadly I don't believe that Perl 5 will
ever be able to provide a performant hand-off mechanism. The internals
assume all over that it's safe for any logical read to actually be a write
behind the scenes (making it awkward to provide any sort of read-only view
of another thread's data), and all interpreter data structures are
implicitly tied to the interpreter that allocated them, which would take a
massive amount of refactoring to attempt to untangle.

I don't think that this is particularly a Perl problem. I'm not aware of any
comparable C-based dynamic language has managed to retrofit true
concurrency. CPython still has a GIL (and Unladen Swallow failed to deliver
on its design to remove that), and my understanding is that Ruby (MRI/YARV)
still single-threads its interpreter, and PHP doesn't even offer threading.
If we had a design to steal, we'd steal it. :-/

Nicholas Clark

* such as the rather nice constructions that Jonathan Worthing demonstrated
  for Perl 6: http://jnthn.net/papers/2013-yapceu-conc.pdf
  (Video not yet online)

Thread Previous | Thread Next


nntp.perl.org: Perl Programming lists via nntp and http.
Comments to Ask Bjørn Hansen at ask@perl.org | Group listing | About