Front page | perl.perl5.porters |
Postings from August 2013
Re: [perl #119445] performance bug: perl Thread::Queue is 20x slower than Unix pipe
Thread Previous
|
Thread Next
From:
Elizabeth Mattijsen
Date:
August 26, 2013 21:56
Subject:
Re: [perl #119445] performance bug: perl Thread::Queue is 20x slower than Unix pipe
Message ID:
679A97B4-904B-4381-817E-E3B7CBD86C8C@dijkmat.nl
On Aug 26, 2013, at 5:58 PM, John Heidemann <johnh@isi.edu> wrote:
> On Mon, 26 Aug 2013 08:11:12 -0700, "Dave Mitchell via RT" wrote:
>> On Sun, Aug 25, 2013 at 05:37:39PM -0700, James E Keenan via RT wrote:
>>> On Fri Aug 23 17:28:00 2013, johnh@isi.edu wrote:
>>>> Why is Thread::Queue *so* slow?
>>>>
>>>> I understand it has to do locking and be careful about data
>>>> structures, but it seems like it is about 20x slower than opening up a
>>>> Unix pipe, printing to that, reading it back and parsing the result.
>> Because it is nothing like a UNIX pipe.
>>
>> A UNIX pipe takes a stream of bytes, and read and writes chunks of them
>> into a shared buffer.
>>
>> A T::Q buffer takes a stream of perl "things", which might be objects or
>> other such complex structures, and ensures they they are accessible by
>> both the originating thread and any potential consumer thread. Migrating a
>> perl "thing" across a thread boundary is considerably more complex than
>> copying a byte across.
>>
>>
>>>> To speculate, I'm thinking the cost is in making all IPC data shared.
>>>> It would be great if one could have data that is sent over
>>>> Thread::Queue that is copied, not shared.
>>
>> But T::Q is build upon a shared array, and is designed to handled shared
>> data.
>>
>> I think the performance you are seeing is the performance I would expect,
>> and that this is not a bug.
>
> I understand that Thread::Queue and perl threads allow shared data, and that
> that's much more than a pipe.
>
> My concern is that Thread::Queue also *forces* shared data, even when
> it's not rqeuired. If that sharing comes with a 20x performance hit,
> that should be clear.
You should realize that the perl ithreads implementation does *not* have any real shared variables at all. Each thread has its own *copy* of the world.
Variables with the :shared trait, are simply tied() variables to some internal logic that will STORE values in yet another, hidden thread. And will FETCH them from that hidden thread again when needed. There is some locking involved there, I would assume. But I think the biggest bottleneck is really that the slow tie() interface is used for shared variables.
The forks module does not do this differently. However, instead of making a copy of the world each time a thread is started, the forks module just does a fork() and let's the OS take care of any Copy-On-Write needed. This makes starting a thread *much* faster, especially if you have something like Moose and its dependencies loaded. Reading and writing shared variables are done by using pipes, Unix pipes if possible.
Thread::Queue::Any is simply a wrapper around Thread::Queue, and thus suffers from the same performance issues.
In other words: don't use Perl 5's ithreads for performance, use it for asynchronous jobs only where not having to wait for something slow,
Liz
Thread Previous
|
Thread Next