develooper Front page | perl.perl5.porters | Postings from January 2005

Re: Detecting thread exits

Thread Previous | Thread Next
Eric Garland
January 25, 2005 08:56
Re: Detecting thread exits
Message ID:
Elizabeth Mattijsen wrote:
> At 11:18 AM -0500 1/24/05, Eric Garland wrote:
>> I'm setting up a Boss/Worker threaded program that runs for a very 
>> long time.  There are times where a worker thread will run into errors 
>> and exit.  The obvios solution would be to prevent that from happening 
>> but I would like to create a fault tolerant framework that doesn't tip 
>> over at the slightest hint of a problem.  I would like to have the 
>> Boss detect when a worker exits and restart a worker thread in its place.
>> So far, I find no functions that will determine if a thread has exited 
>> outside of join().  The obvious issue with join is that it blocks so I 
>> need a join thread for each worker thread so that I can send a message 
>> back to the Boss to restart the thread.  This seems to wildly increase 
>> the memory usage.   There are large shared data structures in this 
>> program and it already seems to be at the memory limits of the machine 
>> without these join threads.
> You might want to have a look at Thread::Running on CPAN, by yours truly.

Interesting approach.  I found an issue in that an immediate call to 
running after starting a thread returns false ala:

my $thread = threads->new( \&worker_thread );
while ($thread->running) {
     print "Still running\n";

will fall right through the while loop. Adding a yield or a pause 
above the while loop seems to fix the problem.

Looking through the code, the trick you use seems to be running the 
entire thread in an eval so that the exit can be trapped by a wrapper 
subroutine.  That's great!  I think I'll just use that directly.

>> Is there a way to remove the shared variables from the join threads so 
>> that they take up as little memory as possible?
> Not sure what you mean by that.

I don't really understand much about shared variable memory usage in 
ithreads but it seems that when you have a shared data structure, it 
takes up memory for each thread.  I'm dealing with a large (100,000 
entry) multi dimensional array where each element is approximately 20 
elements and several of them are arrays themselves.  The process tends 
to use up about 400 MB of ram when running (not all perl) and I'm 
testing on a machine with 512mb of RAM.  Adding the watcher threads 
slowed the machine to a crawl.  It may have just been the overhead of 
the interpreters that did that since I'm so close to using all my ram. 
  I assumed that if I could remove the large shared data structure 
from the watcher threads, it would make them more memory efficient. 
Is that true?

If so, is it possible?

Thanks Liz.

Thread Previous | Thread Next Perl Programming lists via nntp and http.
Comments to Ask Bjørn Hansen at | Group listing | About