perl.ithreads

perl.ithreads http://www.nntp.perl.org/group/perl.ithreads/ ... Copyright 1998-2013 perl.org Mon, 20 May 2013 04:43:50 +0000 ask@perl.org Issues with process forking when 'multithreaded'. by Sander Smeenk Hi, I'm new on this list, not sure if this is the right place for this, please redirect me if this is not on topic here. ;) I am seeing issues with open3()/waitpid() calls when running the same code as multiple threads - this does not happen with a single thread running the code - i'm not sure why this happens, but i can reproduce it so there must be something wrong somewhere. ;) What am i seeing: 1) waitpid() sometimes returns the $PID for the child process (as if it was reaped) while the child process is still running. 2) waitpid() sometimes returns -1 instead of the correct values for the exit state of a child process while the child process ran perfectly fine. Again, all this only happens when running multiple 'threads'. Refer to https://8n1.org/raw/8750/71a3 for a codesnipet from the subroutine which runs as a thread which in turn spawns rsync. Besides these waitpid() issues, the threaded code seems to run just fine and no other abnormalities are found. This is Perl 5.14.2 with ithreads on Ubuntu GNU/Linux, AMD64. Perhaps this all sounds familiar to any of you? Please let me know if more input or code is required! With regards, -Sander. -- | 1 1 was a racehorse, 2 2 was 1 2, 1 1 1 1 race 1 day, 2 2 1 1 2 | 4096R/20CC6CD2 - 6D40 1A20 B9AA 87D4 84C7 FBD6 F3A9 9442 20CC 6CD2 http://www.nntp.perl.org/group/perl.ithreads/2013/03/msg1397.html Tue, 05 Mar 2013 03:24:43 +0000 Re: Issues with process forking when 'multithreaded'. by David Mertens No, no, I mean the module called "forks" from cpan: https://metacpan.org/module/forks#same-API-as-threads Drop-in replacement for threads, including shared data handling. I but that would solve your problem satisfactorily. David On Mon, Mar 4, 2013 at 10:31 AM, Sander Smeenk <ssmeenk@freshdot.net> wrote: > Quoting David Mertens (dcmertens.perl@gmail.com): > > > Would it be possible to use the "forks" module? It is supposed to > provide a > > drop-in replacement for "threads" on operating systems that support posix > > forks, which would be true in your case. I suspect this would cure your > > problems, since everything would operate using forks instead of mixing > > forks and threads. > > I "need" the threads because this brings me ':shared' scalars/hashes. > > I thought of doing this only with fork() but i'd need to create some > kind of inter-process-communication thingie to get the results back, > which isn't impossible, just requires me to rewrite a large part of > my code. ;) > > And i still don't understand why mixing threads+fork is "not smart". ;) > > -Sndr. > -- > | The short fortune teller who escaped from prison: a small medium at > large. > | 4096R/20CC6CD2 - 6D40 1A20 B9AA 87D4 84C7 FBD6 F3A9 9442 20CC 6CD2 > -- "Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it." -- Brian Kernighan http://www.nntp.perl.org/group/perl.ithreads/2013/03/msg1396.html Mon, 04 Mar 2013 22:38:22 +0000 Re: Issues with process forking when 'multithreaded'. by Paul Robert Marino take a look at what I did in MrTools http://sourceforge.net/projects/mrtools/ specifically mrsync.pl http://mrtools.svn.sourceforge.net/viewvc/mrtools/mrtools/trunk/mrsync.pl?revision=32&view=markup Its not perfect but it works fairly well I stopped looking at return codes and just looked at the stderr output to catch errors it worked pretty much flawlessly for me once I worked out the initial kinks. On Mon, Mar 4, 2013 at 1:52 PM, Sander Smeenk <ssmeenk@freshdot.net> wrote: > Quoting Dave Mitchell (davem@iabyn.com): > >> > So, it's not a fork() from the main process (which would perhaps >> > copy all threads to the fork()ed process, or maybe not, who knows). >> fork() on linux creates a single thread in the child process, but copies >> the memory and state of *all* threads. Which is why you can get in a right >> mess. For example, if one of the other threads had something locked, that >> will be still be locked in the new process, but without a corresponding >> thread still running that can unlock it. So the forked process might >> deadlock or otherwise behave strangely. > > Thanks. This is a clear explanation on why you should take care with > threads and forks. As you might have gathered already i'm not that > experienced with the internals of threads and fork... > > This mail thread (haha!) made clear to me why removing 'lock($sharedfoo)' > calls from the threaded code improved upon the stability. It did indeed > no longer 'lock up' randomly, the above quoted must be exactly why. > I reworked the code so i no longer needed to lock() things and thought > to be done with it. > > >> This is why people say not to mix threads and forks. > > I honestly swear i will rewrite my production code using fork() only. :-) > > Still, out of curiosity, i would like to keep discussing this (and > fiddle with the thread+fork mix some more). I'm fairly certain that my > threaded code is not creating locks anymore. I've ran the code > uncountable times now and it has never locked up (yet). > > Locks aren't really my problem, it is waitpid() that is 'acting up' and > i am unable to find an explanation for that. > > Is it true that even though each thread has it's own 'safe' memory to > work with, fork()ing from two threads at separate times > changes (reinitialises?) memorystructures in either thread? > > For example, can a succesfull fork() (or waitpid()?) in thread2 cause > the waitpid() in thread1 to return too? Even though the childprocess > 'in' thread1 is still running? > > > Thanks a bundle so far! > Still learning every day. :) > > > -Sndr. > -- > | The world is so full of these wonderful things, > | i'm sure we should all be as happy as kings. > | 4096R/20CC6CD2 - 6D40 1A20 B9AA 87D4 84C7 FBD6 F3A9 9442 20CC 6CD2 http://www.nntp.perl.org/group/perl.ithreads/2013/03/msg1395.html Mon, 04 Mar 2013 19:09:07 +0000 Re: Issues with process forking when 'multithreaded'. by Sander Smeenk Quoting Dave Mitchell (davem@iabyn.com): > > So, it's not a fork() from the main process (which would perhaps > > copy all threads to the fork()ed process, or maybe not, who knows). > fork() on linux creates a single thread in the child process, but copies > the memory and state of *all* threads. Which is why you can get in a right > mess. For example, if one of the other threads had something locked, that > will be still be locked in the new process, but without a corresponding > thread still running that can unlock it. So the forked process might > deadlock or otherwise behave strangely. Thanks. This is a clear explanation on why you should take care with threads and forks. As you might have gathered already i'm not that experienced with the internals of threads and fork... This mail thread (haha!) made clear to me why removing 'lock($sharedfoo)' calls from the threaded code improved upon the stability. It did indeed no longer 'lock up' randomly, the above quoted must be exactly why. I reworked the code so i no longer needed to lock() things and thought to be done with it. > This is why people say not to mix threads and forks. I honestly swear i will rewrite my production code using fork() only. :-) Still, out of curiosity, i would like to keep discussing this (and fiddle with the thread+fork mix some more). I'm fairly certain that my threaded code is not creating locks anymore. I've ran the code uncountable times now and it has never locked up (yet). Locks aren't really my problem, it is waitpid() that is 'acting up' and i am unable to find an explanation for that. Is it true that even though each thread has it's own 'safe' memory to work with, fork()ing from two threads at separate times changes (reinitialises?) memorystructures in either thread? For example, can a succesfull fork() (or waitpid()?) in thread2 cause the waitpid() in thread1 to return too? Even though the childprocess 'in' thread1 is still running? Thanks a bundle so far! Still learning every day. :) -Sndr. -- | The world is so full of these wonderful things, | i'm sure we should all be as happy as kings. | 4096R/20CC6CD2 - 6D40 1A20 B9AA 87D4 84C7 FBD6 F3A9 9442 20CC 6CD2 http://www.nntp.perl.org/group/perl.ithreads/2013/03/msg1394.html Mon, 04 Mar 2013 18:52:14 +0000 Re: Issues with process forking when 'multithreaded'. by Sander Smeenk Quoting Mike Pomraning (mjp@pilcrow.madison.wi.us): > > And i still don't understand why mixing threads+fork is "not smart". ;) > Mixing the two makes it easy to introduce damning error. > > Here's someone who learned it the hard way: > http://rachelbythebay.com/w/2011/06/07/forked/ > > Here are some caveats on forking and program state: > http://pubs.opengroup.org/onlinepubs/009696799/functions/pthread_atfork.html#tag_03_486_08 > > Here's an acknowledgement that the standards folks didn't get all the pesky > semantics quite right: > https://collaboration.opengroup.org/external/pasc.org/interpretations/unofficial/db/p1003.1c/pasc-1003.1c-37.html Thanks. This was insightful. It's hard to find these links using Google. All you find is people experiencing deadlocks and other problems. ;) -Sndr. -- | Today is the first day of the rest of your life | 4096R/20CC6CD2 - 6D40 1A20 B9AA 87D4 84C7 FBD6 F3A9 9442 20CC 6CD2 http://www.nntp.perl.org/group/perl.ithreads/2013/03/msg1393.html Mon, 04 Mar 2013 18:13:49 +0000 Re: Issues with process forking when 'multithreaded'. by Dave Mitchell On Mon, Mar 04, 2013 at 04:32:13PM +0100, Sander Smeenk wrote: > So, it's not a fork() from the main process (which would perhaps > copy all threads to the fork()ed process, or maybe not, who knows). fork() on linux creates a single thread in the child process, but copies the memory and state of *all* threads. Which is why you can get in a right mess. For example, if one of the other threads had something locked, that will be still be locked in the new process, but without a corresponding thread still running that can unlock it. So the forked process might deadlock or otherwise behave strangely. This is why people say not to mix threads and forks. -- My Dad used to say 'always fight fire with fire', which is probably why he got thrown out of the fire brigade. http://www.nntp.perl.org/group/perl.ithreads/2013/03/msg1392.html Mon, 04 Mar 2013 17:00:57 +0000 Re: Issues with process forking when 'multithreaded'. by Mike Pomraning On Mon, Mar 4, 2013 at 10:31 AM, Sander Smeenk <ssmeenk@freshdot.net> wrote: > > And i still don't understand why mixing threads+fork is "not smart". ;) > Mixing the two makes it easy to introduce damning error. Here's someone who learned it the hard way: http://rachelbythebay.com/w/2011/06/07/forked/ Here are some caveats on forking and program state: http://pubs.opengroup.org/onlinepubs/009696799/functions/pthread_atfork.html#tag_03_486_08 Here's an acknowledgement that the standards folks didn't get all the pesky semantics quite right: https://collaboration.opengroup.org/external/pasc.org/interpretations/unofficial/db/p1003.1c/pasc-1003.1c-37.html -Mike http://www.nntp.perl.org/group/perl.ithreads/2013/03/msg1391.html Mon, 04 Mar 2013 16:46:22 +0000 Re: Issues with process forking when 'multithreaded'. by Sander Smeenk Quoting David Mertens (dcmertens.perl@gmail.com): > Would it be possible to use the "forks" module? It is supposed to provide a > drop-in replacement for "threads" on operating systems that support posix > forks, which would be true in your case. I suspect this would cure your > problems, since everything would operate using forks instead of mixing > forks and threads. I "need" the threads because this brings me ':shared' scalars/hashes. I thought of doing this only with fork() but i'd need to create some kind of inter-process-communication thingie to get the results back, which isn't impossible, just requires me to rewrite a large part of my code. ;) And i still don't understand why mixing threads+fork is "not smart". ;) -Sndr. -- | The short fortune teller who escaped from prison: a small medium at large. | 4096R/20CC6CD2 - 6D40 1A20 B9AA 87D4 84C7 FBD6 F3A9 9442 20CC 6CD2 http://www.nntp.perl.org/group/perl.ithreads/2013/03/msg1390.html Mon, 04 Mar 2013 16:32:11 +0000 Re: Issues with process forking when 'multithreaded'. by David Mertens Sander, Would it be possible to use the "forks" module? It is supposed to provide a drop-in replacement for "threads" on operating systems that support posix forks, which would be true in your case. I suspect this would cure your problems, since everything would operate using forks instead of mixing forks and threads. As for the details of why this causes trouble, I am not yet familiar enough with the internals of Perl's threading implementation to answer. Sorry. David On Mar 4, 2013 9:32 AM, "Sander Smeenk" <ssmeenk@freshdot.net> wrote: > Quoting Sander Smeenk (ssmeenk@freshdot.net): > > > In fact, it does work, for 95% of all threads it works as designed, only > > a select few (but not the same) threads fail each time, i was hoping to > > find out why only this small portion of threads fail their waitpid() > call. ;) > > Also, this might not be clear from my original message, the main process > is the process that creates the threads, one for each server that needs > to be backed up. Each thread calls fork() only once (through open3) for > the rsync to do the heavy lifting. > > So, it's not a fork() from the main process (which would perhaps > copy all threads to the fork()ed process, or maybe not, who knows). > > Amazing there is so little known about what this does / how this works, > i'll see if i can get a working testcase instead of the snipped i > pasted. > > -Sndr. > -- > | One tequila, two tequila, three tequila, floor. > | 4096R/20CC6CD2 - 6D40 1A20 B9AA 87D4 84C7 FBD6 F3A9 9442 20CC 6CD2 > http://www.nntp.perl.org/group/perl.ithreads/2013/03/msg1389.html Mon, 04 Mar 2013 16:01:20 +0000 Re: Issues with process forking when 'multithreaded'. by Sander Smeenk Quoting Sander Smeenk (ssmeenk@freshdot.net): > In fact, it does work, for 95% of all threads it works as designed, only > a select few (but not the same) threads fail each time, i was hoping to > find out why only this small portion of threads fail their waitpid() call. ;) Also, this might not be clear from my original message, the main process is the process that creates the threads, one for each server that needs to be backed up. Each thread calls fork() only once (through open3) for the rsync to do the heavy lifting. So, it's not a fork() from the main process (which would perhaps copy all threads to the fork()ed process, or maybe not, who knows). Amazing there is so little known about what this does / how this works, i'll see if i can get a working testcase instead of the snipped i pasted. -Sndr. -- | One tequila, two tequila, three tequila, floor. | 4096R/20CC6CD2 - 6D40 1A20 B9AA 87D4 84C7 FBD6 F3A9 9442 20CC 6CD2 http://www.nntp.perl.org/group/perl.ithreads/2013/03/msg1388.html Mon, 04 Mar 2013 15:32:29 +0000 Re: Issues with process forking when 'multithreaded'. by Sander Smeenk Quoting Jerry D. Hedden (jerry@hedden.us): > For the most part, threads and forks are not compatible. Therefore, you > probably cannot use open3()/waitpid() in a threaded application. If you > google 'perl fork threads', you'll find lots more info. I kind of expected this reply. :> I did read about "it not being smart to mix fork() and threads". This comes down to portability issues and unexpected results on different platforms. With this project i don't care about portability. This should run on my GNU/Linux AMD64 system only. :) At the same time fork()ing from a thread is allowed, doesn't raise warnings and no-one can tell me exactly why it can not work (everyone tells me 'it is not smart'). In fact, it does work, for 95% of all threads it works as designed, only a select few (but not the same) threads fail each time, i was hoping to find out why only this small portion of threads fail their waitpid() call. ;) I'll start reworking this with pure fork()s. Have to look into some IPC way of getting the output from the childs back in the main process. Thanks so far, -Sndr. -- | Man who lose watch in toilet bound to have shitty time. | 4096R/20CC6CD2 - 6D40 1A20 B9AA 87D4 84C7 FBD6 F3A9 9442 20CC 6CD2 http://www.nntp.perl.org/group/perl.ithreads/2013/03/msg1387.html Mon, 04 Mar 2013 15:17:28 +0000 Re: Issues with process forking when 'multithreaded'. by Jerry D. Hedden For the most part, threads and forks are not compatible. Therefore, you probably cannot use open3()/waitpid() in a threaded application. If you google 'perl fork threads', you'll find lots more info. On Mon, Mar 4, 2013 at 9:25 AM, Sander Smeenk <ssmeenk@freshdot.net> wrote: > Hi, > > I'm new on this list, not sure if this is the right place for this, > please redirect me if this is not on topic here. ;) > > I am seeing issues with open3()/waitpid() calls when running the same > code as multiple threads - this does not happen with a single thread > running the code - i'm not sure why this happens, but i can reproduce it > so there must be something wrong somewhere. ;) > > What am i seeing: > > 1) waitpid() sometimes returns the $PID for the child process (as if > it was reaped) while the child process is still running. > > 2) waitpid() sometimes returns -1 instead of the correct values for the > exit state of a child process while the child process ran perfectly > fine. > > Again, all this only happens when running multiple 'threads'. > Refer to https://8n1.org/raw/8750/71a3 for a codesnipet from the > subroutine which runs as a thread which in turn spawns rsync. > > Besides these waitpid() issues, the threaded code seems to run just fine > and no other abnormalities are found. > > This is Perl 5.14.2 with ithreads on Ubuntu GNU/Linux, AMD64. > Perhaps this all sounds familiar to any of you? > > Please let me know if more input or code is required! > > With regards, > -Sander. > -- > | 1 1 was a racehorse, 2 2 was 1 2, 1 1 1 1 race 1 day, 2 2 1 1 2 > | 4096R/20CC6CD2 - 6D40 1A20 B9AA 87D4 84C7 FBD6 F3A9 9442 20CC 6CD2 > http://www.nntp.perl.org/group/perl.ithreads/2013/03/msg1386.html Mon, 04 Mar 2013 14:51:51 +0000 Issues with process forking when 'multithreaded'. by Sander Smeenk Hi, I'm new on this list, not sure if this is the right place for this, please redirect me if this is not on topic here. ;) I am seeing issues with open3()/waitpid() calls when running the same code as multiple threads - this does not happen with a single thread running the code - i'm not sure why this happens, but i can reproduce it so there must be something wrong somewhere. ;) What am i seeing: 1) waitpid() sometimes returns the $PID for the child process (as if it was reaped) while the child process is still running. 2) waitpid() sometimes returns -1 instead of the correct values for the exit state of a child process while the child process ran perfectly fine. Again, all this only happens when running multiple 'threads'. Refer to https://8n1.org/raw/8750/71a3 for a codesnipet from the subroutine which runs as a thread which in turn spawns rsync. Besides these waitpid() issues, the threaded code seems to run just fine and no other abnormalities are found. This is Perl 5.14.2 with ithreads on Ubuntu GNU/Linux, AMD64. Perhaps this all sounds familiar to any of you? Please let me know if more input or code is required! With regards, -Sander. -- | 1 1 was a racehorse, 2 2 was 1 2, 1 1 1 1 race 1 day, 2 2 1 1 2 | 4096R/20CC6CD2 - 6D40 1A20 B9AA 87D4 84C7 FBD6 F3A9 9442 20CC 6CD2 http://www.nntp.perl.org/group/perl.ithreads/2013/03/msg1385.html Mon, 04 Mar 2013 14:25:49 +0000 Re: Undefined thread object for main thread (tid zero) even thoughthe main thread exists by David Mertens An update: I added a special case for thread id 0, though this is not ideal. For better or for worse, my module does not provide any thread entry points. It simply provides an unconventional means for sharing data across threads. This means that a user of this module could launch an async{} block that pulls in the shared data and modifies it. It also means that they can never share data that originated in an async block. I've thoroughly documented all of this and I don't think this will be too much of a problem for most users. However, if I could get some way, even an XS-based way, to know if a thread is still alive, that would be quite handy. Thanks! David On Tue, Dec 4, 2012 at 10:02 PM, Mike Pomraning <mjp@pilcrow.madison.wi.us>wrote: > On Tue, Dec 4, 2012 at 9:15 PM, David Mertens <dcmertens.perl@gmail.com>wrote: > >> Thanks for the explanation. Can you think of any way (apart from hackery) >> that I could check that a detached thread with the given tid exists? I do >> not need to manipulate it, I just need to know if it exists. I might >> consider tracking the existing threads through a shared array, but then I >> would have to modify the content of that shared array in an END block, >> which seems to be discouraged <<SNIP>> >> > > You could wrap every user-supplied entry point function in your own > enclosing, anonymous sub for bookkeeping. This sub is the true entry point > passed to threads->create(), and it touches `our %Tids : shared` before and > after invoking the user-supplied sub. Thread 0 is still a special case. > > Liz's Thread::Exit has an example of this approach. > > -Mike > -- "Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it." -- Brian Kernighan http://www.nntp.perl.org/group/perl.ithreads/2013/01/msg1384.html Wed, 02 Jan 2013 23:26:03 +0000 Re: [Pdl-porters] RFC: PDL::Parallel::threads andPDL::Parallel::threads::SIMD by jns-cmarshall Hi David- I'm looking forward to taking a look. :-) This reminds me of the MCE (Many Core Engine) module to support multiple core perl processing. I wonder if it could be used for PDL processing? --Chris ----- David Mertens <dcmertens.perl@gmail.com> wrote: > Hello everybody - > > PDL folks will know that I have a lot of projects running lately, but I > felt compelled to push this one through at the moment. Threads folks are > probably aware that I've been working on something lately and I now have > something that is hopefully worthy of your close scrutiny. I would > appreciate any feedback for the two modules listed in the subject line, > both of which shall be distributed on CPAN under PDL::Parallel::threads. > > PDL::Parallel::threads provides a means for sharing PDL data across Perl > threads. Presently, it is not safe to launch multiple threads in Perl after > loading PDL. (A simple CLONE_SKIP fixes this, as implemented.) It makes > sharing of physical piddles and memory mapped files easy. I have hit some > bumps with memory mapping on Windows, but I expect that'll get ironed out > in the coming weeks and months. I can provide more feedback there if it > would be helpful. > > PDL::Parallel::threads::SIMD provides a very simple interface (not > stress-tested) for launching groups of parallel processes with the > capability of performing barrier synchronization. This could almost serve > as a stand-alone Threads module, but I tend to affiliate SIMD stuff with > numerical computing, plus I use it in one or two of my tests. Perhaps when > PDL::Parallel::threads gets pulled into PDL, I can spin off ...::SIMD into > Threads::SIMD or some such. > > I just uploaded the distribution to PAUSE, so expect it to hit CPAN in the > next 12-24 hours. You can examine the code on github: > https://github.com/run4flat/PDL-Parallel-threads > > Thoughts and comments are welcome. PDL folks, I'd like to eventually move > this stuff into PDL's core, so the more feedback I can get on it, the > better. > > Thanks! > David > > -- > "Debugging is twice as hard as writing the code in the first place. > Therefore, if you write the code as cleverly as possible, you are, > by definition, not smart enough to debug it." -- Brian Kernighan http://www.nntp.perl.org/group/perl.ithreads/2012/12/msg1383.html Sun, 16 Dec 2012 07:14:02 +0000 Re: pool example by Jerry D. Hedden The timeout is 10. A thread may sleep from 5 to 15 seconds. So some will end before getting killed. Also, the timer thread does not know how long the sleep value in a thread is. Again, this is an example of pooling. Obviously, a thread that just sleeps is useless. The WORK section might do DB or web searches. In that case, you end result would be the main thread getting whatever results had been obtained during the timeout period. On Thu, Dec 13, 2012 at 12:15 PM, Alfonso Caponi <alfonso.caponi@gmail.com>wrote: > Hi list, > > I've a question about pool example script ( > http://cpansearch.perl.org/src/JDHEDDEN/threads-1.86/examples/pool.pl). > > Theoretically, should the sub "timer" kill threads immediately when sleep > value is greater than timeout value? > At the moment seems that the threads wait for the sleep value than they got > killed by sub "timer". > > Thank you > http://www.nntp.perl.org/group/perl.ithreads/2012/12/msg1382.html Thu, 13 Dec 2012 17:35:26 +0000 pool example by Alfonso Caponi Hi list, I've a question about pool example script ( http://cpansearch.perl.org/src/JDHEDDEN/threads-1.86/examples/pool.pl). Theoretically, should the sub "timer" kill threads immediately when sleep value is greater than timeout value? At the moment seems that the threads wait for the sleep value than they got killed by sub "timer". Thank you http://www.nntp.perl.org/group/perl.ithreads/2012/12/msg1381.html Thu, 13 Dec 2012 17:15:47 +0000 RFC: PDL::Parallel::threads and PDL::Parallel::threads::SIMD by David Mertens Hello everybody - PDL folks will know that I have a lot of projects running lately, but I felt compelled to push this one through at the moment. Threads folks are probably aware that I've been working on something lately and I now have something that is hopefully worthy of your close scrutiny. I would appreciate any feedback for the two modules listed in the subject line, both of which shall be distributed on CPAN under PDL::Parallel::threads. PDL::Parallel::threads provides a means for sharing PDL data across Perl threads. Presently, it is not safe to launch multiple threads in Perl after loading PDL. (A simple CLONE_SKIP fixes this, as implemented.) It makes sharing of physical piddles and memory mapped files easy. I have hit some bumps with memory mapping on Windows, but I expect that'll get ironed out in the coming weeks and months. I can provide more feedback there if it would be helpful. PDL::Parallel::threads::SIMD provides a very simple interface (not stress-tested) for launching groups of parallel processes with the capability of performing barrier synchronization. This could almost serve as a stand-alone Threads module, but I tend to affiliate SIMD stuff with numerical computing, plus I use it in one or two of my tests. Perhaps when PDL::Parallel::threads gets pulled into PDL, I can spin off ...::SIMD into Threads::SIMD or some such. I just uploaded the distribution to PAUSE, so expect it to hit CPAN in the next 12-24 hours. You can examine the code on github: https://github.com/run4flat/PDL-Parallel-threads Thoughts and comments are welcome. PDL folks, I'd like to eventually move this stuff into PDL's core, so the more feedback I can get on it, the better. Thanks! David -- "Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it." -- Brian Kernighan http://www.nntp.perl.org/group/perl.ithreads/2012/12/msg1380.html Thu, 13 Dec 2012 07:19:40 +0000 Re: Undefined thread object for main thread (tid zero) even thoughthe main thread exists by Mike Pomraning On Tue, Dec 4, 2012 at 9:15 PM, David Mertens <dcmertens.perl@gmail.com>wrote: > Thanks for the explanation. Can you think of any way (apart from hackery) > that I could check that a detached thread with the given tid exists? I do > not need to manipulate it, I just need to know if it exists. I might > consider tracking the existing threads through a shared array, but then I > would have to modify the content of that shared array in an END block, > which seems to be discouraged <<SNIP>> > You could wrap every user-supplied entry point function in your own enclosing, anonymous sub for bookkeeping. This sub is the true entry point passed to threads->create(), and it touches `our %Tids : shared` before and after invoking the user-supplied sub. Thread 0 is still a special case. Liz's Thread::Exit has an example of this approach. -Mike http://www.nntp.perl.org/group/perl.ithreads/2012/12/msg1379.html Wed, 05 Dec 2012 04:02:56 +0000 Re: Undefined thread object for main thread (tid zero) even thoughthe main thread exists by David Mertens Mike - Thanks for the explanation. Can you think of any way (apart from hackery) that I could check that a detached thread with the given tid exists? I do not need to manipulate it, I just need to know if it exists. I might consider tracking the existing threads through a shared array, but then I would have to modify the content of that shared array in an END block, which seems to be discouraged here: http://perldoc.perl.org/threads.html#BUGS-AND-LIMITATIONS. At any rate, I will soften the condition on thread existence to allow for the zeroeth thread to pass through. I think I have enough of an answer to move forward for now. However, ideas and comments are welcome! Also, I would be more than happy to discuss the details if anybody would like to hear. I'm going to begin, however, by presuming that the list doesn't want too much noise. :-) github: https://github.com/run4flat/PDL-Parallel-threads David On Mon, Dec 3, 2012 at 8:47 AM, Mike Pomraning <mjp@pilcrow.madison.wi.us>wrote: > On Sun, Dec 2, 2012 at 10:53 PM, David Mertens <dcmertens.perl@gmail.com>wrote: > >> ----%<---- >> >> defined (threads->object($originating_tid{$name})) >> or croak("retrieve_pdls: '$name' was created in a thread that " >> . "is no longer available"); >> >> ---->%---- >> >> In my test suite, I create and store data in the main thread. For this >> example, then, $originating_tid{$name} == 0 for all cases considered. The >> problem is that only the main thread is capable of retrieving the threads >> object for tid 0. Any other (child) thread gets an undefined value. Does >> that surprise anyone? Are child threads able to get the thread objects for >> their parents or siblings? >> > > I might be misreading this, but I believe that threads->object() ignores > detached threads unless a tid is explicitly passed in *and* that tid is the > tid of the calling thread. The main thread is detached, so no thread other > than tid 0 can look it up via object(). > > (You describe an interesting problem and approach; do you have > demonstration code that others can take a look at?) > > -Mike > -- "Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it." -- Brian Kernighan http://www.nntp.perl.org/group/perl.ithreads/2012/12/msg1378.html Wed, 05 Dec 2012 03:15:21 +0000 Re: Undefined thread object for main thread (tid zero) even thoughthe main thread exists by Mike Pomraning On Sun, Dec 2, 2012 at 10:53 PM, David Mertens <dcmertens.perl@gmail.com>wrote: > ----%<---- > > defined (threads->object($originating_tid{$name})) > or croak("retrieve_pdls: '$name' was created in a thread that " > . "is no longer available"); > > ---->%---- > > In my test suite, I create and store data in the main thread. For this > example, then, $originating_tid{$name} == 0 for all cases considered. The > problem is that only the main thread is capable of retrieving the threads > object for tid 0. Any other (child) thread gets an undefined value. Does > that surprise anyone? Are child threads able to get the thread objects for > their parents or siblings? > I might be misreading this, but I believe that threads->object() ignores detached threads unless a tid is explicitly passed in *and* that tid is the tid of the calling thread. The main thread is detached, so no thread other than tid 0 can look it up via object(). (You describe an interesting problem and approach; do you have demonstration code that others can take a look at?) -Mike http://www.nntp.perl.org/group/perl.ithreads/2012/12/msg1377.html Mon, 03 Dec 2012 15:25:18 +0000 Undefined thread object for main thread (tid zero) even though themain thread exists by David Mertens Hey folks - I am working on a module to share PDL data (C-like arrays of data) across threads. It pretty much works, but it's not quite as rock-solid as I'd like. I'd first like to solve my immediate problem, then perhaps open the larger discussion of how one shares (scalar) objects whose guts are actually C structs. (I am interested in this both for use in Prima as well as PDL, and likely other modules that I will come across or write. But I'm getting ahead of myself.) My current module works by letting users share PDL *data*, rather than sharing the piddle itself. Since I cannot share the actual piddle (due to the difficulties with sharing C structs across threads), I simply tick up the original piddle's refcount and store the data pointer in a shared Perl hash. Then, when the user requests a shared copy of that data in a thread, I create a *very* thin piddle wrapper around the C array of the original piddle. Believe it or not, it actually works! With any luck, I may even be able to use this for real multicore high-performance computing stuff. However, life being what it is, there are corner cases. If that original piddle's thread terminates, the interpreter frees the memory and the data goes away. Any wrapped piddles in other threads are now ticking segmentation faults just waiting for a data read/write. To help prevent that situation, I decided to store the thread id in which a piddle's data is first shared. At the very least, this would let me check that the data's originating thread is still around, and croak if this were not the case. That code looks something like this: ----%<---- defined (threads->object($originating_tid{$name})) or croak("retrieve_pdls: '$name' was created in a thread that " . "is no longer available"); ---->%---- In my test suite, I create and store data in the main thread. For this example, then, $originating_tid{$name} == 0 for all cases considered. The problem is that only the main thread is capable of retrieving the threads object for tid 0. Any other (child) thread gets an undefined value. Does that surprise anyone? Are child threads able to get the thread objects for their parents or siblings? Thanks! David -- "Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it." -- Brian Kernighan http://www.nntp.perl.org/group/perl.ithreads/2012/12/msg1376.html Mon, 03 Dec 2012 04:53:58 +0000 Hello by liz Mail transaction failed. Partial message is available. http://www.nntp.perl.org/group/perl.ithreads/2011/11/msg1375.html Sat, 19 Nov 2011 22:17:00 +0000 Re: timing out threads by Mike Pomraning On Tue, Nov 1, 2011 at 6:09 PM, Danny Wong (dannwong) <dannwong@cisco.com>wrote: > I would like to perform the following, start X (say 10) number > of threads to perform some operation (ex. Copy command), I would like to > exit or return out of the thread(s) that takes too long ( ex. 30 minutes > ). Danny, Can you show us in more detail what the long-running threads are doing, perhaps some pseudocode? How one safely kills/interrupts/shuts down a thread depends on what that thread is doing. As this forum discussed a little over a year ago, threads stuck in blocking system calls present their own design challenges. -Mike http://www.nntp.perl.org/group/perl.ithreads/2011/11/msg1374.html Wed, 02 Nov 2011 06:23:07 +0000 Re: timing out threads by Francesco Nidito Hi Danny, You can just add a counter for each thread and propagate the counter to the main thread at the end of the computation. For instance, a hash local to each thread mapping from "job id" (in your case the name of the file) and the time. At the end of the execution, you can send the counters back to the main thread though a Queue and let the main thread do some statistics and output the results. I pseudocode: task queue Q; counters queue C; thread { create counter c_X; while there is "something" in Q do "something"; c_X{something} = time required to do "something"; enqueue c_X in C; } main { create N threads; load Q with the tasks to be done; wait the N thread to finish (join); get all the counters c_i from C; do something with c_1 ... c_N } Hope this helps! Cheers, nids ------------------------------ «Fear is not what's important; it's how you deal with it. It would be like asking a marathon runner if they feel pain. It's not a matter of whether you feel it; it's how you manage it.» — James Nachtwey On Tue, Nov 1, 2011 at 11:09 PM, Danny Wong (dannwong) <dannwong@cisco.com>wrote: > Hi Thread GURU, > I would like to perform the following, start X (say 10) number > of threads to perform some operation (ex. Copy command), I would like to > exit or return out of the thread(s) that takes too long ( ex. 30 minutes > ). I'm thinking I should spawn an independent thread (a watch thread) > and have that thread watch to see if any of the 10 threads are taking > longer than 30 minutes to complete. Any example how I can accomplish > this? Any other suggesting would be great to. > > BTW, I'm using perl 5.8.8., thread version 1.07. > > Thanks. > http://www.nntp.perl.org/group/perl.ithreads/2011/11/msg1373.html Wed, 02 Nov 2011 05:05:49 +0000 timing out threads by Danny Wong Hi Thread GURU, I would like to perform the following, start X (say 10) number of threads to perform some operation (ex. Copy command), I would like to exit or return out of the thread(s) that takes too long ( ex. 30 minutes ). I'm thinking I should spawn an independent thread (a watch thread) and have that thread watch to see if any of the 10 threads are taking longer than 30 minutes to complete. Any example how I can accomplish this? Any other suggesting would be great to. BTW, I'm using perl 5.8.8., thread version 1.07. Thanks. http://www.nntp.perl.org/group/perl.ithreads/2011/11/msg1372.html Tue, 01 Nov 2011 23:27:31 +0000 RE: threads and alarm by Danny Wong How can you do this using perl 5.8.8? -----Original Message----- From: jdhedden@gmail.com [mailto:jdhedden@gmail.com] On Behalf Of Jerry D. Hedden Sent: Thursday, September 01, 2011 5:48 AM To: alfonso caponi Cc: perl-ithreads@perl.org Subject: Re: threads and alarm Signals are received by thread 0 - the main thread. Therefore, when a thread sets an alarm, the ALRM signal hits the main thread not the thread that set the alarm. I don't know your particular needs, but if you want the thread to get the alarm, then you need to set the alarm handler in the main thread, and then send the ALRM signal to the thread using $thr->kill('ALRM'). Thus, in your example, if you add the following: $thr->detach(); $SIG{ALRM} = sub { $thr->kill('ALRM') }; It'll work. On Thu, Sep 1, 2011 at 06:08, alfonso caponi <alfonso.caponi@gmail.com> wrote: > Hi list, > > I need help! :) How can I use "alarm" without exit from a thread only > (without making the script end!). > > For example this works (without threads): > > while(1){ >         eval { >                 local $SIG{ALRM} = sub { print "alarm!\n"; }; >                 alarm(2); >                 print "hi\n"; >                 sleep 5; >                 alarm(0); >         }; > } > > but the script in attachment doesn't works :( > > Thank you very much, > Al > http://www.nntp.perl.org/group/perl.ithreads/2011/09/msg1371.html Thu, 01 Sep 2011 23:05:35 +0000 Re: threads and alarm by Jerry D. Hedden The ->kill() method for threads was added in threads v1.27 (back in May 2006). If you update the threads module, you'll have this capability. The 'best' way to do this is to install Bundle::Thread from CPAN. On Thu, Sep 1, 2011 at 13:27, Danny Wong (dannwong) <dannwong@cisco.com> wrote: > How can you do this using perl 5.8.8? > > -----Original Message----- > From: jdhedden@gmail.com [mailto:jdhedden@gmail.com] On Behalf Of Jerry D. Hedden > Sent: Thursday, September 01, 2011 5:48 AM > To: alfonso caponi > Cc: perl-ithreads@perl.org > Subject: Re: threads and alarm > > Signals are received by thread 0 - the main thread.  Therefore, when a > thread sets an alarm, the ALRM signal hits the main thread not the > thread that set the alarm.  I don't know your particular needs, but if > you want the thread to get the alarm, then you need to set the alarm > handler in the main thread, and then send the ALRM signal to the > thread using $thr->kill('ALRM').  Thus, in your example, if you add > the following: > > $thr->detach(); > $SIG{ALRM} = sub { $thr->kill('ALRM') }; > > It'll work. > > On Thu, Sep 1, 2011 at 06:08, alfonso caponi <alfonso.caponi@gmail.com> wrote: >> Hi list, >> >> I need help! :) How can I use "alarm" without exit from a thread only >> (without making the script end!). >> >> For example this works (without threads): >> >> while(1){ >>         eval { >>                 local $SIG{ALRM} = sub { print "alarm!\n"; }; >>                 alarm(2); >>                 print "hi\n"; >>                 sleep 5; >>                 alarm(0); >>         }; >> } >> >> but the script in attachment doesn't works :( >> >> Thank you very much, >> Al >> > http://www.nntp.perl.org/group/perl.ithreads/2011/09/msg1370.html Thu, 01 Sep 2011 10:54:13 +0000 Re: threads and alarm by Jerry D. Hedden Signals are received by thread 0 - the main thread. Therefore, when a thread sets an alarm, the ALRM signal hits the main thread not the thread that set the alarm. I don't know your particular needs, but if you want the thread to get the alarm, then you need to set the alarm handler in the main thread, and then send the ALRM signal to the thread using $thr->kill('ALRM'). Thus, in your example, if you add the following: $thr->detach(); $SIG{ALRM} = sub { $thr->kill('ALRM') }; It'll work. On Thu, Sep 1, 2011 at 06:08, alfonso caponi <alfonso.caponi@gmail.com> wrote: > Hi list, > > I need help! :) How can I use "alarm" without exit from a thread only > (without making the script end!). > > For example this works (without threads): > > while(1){ >         eval { >                 local $SIG{ALRM} = sub { print "alarm!\n"; }; >                 alarm(2); >                 print "hi\n"; >                 sleep 5; >                 alarm(0); >         }; > } > > but the script in attachment doesn't works :( > > Thank you very much, > Al > http://www.nntp.perl.org/group/perl.ithreads/2011/09/msg1369.html Thu, 01 Sep 2011 05:48:19 +0000 threads and alarm by alfonso caponi #!/usr/bin/perl use strict; use warnings; use threads; $|=1; $SIG{INT} = sub { threads->exit(); exit(1); }; my $thr = threads->new(\&Testsub); $thr->detach(); while(1){ } exit(0); sub Testsub{ while(1){ eval { local $SIG{ALRM} = sub { print "alarm!\n"; }; alarm(2); sleep 5; alarm(0); }; } } http://www.nntp.perl.org/group/perl.ithreads/2011/09/msg1368.html Thu, 01 Sep 2011 03:08:36 +0000 Re: Load modules in Perl at run time and share it between threads -Stack Overflow Question by Dave Mitchell On Sat, Jul 02, 2011 at 03:45:39AM +0100, Pedro Cirne wrote: > Is it possible to load modules at run time, require '/file_path/file_name.pm > ;' , and share the memory between threads? > > Basically, I have a pool of threads, if thread#1 decides to load a module, I > want the module available to all the threads! > http://stackoverflow.com/q/6553598/758523 No, it's not possible. Either the module must be loaded before the thread is created, or each thread must individually load the module. -- More than any other time in history, mankind faces a crossroads. One path leads to despair and utter hopelessness. The other, to total extinction. Let us pray we have the wisdom to choose correctly. -- Woody Allen http://www.nntp.perl.org/group/perl.ithreads/2011/07/msg1367.html Sat, 02 Jul 2011 01:19:47 +0000 Load modules in Perl at run time and share it between threads - StackOverflow Question by Pedro Cirne Is it possible to load modules at run time, require '/file_path/file_name.pm ;' , and share the memory between threads? Basically, I have a pool of threads, if thread#1 decides to load a module, I want the module available to all the threads! http://stackoverflow.com/q/6553598/758523 http://www.nntp.perl.org/group/perl.ithreads/2011/07/msg1366.html Fri, 01 Jul 2011 20:11:05 +0000 RE: Killing a thread by Danny Wong Hi Ajeet, Do you have an example you can provide? I prefer not to loop around (like sleep for 60 seconds and check again) checking for the total time and then kill the thread if it takes too long. Sincerely yours, Danny H. Wong -----Original Message----- From: Ajeet Kumar [mailto:Ajeet.Kumar@aricent.com] Sent: Wednesday, September 29, 2010 2:34 AM To: Danny Wong (dannwong); perl-ithreads@perl.org Subject: RE: Killing a thread Hello Wong, Following is my approach in this case... --Execute the "ps -A" command -- Observe the third column , which is total time taken by process -- check the process ID from the first column -- send a kill -9 signal to required process ID If any confusion is left, then do mail back... -----Original Message----- From: Danny Wong (dannwong) [mailto:dannwong@cisco.com] Sent: Wednesday, September 29, 2010 4:12 AM To: perl-ithreads@perl.org Subject: Killing a thread Hi Perl Thread GURUS, Here is my situation. I started, say 10 threads, executing some command, 1 or 2 of them is hanging for over 2 hours. I want to put in some logic that if the thread run time passes an hour threshold kill that thread or return and fail the thread. Any ideas on how I can accomplish this task? Thanks. Sincerely yours, Danny H. Wong "DISCLAIMER: This message is proprietary to Aricent and is intended solely for the use of the individual to whom it is addressed. It may contain privileged or confidential information and should not be circulated or used for any purpose other than for what it is intended. If you have received this message in error, please notify the originator immediately. If you are not the intended recipient, you are notified that you are strictly prohibited from using, copying, altering, or disclosing the contents of this message. Aricent accepts no responsibility for loss or damage arising from the use of the information transmitted by this email including damage from virus." http://www.nntp.perl.org/group/perl.ithreads/2010/09/msg1365.html Thu, 30 Sep 2010 01:00:48 +0000 Re: Killing a thread by Jerry D. Hedden To my knowledge, if the thread hangs on a system call, then you'll need to do something similar to what was suggested by the previous poster. On Wed, Sep 29, 2010 at 10:47, Danny Wong (dannwong) <dannwong@cisco.com> wrote: > Hi Jerry, >        Is there an example you or someone can provide, if the thread is > hanging, I would just like the thread to "return failure or timeout" and > ignore the system call. Let the system call hang and I'll clean it up > afterwards at the very end of the script. > > Sincerely yours, > Danny H. Wong > > -----Original Message----- > From: jdhedden@gmail.com [mailto:jdhedden@gmail.com] On Behalf Of Jerry > D. Hedden > Sent: Wednesday, September 29, 2010 5:47 AM > To: Danny Wong (dannwong) > Cc: perl-ithreads@perl.org > Subject: Re: Killing a thread > > Danny Wong wrote: >> Here is my situation. I started, say 10 threads, executing >> some command, 1 or 2 of them is hanging for over 2 hours. >> I want to put in some logic that if the thread run time >> passes an hour threshold kill that thread or return and >> fail the thread. Any ideas on how I can accomplish this >> task? Thanks. > > This problem is addressed in the 'examples/pool.pl' file in > the 'threads' distribution on CPAN.  From the POD (note the > third bullet): > >    NAME >        pool.pl - Simple 'threads' example > >    DESCRIPTION >        A simplistic example illustrating the following: >        *   Management of a pool of threads >        *   Communication between threads using queues >        *   Timing out and cancelling threads >        *   Interrupting a threaded program >        *   Cleaning up threads before terminating > > However, if the 'hung' threads are stuck on some system > call, then the above will not work because of Perl's "safe > signals".  You'll need to read the section called "Deferred > Signals (Safe Signals)" in "perldoc perlipc". > http://www.nntp.perl.org/group/perl.ithreads/2010/09/msg1364.html Wed, 29 Sep 2010 08:11:59 +0000 RE: Killing a thread by Danny Wong Hi Jerry, Is there an example you or someone can provide, if the thread is hanging, I would just like the thread to "return failure or timeout" and ignore the system call. Let the system call hang and I'll clean it up afterwards at the very end of the script. Sincerely yours, Danny H. Wong -----Original Message----- From: jdhedden@gmail.com [mailto:jdhedden@gmail.com] On Behalf Of Jerry D. Hedden Sent: Wednesday, September 29, 2010 5:47 AM To: Danny Wong (dannwong) Cc: perl-ithreads@perl.org Subject: Re: Killing a thread Danny Wong wrote: > Here is my situation. I started, say 10 threads, executing > some command, 1 or 2 of them is hanging for over 2 hours. > I want to put in some logic that if the thread run time > passes an hour threshold kill that thread or return and > fail the thread. Any ideas on how I can accomplish this > task? Thanks. This problem is addressed in the 'examples/pool.pl' file in the 'threads' distribution on CPAN. From the POD (note the third bullet): NAME pool.pl - Simple 'threads' example DESCRIPTION A simplistic example illustrating the following: * Management of a pool of threads * Communication between threads using queues * Timing out and cancelling threads * Interrupting a threaded program * Cleaning up threads before terminating However, if the 'hung' threads are stuck on some system call, then the above will not work because of Perl's "safe signals". You'll need to read the section called "Deferred Signals (Safe Signals)" in "perldoc perlipc". http://www.nntp.perl.org/group/perl.ithreads/2010/09/msg1363.html Wed, 29 Sep 2010 08:02:05 +0000 Re: Killing a thread by Jerry D. Hedden Danny Wong wrote: > Here is my situation. I started, say 10 threads, executing > some command, 1 or 2 of them is hanging for over 2 hours. > I want to put in some logic that if the thread run time > passes an hour threshold kill that thread or return and > fail the thread. Any ideas on how I can accomplish this > task? Thanks. This problem is addressed in the 'examples/pool.pl' file in the 'threads' distribution on CPAN. From the POD (note the third bullet): NAME pool.pl - Simple 'threads' example DESCRIPTION A simplistic example illustrating the following: * Management of a pool of threads * Communication between threads using queues * Timing out and cancelling threads * Interrupting a threaded program * Cleaning up threads before terminating However, if the 'hung' threads are stuck on some system call, then the above will not work because of Perl's "safe signals". You'll need to read the section called "Deferred Signals (Safe Signals)" in "perldoc perlipc". http://www.nntp.perl.org/group/perl.ithreads/2010/09/msg1362.html Wed, 29 Sep 2010 05:48:04 +0000 RE: Killing a thread by Ajeet Kumar Hello Wong, Following is my approach in this case... --Execute the "ps -A" command -- Observe the third column , which is total time taken by process -- check the process ID from the first column -- send a kill -9 signal to required process ID If any confusion is left, then do mail back... -----Original Message----- From: Danny Wong (dannwong) [mailto:dannwong@cisco.com] Sent: Wednesday, September 29, 2010 4:12 AM To: perl-ithreads@perl.org Subject: Killing a thread Hi Perl Thread GURUS, Here is my situation. I started, say 10 threads, executing some command, 1 or 2 of them is hanging for over 2 hours. I want to put in some logic that if the thread run time passes an hour threshold kill that thread or return and fail the thread. Any ideas on how I can accomplish this task? Thanks. Sincerely yours, Danny H. Wong "DISCLAIMER: This message is proprietary to Aricent and is intended solely for the use of the individual to whom it is addressed. It may contain privileged or confidential information and should not be circulated or used for any purpose other than for what it is intended. If you have received this message in error, please notify the originator immediately. If you are not the intended recipient, you are notified that you are strictly prohibited from using, copying, altering, or disclosing the contents of this message. Aricent accepts no responsibility for loss or damage arising from the use of the information transmitted by this email including damage from virus." http://www.nntp.perl.org/group/perl.ithreads/2010/09/msg1361.html Wed, 29 Sep 2010 02:34:01 +0000 Killing a thread by Danny Wong Hi Perl Thread GURUS, Here is my situation. I started, say 10 threads, executing some command, 1 or 2 of them is hanging for over 2 hours. I want to put in some logic that if the thread run time passes an hour threshold kill that thread or return and fail the thread. Any ideas on how I can accomplish this task? Thanks. Sincerely yours, Danny H. Wong http://www.nntp.perl.org/group/perl.ithreads/2010/09/msg1360.html Wed, 29 Sep 2010 02:20:54 +0000 Re: Help with concurrent threads by Jerry D. Hedden I tweaked it a bit. Attached. On Thu, May 6, 2010 at 06:02, alfonso caponi <alfonso.caponi@gmail.com> wrote: > Inspired by the pool.pl example, I wrote a simple multi-thread script for > dns server testing on my LAN. It seems working fine, but I would like your > opinion on any improvements. > > Note: I've commented "#while (! $TERM)" to avoid multiple requests while ! > $TERM :( > > Thank you very much at all! > > 2010/4/16 Jerry D. Hedden <jdhedden@cpan.org> >> >> Get the 'threads' distribution from CPAN and look at the sample >> scripts in the 'examples' directory. >> >> On Fri, Apr 16, 2010 at 11:48, alfonso caponi <alfonso.caponi@gmail.com> >> wrote: >> > Hi list, >> > >> > I've some doubts with a multi-threads script. My goal is to create a >> > script >> > with the same max number of threads running at all times. >> > >> > For example: I've an array with 9 elements and 3 threads to process it; >> > each >> > thread can process an element using a different time (a thread could end >> > before another) and I would like 3 threads always running. >> > >> > My idea is to take an element from a queue, start threads to reach the >> > maximum number of active threads allowed and simultaneously start a >> > separate >> > (but unique) thread to wait every thread in execution. >> > >> > In attachment my "solution". Have you any tips? I would not use other >> > modules as well threads, threads::shared, thread::queue. >> > >> > Thank you very much! >> > Al > > http://www.nntp.perl.org/group/perl.ithreads/2010/05/msg1359.html Thu, 06 May 2010 06:26:40 +0000 Re: Help with concurrent threads by alfonso caponi #!/usr/bin/perl $|=1; use strict; use warnings; use Net::DNS; use threads; use threads::shared; use Thread::Queue; use Uniq; my $ip_list = $ARGV[0]; die "$0 <ip list file>\n" unless $ARGV[0]; ######################## ### Global Variables ### # FQDN list my @fqdn = ("www.nba.com","www.google.com","www.kernel.org","www.linux.org"); # Maximum working threads my $MAX_THREADS = 10; # Maximum thread working time my $TIMEOUT = 10; # Flag to inform all threads that application is terminating my $TERM :shared = 0; # Prevents double detach attempts my $DETACHING :shared; ####################### ### Signal Handling ### # Gracefully terminate application on ^C or command line 'kill' $SIG{'INT'} = $SIG{'TERM'} = sub { print(">>> Terminating <<<\n"); $TERM = 1; }; # This signal handler is called inside threads # that get cancelled by the timer thread $SIG{'KILL'} = sub { # Tell user we've been terminated printf("%3d <- Killed\n", threads->tid()); # Detach and terminate lock($DETACHING); threads->detach() if ! threads->is_detached(); threads->exit(); }; ##### my @hosts; open(IPLIST, "<$ip_list") || die "Error : cannot open the ip address list file: $!\n"; chomp (@hosts = <IPLIST>); close (IPLIST); @hosts = uniq sort @hosts; ### fill the queue our $q :shared; $q = new Thread::Queue; foreach my $host (@hosts) { foreach my $fqdn (@fqdn) { $q->enqueue("$host;$fqdn"); } } ############################### ### Main Processing Section ### MAIN: { # Start timer thread print "[+] Start timer thread\n"; my $queue = Thread::Queue->new(); threads->create('Timer',$queue)->detach(); # Manage the thread pool until signalled to terminate while (! $TERM) { # Keep max threads running for (my $needed = $MAX_THREADS - threads->list(); $needed && ! $TERM; $needed--) { my $element = $q->dequeue_nb; # New thread threads->create('RequestDNS',$queue,$element,$TIMEOUT); } # Wait for any threads to finish sleep(1); } # Wait for max timeout for threads to finish while ((threads->list() > 0) && $TIMEOUT--) { sleep(1); } # Detach and kill any remaining threads foreach my $thr (threads->list()) { lock($DETACHING); $thr->detach() if ! $thr->is_detached(); $thr->kill('KILL'); } } exit(0); sub RequestDNS { my ($queue,$element,$timeout) = @_; my ($dns_server,$fqdn) = split(/;/,$element); # My thread ID my $tid = threads->tid(); printf("$dns_server -> %3d\n", $tid); # Register with timer thread $queue->enqueue($tid,$timeout); # Do some work while monitoring $TERM #while (! $TERM) { print "try $dns_server;$fqdn\n"; my $res = Net::DNS::Resolver->new; $res->nameservers($dns_server); my $query = $res->query($fqdn,"A"); if ($query) { foreach my $rr (grep { $_->type eq 'A' } $query->answer) { my $reply = $rr->address; print "$dns_server;$fqdn;$reply\n"; } }# else { # warn "query failed: $dns_server ($fqdn)", $res->errorstring, "\n"; #} #} # Remove signal handler $SIG{'KILL'} = sub {}; # Unregister with timer thread $queue->enqueue($tid, undef); # Tell user we're done printf("%3d <- Finished\n", $tid); # Detach and terminate lock($DETACHING); threads->detach() if ! threads->is_detached(); threads->exit(); } # The timer thread that monitors other threads for timeout sub Timer { my $queue = shift; # The registration queue my %timers; # Contains threads and timeouts # Loop until told to quit while (! $TERM) { # Check queue while (my $tid = $queue->dequeue_nb()) { if (! ($timers{$tid}{'timeout'} = $queue->dequeue()) || ! ($timers{$tid}{'thread'} = threads->object($tid))) { # No timeout - unregister thread delete($timers{$tid}); } } # Cancel timed out threads foreach my $tid (keys(%timers)) { if (--$timers{$tid}{'timeout'} < 0) { $timers{$tid}{'thread'}->kill('KILL'); delete($timers{$tid}); } } # Tick tock sleep(1); } } http://www.nntp.perl.org/group/perl.ithreads/2010/05/msg1358.html Thu, 06 May 2010 03:02:49 +0000