perl.loop

perl.loop http://www.nntp.perl.org/group/perl.loop/ ... Copyright 1998-2013 perl.org Sat, 25 May 2013 04:42:32 +0000 ask@perl.org Re: events and shared memory by Joshua N Pritikin On Wed, Oct 10, 2012 at 01:40:22AM -0700, Max V. Bouglacoff wrote: > I'm trying to use Event module along with IPC::Shareable. The problem is > that Event does not react on changes of scalar vars placed in shared memory. > The code looks like this: > > ... > > tie $VAR,  'IPC::Shareable',  'SCAL'; > ... > Event->var( >     var => \$VAR, >     poll => 'w', >     cb => sub { >        # some code > >     } >   ); > ... > To clarify: when first process modifies $VAR the > changes are visible in second process immediately, but > there is no reaction on Event facility (in second process). > > Would you be so kind to help me with this? Event's var watcher depends on perl triggering the watcher when modifying the variable. This will not happen when the variable is modified by another process. Rather than using shared memory, I recommend you create a socket pair (like a pipe) between the two processes. Then you can write a byte on one side and use the Event io watcher to notice in the receiving process. Will this work for your use case? > ========= > This is perl, v5.8.8 built for i486-linux-thread-multi > Event-1.20 > IPC-Shareable-0.60 > > Linux breslau 2.6.36 #2 SMP PREEMPT Thu Jan 13 10:56:43 EET 2011 i686 AMD Athlon(tm) 64 X2 Dual Core Processor 5200+ AuthenticAMD GNU/Linux > Slackware 12.1.0 -- Joshua N. Pritikin Department of Psychology University of Virginia Gilmer Hall 102; Charlottesville, VA 22903 http://people.virginia.edu/~jnp3bc http://www.nntp.perl.org/group/perl.loop/2012/10/msg1128.html Wed, 10 Oct 2012 05:34:28 +0000 Re: Event.pm error message by Joshua N Pritikin On Mon, Dec 29, 2008 at 01:41:53PM -0500, Jeff Boes wrote: > Joshua N Pritikin wrote: > > Let us know what happens! > > I am a bit confused, still. Should I make *all* my events reentrant, The reentrant constraint is just for debugging. You don't want to keep re-entering the main event loop infinitely because you'll eventually run out of stack. If you are certain that your code works correctly then you can make all your watchers reentrant to avoid triggering the error. > Or just the ones I expect to be fired up *from* the "sweep" calls? Yah, that's the disciplined way to do it. http://www.nntp.perl.org/group/perl.loop/2008/12/msg1127.html Mon, 29 Dec 2008 11:25:56 +0000 Re: Event.pm error message by Joshua N Pritikin On Mon, Dec 29, 2008 at 09:51:16AM -0500, Jeff Boes wrote: > Hello! I posted this to the mailing list some time ago but never got a > response. I didn't see your post. Are you sure it got through? > Do you have any insights? > > I'm upgrading an ancient (2004-era) server to the modern era: > > 2.6.18-92.1.6.el5 x86_64 x86_64 x86_64 GNU/Linux Perl 5.10 Event.pm 1.11 > > One of my applications throws the following error once it's been running for a > few minutes as a daemon: > > Assertion (((pe_watcher*)wa)->flags & 0x0008) || !wa->running failed: file > "c/queue.c" That looks like c/queue.c line 43. Try making your watcher reentrant with $wa->reentrant(1) before you enter your main loop. After doing that, does it work? http://www.nntp.perl.org/group/perl.loop/2008/12/msg1126.html Mon, 29 Dec 2008 10:03:48 +0000 Event.dylib issue by Thomas Keller Greetings, I'm new to integrating C with Perl as well as the Event module. Trying the example: -------------- use strict; use Inline with => 'Event'; use Inline C => <<'EOC'; void c_callback(pe_event * event) { printf("Here is the C callback!\n"); } EOC Event->timer( desc => 'Perl timer with C callback.', interval => 2, cb => \&c_callback, ); Event::loop; ---------------- I get the following error: $ perl event_timer.pl make: *** No rule to make target `/opt/local/lib/perl5/site_perl/ 5.10.0/darwin-2level/auto/Event/Event.dylib', needed by `subdirs'. Stop. A problem was encountered while attempting to compile and install your Inline C code. The command that failed was: make > out.make 2>&1 ------------------ There does not seem to be an Event.dylib at the expected location: $ ls -l /opt/local/lib/perl5/site_perl/5.10.0/darwin-2level/auto/Event/ total 184 -r--r--r-- 1 root admin 0 Nov 20 08:43 Event.bs -r-xr-xr-x 1 root admin 92992 Nov 20 08:43 Event.bundle Your help would be greatly appreciated. regards, Tom Keller Thomas J Keller, PhD MMI Shared Resource Facility Oregon Health & Science University Portland, OR 97239 503-494-2442 kellert@ohsu.edu http://www.nntp.perl.org/group/perl.loop/2008/11/msg1125.html Thu, 20 Nov 2008 13:10:12 +0000 Run-time failure by Jeff Boes I'm upgrading an ancient (2004-era) server to the modern era: 2.6.18-92.1.6.el5 x86_64 x86_64 x86_64 GNU/Linux Perl 5.10 Event.pm 1.1 One of my applications throws the following error once it's been running for a few minutes as a daemon: Assertion (((pe_watcher*)wa)->flags & 0x0008) || !wa->running failed: file "c/queue.c" Help or pointers to info would be appreciated. -- Jeffery Boes <>< jeff@endpoint.com http://www.nntp.perl.org/group/perl.loop/2008/09/msg1124.html Wed, 10 Sep 2008 05:40:47 +0000 Re: non blocking issue ? by Isabelle Cabrera `Zidane Tribal a écrit : > On 28/05/2008, Isabelle Cabrera <isabelle.cabrera@inria.fr> wrote: > >> Hi, >> >> I have a problem using the Event module. >> >> My goal: I have several sentences that I need to parse. For a faster >> process, I have a dispatcher which sends the sentences to several hosts >> so that the processes are distributed. >> >> In my dispatcher, I register every host. For each host I create a watcher : >> >> $self->{hosts}{$host}{watcher} = >> Event->var(desc =>"Process sentence on $host", >> var => \$client->{free}, >> poll => 'w', >> cb => $self->next_dag($host), >> debug => 4, >> ); >> >> The callback next_dag is called every time all the process of parsing >> the sentence and sending back the results is done (more precisely when >> the host says it's ready to receive a new sentence, with variable >> $client->{free}). >> >> Of course, I would like that every host can work at the same time >> (that's the point...), but my problem is that they do the job >> subsequently, one sentence at the time. >> >> The created event mainly does the following: >> - get the next sentence from a file >> - send this sentence to the parser on the free host (through a server of >> parser accessed with Net::Telnet) >> - get the result of the parsing (yes or no) (still through Net::Telnet...) >> - if yes, connect again to the server of parser to generate one or >> several outputs and write them into one or several files >> - says "I'm ready to get a new sentence to parse", triggering the >> variable 'free' >> >> I thought that creating an event was automatically non blocking and >> several events could happen at the same time. Does the problem come from >> the I/O interactions ? If so, do I need to separate all these stages and >> watch for every I/O interactions ? And do I need to specify somewhere >> that I want non blocking I/O ? >> >> Best regards, >> Isabelle >> >> -- >> Isabelle Cabrera >> Projet Alpage - INRIA Rocquencourt >> http://alpage.inria.fr >> Tel : 01 3963 5270 >> >> >> > > it sounds like each event is blocking, waiting for the computation to > return a result, and then releasing to allow EV to continue it's loop. > i have had a similar problem recently, (a multi-server irc bot), and > found i have a similar problem, each time the event was blocking and > waiting for a reply. > > how i solved it was to create a readable event for each socket, so > rather than send-wait-receive as you have, its send, then assign a > receive callback to trigger when a result is returned. thus, send > commands down each socket, and allow EV to loop until each one > returns, and as each result is returned, send another query, then let > EV wait on the result again, ad infinitum. > > i suspect this is a horribly confusing answer, and unfortunately i'm > away from home for a few days so i cant provide a code example, but > feel free to ask anything further. > > James Clark. > Thank you James. Finally my problem comes from the fact that my I/O interactions are blocking. So instead of using Net::Telnet I'm trying to use IO::Socket to do non blocking I/O. One other problem is that I had a huge callback, so I'm going to split up my code in smaller peaces. Best regards, Isabelle Cabrera -- Isabelle Cabrera Equipe-Projet Alpage - INRIA Paris-Rocquencourt http://alpage.inria.fr Tel : 01 3963 5270 http://www.nntp.perl.org/group/perl.loop/2008/06/msg1123.html Wed, 04 Jun 2008 05:47:34 +0000 Re: non blocking issue ? by `Zidane Tribal On 28/05/2008, Isabelle Cabrera <isabelle.cabrera@inria.fr> wrote: > Hi, > > I have a problem using the Event module. > > My goal: I have several sentences that I need to parse. For a faster > process, I have a dispatcher which sends the sentences to several hosts > so that the processes are distributed. > > In my dispatcher, I register every host. For each host I create a watcher : > > $self->{hosts}{$host}{watcher} = > Event->var(desc =>"Process sentence on $host", > var => \$client->{free}, > poll => 'w', > cb => $self->next_dag($host), > debug => 4, > ); > > The callback next_dag is called every time all the process of parsing > the sentence and sending back the results is done (more precisely when > the host says it's ready to receive a new sentence, with variable > $client->{free}). > > Of course, I would like that every host can work at the same time > (that's the point...), but my problem is that they do the job > subsequently, one sentence at the time. > > The created event mainly does the following: > - get the next sentence from a file > - send this sentence to the parser on the free host (through a server of > parser accessed with Net::Telnet) > - get the result of the parsing (yes or no) (still through Net::Telnet...) > - if yes, connect again to the server of parser to generate one or > several outputs and write them into one or several files > - says "I'm ready to get a new sentence to parse", triggering the > variable 'free' > > I thought that creating an event was automatically non blocking and > several events could happen at the same time. Does the problem come from > the I/O interactions ? If so, do I need to separate all these stages and > watch for every I/O interactions ? And do I need to specify somewhere > that I want non blocking I/O ? > > Best regards, > Isabelle > > -- > Isabelle Cabrera > Projet Alpage - INRIA Rocquencourt > http://alpage.inria.fr > Tel : 01 3963 5270 > > it sounds like each event is blocking, waiting for the computation to return a result, and then releasing to allow EV to continue it's loop. i have had a similar problem recently, (a multi-server irc bot), and found i have a similar problem, each time the event was blocking and waiting for a reply. how i solved it was to create a readable event for each socket, so rather than send-wait-receive as you have, its send, then assign a receive callback to trigger when a result is returned. thus, send commands down each socket, and allow EV to loop until each one returns, and as each result is returned, send another query, then let EV wait on the result again, ad infinitum. i suspect this is a horribly confusing answer, and unfortunately i'm away from home for a few days so i cant provide a code example, but feel free to ask anything further. James Clark. http://www.nntp.perl.org/group/perl.loop/2008/06/msg1122.html Sun, 01 Jun 2008 09:27:51 +0000 non blocking issue ? by Isabelle Cabrera Hi, I have a problem using the Event module. My goal: I have several sentences that I need to parse. For a faster process, I have a dispatcher which sends the sentences to several hosts so that the processes are distributed. In my dispatcher, I register every host. For each host I create a watcher : $self->{hosts}{$host}{watcher} = Event->var(desc =>"Process sentence on $host", var => \$client->{free}, poll => 'w', cb => $self->next_dag($host), debug => 4, ); The callback next_dag is called every time all the process of parsing the sentence and sending back the results is done (more precisely when the host says it's ready to receive a new sentence, with variable $client->{free}). Of course, I would like that every host can work at the same time (that's the point...), but my problem is that they do the job subsequently, one sentence at the time. The created event mainly does the following: - get the next sentence from a file - send this sentence to the parser on the free host (through a server of parser accessed with Net::Telnet) - get the result of the parsing (yes or no) (still through Net::Telnet...) - if yes, connect again to the server of parser to generate one or several outputs and write them into one or several files - says "I'm ready to get a new sentence to parse", triggering the variable 'free' I thought that creating an event was automatically non blocking and several events could happen at the same time. Does the problem come from the I/O interactions ? If so, do I need to separate all these stages and watch for every I/O interactions ? And do I need to specify somewhere that I want non blocking I/O ? Best regards, Isabelle -- Isabelle Cabrera Projet Alpage - INRIA Rocquencourt http://alpage.inria.fr Tel : 01 3963 5270 http://www.nntp.perl.org/group/perl.loop/2008/05/msg1121.html Wed, 28 May 2008 08:48:34 +0000 announcing AnyEvent 4.0 by Marc Lehmann I am pleased to announce AnyEvent 4.0 Why the major version bump? Lots of functionality has been added (and thus some bugs are expected), although the event core is still compatible to earlier versions and AnyEvent is still pure-perl without any (hard) dependencies on any XS modules. So whats new? Apart from a number of bugfixes, the AnyEvent::Socket module has been added. It contains a lot of utility functions to handle IPv4 and IPv6 addresses and tcp connections in a 100% non-blocking way. It supports SRV record lookups. To make the 100% non-blocking a reality, AnyEvent::Socket uses AnyEvent::DNS, a fully asynchronous stub resolver capable of making tens of thousands of DNS requests/second, with TCP and EDNS0 support. And if you have Net::SSLeay installed, then AnyEvent::Handle can transparently manage TLS/SSL connections, too. On the microshit windows side, AnyEvent now tries very hard to avoid the freeze in the testsuite on broken perls (with reduced functionality, but resolver and socket code works fine there, too). The distro should hit the CPAN mirrors in a few hours. Documentation can be found via its homepage, http://software.schmorp.de/pkg/AnyEvent Example: This (pseudo-) code snippet connects to jabber server, by querying any SRV records that might exist and connecting to the resulting addresses in turn - regardless of wether the server uses ip version 4 or ip version 6 (or both): AnyEvent::Socket::tcp_connect "jabber.ietf.org", "xmpp-client", sub { my ($fh) = @_ or die "connect: $!"; my $handle; $handle = new AnyEvent::Handle fh => $fh, on_eof => sub { undef $handle; # destroy handle on eof }, ; # etc.. # when server and client handshake a SSL connection, one can simply # call $handle->starttls ("connect") to start ssl negotiation. # push some writes or soem reads here, or set some callbacks }; This (non-pseudo-code) code snippet asynchronously connects to https://www.google.de and makes a simple GET request: use AnyEvent::Impl::Perl; # use portable perl event loop use AnyEvent::Handle; use AnyEvent::Socket; my $cv = AnyEvent->condvar; AnyEvent::Socket::tcp_connect "www.google.de", 443, sub { my ($fh) = @_ or die "connect: $!"; my $handle; # avoid direct assignment so on_eof has it in scope. $handle = new AnyEvent::Handle fh => $fh, tls => "connect", # start client-side tls negotiation on_eof => sub { undef $handle; # keep it alive till eof warn "done.\n"; $cv->send; }; $handle->push_write ("GET / HTTP/1.0\015\012\015\012"); $handle->push_read_line ("\015\012\015\012", sub { my ($handle, $line) = @_; # print response header print "HEADER\n$line\n\nBODY\n"; $handle->on_read (sub { # print response body print $_[0]->rbuf; $_[0]->rbuf = ""; }); }); }; $cv->recv; -- The choice of a Deliantra, the free code+content MORPG -----==- _GNU_ http://www.deliantra.net ----==-- _ generation ---==---(_)__ __ ____ __ Marc Lehmann --==---/ / _ \/ // /\ \/ / pcg@goof.com -=====/_/_//_/\_,_/ /_/\_\ http://www.nntp.perl.org/group/perl.loop/2008/05/msg1120.html Sat, 24 May 2008 11:28:22 +0000 Re: benchmarking various event loops with and without anyevent by Rocco Caputo On Apr 28, 2008, at 03:21, Marc Lehmann wrote: > > In all fairness, I want to point out that, after _multiple_ rounds of > longish e-mail exchanges, Rocco Caputo could not solve the problems > that > forced AnyEvent to use this design, nor did he enlighten me on how > to work > around the specific problems that I mentioned to him that forced this > design decision(*). Addressed in <629C022B-BF10-4461-9D52-8F305D2051DC@pobox.com>. Please respond there. > He did not come up with any further evidence for a problem, either > (just > repeatedly stating that the design is broken. The only argument he > brought > up was: one of your design goals is to be reasonably efficient, POE > does > not do it reasonably efficient, so your design is broken, which is an > outright absurd logic). Marc and I are disagreeing with what I wrote in the message included in <E5761512-9F7F-45ED-9C45-A8D9FFFC8C9C@pobox.com>. No amount of he- said/no-he-said will resolve it at this point, so I refer the reader back to the actual exchange. Everyone: Your suggestions to improve my communication are greatly appreciated. Please comment off-list, if you can. > In fact, it seems his problem is indeed the AnyEvent API and not the > interface module to POE, i.e. the "broken" means I should not provide > events in the form AnyEvent does, which is of course > counterproductive to > the goal of AnyEvent of being compatible to multiple event loops (I > can't > provide different APIs to different event loops...). I explicitly stated otherwise in the message included in <E5761512-9F7F-45ED-9C45-A8D9FFFC8C9C@pobox.com >. It's the sentence beginning with "We should not need to change AnyEvent::Impl::POE's public interface". > So I conclude that even the POE author is unable to provide a > (strongly) > more efficient approach, which, according to his own words, would > make him > worse then the average first-time POE user. Considering our previous discussions on the matter, I feel that your conclusion is premature. Also, you seem to be saying that one solution can simultaneously perform equally as well as another and worse than it. Which quantum computers have you ported AnyEvent to lately? :) -- Rocco Caputo - rcaputo@pobox.com http://www.nntp.perl.org/group/perl.loop/2008/04/msg1119.html Mon, 28 Apr 2008 14:19:03 +0000 Re: benchmarking various event loops with and without anyevent by Rocco Caputo On Apr 28, 2008, at 06:24, Marc Lehmann wrote: > [most important points first] > >> In your case, I would create a single persistent POE::Session >> instance >> that serviced all the watchers. > > I would, too, but I cannot find a way to do that with POE: sessions > without active watchers will be destroyed, forcing the session to have > active resources will make the program never-ending. > > I already told you that I tried this approach, and why I couldn't > get it > working. I might use something like an explicit reference count to keep the session alive. I would create a proxy object that, when DESTROYed, would post a shutdown message to the session. Or if AnyEvent knows when a program is to exit, I would have it explicitly shutdown the session as part of its finalization. The shutdown handler would remove the explicit reference count, allowing the session to stop. It's similar to the technique you use in AnyEvent::Impl::POE: sub DESTROY { POE::Kernel->post (${${$_[0]}}, "stop"); } Except it would be done once at the very end, rather than for each event watcher. Perhaps this isn't necessary. You're not using run(), so technically you're free to go at any time. If your program must exit while watchers are active, then you could force the issue by sending a global UIDESTROY signal (designed to tell POE when it must unconditionally stop), and calling run(): $poe_kernel->signal($poe_kernel, "UIDESTROY"); $poe_kernel->run(); On the other hand, your AnyEvent::Impl::POE proxy objects could also hold references to the singleton session, and if they release those references when they clean up their POE::Kernel resources, the session should be "empty" by the time they all destruct. In that case, the UIDESTROY signal should not be needed. run() will return after removing the "empty" session. In short, your AnyEvent::Impl::POE objects would be of the form: sub new { # (AnyEvent::Impl::POE setup goes here) # set up the POE::Kernel watcher $poe_kernel->something(something); # make a note of the watcher in this object $self->{something} = $record_of_the_poe_watcher; return $self; } sub DESTROY { my $self = shift; # ... release the POE::Kernel watcher $poe_kernel->something($self->{something}, something); } If you expect the user to be creating their own POE::Session instances, then you'd need to call() AnyEvent::Impl::POE to make sure the watchers are created in the right context. sub new { # (AnyEvent::Impl::POE setup goes here) # set up the POE::Kernel watcher $poe_kernel->call("anyevent_impl_poe", "something", something); # make a note of the watcher in this object $self->{something} = $record_of_the_poe_watcher; return $self; } And DESTROY would tell the session when to clear the watcher. You may need to add a new AnyEvent::Impl method to explicitly stop POE, especially if your public API allows users to exit with active watchers. sub shutdown { $poe_kernel->signal($poe_kernel, "UIDESTROY"); POE::Kernel->run(); } As an added bonus, shutting down this way satisfies the run() warning. I know this isn't a full solution, but I hope you still find it helpful. > You kepe repeating how it could be designed better, but you never > actually > say how to solve the fundamental problems and bugs within POE that > keep it > from being implementable. I would be welcome to discuss the code more than each-other. If we can agree on this, perhaps we can get down to more important matters. See above. > As I said, if possible, I can only imagine the design becoming > vastly more > complex because I would have to create sdessions on demand and be > able to > react to my session beign turned down at unopportune times. I don't understand why this design is necessary. Please help me understand your design constraints, so that I may focus on designs that will work. > What we seem to agree on by now is that such a design is not trivial > to do > with POE. > > Also, remember that the benchmarks show that session creation is not > the big problem, running the sesions is - of course, there could be > inefficiencies in POE handling large number of sessions, but that > means > just that - POE doesn't scale well. While my suggestions are not as trivial as your current design, I don't think the end design will be as complex as you expect. Thank you for your feedback. I'm sorry that POE doesn't meet your needs. When I have the chance, I'll profile POE while running your benchmark and see what I can do. > As the documentation mentions, AnyEvent doesn't enforce itself on a > module, unlike POE - a module using POE is not going to work with > other > event loops, because it monopolises the process. > > This means that a module using POE forces all its users to also use > POE. This is factually incorrect. For example, POE::Loop::Glib allows POE to be embedded into applications like vim and irssi. The application's functionality is not impaired, nor is communication between the native scripting elements and the embedded POE code. POE defers to the native event loop whenever possible. For example, the main loop from POE::Loop::Gtk: sub loop_run { unless (defined $_watcher_timer) { $_watcher_timer = Gtk->idle_add(\&_loop_resume_timer); } Gtk->main; } In cases where issues prevent POE from using the native loop, a custom loop is written around the native DoOneEvent() (or equivalent) dispatcher. > This is a fundamental difference between POE and AnyEvent, it has > nothing > to do with event loop backend modules, of which POE also emplys a > few, but > comes form the fact that you have to call POE::Kernel->run and give up > your process to it (just like with EV::loop etc.) I believe the above loop_run() above shows this to be factually incorrect. Even when POE::Kernel->run() is called, execution is passed back to the native event loop. AnyEvent::Impl::POE doesn't relinquish control to POE::Kernel->run(), so it seems to conflict with your claim. > On Mon, Apr 28, 2008 at 04:36:42AM -0400, Rocco Caputo <rcaputo@pobox.com > > wrote: > >> Are you aware that I'm gradually rewriting POE's documentation? If >> you could describe what you don't like in a useful way, I may be able >> to do something about it. > > Since I described it already (and you know that) it means you find > the way > I did it "not useful". Thats a strawman argument. If you don't like my > criticism or don't understand it, ask. Okay, I will ask. In all cases, please include sufficient information in your reply so that I can find the documentation in question. What is inaccurate? How is it inaccurate? What parts could I improve? How could I improve them? What is missing? What isn't necessary? If I've overlooked a question to ask, please let me know, and I will ask it. > It is fine with me if you don't understand AnyEvent, it is somewhat > fine > with me if you make strong (But wrong) statements about it, but don't > expect anybody to put much faith in them, or you ability to make > useful > statements. Back at you. See above for some examples where your strong statements against POE have been incorrect. As I've said before, I'd like to resolve these misunderstandings so that we can move forward in a more amicable and efficient fashion. If we can eventually exhaust our misconceptions by thoroughly addressing them, then perhaps we can move on to more productive things. >> Assuming that N is the same between the equivalent POE and >> AnyEvent::Impl::POE program: >> >> S(N*M) > S(N) for M > 1. >> >> QED :P > > I couldn't really follow you here, and I am not sure what you have > proven. To > me it certainly looks as if it was "POE cannot support the AnyEvent > API > efficiently" (at leats not in a simple and straightforward way). I don't know how to be more explicit, unambiguous, and emotionally neutral than by modeling the overhead mathematically. If that's insufficient, then I'm at a loss for how to proceed. >>> I can only imagine making some very complex on-demand >>> instantiating and >>> re- check wether the session still exists on each watcher creation. >> >> Your imagination comes up with such incredible things. Don't lose >> that. :) > > Overbearing words after being wrong before. My bad. It's a long message, and I had to let off some steam. On a more positive note, you do have an inventive mind, and I really do hope that you never lose that quality. >> I use this design in POE::Stage. > > Does it survice multiple run's? If yes, how so? I'm not sure that multiple run()s is a relevant issue. Your code seems to get along without calling run() at all. The number of POE::Session instances has no bearing on whether you can continue using loop_do_timeslice(). I am obviously overlooking something. If you could explain what that is, we could put this detail to rest and move on. >>> (the race condition with sigchld is really annoying, >> >> Please describe the race condition in a useful way, so that I can >> investigate and possibly fix it. > > Again the red herring "useful" - it obviously was useful enough for > other > people to understand the issue. The problem is that rgeistering a > watcher > after the child process already exited causes POE to introduce a > delay in > reapign the child (it does seem to do polling to do so, and seems to > do so > once per second, so the delay is at most a second, still, thats a > pretty > long time). [more specifics snipped] A useful report is one that can be use to replicate or isolate the problem. Saying "the race condition with sigchld" is not useful. Being more specific, as you have just done, is VERY useful. Thank you! An experimental fix for what you describe has already been released. Because it's experimental, you will need to enable it manually. Please verify you're using the most recent POE, and check POE::Kernel's documentation for "USE_SIGCHLD". It appears that the sig_child() documentation conflicts with the USE_SIGCHLD documentation. I have fixed it for the next release. >>> The "run() method was never called" message can be silenced by >>> calling >> run(). Not all applications require run(). But if you call it >> without any POE::Session instances, it will return immediately and >> satisfy POE::Kernel. > > This is impossible it guarentee from within AnyEvent. > > So there is no way to silence just the message. [...] > AnyEvent::Impl::POE already invokes the run workaround in its > manpage, and > why it is undesirable. This puts an unecessary burden on the user, > and is > certainly an issue. Your code already silences this warning, so I don't know why you keep saying it can't be done: # have to do this to keep POE from spilling ugly messages POE::Session->create (inline_states => { _start => sub { @_[KERNEL]- >stop } }); POE::Kernel->run; Unless AnyEvent is doing something I don't know, you should be able to drop the POE::Session->create() line without any harm. >> You complain about a message that occurs when a program leaks a child >> process. Programs that leak processes are broken in a way that is >> difficult to detect without the warning. In extreme cases, process >> leakage can quietly strangle a machine to death. > > No, I complain about the emssage complaining about POE reaping a > child. No > leak is involved anywhere, and nothign will "quietly" "strangle" > anything > to death. The message I'm thinking of occurs at the end of the program when POE checks whether any child processes remain. Exiting without reaping children is considered bad form and can lead to orphan processes on some systems, so POE enforces this behavior. The warning indicates that the application has not been looking after its children. POE doesn't reap children unless there is a sig_child() watcher. Therefore, if your program is long-running, spawns processes during its normal operation, and fails to trigger reaping, it will eventually exhaust its process resources. If the application runs as root, there will often be no ulimit guard preventing zombie processes from filling up the system's process table. If this happens, the system will most likely become unresponsive and need to be restarted. I consider such applications to be broken. You are welcome to think otherwise, but POE will continue to point out when this happens. The most direct solution is for an application to set sig_child() handlers for the children it creates. This will trigger reaping, and the warnings (and potential for catastrophe) will go away. >> OMG, we agree on something! The end times are here: > > sorry? It's a joke. You see, we agree on so little, so actually agreeing on something is a momentous event. I was humorously drawing parallels to events that signal the impending end of the world in some religions. The "end times" are also conveniently where I finished my e-mail. It's a bit of a pun, really. -- Rocco Caputo - rcaputo@pobox.com http://www.nntp.perl.org/group/perl.loop/2008/04/msg1118.html Mon, 28 Apr 2008 12:51:45 +0000 Re: benchmarking various event loops with and without anyevent by Marc Lehmann [most important points first] > In your case, I would create a single persistent POE::Session instance > that serviced all the watchers. I would, too, but I cannot find a way to do that with POE: sessions without active watchers will be destroyed, forcing the session to have active resources will make the program never-ending. I already told you that I tried this approach, and why I couldn't get it working. You kepe repeating how it could be designed better, but you never actually say how to solve the fundamental problems and bugs within POE that keep it from being implementable. As I said, if possible, I can only imagine the design becoming vastly more complex because I would have to create sdessions on demand and be able to react to my session beign turned down at unopportune times. What we seem to agree on by now is that such a design is not trivial to do with POE. Also, remember that the benchmarks show that session creation is not the big problem, running the sesions is - of course, there could be inefficiencies in POE handling large number of sessions, but that means just that - POE doesn't scale well. On Mon, Apr 28, 2008 at 04:36:42AM -0400, Rocco Caputo <rcaputo@pobox.com> wrote: > Most people on the planet don't know Perl, or even how to program a > computer. No amount of documentation will help them. :) Good (or any) documentation is widely known to be almost a _requirement_ for learning how to program or use a software package. > In the future, you may wish to include me on your list of people to > consult about designing applications for POE. I may known a thing or > two about the topic. :) I am not interested in designing applications for POE - I am interested in making it possible to write event-based modules that are interoperable between various event loops such as POE. As the documentation mentions, AnyEvent doesn't enforce itself on a module, unlike POE - a module using POE is not going to work with other event loops, because it monopolises the process. This means that a module using POE forces all its users to also use POE. AnyEvent does not do that, as long as it supports the event model actually in use (it is not an event loop itself!): A module that uses AnyEvent works with both Qt, POE, IO::Async (once its backend is implemented), EV and so on. This is a fundamental difference between POE and AnyEvent, it has nothing to do with event loop backend modules, of which POE also emplys a few, but comes form the fact that you have to call POE::Kernel->run and give up your process to it (just like with EV::loop etc.) > Are you aware that I'm gradually rewriting POE's documentation? If > you could describe what you don't like in a useful way, I may be able > to do something about it. Since I described it already (and you know that) it means you find the way I did it "not useful". Thats a strawman argument. If you don't like my criticism or don't understand it, ask. > I would love to have the opportunity to suggest a different design, You always ahd the opportunity, you are not using it. > Obviously I cannot expect you to know everything about POE. Likewise, > you cannot expect me to magically know when you started writing > AnyEvent::Impl::POE. It doesn't matter when I started writing AnyEvent::Impl::POE at all. What matters is that you made unfounded (and as you now admit, wrong) statements about it (and its author). It is fine with me if you don't understand AnyEvent, it is somewhat fine with me if you make strong (But wrong) statements about it, but don't expect anybody to put much faith in them, or you ability to make useful statements. > Even if you announced it somewhere, I may not have been looking. The > first I heard of it was here, when you announced your benchmarks. Yes, so? > In general, if you need someone's attention online, the most effective > and polite way is to contact them directly. I did not need your attention? > about this, then I'm sorry that I missed it. Are you sure your > message wasn't lost in transit? Which message? I was only conveying some benchmark results, to give people an idea of the overheads of AnyEvent of various event loop implementations in the hope of beign useful. I was also hoping that people might give AnyEvent some try, as it's design doesn't force a module author using it into a specific event loop. > Assuming that N is the same between the equivalent POE and > AnyEvent::Impl::POE program: > > S(N*M) > S(N) for M > 1. > > QED :P I couldn't really follow you here, and I am not sure what you have proven. To me it certainly looks as if it was "POE cannot support the AnyEvent API efficiently" (at leats not in a simple and straightforward way). I knew that already. > >I can only imagine making some very complex on-demand instantiating and > >re- check wether the session still exists on each watcher creation. > > Your imagination comes up with such incredible things. Don't lose > that. :) Overbearing words after being wrong before. > The "trick" is to minimize the number of sessions used. I already said that, and explaiend why it didn't work. > that sessions imposed overhead. I wrongly assumed the solution would > be obvious. Yes. > I use this design in POE::Stage. Does it survice multiple run's? If yes, how so? > experimental code, you can find it on the CPAN. It also does a lot of > other, unrelated things, so you may have difficulty separating the > magic you need from the voodoo you don't. As I said, less magic and vodoo and better documentation could be more helpful in the long run. I don't see how POE::Stage solves the problem of multiple run's stopping sessions. If it does, couldn't you just advise me on how to do it insteadf of referring to bugloads of other code? Either you know how to solve the issues, then just tell us, or you don't, in which case going through large amounts of code won't help. > I won't blame you. But I will point out that your documentation says > you're already familiar with using undocumented POE features. :) Yes indeed :/ > >Instead of going around and accuse people of making bad designs, it > >would be much better to improve the documentation for POE, so that said > >people don't have to go around and start guessing... > I don't appreciate your vague, negative comments about the > documentation. Please describe problems in a useful way, or your > expectation that they be fixed is unreasonable. I did, in a long document that you admit having read already. If you don't find the comments useful, thats your problem - you could aks me to clarify them, for example. _I_ don't appreciate your continuous implications that my comemnts were in some way flawed without every mentioning a single piece of evidence. > You seem to be saying that your design for AnyEvent::Impl::POE is > based on guesswork. Yes. > If so, I overestimated your knowledge of POE based on your > documentation. I have no idea how you estimate my knowledge. I also fail to see what my person has to do with anything. > Therefore, please consider the chance that you may know less about POE > than your documentation implies. If there is any meaning in this sentence, it escapes me. I quited examples form the documentation and explained all my arguments. It doesn't matter how much I know or imply to know, as I *explain* the reasons behind what I said. That is in stark contrast to you - you only ever imply the documentation is rude, the module is fundamentally flawed, the comments are not useful etc., without ever trying to solve the problems with POE that lead to this design and the comments, and which are explained clearly. > >Oh, and while you fix the docs, could you fix the other bugs as well > > I certainly can, but you'll need to describe them in a useful way Done a long time ago and given to you. It might not be in a form that you like, but it is doubtless that they are useful. > >(the race condition with sigchld is really annoying, > > Please describe the race condition in a useful way, so that I can > investigate and possibly fix it. Again the red herring "useful" - it obviously was useful enough for other people to understand the issue. The problem is that rgeistering a watcher after the child process already exited causes POE to introduce a delay in reapign the child (it does seem to do polling to do so, and seems to do so once per second, so the delay is at most a second, still, thats a pretty long time). > POE checks for child processes between calls to application code. If > you fork your process and set sig_child() in the same event handler, > POE won't be able to call waitpid() between them. If this is the race > condition you mean, then it simply shouldn't be possible. The race condition is quite simple: 1. fork 2. child exits 3. parents gets sigchld (but no sigchld handler is registered yet) 4. sigchld watchers gets created [time passes, POE select's many times without noticing the child exit] 5. POE eventually waitpids and finally detetcs the child > On the other hand, I haven't actually documented this behavior. It > hadn't occurred to me because all my programs follow the reliable > pattern. I have no clue what you mean, but POE ahs to support it, just as with any other event loop - you cnanot tregister a child handler BEFORE you fork because you don't know the pid. You cannot guarantee that the child exits only afetr you registered the watcher, so POE has to deal with it already having exited. AnyEvent and EV certainly deal with this issue cirretcly and don't miss the child exit. > "Programs that wish to reliably reap child processes should be sure to > call sig_child() before returning from the event handler that forked > the process. Otherwise POE::Kernel may have an opportunity to call > waitpid() before an appropriate event watcher has been registered." No, it is a race in POE, not something the application can do anything about. > In the event that you've found an actual bug, I regret that your > description of the problem is not useful. Well, I can declare anything I want "not useful" - if you are incapable of understanding, you should ask - I don't know how wlel your knowledge is w.r.t. to process handling on unix, and I cannot write full novels and programming manuals just in case I need to cover something you don't understand. > <purl> Look buddy, doesn't work is a strong statement. Does it sit on "doesn't work" is certainly not how I described it. > >and the nag messages are as well). > > I understand from your documentation that you dislike the "ugly" "nag" > messages. I'm sorry, but I cannot help you there. They were added as > the result of useful feedback from other users. And contrary to your > documentation, they can be silenced. Possible, but it is not documented (yes, run silences them, but it has other undesirable effects as well, which is documented by AnyEvent's adaptor module). > The "run() method was never called" message can be silenced by calling > run(). Not all applications require run(). But if you call it > without any POE::Session instances, it will return immediately and > satisfy POE::Kernel. This is impossible it guarentee from within AnyEvent. So there is no way to silence just the message. > While this does not: > > perl -wle 'use POE; POE::Kernel->run; POE::Session- > >create(inline_states => { _start => sub{} });' As I said, for design reasons, Anyevent doesn't enforce itself on its users, so the requirement to call ->run is not implementable. AnyEvent::Impl::POE already invokes the run workaround in its manpage, and why it is undesirable. This puts an unecessary burden on the user, and is certainly an issue. > I hadn't considered to document the technique because it seemed > obvious to me. It is obvious to me. But there is now ay to do that from within a module: a module has no control over when it was loaded. It is just one of the many small design issues (some of which I documented) that make it very hard to use POE in a generic way. I don't claim more, I don't claim less. > obvious to me than are actually obvious to the average user. You > should probably call them to my attention if you'd like them to be > documented. I only need a way to silence them withotu potentially freezing the program, clearing the event queue, stopping running sessions etc. > Speaking of which, I added this to the docs. Please let me know how/ > if I can improve it: No amount of documentation can improve this, it is a problem within the POE code. > You complain about a message that occurs when a program leaks a child > process. Programs that leak processes are broken in a way that is > difficult to detect without the warning. In extreme cases, process > leakage can quietly strangle a machine to death. No, I complain about the emssage complaining about POE reaping a child. No leak is involved anywhere, and nothign will "quietly" "strangle" anything to death. > POE can detect this silent but deadly condition. There is nothing deadly about it. > If you receive this message, it can be silenced by fixing the process > leak. ... which is nonexistant. > >Not sure why it would be a courtesy to me, do you have anything to > >hide > >or do you want to insult me even more in private so nobody sees your > >real > >self or something? (just guessing... :) > > Wow, you really do expect the worst from me. I don't know where that > came from. I expect you to back up your statements you made publicly (and in private). As I told you before, it *is* bad (or worse) to make statements without backing them up. > The courtesy to you would be that I brought the issues to you > directly. It would be, if you had done that. But in private you just insulted me more directly, but despite reqpeatedly askign you how to solve the probelms with session lifetime, you never came up with any evidence. > I could post them somewhere public and unexpected and wait > for you to find them, but that's just rude. Thats much better then making public statements and not tellign anybody about it, especially not me. > Likewise, posting long, limited-interest e-mail to a public > distribution list is a violation of common etiquette. To those who > are still reading, I'm sorry. Discussing various event loops for perl is certainly within the charter of this mailing list. Also note that you started it - I backed up all my claims with evidence (even if it is wrong, one can at least verify my claims). All you did is make broad statements such as "your module has a broken design" without ever explaining how to work around the problems I documented that caused this allegedly "broken design". > I kind of thought those points would be obvious. It is possible that everything is obvious to you, but if you make broad statements like the above you *need to back them up with evidence*. > If you still insist on doing this publicly, please indicate your > intent by posting your documentation for AnyEvent::Impl::POE to the > mailing list (new thread, please). The docoumentation always was publicly available from the AnyEvent site, also where the benchmakr results have bene published. Anyevent 3.3 has now been released with the POE module, though, so it should not be a problem to find it anymore, it is on CPAN. > OMG, we agree on something! The end times are here: sorry? -- The choice of a Deliantra, the free code+content MORPG -----==- _GNU_ http://www.deliantra.net ----==-- _ generation ---==---(_)__ __ ____ __ Marc Lehmann --==---/ / _ \/ // /\ \/ / pcg@goof.com -=====/_/_//_/\_,_/ /_/\_\ http://www.nntp.perl.org/group/perl.loop/2008/04/msg1117.html Mon, 28 Apr 2008 03:24:37 +0000 Re: benchmarking various event loops with and without anyevent by Rocco Caputo On Apr 28, 2008, at 03:21, Marc Lehmann wrote: > On Sat, Apr 26, 2008 at 08:13:27AM -0400, Rocco Caputo <rcaputo@pobox.com > > wrote: >> each event watcher. Anyone who knows POE can tell you this is one of >> the least efficient designs possible. In fact, this design is worse >> than the average for first-time POE users. > > [He then called my model fundamentally broken in private mail and the > documentation rude and unprofessional, without bringing up any > evidence] Dear perl-loop@perl.org, and anyone reading this in the archives. I apologize for my part in this unfolding thread. As Marc mentions, I have been trying to take it to private e-mail. However I feel that Marc has misrepresented to the list what I said to him in private. At this point, it's easier for me to repost what I actually said rather than paraphrase it. Again, I'm sorry you had to be involved. -- Rocco Caputo - rcaputo@pobox.com Begin forwarded message: > From: Rocco Caputo <rcaputo@pobox.com> > Date: April 27, 2008 23:29:18 EDT > To: Marc Lehmann <schmorp@schmorp.de> > Subject: Re: benchmarking various event loops with and without > anyevent > > On Apr 27, 2008, at 01:53, Marc Lehmann wrote: > >> On Sun, Apr 27, 2008 at 01:15:49AM -0400, Rocco Caputo <rcaputo@pobox.com >> > wrote: >>> I have read your code and documentation for AnyEvent::Impl::POE. >>> Your >>> module's design is fundamentally broken, and your code is probably >>> more to blame than POE. >> >> Oh, btw, be careful with such strong idioms such as "fundemantally >> broken": >> so far, there is no evidence that it is broken at all, only >> inefficient. >> >> (If you think it really is fundamentally _broken_ then you better >> back up >> your statements). > > I hope to show that I intended no offense. > > Reasonable scalability (CPU and memory) seems to be one of > AnyEvent's design goals. I base this impression on the fact that > you're benchmarking your code in terms of speed and size. > "Reasonable" is subject to interpretation, but I think we agree that > AnyEvent::Impl::POE is neither as fast nor as small as it should be. > > Therefore, while AnyEvent::Impl::POE operates correctly, it does not > fulfill some of AnyEvent's design goals. > > AnyEvent::Impl::POE's greatest inefficiencies stem from one > fundamental design choice: the 1:1 relationship between watcher > instances and POE::Session instances. In your own words: "AnyEvent > has to create one POE::Session per event watcher, which is immensely > slow and makes watchers very large." > > One point of contention may be whether this is a design or > implementation flaw. The problem is inherent in the way one class > (AnyEvent::Impl::POE) interacts with another (POE::Session). Class > interaction is a software design issue. It can be modeled in > software design languages such as UML. Re-implementing the same > entity relationship more efficiently, or in a faster language such > as C, would not resolve the scalability problem. > > Therefore, AnyEvent::Impl::POE is flawed in design rather than > implementation. > > Therefore, AnyEvent::Impl::POE's design prevents it from meeting > some of AnyEvent's design goals. > > Therefore, AnyEvent::Impl::POE's design is broken. > > Unfortunately most of AnyEvent::Impl::POE's design stems from its > flawed interaction with POE::Session. We should not need to change > AnyEvent::Impl::POE's public interface, but we will need to rethink > and revise nearly all aspects of its interaction with POE. > > Therefore, AnyEvent::Impl::POE's design flaw is a fundamental one. > > Therefore, AnyEvent::Impl::POE's design is fundamentally broken. > >> Again, let me repeat that empty insults that obviously are founded by >> paranoia will not have any positive effect on your standing with me >> (I >> mean, I won't hate you or anything, but you make yourself an idiot >> in my >> eyes very quickly by repeatedly not beinging up any evidence...). >> >> Of course, I understand that if you mistook my comments about POE >> as rude, >> there is a natural tendency to "insult back". > > > I hope I have shown reasonable evidence that my assertion is neither > empty nor intrinsically insulting. Without the intent to insult, > there can be no intent to "insult back". In the end, any offense > you have taken may be of your own manufacture. Could there be > cultural differences to overcome? > > In this light, your assertion of my paranoia is unfounded, unjust > and offensive. Your view that I'm acting like an idiot is no > better. You are of course entitled to your opinions, but those two > are not appropriate for polite conversation. > > ... > > I'm sorry that I haven't responded promptly to your e-mail. I get > the impression that you expect the worst from me, so I feel the need > to choose my words in a slow and painful (and often futile) effort > to minimize being misunderstood. As a result, I cannot write to you > as often as I would like. Nor have I had the opportunity to work > with you on the module in question (something I would much rather be > doing). > > I hope we can reach a mutual understanding, if only so that we may > correspond and collaborate more effectively. > > Until next time, > > -- > Rocco Caputo - rcaputo@pobox.com http://www.nntp.perl.org/group/perl.loop/2008/04/msg1116.html Mon, 28 Apr 2008 02:21:09 +0000 Re: benchmarking various event loops with and without anyevent by Rocco Caputo On Apr 27, 2008, at 00:56, Marc Lehmann wrote: > On Sat, Apr 26, 2008 at 08:13:27AM -0400, Rocco Caputo <rcaputo@pobox.com > > wrote: >> each event watcher. Anyone who knows POE can tell you this is one of >> the least efficient designs possible. > > It is the only design that I could get working, even after > consulting a > few people and implementing some workarounds for the bugs in POE. > > In any case, you have to consider that most people on this planet > don't know POE, and even if, they don't know it that well. Since the > documentation for POE is in such a bad state, thats the obvious way > to fix > that. Most people on the planet don't know Perl, or even how to program a computer. No amount of documentation will help them. :) In the future, you may wish to include me on your list of people to consult about designing applications for POE. I may known a thing or two about the topic. :) Are you aware that I'm gradually rewriting POE's documentation? If you could describe what you don't like in a useful way, I may be able to do something about it. >> In fact, this design is worse than the average for first-time POE >> users. > > If a better design is possible, it is not known to me, and you haven't > suggested one either, so talk is cheap. I'd be happy to get a more > efficient design for POE but nobody could come up with one that also > worked reliably through multiple iterations of run and also does not > keep > the POE kernel from returning. I would love to have the opportunity to suggest a different design, but most of my discretionary time is spent addressing the constant misunderstandings between us. If we can first resolve them, I'll have that much more time to work on the design. Obviously I cannot expect you to know everything about POE. Likewise, you cannot expect me to magically know when you started writing AnyEvent::Impl::POE. Even if you announced it somewhere, I may not have been looking. The first I heard of it was here, when you announced your benchmarks. In general, if you need someone's attention online, the most effective and polite way is to contact them directly. If you did contact me about this, then I'm sorry that I missed it. Are you sure your message wasn't lost in transit? As for the design: First-time POE users tend to design programs where the number of sessions scales linearly with the number of objects that handle events. If S(1) is the total overhead imposed by a single session, then S(N) is the overhead imposed by the average naïve POE user. N is the number of objects handling events. AnyEvent::Impl::POE creates a new POE::Session for every event watcher. It's not uncommon for an object to use more than one event watcher (I/O and timeout, for example). So we can model the session overhead in an AnyEvent::Impl::POE program as S(N*M), where N is the number of objects handling events, and M is the average number of event watchers per object. Assuming that N is the same between the equivalent POE and AnyEvent::Impl::POE program: S(N*M) > S(N) for M > 1. QED :P > If a better design *is* possible (which I don't really doubt), then it > needs to be vastly more complex, or it needs some non-obvious trick. > I can > only imagine making some very complex on-demand instantiating and re- > check > wether the session still exists on each watcher creation. Your imagination comes up with such incredible things. Don't lose that. :) The "trick" is to minimize the number of sessions used. Your benchmarks and comments in your documentation implied that you knew that sessions imposed overhead. I wrongly assumed the solution would be obvious. In your case, I would create a single persistent POE::Session instance that serviced all the watchers. The watchers themselves would be small proxies that controlled POE::Kernel watchers within that session's context. I use this design in POE::Stage. If you're not averse to looking at experimental code, you can find it on the CPAN. It also does a lot of other, unrelated things, so you may have difficulty separating the magic you need from the voodoo you don't. Comments are welcome, if they're useful. > (It is possible that I was fooled by the docs as well, so if there > is a > better way, it likely isn't documented, so who could blame me). I won't blame you. But I will point out that your documentation says you're already familiar with using undocumented POE features. :) > Instead of going around and accuse people of making bad designs, it > would > be much better to improve the documentation for POE, so that said > people > don't have to go around and start guessing... I don't appreciate your vague, negative comments about the documentation. Please describe problems in a useful way, or your expectation that they be fixed is unreasonable. You seem to be saying that your design for AnyEvent::Impl::POE is based on guesswork. If so, I overestimated your knowledge of POE based on your documentation. Therefore, please consider the chance that you may know less about POE than your documentation implies. > Oh, and while you fix the docs, could you fix the other bugs as well I certainly can, but you'll need to describe them in a useful way first. http://search.yahoo.com/search?p=useful+bug+reports seems to link to good tips. And please post them to POE's rt.cpan.org queue. The Perl community has been kind enough to establish a convenient, central place to report problems with CPAN distributions. It would be most cooperative of you to use it. > (the race condition with sigchld is really annoying, Please describe the race condition in a useful way, so that I can investigate and possibly fix it. POE checks for child processes between calls to application code. If you fork your process and set sig_child() in the same event handler, POE won't be able to call waitpid() between them. If this is the race condition you mean, then it simply shouldn't be possible. On the other hand, I haven't actually documented this behavior. It hadn't occurred to me because all my programs follow the reliable pattern. Based on your feedback, I added this to the description of sig_child(). Please let me know how/if I can improve it: "Programs that wish to reliably reap child processes should be sure to call sig_child() before returning from the event handler that forked the process. Otherwise POE::Kernel may have an opportunity to call waitpid() before an appropriate event watcher has been registered." In the event that you've found an actual bug, I regret that your description of the problem is not useful. As we say in IRC: <purl> Look buddy, doesn't work is a strong statement. Does it sit on the couch all day? Is it making faces at you? Does it want more money? Is it sleeping with your girlfriend? Please be specific! > and the nag messages are as well). I understand from your documentation that you dislike the "ugly" "nag" messages. I'm sorry, but I cannot help you there. They were added as the result of useful feedback from other users. And contrary to your documentation, they can be silenced. The "run() method was never called" message can be silenced by calling run(). Not all applications require run(). But if you call it without any POE::Session instances, it will return immediately and satisfy POE::Kernel. For example, this generates the warning: perl -wle 'use POE; POE::Session->create(inline_states => { _start => sub{} });' While this does not: perl -wle 'use POE; POE::Kernel->run; POE::Session- >create(inline_states => { _start => sub{} });' I hadn't considered to document the technique because it seemed obvious to me. By now I hope it's clear that a lot more things seem obvious to me than are actually obvious to the average user. You should probably call them to my attention if you'd like them to be documented. Speaking of which, I added this to the docs. Please let me know how/ if I can improve it: "POE::Kernel will print a strong message if a program creates sessions but fails to call run(). If the lack of a run() call is deliberate, you can avoid the message by calling it before creating a session. run() at that point will return immediately, and POE::Kernel will be satisfied." ... You complain about a message that occurs when a program leaks a child process. Programs that leak processes are broken in a way that is difficult to detect without the warning. In extreme cases, process leakage can quietly strangle a machine to death. POE can detect this silent but deadly condition. It notifies users out of courtesy to them, and also because I was tired of handling "POE crashed my machine!" reports. In this case, the one who smelled it did not deal it. If you receive this message, it can be silenced by fixing the process leak. >> I have specific issues with your docs. As a courtesy to you and the >> list, I'll send them to you directly. > > Not sure why it would be a courtesy to me, do you have anything to > hide > or do you want to insult me even more in private so nobody sees your > real > self or something? (just guessing... :) Wow, you really do expect the worst from me. I don't know where that came from. The courtesy to you would be that I brought the issues to you directly. I could post them somewhere public and unexpected and wait for you to find them, but that's just rude. Likewise, posting long, limited-interest e-mail to a public distribution list is a violation of common etiquette. To those who are still reading, I'm sorry. I kind of thought those points would be obvious. If you still insist on doing this publicly, please indicate your intent by posting your documentation for AnyEvent::Impl::POE to the mailing list (new thread, please). That will make it easier for me to respond. For the record, I still think this is a bad idea. > I do fix my bugs and am open to suggestions and improvements. But > correctness comes first, performance second. OMG, we agree on something! The end times are here: -- Rocco Caputo - rcaputo@pobox.com http://www.nntp.perl.org/group/perl.loop/2008/04/msg1115.html Mon, 28 Apr 2008 01:37:00 +0000 Re: benchmarking various event loops with and without anyevent by Marc Lehmann On Sat, Apr 26, 2008 at 08:13:27AM -0400, Rocco Caputo <rcaputo@pobox.com> wrote: > each event watcher. Anyone who knows POE can tell you this is one of > the least efficient designs possible. In fact, this design is worse > than the average for first-time POE users. [He then called my model fundamentally broken in private mail and the documentation rude and unprofessional, without bringing up any evidence] In all fairness, I want to point out that, after _multiple_ rounds of longish e-mail exchanges, Rocco Caputo could not solve the problems that forced AnyEvent to use this design, nor did he enlighten me on how to work around the specific problems that I mentioned to him that forced this design decision(*). He did not come up with any further evidence for a problem, either (just repeatedly stating that the design is broken. The only argument he brought up was: one of your design goals is to be reasonably efficient, POE does not do it reasonably efficient, so your design is broken, which is an outright absurd logic). In fact, it seems his problem is indeed the AnyEvent API and not the interface module to POE, i.e. the "broken" means I should not provide events in the form AnyEvent does, which is of course counterproductive to the goal of AnyEvent of being compatible to multiple event loops (I can't provide different APIs to different event loops...). So I conclude that even the POE author is unable to provide a (strongly) more efficient approach, which, according to his own words, would make him worse then the average first-time POE user. This is absurd, so I conclude that the original claim has been disproven. (And yes, I did ask multiple times to come up with how to better design the interface to POE, or how to solve the lifetime-issues with POE). (*) the specific problems are (taken directly from my mail to rocco): - the session must not go away, or there must be an easy way to recreate it when the kernel kills it. - the session itself must not keep the kernel "alive"/running (preferably without going away). -- The choice of a Deliantra, the free code+content MORPG -----==- _GNU_ http://www.deliantra.net ----==-- _ generation ---==---(_)__ __ ____ __ Marc Lehmann --==---/ / _ \/ // /\ \/ / pcg@goof.com -=====/_/_//_/\_,_/ /_/\_\ http://www.nntp.perl.org/group/perl.loop/2008/04/msg1114.html Mon, 28 Apr 2008 00:22:12 +0000 Re: benchmarking various event loops with and without anyevent by Marc Lehmann On Sat, Apr 26, 2008 at 08:13:27AM -0400, Rocco Caputo <rcaputo@pobox.com> wrote: > each event watcher. Anyone who knows POE can tell you this is one of > the least efficient designs possible. It is the only design that I could get working, even after consulting a few people and implementing some workarounds for the bugs in POE. In any case, you have to consider that most people on this planet don't know POE, and even if, they don't know it that well. Since the documentation for POE is in such a bad state, thats the obvious way to fix that. > In fact, this design is worse than the average for first-time POE users. If a better design is possible, it is not known to me, and you haven't suggested one either, so talk is cheap. I'd be happy to get a more efficient design for POE but nobody could come up with one that also worked reliably through multiple iterations of run and also does not keep the POE kernel from returning. If a better design *is* possible (which I don't really doubt), then it needs to be vastly more complex, or it needs some non-obvious trick. I can only imagine making some very complex on-demand instantiating and re-check wether the session still exists on each watcher creation. (It is possible that I was fooled by the docs as well, so if there is a better way, it likely isn't documented, so who could blame me). Instead of going around and accuse people of making bad designs, it would be much better to improve the documentation for POE, so that said people don't have to go around and start guessing... Oh, and while you fix the docs, could you fix the other bugs as well (the race condition with sigchld is really annoying, and the nag messages are as well). > I have specific issues with your docs. As a courtesy to you and the > list, I'll send them to you directly. Not sure why it would be a courtesy to me, do you have anything to hide or do you want to insult me even more in private so nobody sees your real self or something? (just guessing... :) I do fix my bugs and am open to suggestions and improvements. But correctness comes first, performance second. Greetings, -- The choice of a Deliantra, the free code+content MORPG -----==- _GNU_ http://www.deliantra.net ----==-- _ generation ---==---(_)__ __ ____ __ Marc Lehmann --==---/ / _ \/ // /\ \/ / pcg@goof.com -=====/_/_//_/\_,_/ /_/\_\ http://www.nntp.perl.org/group/perl.loop/2008/04/msg1113.html Sat, 26 Apr 2008 21:56:31 +0000 Re: benchmarking various event loops with and without anyevent by Rocco Caputo On Apr 25, 2008, at 22:46, Marc Lehmann wrote: > > The next release of AnyEvent contains support for a few more > "backends", > notably POE, so AnyEvent is now by definition compatible to POE > (before it > was only compatible when using an even loop used by POE, such as > Event or > EV that could be shared). > The results were mostly as expected, with EV leading and POE being > abysmal. It's no wonder that AnyEvent::Impl::POE runs so slowly. According to your documentation, you designed it to use a separate POE::Session for each event watcher. Anyone who knows POE can tell you this is one of the least efficient designs possible. In fact, this design is worse than the average for first-time POE users. On the bright side, you could still have done worse. You could have instantiated and destroyed a POE::Session for each event. Perhaps you're saving that for a future release. :) I have specific issues with your docs. As a courtesy to you and the list, I'll send them to you directly. -- Rocco Caputo - rcaputo@pobox.com http://www.nntp.perl.org/group/perl.loop/2008/04/msg1112.html Sat, 26 Apr 2008 05:13:43 +0000 Re: benchmarking various event loops with and without anyevent by Uri Guttman >>>>> "ML" == Marc Lehmann <schmorp@schmorp.de> writes: ML> The surprising one was the pure perl implementation, which was quite on ML> par with C-based event loops such as Event or Glib. I did expect the pure ML> perl implementatioon to be at least a factor of three slower than Event or ML> Glib. ML> As the pure perl loop wasn't written with high performance in mind, this ML> prompted me to optimise it for some important cases (mostly to get rid of ML> the O(n²) degenerate cases and improving the select bitmask parsing for ML> the sparse case). check out stem's pure perl event loop. there are examples in the /sessions dir on how to use that directly without the rest of the modules. it does things in a different direction and doesn't scan select's bit masks but instead it scans the interesting handles and see whether their bits are set. it should exhibit good behavior under growth as all the data are managed in hashes. ML> I then made a second benchmark, designed not to measure anyevent overhead, ML> but to measure real-world performance of a socket server. and that /sessions code also shows use of the asyncio module. if you can benchmark that i would be interested in the results. ML> The result is that the pure perl event loop used as fallback in AnyEvent ML> single-handedly beats Glib by a large margin, and even event by a factor ML> of two. ML> For small servers, the overhead introduced by running a lot of perl ML> opcodes per iteration dominates, however, reflected in the last benchmark. in a heavily loaded server most of the work in in the i/o and should overwhelm the event loop itself. that is the whole purpose of event loops as we all know here. ML> However, the net result is that the pure perl event loop performs ML> better than almost all other event loops (EV being the only exception) ML> ins erious/medium-sized cases, while I originally expected it to fail ML> completely w.r.t. performance and being only usable as a workaround when ML> no "better" event module is installed. i don't find that surprising. perl's i/o is decent and as i said above, a loaded server is doing mostly i/o. ML> All the benchmark data and explanations can be found here: ML> http://pod.tst.eu/http://cvs.schmorp.de/AnyEvent/lib/AnyEvent.pm#BENCHMARKS ML> The code is not yet released and likely still buggy (the question is ML> whether any bugs affect the benchmark results). It is only available via ML> CVS: http://software.schmorp.de/pkg/AnyEvent i will take a gander and see if i can play with it and add stem's loop to it. if you want to work on this with me, i wouldn't mind the help. thanx, uri -- Uri Guttman ------ uri@stemsystems.com -------- http://www.sysarch.com -- ----- Perl Code Review , Architecture, Development, Training, Support ------ --------- Free Perl Training --- http://perlhunter.com/college.html --------- --------- Gourmet Hot Cocoa Mix ---- http://bestfriendscocoa.com --------- http://www.nntp.perl.org/group/perl.loop/2008/04/msg1111.html Sat, 26 Apr 2008 02:34:46 +0000 Re: benchmarking various event loops with and without anyevent by Uri Guttman >>>>> "ML" == Marc Lehmann <schmorp@schmorp.de> writes: ML> On Fri, Apr 25, 2008 at 11:40:03PM -0400, Uri Guttman <uri@stemsystems.com> wrote: >> check out stem's pure perl event loop. there are examples in the ML> Maybe I'll provide a backend for stem. actually it makes more sense to me to wrap anyevent in stem. it already has several event wrappers (pure perl, event.pm and tk) and wrapping is very easy to do. not much different than the code i see in anyevent. >> modules. it does things in a different direction and doesn't scan >> select's bit masks but instead it scans the interesting handles and see >> whether their bits are set. it should exhibit good behavior under growth >> as all the data are managed in hashes. ML> That is exactly as the anyevent perl backend did in the previous ML> version. I don't see how that should exhibit good behaviour - file ML> descristpors are small integers, so hashes only add overhead over arrays, ML> and checking every file descriptor that way is much slower than scanning ML> the bitmask. ML> Especially in the important case of many handles/few active ones, an approach ML> of "scan all handles" is very slow. my code doesn't scan all handle, but scan all writeable events to see if their handle bits are set. descriptors increment and can cause bloat of the array if you have many in use (and not all of them in the event loop). ML> (The benchmarks reflect this, you could try with an older anyevent ML> release where we just check the keys of the hash storing fd ML> information: performance is much lower). ML> I then made a second benchmark, designed not to measure anyevent overhead, ML> but to measure real-world performance of a socket server. >> >> and that /sessions code also shows use of the asyncio module. if you can >> benchmark that i would be interested in the results. ML> Hmm, tt seems to have become fashionable lately to call synchronous I/O ML> asynchronous (see IO::Async, another 100% synchronous framework). What ML> exactly is asynchronous about the I/O in that example, it seems to be ML> fully synchronous to me (just event-driven). event loops are async in that you get callbacks as they are needed. sure you can't get true async behavior from any cpu/os combo but calling it async is no different than calling it parallel processing when it is just context switching among processes on a single cpu. the illusion and api are what matters. a better term may be non-blocking i/o (and socket connections) but that needs more explanation in some ways. my view is that threads are the real sync app style and event loops are the async style. but this is not the time nor place to discuss that. >> i don't find that surprising. perl's i/o is decent and as i said above, >> a loaded server is doing mostly i/o. ML> Well, since the server in this benchmark hardly does anything, and as you ML> can see by the results, the event loop completely dominates, and bad event ML> loops are a thousand times slower than good ones. i haven't looked at the benchmark stuff yet. i was browsing the code earlier. ML> When put into requests/s, this means that with POE, you are limited ML> to <1000 requests/s, while with EV you can easily reach hundreds of ML> thousands. ML> Also, this does not explain at all why Event is so slow, and why Glib scales ML> so extremely badly. Most of the stuff that slows down the perl-based event ML> loop is clearly stuff that is much much faster in C. poor memory management in the c code? i have done pure c event loops in a framework where memory management was fairly fast due to cached queues of known block sizes. alloc/free were usually a trivial call and the event code had it own private fixed size buffer queues. it had no problem doing 2k parallel web fetches including all the url parsing all on a single sparc. we had to throttle it down to keep up with the slower indexer. ML> For a reasonable event loop implemented in C, the overhead of ML> calling select should completely dominate the rest of the ML> processing (it does so in EV). true, but bad c (and perl) code is all around us. >> i will take a gander and see if i can play with it and add stem's loop >> to it. if you want to work on this with me, i wouldn't mind the help. ML> Well, can't say thereis much demand for it, but if you cna give me a pointer ML> on these things in the docs I can probably come up with one before the next ML> release: ML> - how can I do one iteration of the event loop? This is important i don't have an api for oneevent. i still haven't seen a need for it. hence having stem wrap anyevent may be the better way. the goal for me is to have stem support more (and faster) event loops. if you are doing a stem app, you should use the stem event api. ML> for condvars: the function must not block after one or more events ML> have been handled. ?? ML> - how do I register a callback to be called when an fh or fd becomes ML> "readable" or "writable"? the sessions examples show plenty of that. very similar to event.pm overall. ML> - how can I register a relative timer? relative? is that a delay timer (relative to now)? event.pm has hard (timing from before callback to next trigger) and soft (time from after callback to next trigger) timers. ML> - are there any limitations, for example, what happens when I register ML> two read watchers for one fh (or fd)? shouldn't be a problem if select handles it. the stem perl event loop is a wrapper around select. ML> - does stem provide something to catch signals synchronously? either use event.pm as the core loop or the pure perl one can do it under 5.8 and safe signals. ML> - how about child status changes or exits? signals are signals. it handles sigchld and the stem::proc module deals with reaping as needed. ML> The EV or Event implementation modules should give a good idea of whats ML> required. ML> And for the documentation: ML> - how does stem handle time jumps, if at all? ?? ML> - are it's timers based on absolute or wallclock time or relative/monotonic time? iirc it checks wallclock (HiRes::time()) after each call to select and gets its current delta from that. you can't trust a relative clock as it will skew around. uri -- Uri Guttman ------ uri@stemsystems.com -------- http://www.sysarch.com -- ----- Perl Code Review , Architecture, Development, Training, Support ------ --------- Free Perl Training --- http://perlhunter.com/college.html --------- --------- Gourmet Hot Cocoa Mix ---- http://bestfriendscocoa.com --------- http://www.nntp.perl.org/group/perl.loop/2008/04/msg1110.html Sat, 26 Apr 2008 02:34:41 +0000 Benchmarking Stem :) by Marc Lehmann Ok, I have enough for a benchmark, just short of being able to provide a working implementation. I actually ran out of memory with the Stem implementation, it seems to use a lot of additional memory when running the event loop. I had to reduce the number of handles to 20000. Otherwise, it looks quite good (this is the synthetic benchmark that doesn't test event loop performance, just overheads): name watchers bytes create invoke EV/EV 100000 244 0.54 0.45 Perl/Any 100000 513 4.61 0.87 Stem/Any 20000 1159 31.51 9.12 Event/Any 16000 936 38.80 31.91 POE/Any 2000 6658 108.61 726.39 It is only 10 times slower than anyevents pure perl backend, but three times faster than Event. For the large server benchmark, it still looks very good: name sockets create request EV 20000 68.51 10.65 Perl 20000 73.58 112.90 Event 20000 199.29 252.45 Stem 20000 128.11 660.92 Glib 20000 632.13 1818.40 POE 20000 344.56 12154.89 Only 6 times slower than anyevents pure perl backend, but faster than Glib (ok, glib clearly has issues :) So it is actually quite good, and quite usable, except for the out of memory issue. (This is without the multiple watchers per fd/combo workaround, btw., which would only affect create speed). It hurts a bit to come so far and then to only fail because stem has no one_iteration-like functionality, though (I would go with busy waiting, if that were implementable, but thats not reasonable to implement either). -- The choice of a Deliantra, the free code+content MORPG -----==- _GNU_ http://www.deliantra.net ----==-- _ generation ---==---(_)__ __ ____ __ Marc Lehmann --==---/ / _ \/ // /\ \/ / pcg@goof.com -=====/_/_//_/\_,_/ /_/\_\ http://www.nntp.perl.org/group/perl.loop/2008/04/msg1109.html Sat, 26 Apr 2008 00:15:11 +0000 Re: benchmarking various event loops with and without anyevent by Marc Lehmann On Sat, Apr 26, 2008 at 01:14:07AM -0400, Uri Guttman <uri@stemsystems.com> wrote: > actually it makes more sense to me to wrap anyevent in stem. it already Yes, after it turned out that stopping the loop also seems to clear events it became clear that Stem cannot provide even the stripped down interface of anyevent (even when one works aorund the bugs that make it unusable). Too bad, I'd liked to benchmark it, but thats almost out of the question (maybe with a quick hack...). -- The choice of a Deliantra, the free code+content MORPG -----==- _GNU_ http://www.deliantra.net ----==-- _ generation ---==---(_)__ __ ____ __ Marc Lehmann --==---/ / _ \/ // /\ \/ / pcg@goof.com -=====/_/_//_/\_,_/ /_/\_\ http://www.nntp.perl.org/group/perl.loop/2008/04/msg1108.html Sat, 26 Apr 2008 00:00:39 +0000 Re: benchmarking various event loops with and without anyevent by Marc Lehmann Hmm, more bugs: sub stopbusywaiting { Stem::Event::stop_loop; } my $stopper = new Stem::Event::Timer object => $dummy, method => "stopbusywaiting", delay => 0.05, interval => 0.05; # bug workaround warn "a\n"; $stopper->start; Stem::Event::start_loop; $stopper->cancel; warn "b\n"; This always takes a second, here is an strace: http://ue.tst.eu/57e55320bf26d8c4ae403e722bd7e268.txt adding "use Time::HiRes 'time'" to Perl.pm makes it run fast (yes, diretcly below the code that should handle this, something is broken witht he time::hires loading code apparently, maybe its this weird core-overwriting issue). -- The choice of a Deliantra, the free code+content MORPG -----==- _GNU_ http://www.deliantra.net ----==-- _ generation ---==---(_)__ __ ____ __ Marc Lehmann --==---/ / _ \/ // /\ \/ / pcg@goof.com -=====/_/_//_/\_,_/ /_/\_\ http://www.nntp.perl.org/group/perl.loop/2008/04/msg1107.html Fri, 25 Apr 2008 23:53:49 +0000 Re: benchmarking various event loops with and without anyevent by Marc Lehmann On Sat, Apr 26, 2008 at 01:14:07AM -0400, Uri Guttman <uri@stemsystems.com> wrote: > >>>>> "ML" == Marc Lehmann <schmorp@schmorp.de> writes: > > ML> On Fri, Apr 25, 2008 at 11:40:03PM -0400, Uri Guttman <uri@stemsystems.com> wrote: > >> check out stem's pure perl event loop. there are examples in the > > ML> Maybe I'll provide a backend for stem. > > actually it makes more sense to me to wrap anyevent in stem. it already > has several event wrappers (pure perl, event.pm and tk) and wrapping is > very easy to do. not much different than the code i see in anyevent. FYI: I think I found a bug, one-shot timer events never get reported: Pelr.pm: - calls timer_triggered, - which cancels, - which sets active to 0, - then calls trigger - which returns becasue active is 0 and callback never gets invoked. Maybe my reading of the sourcecode is wrong, but I can't for the life of it get any timer event out of stem. -- The choice of a Deliantra, the free code+content MORPG -----==- _GNU_ http://www.deliantra.net ----==-- _ generation ---==---(_)__ __ ____ __ Marc Lehmann --==---/ / _ \/ // /\ \/ / pcg@goof.com -=====/_/_//_/\_,_/ /_/\_\ http://www.nntp.perl.org/group/perl.loop/2008/04/msg1106.html Fri, 25 Apr 2008 23:43:57 +0000 Re: benchmarking various event loops with and without anyevent by Marc Lehmann On Sat, Apr 26, 2008 at 01:14:07AM -0400, Uri Guttman <uri@stemsystems.com> wrote: > ML> Maybe I'll provide a backend for stem. > > actually it makes more sense to me to wrap anyevent in stem. it already of course, using anyevent always makes sense. however, using anyevent doesn't solve the interoeprability problem: you still cannot use modules using anyevent when you don't use the anyevent but the wx interface, for example. > has several event wrappers (pure perl, event.pm and tk) and wrapping is > very easy to do. not much different than the code i see in anyevent. Yes, but that doesn't give you much advantage - as long as your module isn't good citizen and plays nicely with other modules by monopolising the process it is not interoperable. Making anyevent compatible to stem enables anyevent modules to be used in stem. making stem use anyevent doesn't achieve that aslong as it doesn't use it's anyevent backend, and it cnanot be used in other programs as well, as it isn't event-loop agnostic because it forces the user to use "stem" as the event loop. Also, this would make it impossible to benchmark the pure perl event loop - I would *prepdict* (but i am bad at that) that it will be the slowest one, ignoring POE, which I expect to be much slower still. > ML> Especially in the important case of many handles/few active ones, an approach > ML> of "scan all handles" is very slow. > > my code doesn't scan all handle, but scan all writeable events to see if > their handle bits are set. That's even worse, as there can be many more events (watchers?) then file descriptors (in the first benchmark you would scan, say, 10000 event watchers for the same fd). even if not, it's slow O(n), compared to fast O(n) that you can achieve with scanning the bitmasks. It is only fast if you have a lot of handles and very few active watchers, which isn't too typical (who uses all those extra fd's if not your program?). In any case, the "scan mask" approach is about many times faster in the actual benchmark with 10000 handles and 100 active ones, simply because scanning a mask is so much faster. Besides, select returns the number of active handles, so one could use both approachs and select between them, e.g. when more than 80% of the handles are active, use the scan-all-handles method, otherwise the bitmask method. > descriptors increment and can cause bloat of the array if you have many > in use (and not all of them in the event loop). That is not a common case, besides, arrays are very compact, unlike hashes, so it's not a clear win (note how the pure perl backend in anyevent comes out as one of the backends using the leats amount of memory). In any case, a lot of technology in the kernel goes into providing "small integers" as fd's, not taking advantage of that gives away optimisaiton opportunities. In this case, unix guarantees that the memory use is bounded (there is even a resource limit for it, and reserving 4 or 8 bytes/file descriptor is nothing really). Trying to avoid bloat on that side is the wrogn side to optimise for. > event loops are async in that you get callbacks as they are needed. sure Yes, but the I/O isn't async, which was my point. asynchronous I/O is quite a different beast, but few people really use it. (which is a pity, but only pelr ahs a IMnsHO decent module for it). > api are what matters. a better term may be non-blocking i/o (and socket It's actually the only correct term, as no I/O is done in the event loop. > ML> Also, this does not explain at all why Event is so slow, and why Glib scales > ML> so extremely badly. Most of the stuff that slows down the perl-based event > ML> loop is clearly stuff that is much much faster in C. > > poor memory management in the c code? Perl's memory management is quite good, yeah. I do suspect that it has somethign to do with glib scanning its watcher list (ALL of them) repeatedly, and when removing, who knows, it might run a singly-linked list to its end to remove the watcher. As for Event, I think it simply does way too much around simple callback invocation, for example it uses its event queue and adds events at the end (walking the list each time). All that the event queue has done for me, hwoever, was causing infite memory growth when it added multiple events for the same fd again and again becasue some high-priority watcher got precedence. (I have nothing against event queues, you need one, but one *can* manage it abysmally). > a framework where memory management was fairly fast due to cached queues > of known block sizes. alloc/free were usually a trivial call and the > event code had it own private fixed size buffer queues. it had no > problem doing 2k parallel web fetches including all the url parsing all > on a single sparc. we had to throttle it down to keep up with the slower hehe :) > ML> For a reasonable event loop implemented in C, the overhead of > ML> calling select should completely dominate the rest of the > ML> processing (it does so in EV). > > true, but bad c (and perl) code is all around us. hehe :/ > ML> - how can I do one iteration of the event loop? This is important > > i don't have an api for oneevent. i still haven't seen a need for > it. > hence having stem wrap anyevent may be the better way. Its the easier way, because you don't have to provide the common interface anyevent offers, but it is also far less useful - since Stem seems to be like POE in that it dominates the process (it's me and nothing else), there is liekly little pratical need to interoperability, but it sure would be nice :) > the goal for me is to have stem support more (and faster) event loops. > if you are doing a stem app, you should use the stem event api. *sigh*, I think the world would be a much better place if not every module author emplyong events insisted on ruling out all others :) > ML> for condvars: the function must not block after one or more events > ML> have been handled. > > ?? Some event models had bugs in that they: 1. handled some timer events 2. blocked waiting for some i/o events This must not happen in the "one event loop iteration" function. It can handle as many events as it likes, but when it handles at least one event, it must not block. > ML> - how do I register a callback to be called when an fh or fd becomes > ML> "readable" or "writable"? > > the sessions examples show plenty of that. very similar to event.pm overall. Actually, I tried, but I found the testsuite is much clearer on that. > ML> - how can I register a relative timer? > > relative? is that a delay timer (relative to now)? event.pm has hard > (timing from before callback to next trigger) and soft (time from after > callback to next trigger) timers. I found it, it only supports relative timers (but internally uses absolute timing). > ML> - are there any limitations, for example, what happens when I register > ML> two read watchers for one fh (or fd)? > > shouldn't be a problem if select handles it. the stem perl event loop is > a wrapper around select. Well, the Tk backend doesn't handle it, unless I missed something, and the lowest common denominator counts. (Since AnyEvent prefers using its Tk backend this is not an issue unles sa user would force the Stem backend). > ML> - how about child status changes or exits? > > signals are signals. it handles sigchld and the stem::proc module deals > with reaping as needed. Well, signals are signals, but we are talking about waitpid here - how do I get the info that stem::proc reaps in my sigchld handler? > ML> - how does stem handle time jumps, if at all? > > ?? bios sets the time to 2010, ntp resets it to 2005, do stem programs then sleep for 5 years? answer is: likely ses. > ML> - are it's timers based on absolute or wallclock time or relative/monotonic time? > > iirc it checks wallclock (HiRes::time()) after each call to select and > gets its current delta from that. you can't trust a relative clock as it > will skew around. It is the othere way around, a relative clock does exactly the same thing as your timers (you tell it to delay 5 seconds, it will delay 5 seconds), an absoltue clock does not, for example, a user who registers a 5 second delay with Stem might find out that it takes years to complete because it uses wallclock time which can change, unlike for example,a true monotonic clock, or some other time source that allows me to build relative timing. But that is ok, as far as I know: - libevent and Event::Lib use a monotonic clock ti gte relative timers - Event has some crude checking for time jumps - EV handles relative and absolute timers about perfectly - all else just suffers from timejumps So stem is in good company, its' a perfectionist issue (although each one of my programs that didn't handle relative timers correctly got a bug report in the past because of that, so it happens quite often, even on servers). -- The choice of a Deliantra, the free code+content MORPG -----==- _GNU_ http://www.deliantra.net ----==-- _ generation ---==---(_)__ __ ____ __ Marc Lehmann --==---/ / _ \/ // /\ \/ / pcg@goof.com -=====/_/_//_/\_,_/ /_/\_\ http://www.nntp.perl.org/group/perl.loop/2008/04/msg1105.html Fri, 25 Apr 2008 23:31:03 +0000 Re: benchmarking various event loops with and without anyevent by Marc Lehmann On Sat, Apr 26, 2008 at 07:01:39AM +0200, Marc Lehmann <schmorp@schmorp.de> wrote: > And another one: does Stem deal with subsecond delay values? Under what circumstances? AnyEvent guarantees subsecond accuracy currently. Also: The ’hard’ attribute means that the next interval delay starts before the callback to the object is made. If a soft timer is selected (hard is 0), the delay starts after the callback returns. So the hard timer ignores the time taken by the callback and so it is a more accurate timer. The accuracy a soft timer is affected by how much time the callback takes. It is nice to have a hard timer, but please implement clumping. Not implementing clumping makes this type of timer completely unusable in practise, as a time jump or stopping the program for a long time has makes it potentially unusable. (In fact, this inability to use Event's hard timers was one of the reasons I was looking for a different event loop. EV doesn't do clumping, however, as it has two timer types that solve time-jump and stopping-related problems diferently). -- The choice of a Deliantra, the free code+content MORPG -----==- _GNU_ http://www.deliantra.net ----==-- _ generation ---==---(_)__ __ ____ __ Marc Lehmann --==---/ / _ \/ // /\ \/ / pcg@goof.com -=====/_/_//_/\_,_/ /_/\_\ http://www.nntp.perl.org/group/perl.loop/2008/04/msg1104.html Fri, 25 Apr 2008 22:05:08 +0000 Re: benchmarking various event loops with and without anyevent by Marc Lehmann On Sat, Apr 26, 2008 at 06:48:55AM +0200, Marc Lehmann <schmorp@schmorp.de> wrote: > Well, can't say thereis much demand for it, but if you cna give me a pointer > on these things in the docs I can probably come up with one before the next > release: Looking at Stem::Event, which hopefully is the right place to look, I cna come up with soem answres and some more refined questions :) Here is a new question: - How do I select a specific event loop backend to be used? > - does stem provide something to catch signals synchronously? doesn't seem to be documented, I assume so. > - how can I do one iteration of the event loop? This is important Interestingly, the manpage for "Stem::Event # event loop" doesn't mention any way to start said event loop. The source code has init_loop, start_loop and, interestingly, stop_loop. Does that mena I have to initialsie the event loop myself? Is initialising it multiple times a problem (obviously, code using anyevent can't be bothered to implement workarounds for stem, so if that were a problem, stem couldn't be used by default. Thats not too bad, however, as it would sitll be used if the main program uses stem). > - how do I register a callback to be called when an fh or fd becomes > "readable" or "writable"? This is also not documented. All it says is: This class creates an event that will trigger a callback whenever its file descriptor has data to be read. [...] It doesn't say how to pass "its dile descriptor", or wether it also works with file handles. > - how can I register a relative timer? Solved :) Whats missing is how I cancel that time, does it automatically cancel itself when the returned object gets forgotten by the calling code? > - are there any limitations, for example, what happens when I register > two read watchers for one fh (or fd)? The answre seems to be: stem needs the same workaround as Tk, unless I overlooked some internal layer. > - how about child status changes or exits? Doesn't seem to be supported, so AnyEvent will fall back to its own code (there is nothing wrong with that, POE is much worse, as its code is broken but insists on reaping children so one cnanot even use one's own implementation). > - how does stem handle time jumps, if at all? Can't really find anything here either, I guess changing the clock requires a restart. > - are it's timers based on absolute or wallclock time or relative/monotonic time? Seems to be absolute time. -- The choice of a Deliantra, the free code+content MORPG -----==- _GNU_ http://www.deliantra.net ----==-- _ generation ---==---(_)__ __ ____ __ Marc Lehmann --==---/ / _ \/ // /\ \/ / pcg@goof.com -=====/_/_//_/\_,_/ /_/\_\ http://www.nntp.perl.org/group/perl.loop/2008/04/msg1103.html Fri, 25 Apr 2008 22:01:47 +0000 Re: benchmarking various event loops with and without anyevent by Marc Lehmann On Fri, Apr 25, 2008 at 11:40:03PM -0400, Uri Guttman <uri@stemsystems.com> wrote: > check out stem's pure perl event loop. there are examples in the Maybe I'll provide a backend for stem. > modules. it does things in a different direction and doesn't scan > select's bit masks but instead it scans the interesting handles and see > whether their bits are set. it should exhibit good behavior under growth > as all the data are managed in hashes. That is exactly as the anyevent perl backend did in the previous version. I don't see how that should exhibit good behaviour - file descristpors are small integers, so hashes only add overhead over arrays, and checking every file descriptor that way is much slower than scanning the bitmask. Especially in the important case of many handles/few active ones, an approach of "scan all handles" is very slow. (The benchmarks reflect this, you could try with an older anyevent release where we just check the keys of the hash storing fd information: performance is much lower). > ML> I then made a second benchmark, designed not to measure anyevent overhead, > ML> but to measure real-world performance of a socket server. > > and that /sessions code also shows use of the asyncio module. if you can > benchmark that i would be interested in the results. Hmm, tt seems to have become fashionable lately to call synchronous I/O asynchronous (see IO::Async, another 100% synchronous framework). What exactly is asynchronous about the I/O in that example, it seems to be fully synchronous to me (just event-driven). > ML> For small servers, the overhead introduced by running a lot of perl > ML> opcodes per iteration dominates, however, reflected in the last benchmark. > > in a heavily loaded server most of the work in in the i/o and should > overwhelm the event loop itself. that is the whole purpose of event > loops as we all know here. I actually don't know that. When I write response data (such as a file in my anime server), the performance of the event loop completely dominates (ignoring IO::AIO overhead in this case, which also of coruse relies on the event loop). Of course, select/poll don't scale at all, unless the majority of handles is active, also not very typical of heavily loaded servers. > ML> However, the net result is that the pure perl event loop performs > ML> better than almost all other event loops (EV being the only exception) > ML> ins erious/medium-sized cases, while I originally expected it to fail > ML> completely w.r.t. performance and being only usable as a workaround when > ML> no "better" event module is installed. > > i don't find that surprising. perl's i/o is decent and as i said above, > a loaded server is doing mostly i/o. Well, since the server in this benchmark hardly does anything, and as you can see by the results, the event loop completely dominates, and bad event loops are a thousand times slower than good ones. When put into requests/s, this means that with POE, you are limited to <1000 requests/s, while with EV you can easily reach hundreds of thousands. Also, this does not explain at all why Event is so slow, and why Glib scales so extremely badly. Most of the stuff that slows down the perl-based event loop is clearly stuff that is much much faster in C. For a reasonable event loop implemented in C, the overhead of calling select should completely dominate the rest of the processing (it does so in EV). This is not the case for any of the event loops I tested with the exception of EV and Event (which "only" has high overhead, but it doesn't grow beyond reason). > i will take a gander and see if i can play with it and add stem's loop > to it. if you want to work on this with me, i wouldn't mind the help. Well, can't say thereis much demand for it, but if you cna give me a pointer on these things in the docs I can probably come up with one before the next release: - how can I do one iteration of the event loop? This is important for condvars: the function must not block after one or more events have been handled. - how do I register a callback to be called when an fh or fd becomes "readable" or "writable"? - how can I register a relative timer? - are there any limitations, for example, what happens when I register two read watchers for one fh (or fd)? - does stem provide something to catch signals synchronously? - how about child status changes or exits? The EV or Event implementation modules should give a good idea of whats required. And for the documentation: - how does stem handle time jumps, if at all? - are it's timers based on absolute or wallclock time or relative/monotonic time? -- The choice of a Deliantra, the free code+content MORPG -----==- _GNU_ http://www.deliantra.net ----==-- _ generation ---==---(_)__ __ ____ __ Marc Lehmann --==---/ / _ \/ // /\ \/ / pcg@goof.com -=====/_/_//_/\_,_/ /_/\_\ http://www.nntp.perl.org/group/perl.loop/2008/04/msg1102.html Fri, 25 Apr 2008 21:49:06 +0000 benchmarking various event loops with and without anyevent by Marc Lehmann Hi! The next release of AnyEvent contains support for a few more "backends", notably POE, so AnyEvent is now by definition compatible to POE (before it was only compatible when using an even loop used by POE, such as Event or EV that could be shared). To get a feeling about the overheads imposed by the extra layer of AnyEvent and especially of the performance of the POE backend (which is severely limited by a number of design problems in POE), I made some benchmarks. I started with benchmarking watcher creation/destruction time and callback invocation time. The results were mostly as expected, with EV leading and POE being abysmal. The surprising one was the pure perl implementation, which was quite on par with C-based event loops such as Event or Glib. I did expect the pure perl implementatioon to be at least a factor of three slower than Event or Glib. As the pure perl loop wasn't written with high performance in mind, this prompted me to optimise it for some important cases (mostly to get rid of the O(n²) degenerate cases and improving the select bitmask parsing for the sparse case). I then made a second benchmark, designed not to measure anyevent overhead, but to measure real-world performance of a socket server. The result is that the pure perl event loop used as fallback in AnyEvent single-handedly beats Glib by a large margin, and even event by a factor of two. For small servers, the overhead introduced by running a lot of perl opcodes per iteration dominates, however, reflected in the last benchmark. However, the net result is that the pure perl event loop performs better than almost all other event loops (EV being the only exception) ins erious/medium-sized cases, while I originally expected it to fail completely w.r.t. performance and being only usable as a workaround when no "better" event module is installed. All the benchmark data and explanations can be found here: http://pod.tst.eu/http://cvs.schmorp.de/AnyEvent/lib/AnyEvent.pm#BENCHMARKS The code is not yet released and likely still buggy (the question is whether any bugs affect the benchmark results). It is only available via CVS: http://software.schmorp.de/pkg/AnyEvent Of course, I am very pleased by the performance of perl 5.10 in particular and perl in general. This case also shows that one can beat C code in many important cases by using better algorithms (i.e. by writing it the "perlish" way). It also means one can write medium-sized servers without relying on any XS modules for performance, and that AnyEvent adds no serious overhead to programs. Feedback would be appreciated, otherwise: enjoy :) -- The choice of a Deliantra, the free code+content MORPG -----==- _GNU_ http://www.deliantra.net ----==-- _ generation ---==---(_)__ __ ____ __ Marc Lehmann --==---/ / _ \/ // /\ \/ / pcg@goof.com -=====/_/_//_/\_,_/ /_/\_\ http://www.nntp.perl.org/group/perl.loop/2008/04/msg1101.html Fri, 25 Apr 2008 19:46:30 +0000 EV module questions by Peter Maynard Using EV for a daemon re-write and have run into a little trouble. Original daemon redirected STDERR to a variable $stderr and each time through the daemon's while loop, I simply checked if the variable had data and wrote it to syslog as an error if it did. Then I'd clear the variable and carry on. I'm now trying to re-create this using EV and I'm having problemss. NOTE: I can't write the output to a file and watch the file. Due to operational restrictions, STDERR must go directly to syslog. I've managed to setup a periodic watcher to write the $stderr variable to syslog, but it won't clear. I also tried to setup a watcher directly on STDERR, but that didn't seem to work at all. Any suggestions or help would be greatly appreciated. PS. Thanks for EV and Net::SNMP::EV. I was fighting to do this with POE and EV has been a life saver! Peter Maynard http://www.nntp.perl.org/group/perl.loop/2008/02/msg1100.html Sat, 02 Feb 2008 01:51:21 +0000 overview over the EV module family by Marc Lehmann It's been a while since I announced EV, and in the meantime, not only has EV proven itself in a lot of apps, the module family around it grew considerably. With version 2.0, EV gained support for multiple event loops and embed watchers (a bit scarcely documented), so one can now use e.g. kqueue for sockets only and poll for overall event polling. A number of new modules are now available, separately from the main EV distro: - EV::ADNS This is an interface to libadns, an asynchronous dns resolver library, that replaces EV::DNS which was removed from EV. It is written in C and is a nice example on how to access EV from the C level, and how to effectively integrate foreign event loop users (such as libadns). - Net::SNMP::EV This is an adaptor that makes non-blocking Net::SNMP requests run as part of the EV event loop. It is written fully in Perl and has no public interface, just loading it makes Net::SNMP requests use EV. This is a nice example on how to integrate a foreign event loop user using perl only. - EV::Glib This integrates the default glib maincontext into EV. That means that watchers registered via Glib (e.g. in gtk+ programs) are handled together with EV watchers. This doesn't make glib any faster (it's easily the slowest event loop I have seen), but it allows you to integrate glib/gtk+ programs into EV-based programs. - Glib::EV This is the opposite of EV::Glib - it uses EV as the polling backend for Glib. This, too, doesn't make glib any more efficient (its usually a bit less efficient with epoll for example), but allows one to use EV and EV watchers in glib/gtk+ programs (i.e. the main loop that runs is the glib one, while with EV::Glib the main loop would be the EV one). If one has a lot of IO watchers or timers, then using this module might result in a drastic performance increase over glib (the "lot of" comes from the fact that epoll cannot efficiently emulate poll, but in practise this can already pay off when watching two or more file descriptors). - AnyEvent::Impl::EV and AnyEvent::Impl::CoroEV AnyEvent has had EV support for quite some time, thought I'd mention this. - Coro::EV, Coro::Util When Coro detects that EV is used, it will use Coro::EV to efficiently hook into the event loop for non-blocking socket access. Likewise, Coro::Util will take advantage of EV::ADNS if available, when resolving dns domains. Most of these were written to satisfy my most immediate personal and company needs, which explains the weird coverage. Hope it is useful to somebody else, too. -- The choice of a Deliantra, the free code+content MORPG -----==- _GNU_ http://www.deliantra.net ----==-- _ generation ---==---(_)__ __ ____ __ Marc Lehmann --==---/ / _ \/ // /\ \/ / pcg@goof.com -=====/_/_//_/\_,_/ /_/\_\ http://www.nntp.perl.org/group/perl.loop/2007/12/msg1099.html Sun, 30 Dec 2007 17:16:48 +0000 Re: announcing the new EV module, a modern redesign of Event by Marc Lehmann On Fri, Dec 14, 2007 at 12:22:00AM +0100, Jochen Stenzel <perl@jochen-stenzel.de> wrote: > > So far, it seems everything works right. While EV is young and certainly > > has some bugs, it is used in a lot of projects, and really can't be > > _totally_ broken. > > I never thought it was broken. If there was an impression as if I wanted > to express something like that, or even "totally broken", I'm very > sorry. Nono, I didn't get that impression at all. "totally broken" were only my words :) > I was curious, I played a bit, I hoped to use EV in a project > (and actually started to do so now), some things didn't seem to work > perfectly and so I made some notes, and yes given that I was under heavy > time pressure I agree it was unwise to quickly send first impressions in > between spontaneously. Good. Happens to me, too (the reporting-too-early). I was actually surprised to see the debugger take a reference btw.... Its better this way than knowing about a bug and being too shy to report it :) > There was no adrenaline involved here ;-) As I said before: I consider Your mail was worded quite neutrally, if that helps :) I said that mostly for the benefit of other people following the discussion. -- The choice of a Deliantra, the free code+content MORPG -----==- _GNU_ http://www.deliantra.net ----==-- _ generation ---==---(_)__ __ ____ __ Marc Lehmann --==---/ / _ \/ // /\ \/ / pcg@goof.com -=====/_/_//_/\_,_/ /_/\_\ http://www.nntp.perl.org/group/perl.loop/2007/12/msg1098.html Thu, 13 Dec 2007 22:56:45 +0000 Re: announcing the new EV module, a modern redesign of Event by Jochen Stenzel Hi Marc, > So far, it seems everything works right. While EV is young and certainly > has some bugs, it is used in a lot of projects, and really can't be > _totally_ broken. I never thought it was broken. If there was an impression as if I wanted to express something like that, or even "totally broken", I'm very sorry. I was curious, I played a bit, I hoped to use EV in a project (and actually started to do so now), some things didn't seem to work perfectly and so I made some notes, and yes given that I was under heavy time pressure I agree it was unwise to quickly send first impressions in between spontaneously. Time pressure continues and so it could take a while until I can return to the test scripts and your detailed answer (thanks for that), but I just want to make clear now that there was absolutely no intention to blame you or your work. *If* there are bugs they can probably be fixed. There was no adrenaline involved here ;-) As I said before: I consider it great news you wrote and maintain EV, and I think you are the right man for such a project, according to my impressions. Sorry for any other impression. Greetings Jochen http://www.nntp.perl.org/group/perl.loop/2007/12/msg1097.html Thu, 13 Dec 2007 15:22:14 +0000 io/timer combo examples [Was: Re: Replacement of Event with EV] by Marc Lehmann > As it is trivial to create a separate timeout (and much faster than with > Event, too), there is no drawback to this design. As it might not be too obvious, here is (in pseudocode) an example of an idle timeout, i.e. a timeout that triggers afetr some time of inactivity. It uses two watchers (which is not required but customary): # 60 seconds idle timeout my $to = EV::timer_ns 0, 60, sub { close $fh; ... log error; }; $fh = ... new socket; $to->again; my $wr = EV::io $fh, EV::READ, sub { $to->again; # reset timer # do stuff }; my $ww = EV::io $fh, EV::WRITE, sub { $to->again; # reset timer # do stuff }; The read and write watchers can easily be started and stopped independently of each other this way, and also independent of the timeout ($to->stop ... $to->again is a valid sequence to temporarily disable a timeout, e.g. when the connection is not dpoing any work and can be kept open). If you try to do this in event and re-use one watcher's timeout, you will need to stop/start it each time you get activity, which is rather costly (and costs a lot when one uses e.g. the epoll backend). Of course one can use a separate watcher with Event, too. If an overall timeout is required (e.g. a single request must be done afetr 90 seconds, no matter what), the code becomes simpler (basiclaly its then using a simple timer and never resets it): # 90 seconds idle timeout $fh = ... new socket; my $to = EV::timer_ns 90, 0, sub { close $fh; ... log error; }; my $wr = EV::io $fh, EV::READ, sub { # do stuff }; my $ww = EV::io $fh, EV::WRITE, sub { # do stuff }; -- The choice of a Deliantra, the free code+content MORPG -----==- _GNU_ http://www.deliantra.net ----==-- _ generation ---==---(_)__ __ ____ __ Marc Lehmann --==---/ / _ \/ // /\ \/ / pcg@goof.com -=====/_/_//_/\_,_/ /_/\_\ http://www.nntp.perl.org/group/perl.loop/2007/12/msg1096.html Thu, 06 Dec 2007 15:28:23 +0000 Re: Replacement of Event with EV by Uri Guttman >>>>> "ML" == Marc Lehmann <schmorp@schmorp.de> writes: ML> Its a design decision. Combining timers with io watchers is unnatural, ML> because: ML> - you often don't need both, so shouldn't be forced to pay for them ML> - you can't choose the type of timeout (event only has absolute timers ML> anyways, though, which are unsuitable for many tasks such as idle timeouts ML> for i/o, where you need relative timeouts). ML> - you have to manage two timeouts or weirder stuff when you have ML> two separate i/o watchers, or use a third watcher (paying even more). ML> - it is often unclear on which events exactly the timeout gets reset ML> (if at all), and even if not, there is no flexibility (how do you ML> chose between an idle timeout and an overall timeout ML> for completion with Event? you cannot). ML> As it is trivial to create a separate timeout (and much faster than with ML> Event, too), there is no drawback to this design. i do that in stem's event layer too. i/o watchers are created and an optional timer is added if desired (a separate event with a different method to callback into the object). this means all the event layer needs is simple i/o watchers and a simple timer. my code builds up the combined i/o with timer as needed. much cleaner and easier to manage at all levels. as marc said it is also simpler in the lower levels to code up. and i bet most i/o watchers don't need timers which saves on the storage and coding. uri -- Uri Guttman ------ uri@stemsystems.com -------- http://www.stemsystems.com --Perl Consulting, Stem Development, Systems Architecture, Design and Coding- Search or Offer Perl Jobs ---------------------------- http://jobs.perl.org http://www.nntp.perl.org/group/perl.loop/2007/12/msg1095.html Thu, 06 Dec 2007 07:04:49 +0000 Re: FAIL Event-1.09 i686-linux 2.4.20-8smp by Joshua N Pritikin On Thu, Dec 06, 2007 at 12:35:12PM +0200, Alexandr Ciornii wrote: > Yes, I can build many other modules fine. > You can see similar failures here: > http://www.nntp.perl.org/group/perl.cpan.testers/2007/10/msg712278.html > http://www.nntp.perl.org/group/perl.cpan.testers/2007/10/msg670107.html Oh, that helps. All these failures occurred with perl 5.5.5. That's a really old version. > In general all testing errors are shown here: > http://cpantesters.perl.org/show/Event.html#Event-1.09 Can you retry with a more recent version? Even perl 5.6 is really old by now. My development box has 5.8.8. http://www.nntp.perl.org/group/perl.loop/2007/12/msg1094.html Thu, 06 Dec 2007 06:14:57 +0000 Re: announcing the new EV module, a modern redesign of Event by Marc Lehmann On Thu, Dec 06, 2007 at 01:07:21AM +0100, Jochen Stenzel <perl@jochen-stenzel.de> wrote: > * I *could* stop the loop in the debugger today, but it took a while: Although the behaviour is consistent with any other extension module for perl, I tried to document this in the next EV release: =head1 PERL SIGNALS While Perl signal handling (C<%SIG>) is not affected by EV, the behaviour with EV is as with any other C library: signals will only be handled when Perl runs, which means your signal handler might be invoked only the next time an event gets handles. The solution is to use the the EV signal handlers (see C<EV::signal>). If you cannot do this for whatever reason, you can also force a watcher to be called on every iteration by installing a C<EV::check> watcher: my $async_check = EV::check sub { }; This ensures that perl shortly gets into control for a short time. Note, however, that this does not work for the perl debugger as the perl debugger can only stop at the next perl statement to be executed - this is a limitation of perl (or more precisely the debugger). As this is also the exact same behaviour as with Event (or other event loops), so I find it rather surprising that it needs mentioning, but I hope the additional documentation is not wasted and actually proves helpful to somebody. (But then, signal handling does require knowledge of the underlying operating system handling of signals). -- The choice of a -----==- _GNU_ ----==-- _ generation Marc Lehmann ---==---(_)__ __ ____ __ pcg@goof.com --==---/ / _ \/ // /\ \/ / http://schmorp.de/ -=====/_/_//_/\_,_/ /_/\_\ XX11-RIPE http://www.nntp.perl.org/group/perl.loop/2007/12/msg1093.html Wed, 05 Dec 2007 19:09:14 +0000 Re: announcing the new EV module, a modern redesign of Event by Marc Lehmann On Thu, Dec 06, 2007 at 01:07:21AM +0100, Jochen Stenzel <perl@jochen-stenzel.de> wrote: > > any signals, and in fatc, I cannot reproduce this. Are you sure that your > > tty settings actually do anything with CTRL-C (such as sending an INT)? > > I am - I retried it today, other programs receive the signal and the > effect is the same if I send the signal via kill from another terminal. Here is what happens on windows btw: at least activestate perl overrides the system console handler and thus disables those signals. As it doesn't provide signal emulation for other programs you cnanot catch those signals anymore, and there is nothing one can do about it. Or to put it in other words, perl actually emulates those signals in perl without emulating them in the C level - nothing an event loop can do about that, thats a limitation of at leats activestate perl. > I know the following is an incomplete report as it does not contain > exact system information (in general, Solaris 5.8) and perl version (in Since there is no solaris 5.8 version in existance, I guess you mean solaris 8 (that could make a difference as I only test on solaris 10). > * By the way, there *were* cases today when the signal was ignored > completely, this time in a script where I had to add an explicit > signal watcher to make it interruptable at all. I hope to shrink > this down to a short demo script to send. Indeed, that would be helpful. However, are you sure you don't provide your own signal handlers via %SIG and you really have a signal handler installed for SIGINT? Either would actually perfectly explain the behaviour you are seeing. %SIG cannot work because in newer perls this is only handled when the perl interpreter gets control (which cannot be the case unless you run some watchers). The whole point of an event loop is to shield you from asynchronous signals, you have to embed them into the event loop (thats not specific to EV, other event loops work the same way). If you cannot install signal handlers properly you could try to install a check watcher, which runs at every loop iteration, and hope that whoever installs its own signal handler doesn't make syscalls restartable, in which case you have lost anyways. > DB<2> $p=EV::periodic(0, 2, 0, sub {print "Periodic call with short intervall at ", time, ".\n"}) Here you store one reference into $p, the other is kept by the debugger. > DB<<4>> $p=EV::periodic(0, 20, 0, sub {print "Periodic call with > longer intervall at ", time, ".\n"}) > Periodic call with longer intervall at 1196850145. > Periodic call with short intervall at 1196850145. Since you only overwrote one reference here, both watchers are still active. Watchers only get destroyed when _all_ refererences are broken, as with any other perl object (the same is true with Event, except that you additionally need an additional explicit cancel call with Event). > was no further reference to the first watcher object. But when running > the loop again *both* watchers were present. Yup, because both are alive. Try outside the debugger, that will be less confusing (also, it would help if you wouldn't wrap codelines in your emails :). Try e.g.: use EV; $p=EV::periodic(0, 2, 0, sub {print "Periodic call with short intervall at ", EV::now, ".\n"}); $t = EV::timer 3,0,sub { EV::unloop }; EV::loop; $p = EV::periodic(0, 20, 0, sub {print "Periodic call with longer intervall at ", EV::now, ".\n"}); EV::loop; Here it is easy to see when values get destroyed. As an additional hint, you should give EV::now a try: it is faster then time (no syscall) and more accurate in the sense that it returns the event timestamp (while time returns the current time, which might or might not be what you are interested in). > Finally, in both traces one sees a first callback invocation that does > not fit into the periodic scheme. That "one" must have rather magical abilities to see anything when you round your time and do not use the event time but some other time :-> Try using EV::now and you will see that the timestamps are actually correct. Here are a few sample runs with both EV::now and time to elaborate this: cerebro /tmp# perl x.pl Periodic call with short intervall at 1196906694.00098 1196906693. Periodic call with longer intervall at 1196906700.00108 1196906699. cerebro /tmp# perl x.pl Periodic call with short intervall at 1196906706.00513 1196906706. Periodic call with short intervall at 1196906708.00117 1196906707. Periodic call with longer intervall at 1196906720.00134 1196906719. Periodic call with longer intervall at 1196906740.0056 1196906739. You could alternatively use Time::Hires, which also doesn't round: cerebro /tmp# perl x.pl Periodic call with short intervall at 1196906782.00217 1196906782.00224. Periodic call with short intervall at 1196906784.00219 1196906784.00222. cerebro /tmp# perl x.pl Periodic call with short intervall at 1196906796.00233 1196906796.00241. Periodic call with longer intervall at 1196906800.00238 1196906800.00242. Periodic call with longer intervall at 1196906820.00265 1196906820.00268. In any case, you would see the same behaviour with Event, as it uses Time::HiRes internally and is therefore also more accurate than time(). > I am not sure when I wil have time to send a demo script for the > mentioned case of an ignored interrupt but I hope the traces give you a > first impression. So far, it seems everything works right. While EV is young and certainly has some bugs, it is used in a lot of projects, and really can't be _totally_ broken. So, when you see totally off behaviour like above, first give your test script a bit of thinking, and try to find out wether your test actually can give accurate results (a good example where you should is the time call and your conclusion that its resolution must be enough to actually check the scheduling accuracy of EV, while forgetting that its resolution is very bad indeed). That keeps adrenaline levels down on all sides :) -- The choice of a Deliantra, the free code+content MORPG -----==- _GNU_ http://www.deliantra.net ----==-- _ generation ---==---(_)__ __ ____ __ Marc Lehmann --==---/ / _ \/ // /\ \/ / pcg@goof.com -=====/_/_//_/\_,_/ /_/\_\ http://www.nntp.perl.org/group/perl.loop/2007/12/msg1092.html Wed, 05 Dec 2007 18:21:35 +0000 Re: Replacement of Event with EV by Marc Lehmann On Wed, Dec 05, 2007 at 05:40:47PM -0700, Rusty Conover <rconover@infogears.com> wrote: > I'm looking into using EV rather then Event for my needs but it seems > the I/O timers don't have a nice way to schedule both a timeout and a > I/O callback at the same time into the same callback. Thats a feature, and compared to Event, actually a bugfix. > Such as setting the timeout parameter to the IO watcher. > > Is this just my oversight or a design decision? Its a design decision. Combining timers with io watchers is unnatural, because: - you often don't need both, so shouldn't be forced to pay for them - you can't choose the type of timeout (event only has absolute timers anyways, though, which are unsuitable for many tasks such as idle timeouts for i/o, where you need relative timeouts). - you have to manage two timeouts or weirder stuff when you have two separate i/o watchers, or use a third watcher (paying even more). - it is often unclear on which events exactly the timeout gets reset (if at all), and even if not, there is no flexibility (how do you chose between an idle timeout and an overall timeout for completion with Event? you cannot). As it is trivial to create a separate timeout (and much faster than with Event, too), there is no drawback to this design. It also usually leads to smaller and simpler code, once one understands ones own needs... -- The choice of a Deliantra, the free code+content MORPG -----==- _GNU_ http://www.deliantra.net ----==-- _ generation ---==---(_)__ __ ____ __ Marc Lehmann --==---/ / _ \/ // /\ \/ / pcg@goof.com -=====/_/_//_/\_,_/ /_/\_\ http://www.nntp.perl.org/group/perl.loop/2007/12/msg1091.html Wed, 05 Dec 2007 17:19:43 +0000 Replacement of Event with EV by Rusty Conover I'm looking into using EV rather then Event for my needs but it seems the I/O timers don't have a nice way to schedule both a timeout and a I/O callback at the same time into the same callback. Such as setting the timeout parameter to the IO watcher. Is this just my oversight or a design decision? Thanks, Rusty http://www.nntp.perl.org/group/perl.loop/2007/12/msg1090.html Wed, 05 Dec 2007 16:40:58 +0000 Re: announcing the new EV module, a modern redesign of Event by Jochen Stenzel Hello Marc, >> Then I run a simple loop in the debugger (on the same system), with just >> one periodic watcher. Surprisingly, once started I could not interrupt >> the loop via CTRL-C. > > That is interesting. I am not aware that solaris event ports would block > any signals, and in fatc, I cannot reproduce this. Are you sure that your > tty settings actually do anything with CTRL-C (such as sending an INT)? I am - I retried it today, other programs receive the signal and the effect is the same if I send the signal via kill from another terminal. I know the following is an incomplete report as it does not contain exact system information (in general, Solaris 5.8) and perl version (in general, 5.8.6 without thread support), but today the picture was a bit different: * I *could* stop the loop in the debugger today, but it took a while: DB<2> sub d {print "===> TIME: ", time, "\n";} DB<3> d ===> TIME: 1196849803 DB<4> use EV DB<5> $p=EV::periodic(0, 10, 0, sub {print "Periodic call at ", time, ".\n"}) DB<6> d(); EV::loop ===> TIME: 1196849839 Periodic call at 1196849839. Periodic call at 1196849840. Periodic call at 1196849850. Periodic call at 1196849860. Periodic call at 1196849870. ^Cd Periodic call at 1196849880. EV::loop((eval 32)[.../perl5db.pl:628]:2): 2: DB<<7>> d ===> TIME: 1196849890 * so, CTRL-C worked this time ... * ... but not immediately (as usual, instead it took until the next watcher event, plus the time to the next event check (as it seems)). This was verified with longer intervals and it was reproducable. * In contrast, with an additional signal watcher the signal is handled instantly. * By the way, there *were* cases today when the signal was ignored completely, this time in a script where I had to add an explicit signal watcher to make it interruptable at all. I hope to shrink this down to a short demo script to send. Another interesting effect you might be interested in was this: DB<1> use EV DB<2> $p=EV::periodic(0, 2, 0, sub {print "Periodic call with short intervall at ", time, ".\n"}) DB<3> EV::loop Periodic call with short intervall at 1196850121. Periodic call with short intervall at 1196850122. Periodic call with short intervall at 1196850124. Periodic call with short intervall at 1196850126. ^CPeriodic call with short intervall at 1196850128. EV::loop((eval 23)[.../perl5db.pl:628]:2): 2: DB<<4>> $p=EV::periodic(0, 20, 0, sub {print "Periodic call with longer intervall at ", time, ".\n"}) DB<<5>> EV::loop Periodic call with longer intervall at 1196850145. Periodic call with short intervall at 1196850145. Periodic call with short intervall at 1196850146. Periodic call with short intervall at 1196850148. Periodic call with short intervall at 1196850150. Periodic call with short intervall at 1196850152. Periodic call with short intervall at 1196850154. Periodic call with short intervall at 1196850156. Periodic call with short intervall at 1196850158. Periodic call with longer intervall at 1196850160. Periodic call with short intervall at 1196850160. ^CPeriodic call with short intervall at 1196850162. So, a periodic watcher with a short interval was assigned to a variable, then the loop was started and interrupted via CTRL-C. After that a new watcher was installed and assigned to the same variable, which - as I understood the docs - should uninstall the first watcher as now there was no further reference to the first watcher object. But when running the loop again *both* watchers were present. Finally, in both traces one sees a first callback invocation that does not fit into the periodic scheme. I am not sure when I wil have time to send a demo script for the mentioned case of an ignored interrupt but I hope the traces give you a first impression. Jochen http://www.nntp.perl.org/group/perl.loop/2007/12/msg1089.html Wed, 05 Dec 2007 16:07:49 +0000