perl.loop

perl.loop http://www.nntp.perl.org/group/perl.loop/ ... Copyright 1998-2008 perl.org Mon, 13 Oct 2008 13:21:39 +0000 ask@perl.org Run-time failure by Jeff Boes I'm upgrading an ancient (2004-era) server to the modern era: 2.6.18-92.1.6.el5 x86_64 x86_64 x86_64 GNU/Linux Perl 5.10 Event.pm 1.1 One of my applications throws the following error once it's been running for a few minutes as a daemon: Assertion (((pe_watcher*)wa)->flags & 0x0008) || !wa->running failed: file "c/queue.c" Help or pointers to info would be appreciated. -- Jeffery Boes <>< jeff@endpoint.com http://www.nntp.perl.org/group/perl.loop/2008/09/msg1124.html Wed, 10 Sep 2008 05:40:47 +0000 Re: non blocking issue ? by Isabelle Cabrera `Zidane Tribal a écrit : > On 28/05/2008, Isabelle Cabrera <isabelle.cabrera@inria.fr> wrote: > >> Hi, >> >> I have a problem using the Event module. >> >> My goal: I have several sentences that I need to parse. For a faster >> process, I have a dispatcher which sends the sentences to several hosts >> so that the processes are distributed. >> >> In my dispatcher, I register every host. For each host I create a watcher : >> >> $self->{hosts}{$host}{watcher} = >> Event->var(desc =>"Process sentence on $host", >> var => \$client->{free}, >> poll => 'w', >> cb => $self->next_dag($host), >> debug => 4, >> ); >> >> The callback next_dag is called every time all the process of parsing >> the sentence and sending back the results is done (more precisely when >> the host says it's ready to receive a new sentence, with variable >> $client->{free}). >> >> Of course, I would like that every host can work at the same time >> (that's the point...), but my problem is that they do the job >> subsequently, one sentence at the time. >> >> The created event mainly does the following: >> - get the next sentence from a file >> - send this sentence to the parser on the free host (through a server of >> parser accessed with Net::Telnet) >> - get the result of the parsing (yes or no) (still through Net::Telnet...) >> - if yes, connect again to the server of parser to generate one or >> several outputs and write them into one or several files >> - says "I'm ready to get a new sentence to parse", triggering the >> variable 'free' >> >> I thought that creating an event was automatically non blocking and >> several events could happen at the same time. Does the problem come from >> the I/O interactions ? If so, do I need to separate all these stages and >> watch for every I/O interactions ? And do I need to specify somewhere >> that I want non blocking I/O ? >> >> Best regards, >> Isabelle >> >> -- >> Isabelle Cabrera >> Projet Alpage - INRIA Rocquencourt >> http://alpage.inria.fr >> Tel : 01 3963 5270 >> >> >> > > it sounds like each event is blocking, waiting for the computation to > return a result, and then releasing to allow EV to continue it's loop. > i have had a similar problem recently, (a multi-server irc bot), and > found i have a similar problem, each time the event was blocking and > waiting for a reply. > > how i solved it was to create a readable event for each socket, so > rather than send-wait-receive as you have, its send, then assign a > receive callback to trigger when a result is returned. thus, send > commands down each socket, and allow EV to loop until each one > returns, and as each result is returned, send another query, then let > EV wait on the result again, ad infinitum. > > i suspect this is a horribly confusing answer, and unfortunately i'm > away from home for a few days so i cant provide a code example, but > feel free to ask anything further. > > James Clark. > Thank you James. Finally my problem comes from the fact that my I/O interactions are blocking. So instead of using Net::Telnet I'm trying to use IO::Socket to do non blocking I/O. One other problem is that I had a huge callback, so I'm going to split up my code in smaller peaces. Best regards, Isabelle Cabrera -- Isabelle Cabrera Equipe-Projet Alpage - INRIA Paris-Rocquencourt http://alpage.inria.fr Tel : 01 3963 5270 http://www.nntp.perl.org/group/perl.loop/2008/06/msg1123.html Wed, 04 Jun 2008 05:47:34 +0000 Re: non blocking issue ? by `Zidane Tribal On 28/05/2008, Isabelle Cabrera <isabelle.cabrera@inria.fr> wrote: > Hi, > > I have a problem using the Event module. > > My goal: I have several sentences that I need to parse. For a faster > process, I have a dispatcher which sends the sentences to several hosts > so that the processes are distributed. > > In my dispatcher, I register every host. For each host I create a watcher : > > $self->{hosts}{$host}{watcher} = > Event->var(desc =>"Process sentence on $host", > var => \$client->{free}, > poll => 'w', > cb => $self->next_dag($host), > debug => 4, > ); > > The callback next_dag is called every time all the process of parsing > the sentence and sending back the results is done (more precisely when > the host says it's ready to receive a new sentence, with variable > $client->{free}). > > Of course, I would like that every host can work at the same time > (that's the point...), but my problem is that they do the job > subsequently, one sentence at the time. > > The created event mainly does the following: > - get the next sentence from a file > - send this sentence to the parser on the free host (through a server of > parser accessed with Net::Telnet) > - get the result of the parsing (yes or no) (still through Net::Telnet...) > - if yes, connect again to the server of parser to generate one or > several outputs and write them into one or several files > - says "I'm ready to get a new sentence to parse", triggering the > variable 'free' > > I thought that creating an event was automatically non blocking and > several events could happen at the same time. Does the problem come from > the I/O interactions ? If so, do I need to separate all these stages and > watch for every I/O interactions ? And do I need to specify somewhere > that I want non blocking I/O ? > > Best regards, > Isabelle > > -- > Isabelle Cabrera > Projet Alpage - INRIA Rocquencourt > http://alpage.inria.fr > Tel : 01 3963 5270 > > it sounds like each event is blocking, waiting for the computation to return a result, and then releasing to allow EV to continue it's loop. i have had a similar problem recently, (a multi-server irc bot), and found i have a similar problem, each time the event was blocking and waiting for a reply. how i solved it was to create a readable event for each socket, so rather than send-wait-receive as you have, its send, then assign a receive callback to trigger when a result is returned. thus, send commands down each socket, and allow EV to loop until each one returns, and as each result is returned, send another query, then let EV wait on the result again, ad infinitum. i suspect this is a horribly confusing answer, and unfortunately i'm away from home for a few days so i cant provide a code example, but feel free to ask anything further. James Clark. http://www.nntp.perl.org/group/perl.loop/2008/06/msg1122.html Sun, 01 Jun 2008 09:27:51 +0000 non blocking issue ? by Isabelle Cabrera Hi, I have a problem using the Event module. My goal: I have several sentences that I need to parse. For a faster process, I have a dispatcher which sends the sentences to several hosts so that the processes are distributed. In my dispatcher, I register every host. For each host I create a watcher : $self->{hosts}{$host}{watcher} = Event->var(desc =>"Process sentence on $host", var => \$client->{free}, poll => 'w', cb => $self->next_dag($host), debug => 4, ); The callback next_dag is called every time all the process of parsing the sentence and sending back the results is done (more precisely when the host says it's ready to receive a new sentence, with variable $client->{free}). Of course, I would like that every host can work at the same time (that's the point...), but my problem is that they do the job subsequently, one sentence at the time. The created event mainly does the following: - get the next sentence from a file - send this sentence to the parser on the free host (through a server of parser accessed with Net::Telnet) - get the result of the parsing (yes or no) (still through Net::Telnet...) - if yes, connect again to the server of parser to generate one or several outputs and write them into one or several files - says "I'm ready to get a new sentence to parse", triggering the variable 'free' I thought that creating an event was automatically non blocking and several events could happen at the same time. Does the problem come from the I/O interactions ? If so, do I need to separate all these stages and watch for every I/O interactions ? And do I need to specify somewhere that I want non blocking I/O ? Best regards, Isabelle -- Isabelle Cabrera Projet Alpage - INRIA Rocquencourt http://alpage.inria.fr Tel : 01 3963 5270 http://www.nntp.perl.org/group/perl.loop/2008/05/msg1121.html Wed, 28 May 2008 08:48:34 +0000 announcing AnyEvent 4.0 by Marc Lehmann I am pleased to announce AnyEvent 4.0 Why the major version bump? Lots of functionality has been added (and thus some bugs are expected), although the event core is still compatible to earlier versions and AnyEvent is still pure-perl without any (hard) dependencies on any XS modules. So whats new? Apart from a number of bugfixes, the AnyEvent::Socket module has been added. It contains a lot of utility functions to handle IPv4 and IPv6 addresses and tcp connections in a 100% non-blocking way. It supports SRV record lookups. To make the 100% non-blocking a reality, AnyEvent::Socket uses AnyEvent::DNS, a fully asynchronous stub resolver capable of making tens of thousands of DNS requests/second, with TCP and EDNS0 support. And if you have Net::SSLeay installed, then AnyEvent::Handle can transparently manage TLS/SSL connections, too. On the microshit windows side, AnyEvent now tries very hard to avoid the freeze in the testsuite on broken perls (with reduced functionality, but resolver and socket code works fine there, too). The distro should hit the CPAN mirrors in a few hours. Documentation can be found via its homepage, http://software.schmorp.de/pkg/AnyEvent Example: This (pseudo-) code snippet connects to jabber server, by querying any SRV records that might exist and connecting to the resulting addresses in turn - regardless of wether the server uses ip version 4 or ip version 6 (or both): AnyEvent::Socket::tcp_connect "jabber.ietf.org", "xmpp-client", sub { my ($fh) = @_ or die "connect: $!"; my $handle; $handle = new AnyEvent::Handle fh => $fh, on_eof => sub { undef $handle; # destroy handle on eof }, ; # etc.. # when server and client handshake a SSL connection, one can simply # call $handle->starttls ("connect") to start ssl negotiation. # push some writes or soem reads here, or set some callbacks }; This (non-pseudo-code) code snippet asynchronously connects to https://www.google.de and makes a simple GET request: use AnyEvent::Impl::Perl; # use portable perl event loop use AnyEvent::Handle; use AnyEvent::Socket; my $cv = AnyEvent->condvar; AnyEvent::Socket::tcp_connect "www.google.de", 443, sub { my ($fh) = @_ or die "connect: $!"; my $handle; # avoid direct assignment so on_eof has it in scope. $handle = new AnyEvent::Handle fh => $fh, tls => "connect", # start client-side tls negotiation on_eof => sub { undef $handle; # keep it alive till eof warn "done.\n"; $cv->send; }; $handle->push_write ("GET / HTTP/1.0\015\012\015\012"); $handle->push_read_line ("\015\012\015\012", sub { my ($handle, $line) = @_; # print response header print "HEADER\n$line\n\nBODY\n"; $handle->on_read (sub { # print response body print $_[0]->rbuf; $_[0]->rbuf = ""; }); }); }; $cv->recv; -- The choice of a Deliantra, the free code+content MORPG -----==- _GNU_ http://www.deliantra.net ----==-- _ generation ---==---(_)__ __ ____ __ Marc Lehmann --==---/ / _ \/ // /\ \/ / pcg@goof.com -=====/_/_//_/\_,_/ /_/\_\ http://www.nntp.perl.org/group/perl.loop/2008/05/msg1120.html Sat, 24 May 2008 11:28:22 +0000 Re: benchmarking various event loops with and without anyevent by Rocco Caputo On Apr 28, 2008, at 03:21, Marc Lehmann wrote: > > In all fairness, I want to point out that, after _multiple_ rounds of > longish e-mail exchanges, Rocco Caputo could not solve the problems > that > forced AnyEvent to use this design, nor did he enlighten me on how > to work > around the specific problems that I mentioned to him that forced this > design decision(*). Addressed in <629C022B-BF10-4461-9D52-8F305D2051DC@pobox.com>. Please respond there. > He did not come up with any further evidence for a problem, either > (just > repeatedly stating that the design is broken. The only argument he > brought > up was: one of your design goals is to be reasonably efficient, POE > does > not do it reasonably efficient, so your design is broken, which is an > outright absurd logic). Marc and I are disagreeing with what I wrote in the message included in <E5761512-9F7F-45ED-9C45-A8D9FFFC8C9C@pobox.com>. No amount of he- said/no-he-said will resolve it at this point, so I refer the reader back to the actual exchange. Everyone: Your suggestions to improve my communication are greatly appreciated. Please comment off-list, if you can. > In fact, it seems his problem is indeed the AnyEvent API and not the > interface module to POE, i.e. the "broken" means I should not provide > events in the form AnyEvent does, which is of course > counterproductive to > the goal of AnyEvent of being compatible to multiple event loops (I > can't > provide different APIs to different event loops...). I explicitly stated otherwise in the message included in <E5761512-9F7F-45ED-9C45-A8D9FFFC8C9C@pobox.com >. It's the sentence beginning with "We should not need to change AnyEvent::Impl::POE's public interface". > So I conclude that even the POE author is unable to provide a > (strongly) > more efficient approach, which, according to his own words, would > make him > worse then the average first-time POE user. Considering our previous discussions on the matter, I feel that your conclusion is premature. Also, you seem to be saying that one solution can simultaneously perform equally as well as another and worse than it. Which quantum computers have you ported AnyEvent to lately? :) -- Rocco Caputo - rcaputo@pobox.com http://www.nntp.perl.org/group/perl.loop/2008/04/msg1119.html Mon, 28 Apr 2008 14:19:03 +0000 Re: benchmarking various event loops with and without anyevent by Rocco Caputo On Apr 28, 2008, at 06:24, Marc Lehmann wrote: > [most important points first] > >> In your case, I would create a single persistent POE::Session >> instance >> that serviced all the watchers. > > I would, too, but I cannot find a way to do that with POE: sessions > without active watchers will be destroyed, forcing the session to have > active resources will make the program never-ending. > > I already told you that I tried this approach, and why I couldn't > get it > working. I might use something like an explicit reference count to keep the session alive. I would create a proxy object that, when DESTROYed, would post a shutdown message to the session. Or if AnyEvent knows when a program is to exit, I would have it explicitly shutdown the session as part of its finalization. The shutdown handler would remove the explicit reference count, allowing the session to stop. It's similar to the technique you use in AnyEvent::Impl::POE: sub DESTROY { POE::Kernel->post (${${$_[0]}}, "stop"); } Except it would be done once at the very end, rather than for each event watcher. Perhaps this isn't necessary. You're not using run(), so technically you're free to go at any time. If your program must exit while watchers are active, then you could force the issue by sending a global UIDESTROY signal (designed to tell POE when it must unconditionally stop), and calling run(): $poe_kernel->signal($poe_kernel, "UIDESTROY"); $poe_kernel->run(); On the other hand, your AnyEvent::Impl::POE proxy objects could also hold references to the singleton session, and if they release those references when they clean up their POE::Kernel resources, the session should be "empty" by the time they all destruct. In that case, the UIDESTROY signal should not be needed. run() will return after removing the "empty" session. In short, your AnyEvent::Impl::POE objects would be of the form: sub new { # (AnyEvent::Impl::POE setup goes here) # set up the POE::Kernel watcher $poe_kernel->something(something); # make a note of the watcher in this object $self->{something} = $record_of_the_poe_watcher; return $self; } sub DESTROY { my $self = shift; # ... release the POE::Kernel watcher $poe_kernel->something($self->{something}, something); } If you expect the user to be creating their own POE::Session instances, then you'd need to call() AnyEvent::Impl::POE to make sure the watchers are created in the right context. sub new { # (AnyEvent::Impl::POE setup goes here) # set up the POE::Kernel watcher $poe_kernel->call("anyevent_impl_poe", "something", something); # make a note of the watcher in this object $self->{something} = $record_of_the_poe_watcher; return $self; } And DESTROY would tell the session when to clear the watcher. You may need to add a new AnyEvent::Impl method to explicitly stop POE, especially if your public API allows users to exit with active watchers. sub shutdown { $poe_kernel->signal($poe_kernel, "UIDESTROY"); POE::Kernel->run(); } As an added bonus, shutting down this way satisfies the run() warning. I know this isn't a full solution, but I hope you still find it helpful. > You kepe repeating how it could be designed better, but you never > actually > say how to solve the fundamental problems and bugs within POE that > keep it > from being implementable. I would be welcome to discuss the code more than each-other. If we can agree on this, perhaps we can get down to more important matters. See above. > As I said, if possible, I can only imagine the design becoming > vastly more > complex because I would have to create sdessions on demand and be > able to > react to my session beign turned down at unopportune times. I don't understand why this design is necessary. Please help me understand your design constraints, so that I may focus on designs that will work. > What we seem to agree on by now is that such a design is not trivial > to do > with POE. > > Also, remember that the benchmarks show that session creation is not > the big problem, running the sesions is - of course, there could be > inefficiencies in POE handling large number of sessions, but that > means > just that - POE doesn't scale well. While my suggestions are not as trivial as your current design, I don't think the end design will be as complex as you expect. Thank you for your feedback. I'm sorry that POE doesn't meet your needs. When I have the chance, I'll profile POE while running your benchmark and see what I can do. > As the documentation mentions, AnyEvent doesn't enforce itself on a > module, unlike POE - a module using POE is not going to work with > other > event loops, because it monopolises the process. > > This means that a module using POE forces all its users to also use > POE. This is factually incorrect. For example, POE::Loop::Glib allows POE to be embedded into applications like vim and irssi. The application's functionality is not impaired, nor is communication between the native scripting elements and the embedded POE code. POE defers to the native event loop whenever possible. For example, the main loop from POE::Loop::Gtk: sub loop_run { unless (defined $_watcher_timer) { $_watcher_timer = Gtk->idle_add(\&_loop_resume_timer); } Gtk->main; } In cases where issues prevent POE from using the native loop, a custom loop is written around the native DoOneEvent() (or equivalent) dispatcher. > This is a fundamental difference between POE and AnyEvent, it has > nothing > to do with event loop backend modules, of which POE also emplys a > few, but > comes form the fact that you have to call POE::Kernel->run and give up > your process to it (just like with EV::loop etc.) I believe the above loop_run() above shows this to be factually incorrect. Even when POE::Kernel->run() is called, execution is passed back to the native event loop. AnyEvent::Impl::POE doesn't relinquish control to POE::Kernel->run(), so it seems to conflict with your claim. > On Mon, Apr 28, 2008 at 04:36:42AM -0400, Rocco Caputo <rcaputo@pobox.com > > wrote: > >> Are you aware that I'm gradually rewriting POE's documentation? If >> you could describe what you don't like in a useful way, I may be able >> to do something about it. > > Since I described it already (and you know that) it means you find > the way > I did it "not useful". Thats a strawman argument. If you don't like my > criticism or don't understand it, ask. Okay, I will ask. In all cases, please include sufficient information in your reply so that I can find the documentation in question. What is inaccurate? How is it inaccurate? What parts could I improve? How could I improve them? What is missing? What isn't necessary? If I've overlooked a question to ask, please let me know, and I will ask it. > It is fine with me if you don't understand AnyEvent, it is somewhat > fine > with me if you make strong (But wrong) statements about it, but don't > expect anybody to put much faith in them, or you ability to make > useful > statements. Back at you. See above for some examples where your strong statements against POE have been incorrect. As I've said before, I'd like to resolve these misunderstandings so that we can move forward in a more amicable and efficient fashion. If we can eventually exhaust our misconceptions by thoroughly addressing them, then perhaps we can move on to more productive things. >> Assuming that N is the same between the equivalent POE and >> AnyEvent::Impl::POE program: >> >> S(N*M) > S(N) for M > 1. >> >> QED :P > > I couldn't really follow you here, and I am not sure what you have > proven. To > me it certainly looks as if it was "POE cannot support the AnyEvent > API > efficiently" (at leats not in a simple and straightforward way). I don't know how to be more explicit, unambiguous, and emotionally neutral than by modeling the overhead mathematically. If that's insufficient, then I'm at a loss for how to proceed. >>> I can only imagine making some very complex on-demand >>> instantiating and >>> re- check wether the session still exists on each watcher creation. >> >> Your imagination comes up with such incredible things. Don't lose >> that. :) > > Overbearing words after being wrong before. My bad. It's a long message, and I had to let off some steam. On a more positive note, you do have an inventive mind, and I really do hope that you never lose that quality. >> I use this design in POE::Stage. > > Does it survice multiple run's? If yes, how so? I'm not sure that multiple run()s is a relevant issue. Your code seems to get along without calling run() at all. The number of POE::Session instances has no bearing on whether you can continue using loop_do_timeslice(). I am obviously overlooking something. If you could explain what that is, we could put this detail to rest and move on. >>> (the race condition with sigchld is really annoying, >> >> Please describe the race condition in a useful way, so that I can >> investigate and possibly fix it. > > Again the red herring "useful" - it obviously was useful enough for > other > people to understand the issue. The problem is that rgeistering a > watcher > after the child process already exited causes POE to introduce a > delay in > reapign the child (it does seem to do polling to do so, and seems to > do so > once per second, so the delay is at most a second, still, thats a > pretty > long time). [more specifics snipped] A useful report is one that can be use to replicate or isolate the problem. Saying "the race condition with sigchld" is not useful. Being more specific, as you have just done, is VERY useful. Thank you! An experimental fix for what you describe has already been released. Because it's experimental, you will need to enable it manually. Please verify you're using the most recent POE, and check POE::Kernel's documentation for "USE_SIGCHLD". It appears that the sig_child() documentation conflicts with the USE_SIGCHLD documentation. I have fixed it for the next release. >>> The "run() method was never called" message can be silenced by >>> calling >> run(). Not all applications require run(). But if you call it >> without any POE::Session instances, it will return immediately and >> satisfy POE::Kernel. > > This is impossible it guarentee from within AnyEvent. > > So there is no way to silence just the message. [...] > AnyEvent::Impl::POE already invokes the run workaround in its > manpage, and > why it is undesirable. This puts an unecessary burden on the user, > and is > certainly an issue. Your code already silences this warning, so I don't know why you keep saying it can't be done: # have to do this to keep POE from spilling ugly messages POE::Session->create (inline_states => { _start => sub { @_[KERNEL]- >stop } }); POE::Kernel->run; Unless AnyEvent is doing something I don't know, you should be able to drop the POE::Session->create() line without any harm. >> You complain about a message that occurs when a program leaks a child >> process. Programs that leak processes are broken in a way that is >> difficult to detect without the warning. In extreme cases, process >> leakage can quietly strangle a machine to death. > > No, I complain about the emssage complaining about POE reaping a > child. No > leak is involved anywhere, and nothign will "quietly" "strangle" > anything > to death. The message I'm thinking of occurs at the end of the program when POE checks whether any child processes remain. Exiting without reaping children is considered bad form and can lead to orphan processes on some systems, so POE enforces this behavior. The warning indicates that the application has not been looking after its children. POE doesn't reap children unless there is a sig_child() watcher. Therefore, if your program is long-running, spawns processes during its normal operation, and fails to trigger reaping, it will eventually exhaust its process resources. If the application runs as root, there will often be no ulimit guard preventing zombie processes from filling up the system's process table. If this happens, the system will most likely become unresponsive and need to be restarted. I consider such applications to be broken. You are welcome to think otherwise, but POE will continue to point out when this happens. The most direct solution is for an application to set sig_child() handlers for the children it creates. This will trigger reaping, and the warnings (and potential for catastrophe) will go away. >> OMG, we agree on something! The end times are here: > > sorry? It's a joke. You see, we agree on so little, so actually agreeing on something is a momentous event. I was humorously drawing parallels to events that signal the impending end of the world in some religions. The "end times" are also conveniently where I finished my e-mail. It's a bit of a pun, really. -- Rocco Caputo - rcaputo@pobox.com http://www.nntp.perl.org/group/perl.loop/2008/04/msg1118.html Mon, 28 Apr 2008 12:51:45 +0000 Re: benchmarking various event loops with and without anyevent by Marc Lehmann [most important points first] > In your case, I would create a single persistent POE::Session instance > that serviced all the watchers. I would, too, but I cannot find a way to do that with POE: sessions without active watchers will be destroyed, forcing the session to have active resources will make the program never-ending. I already told you that I tried this approach, and why I couldn't get it working. You kepe repeating how it could be designed better, but you never actually say how to solve the fundamental problems and bugs within POE that keep it from being implementable. As I said, if possible, I can only imagine the design becoming vastly more complex because I would have to create sdessions on demand and be able to react to my session beign turned down at unopportune times. What we seem to agree on by now is that such a design is not trivial to do with POE. Also, remember that the benchmarks show that session creation is not the big problem, running the sesions is - of course, there could be inefficiencies in POE handling large number of sessions, but that means just that - POE doesn't scale well. On Mon, Apr 28, 2008 at 04:36:42AM -0400, Rocco Caputo <rcaputo@pobox.com> wrote: > Most people on the planet don't know Perl, or even how to program a > computer. No amount of documentation will help them. :) Good (or any) documentation is widely known to be almost a _requirement_ for learning how to program or use a software package. > In the future, you may wish to include me on your list of people to > consult about designing applications for POE. I may known a thing or > two about the topic. :) I am not interested in designing applications for POE - I am interested in making it possible to write event-based modules that are interoperable between various event loops such as POE. As the documentation mentions, AnyEvent doesn't enforce itself on a module, unlike POE - a module using POE is not going to work with other event loops, because it monopolises the process. This means that a module using POE forces all its users to also use POE. AnyEvent does not do that, as long as it supports the event model actually in use (it is not an event loop itself!): A module that uses AnyEvent works with both Qt, POE, IO::Async (once its backend is implemented), EV and so on. This is a fundamental difference between POE and AnyEvent, it has nothing to do with event loop backend modules, of which POE also emplys a few, but comes form the fact that you have to call POE::Kernel->run and give up your process to it (just like with EV::loop etc.) > Are you aware that I'm gradually rewriting POE's documentation? If > you could describe what you don't like in a useful way, I may be able > to do something about it. Since I described it already (and you know that) it means you find the way I did it "not useful". Thats a strawman argument. If you don't like my criticism or don't understand it, ask. > I would love to have the opportunity to suggest a different design, You always ahd the opportunity, you are not using it. > Obviously I cannot expect you to know everything about POE. Likewise, > you cannot expect me to magically know when you started writing > AnyEvent::Impl::POE. It doesn't matter when I started writing AnyEvent::Impl::POE at all. What matters is that you made unfounded (and as you now admit, wrong) statements about it (and its author). It is fine with me if you don't understand AnyEvent, it is somewhat fine with me if you make strong (But wrong) statements about it, but don't expect anybody to put much faith in them, or you ability to make useful statements. > Even if you announced it somewhere, I may not have been looking. The > first I heard of it was here, when you announced your benchmarks. Yes, so? > In general, if you need someone's attention online, the most effective > and polite way is to contact them directly. I did not need your attention? > about this, then I'm sorry that I missed it. Are you sure your > message wasn't lost in transit? Which message? I was only conveying some benchmark results, to give people an idea of the overheads of AnyEvent of various event loop implementations in the hope of beign useful. I was also hoping that people might give AnyEvent some try, as it's design doesn't force a module author using it into a specific event loop. > Assuming that N is the same between the equivalent POE and > AnyEvent::Impl::POE program: > > S(N*M) > S(N) for M > 1. > > QED :P I couldn't really follow you here, and I am not sure what you have proven. To me it certainly looks as if it was "POE cannot support the AnyEvent API efficiently" (at leats not in a simple and straightforward way). I knew that already. > >I can only imagine making some very complex on-demand instantiating and > >re- check wether the session still exists on each watcher creation. > > Your imagination comes up with such incredible things. Don't lose > that. :) Overbearing words after being wrong before. > The "trick" is to minimize the number of sessions used. I already said that, and explaiend why it didn't work. > that sessions imposed overhead. I wrongly assumed the solution would > be obvious. Yes. > I use this design in POE::Stage. Does it survice multiple run's? If yes, how so? > experimental code, you can find it on the CPAN. It also does a lot of > other, unrelated things, so you may have difficulty separating the > magic you need from the voodoo you don't. As I said, less magic and vodoo and better documentation could be more helpful in the long run. I don't see how POE::Stage solves the problem of multiple run's stopping sessions. If it does, couldn't you just advise me on how to do it insteadf of referring to bugloads of other code? Either you know how to solve the issues, then just tell us, or you don't, in which case going through large amounts of code won't help. > I won't blame you. But I will point out that your documentation says > you're already familiar with using undocumented POE features. :) Yes indeed :/ > >Instead of going around and accuse people of making bad designs, it > >would be much better to improve the documentation for POE, so that said > >people don't have to go around and start guessing... > I don't appreciate your vague, negative comments about the > documentation. Please describe problems in a useful way, or your > expectation that they be fixed is unreasonable. I did, in a long document that you admit having read already. If you don't find the comments useful, thats your problem - you could aks me to clarify them, for example. _I_ don't appreciate your continuous implications that my comemnts were in some way flawed without every mentioning a single piece of evidence. > You seem to be saying that your design for AnyEvent::Impl::POE is > based on guesswork. Yes. > If so, I overestimated your knowledge of POE based on your > documentation. I have no idea how you estimate my knowledge. I also fail to see what my person has to do with anything. > Therefore, please consider the chance that you may know less about POE > than your documentation implies. If there is any meaning in this sentence, it escapes me. I quited examples form the documentation and explained all my arguments. It doesn't matter how much I know or imply to know, as I *explain* the reasons behind what I said. That is in stark contrast to you - you only ever imply the documentation is rude, the module is fundamentally flawed, the comments are not useful etc., without ever trying to solve the problems with POE that lead to this design and the comments, and which are explained clearly. > >Oh, and while you fix the docs, could you fix the other bugs as well > > I certainly can, but you'll need to describe them in a useful way Done a long time ago and given to you. It might not be in a form that you like, but it is doubtless that they are useful. > >(the race condition with sigchld is really annoying, > > Please describe the race condition in a useful way, so that I can > investigate and possibly fix it. Again the red herring "useful" - it obviously was useful enough for other people to understand the issue. The problem is that rgeistering a watcher after the child process already exited causes POE to introduce a delay in reapign the child (it does seem to do polling to do so, and seems to do so once per second, so the delay is at most a second, still, thats a pretty long time). > POE checks for child processes between calls to application code. If > you fork your process and set sig_child() in the same event handler, > POE won't be able to call waitpid() between them. If this is the race > condition you mean, then it simply shouldn't be possible. The race condition is quite simple: 1. fork 2. child exits 3. parents gets sigchld (but no sigchld handler is registered yet) 4. sigchld watchers gets created [time passes, POE select's many times without noticing the child exit] 5. POE eventually waitpids and finally detetcs the child > On the other hand, I haven't actually documented this behavior. It > hadn't occurred to me because all my programs follow the reliable > pattern. I have no clue what you mean, but POE ahs to support it, just as with any other event loop - you cnanot tregister a child handler BEFORE you fork because you don't know the pid. You cannot guarantee that the child exits only afetr you registered the watcher, so POE has to deal with it already having exited. AnyEvent and EV certainly deal with this issue cirretcly and don't miss the child exit. > "Programs that wish to reliably reap child processes should be sure to > call sig_child() before returning from the event handler that forked > the process. Otherwise POE::Kernel may have an opportunity to call > waitpid() before an appropriate event watcher has been registered." No, it is a race in POE, not something the application can do anything about. > In the event that you've found an actual bug, I regret that your > description of the problem is not useful. Well, I can declare anything I want "not useful" - if you are incapable of understanding, you should ask - I don't know how wlel your knowledge is w.r.t. to process handling on unix, and I cannot write full novels and programming manuals just in case I need to cover something you don't understand. > <purl> Look buddy, doesn't work is a strong statement. Does it sit on "doesn't work" is certainly not how I described it. > >and the nag messages are as well). > > I understand from your documentation that you dislike the "ugly" "nag" > messages. I'm sorry, but I cannot help you there. They were added as > the result of useful feedback from other users. And contrary to your > documentation, they can be silenced. Possible, but it is not documented (yes, run silences them, but it has other undesirable effects as well, which is documented by AnyEvent's adaptor module). > The "run() method was never called" message can be silenced by calling > run(). Not all applications require run(). But if you call it > without any POE::Session instances, it will return immediately and > satisfy POE::Kernel. This is impossible it guarentee from within AnyEvent. So there is no way to silence just the message. > While this does not: > > perl -wle 'use POE; POE::Kernel->run; POE::Session- > >create(inline_states => { _start => sub{} });' As I said, for design reasons, Anyevent doesn't enforce itself on its users, so the requirement to call ->run is not implementable. AnyEvent::Impl::POE already invokes the run workaround in its manpage, and why it is undesirable. This puts an unecessary burden on the user, and is certainly an issue. > I hadn't considered to document the technique because it seemed > obvious to me. It is obvious to me. But there is now ay to do that from within a module: a module has no control over when it was loaded. It is just one of the many small design issues (some of which I documented) that make it very hard to use POE in a generic way. I don't claim more, I don't claim less. > obvious to me than are actually obvious to the average user. You > should probably call them to my attention if you'd like them to be > documented. I only need a way to silence them withotu potentially freezing the program, clearing the event queue, stopping running sessions etc. > Speaking of which, I added this to the docs. Please let me know how/ > if I can improve it: No amount of documentation can improve this, it is a problem within the POE code. > You complain about a message that occurs when a program leaks a child > process. Programs that leak processes are broken in a way that is > difficult to detect without the warning. In extreme cases, process > leakage can quietly strangle a machine to death. No, I complain about the emssage complaining about POE reaping a child. No leak is involved anywhere, and nothign will "quietly" "strangle" anything to death. > POE can detect this silent but deadly condition. There is nothing deadly about it. > If you receive this message, it can be silenced by fixing the process > leak. ... which is nonexistant. > >Not sure why it would be a courtesy to me, do you have anything to > >hide > >or do you want to insult me even more in private so nobody sees your > >real > >self or something? (just guessing... :) > > Wow, you really do expect the worst from me. I don't know where that > came from. I expect you to back up your statements you made publicly (and in private). As I told you before, it *is* bad (or worse) to make statements without backing them up. > The courtesy to you would be that I brought the issues to you > directly. It would be, if you had done that. But in private you just insulted me more directly, but despite reqpeatedly askign you how to solve the probelms with session lifetime, you never came up with any evidence. > I could post them somewhere public and unexpected and wait > for you to find them, but that's just rude. Thats much better then making public statements and not tellign anybody about it, especially not me. > Likewise, posting long, limited-interest e-mail to a public > distribution list is a violation of common etiquette. To those who > are still reading, I'm sorry. Discussing various event loops for perl is certainly within the charter of this mailing list. Also note that you started it - I backed up all my claims with evidence (even if it is wrong, one can at least verify my claims). All you did is make broad statements such as "your module has a broken design" without ever explaining how to work around the problems I documented that caused this allegedly "broken design". > I kind of thought those points would be obvious. It is possible that everything is obvious to you, but if you make broad statements like the above you *need to back them up with evidence*. > If you still insist on doing this publicly, please indicate your > intent by posting your documentation for AnyEvent::Impl::POE to the > mailing list (new thread, please). The docoumentation always was publicly available from the AnyEvent site, also where the benchmakr results have bene published. Anyevent 3.3 has now been released with the POE module, though, so it should not be a problem to find it anymore, it is on CPAN. > OMG, we agree on something! The end times are here: sorry? -- The choice of a Deliantra, the free code+content MORPG -----==- _GNU_ http://www.deliantra.net ----==-- _ generation ---==---(_)__ __ ____ __ Marc Lehmann --==---/ / _ \/ // /\ \/ / pcg@goof.com -=====/_/_//_/\_,_/ /_/\_\ http://www.nntp.perl.org/group/perl.loop/2008/04/msg1117.html Mon, 28 Apr 2008 03:24:37 +0000 Re: benchmarking various event loops with and without anyevent by Rocco Caputo On Apr 28, 2008, at 03:21, Marc Lehmann wrote: > On Sat, Apr 26, 2008 at 08:13:27AM -0400, Rocco Caputo <rcaputo@pobox.com > > wrote: >> each event watcher. Anyone who knows POE can tell you this is one of >> the least efficient designs possible. In fact, this design is worse >> than the average for first-time POE users. > > [He then called my model fundamentally broken in private mail and the > documentation rude and unprofessional, without bringing up any > evidence] Dear perl-loop@perl.org, and anyone reading this in the archives. I apologize for my part in this unfolding thread. As Marc mentions, I have been trying to take it to private e-mail. However I feel that Marc has misrepresented to the list what I said to him in private. At this point, it's easier for me to repost what I actually said rather than paraphrase it. Again, I'm sorry you had to be involved. -- Rocco Caputo - rcaputo@pobox.com Begin forwarded message: > From: Rocco Caputo <rcaputo@pobox.com> > Date: April 27, 2008 23:29:18 EDT > To: Marc Lehmann <schmorp@schmorp.de> > Subject: Re: benchmarking various event loops with and without > anyevent > > On Apr 27, 2008, at 01:53, Marc Lehmann wrote: > >> On Sun, Apr 27, 2008 at 01:15:49AM -0400, Rocco Caputo <rcaputo@pobox.com >> > wrote: >>> I have read your code and documentation for AnyEvent::Impl::POE. >>> Your >>> module's design is fundamentally broken, and your code is probably >>> more to blame than POE. >> >> Oh, btw, be careful with such strong idioms such as "fundemantally >> broken": >> so far, there is no evidence that it is broken at all, only >> inefficient. >> >> (If you think it really is fundamentally _broken_ then you better >> back up >> your statements). > > I hope to show that I intended no offense. > > Reasonable scalability (CPU and memory) seems to be one of > AnyEvent's design goals. I base this impression on the fact that > you're benchmarking your code in terms of speed and size. > "Reasonable" is subject to interpretation, but I think we agree that > AnyEvent::Impl::POE is neither as fast nor as small as it should be. > > Therefore, while AnyEvent::Impl::POE operates correctly, it does not > fulfill some of AnyEvent's design goals. > > AnyEvent::Impl::POE's greatest inefficiencies stem from one > fundamental design choice: the 1:1 relationship between watcher > instances and POE::Session instances. In your own words: "AnyEvent > has to create one POE::Session per event watcher, which is immensely > slow and makes watchers very large." > > One point of contention may be whether this is a design or > implementation flaw. The problem is inherent in the way one class > (AnyEvent::Impl::POE) interacts with another (POE::Session). Class > interaction is a software design issue. It can be modeled in > software design languages such as UML. Re-implementing the same > entity relationship more efficiently, or in a faster language such > as C, would not resolve the scalability problem. > > Therefore, AnyEvent::Impl::POE is flawed in design rather than > implementation. > > Therefore, AnyEvent::Impl::POE's design prevents it from meeting > some of AnyEvent's design goals. > > Therefore, AnyEvent::Impl::POE's design is broken. > > Unfortunately most of AnyEvent::Impl::POE's design stems from its > flawed interaction with POE::Session. We should not need to change > AnyEvent::Impl::POE's public interface, but we will need to rethink > and revise nearly all aspects of its interaction with POE. > > Therefore, AnyEvent::Impl::POE's design flaw is a fundamental one. > > Therefore, AnyEvent::Impl::POE's design is fundamentally broken. > >> Again, let me repeat that empty insults that obviously are founded by >> paranoia will not have any positive effect on your standing with me >> (I >> mean, I won't hate you or anything, but you make yourself an idiot >> in my >> eyes very quickly by repeatedly not beinging up any evidence...). >> >> Of course, I understand that if you mistook my comments about POE >> as rude, >> there is a natural tendency to "insult back". > > > I hope I have shown reasonable evidence that my assertion is neither > empty nor intrinsically insulting. Without the intent to insult, > there can be no intent to "insult back". In the end, any offense > you have taken may be of your own manufacture. Could there be > cultural differences to overcome? > > In this light, your assertion of my paranoia is unfounded, unjust > and offensive. Your view that I'm acting like an idiot is no > better. You are of course entitled to your opinions, but those two > are not appropriate for polite conversation. > > ... > > I'm sorry that I haven't responded promptly to your e-mail. I get > the impression that you expect the worst from me, so I feel the need > to choose my words in a slow and painful (and often futile) effort > to minimize being misunderstood. As a result, I cannot write to you > as often as I would like. Nor have I had the opportunity to work > with you on the module in question (something I would much rather be > doing). > > I hope we can reach a mutual understanding, if only so that we may > correspond and collaborate more effectively. > > Until next time, > > -- > Rocco Caputo - rcaputo@pobox.com http://www.nntp.perl.org/group/perl.loop/2008/04/msg1116.html Mon, 28 Apr 2008 02:21:09 +0000 Re: benchmarking various event loops with and without anyevent by Rocco Caputo On Apr 27, 2008, at 00:56, Marc Lehmann wrote: > On Sat, Apr 26, 2008 at 08:13:27AM -0400, Rocco Caputo <rcaputo@pobox.com > > wrote: >> each event watcher. Anyone who knows POE can tell you this is one of >> the least efficient designs possible. > > It is the only design that I could get working, even after > consulting a > few people and implementing some workarounds for the bugs in POE. > > In any case, you have to consider that most people on this planet > don't know POE, and even if, they don't know it that well. Since the > documentation for POE is in such a bad state, thats the obvious way > to fix > that. Most people on the planet don't know Perl, or even how to program a computer. No amount of documentation will help them. :) In the future, you may wish to include me on your list of people to consult about designing applications for POE. I may known a thing or two about the topic. :) Are you aware that I'm gradually rewriting POE's documentation? If you could describe what you don't like in a useful way, I may be able to do something about it. >> In fact, this design is worse than the average for first-time POE >> users. > > If a better design is possible, it is not known to me, and you haven't > suggested one either, so talk is cheap. I'd be happy to get a more > efficient design for POE but nobody could come up with one that also > worked reliably through multiple iterations of run and also does not > keep > the POE kernel from returning. I would love to have the opportunity to suggest a different design, but most of my discretionary time is spent addressing the constant misunderstandings between us. If we can first resolve them, I'll have that much more time to work on the design. Obviously I cannot expect you to know everything about POE. Likewise, you cannot expect me to magically know when you started writing AnyEvent::Impl::POE. Even if you announced it somewhere, I may not have been looking. The first I heard of it was here, when you announced your benchmarks. In general, if you need someone's attention online, the most effective and polite way is to contact them directly. If you did contact me about this, then I'm sorry that I missed it. Are you sure your message wasn't lost in transit? As for the design: First-time POE users tend to design programs where the number of sessions scales linearly with the number of objects that handle events. If S(1) is the total overhead imposed by a single session, then S(N) is the overhead imposed by the average naïve POE user. N is the number of objects handling events. AnyEvent::Impl::POE creates a new POE::Session for every event watcher. It's not uncommon for an object to use more than one event watcher (I/O and timeout, for example). So we can model the session overhead in an AnyEvent::Impl::POE program as S(N*M), where N is the number of objects handling events, and M is the average number of event watchers per object. Assuming that N is the same between the equivalent POE and AnyEvent::Impl::POE program: S(N*M) > S(N) for M > 1. QED :P > If a better design *is* possible (which I don't really doubt), then it > needs to be vastly more complex, or it needs some non-obvious trick. > I can > only imagine making some very complex on-demand instantiating and re- > check > wether the session still exists on each watcher creation. Your imagination comes up with such incredible things. Don't lose that. :) The "trick" is to minimize the number of sessions used. Your benchmarks and comments in your documentation implied that you knew that sessions imposed overhead. I wrongly assumed the solution would be obvious. In your case, I would create a single persistent POE::Session instance that serviced all the watchers. The watchers themselves would be small proxies that controlled POE::Kernel watchers within that session's context. I use this design in POE::Stage. If you're not averse to looking at experimental code, you can find it on the CPAN. It also does a lot of other, unrelated things, so you may have difficulty separating the magic you need from the voodoo you don't. Comments are welcome, if they're useful. > (It is possible that I was fooled by the docs as well, so if there > is a > better way, it likely isn't documented, so who could blame me). I won't blame you. But I will point out that your documentation says you're already familiar with using undocumented POE features. :) > Instead of going around and accuse people of making bad designs, it > would > be much better to improve the documentation for POE, so that said > people > don't have to go around and start guessing... I don't appreciate your vague, negative comments about the documentation. Please describe problems in a useful way, or your expectation that they be fixed is unreasonable. You seem to be saying that your design for AnyEvent::Impl::POE is based on guesswork. If so, I overestimated your knowledge of POE based on your documentation. Therefore, please consider the chance that you may know less about POE than your documentation implies. > Oh, and while you fix the docs, could you fix the other bugs as well I certainly can, but you'll need to describe them in a useful way first. http://search.yahoo.com/search?p=useful+bug+reports seems to link to good tips. And please post them to POE's rt.cpan.org queue. The Perl community has been kind enough to establish a convenient, central place to report problems with CPAN distributions. It would be most cooperative of you to use it. > (the race condition with sigchld is really annoying, Please describe the race condition in a useful way, so that I can investigate and possibly fix it. POE checks for child processes between calls to application code. If you fork your process and set sig_child() in the same event handler, POE won't be able to call waitpid() between them. If this is the race condition you mean, then it simply shouldn't be possible. On the other hand, I haven't actually documented this behavior. It hadn't occurred to me because all my programs follow the reliable pattern. Based on your feedback, I added this to the description of sig_child(). Please let me know how/if I can improve it: "Programs that wish to reliably reap child processes should be sure to call sig_child() before returning from the event handler that forked the process. Otherwise POE::Kernel may have an opportunity to call waitpid() before an appropriate event watcher has been registered." In the event that you've found an actual bug, I regret that your description of the problem is not useful. As we say in IRC: <purl> Look buddy, doesn't work is a strong statement. Does it sit on the couch all day? Is it making faces at you? Does it want more money? Is it sleeping with your girlfriend? Please be specific! > and the nag messages are as well). I understand from your documentation that you dislike the "ugly" "nag" messages. I'm sorry, but I cannot help you there. They were added as the result of useful feedback from other users. And contrary to your documentation, they can be silenced. The "run() method was never called" message can be silenced by calling run(). Not all applications require run(). But if you call it without any POE::Session instances, it will return immediately and satisfy POE::Kernel. For example, this generates the warning: perl -wle 'use POE; POE::Session->create(inline_states => { _start => sub{} });' While this does not: perl -wle 'use POE; POE::Kernel->run; POE::Session- >create(inline_states => { _start => sub{} });' I hadn't considered to document the technique because it seemed obvious to me. By now I hope it's clear that a lot more things seem obvious to me than are actually obvious to the average user. You should probably call them to my attention if you'd like them to be documented. Speaking of which, I added this to the docs. Please let me know how/ if I can improve it: "POE::Kernel will print a strong message if a program creates sessions but fails to call run(). If the lack of a run() call is deliberate, you can avoid the message by calling it before creating a session. run() at that point will return immediately, and POE::Kernel will be satisfied." ... You complain about a message that occurs when a program leaks a child process. Programs that leak processes are broken in a way that is difficult to detect without the warning. In extreme cases, process leakage can quietly strangle a machine to death. POE can detect this silent but deadly condition. It notifies users out of courtesy to them, and also because I was tired of handling "POE crashed my machine!" reports. In this case, the one who smelled it did not deal it. If you receive this message, it can be silenced by fixing the process leak. >> I have specific issues with your docs. As a courtesy to you and the >> list, I'll send them to you directly. > > Not sure why it would be a courtesy to me, do you have anything to > hide > or do you want to insult me even more in private so nobody sees your > real > self or something? (just guessing... :) Wow, you really do expect the worst from me. I don't know where that came from. The courtesy to you would be that I brought the issues to you directly. I could post them somewhere public and unexpected and wait for you to find them, but that's just rude. Likewise, posting long, limited-interest e-mail to a public distribution list is a violation of common etiquette. To those who are still reading, I'm sorry. I kind of thought those points would be obvious. If you still insist on doing this publicly, please indicate your intent by posting your documentation for AnyEvent::Impl::POE to the mailing list (new thread, please). That will make it easier for me to respond. For the record, I still think this is a bad idea. > I do fix my bugs and am open to suggestions and improvements. But > correctness comes first, performance second. OMG, we agree on something! The end times are here: -- Rocco Caputo - rcaputo@pobox.com http://www.nntp.perl.org/group/perl.loop/2008/04/msg1115.html Mon, 28 Apr 2008 01:37:00 +0000 Re: benchmarking various event loops with and without anyevent by Marc Lehmann On Sat, Apr 26, 2008 at 08:13:27AM -0400, Rocco Caputo <rcaputo@pobox.com> wrote: > each event watcher. Anyone who knows POE can tell you this is one of > the least efficient designs possible. In fact, this design is worse > than the average for first-time POE users. [He then called my model fundamentally broken in private mail and the documentation rude and unprofessional, without bringing up any evidence] In all fairness, I want to point out that, after _multiple_ rounds of longish e-mail exchanges, Rocco Caputo could not solve the problems that forced AnyEvent to use this design, nor did he enlighten me on how to work around the specific problems that I mentioned to him that forced this design decision(*). He did not come up with any further evidence for a problem, either (just repeatedly stating that the design is broken. The only argument he brought up was: one of your design goals is to be reasonably efficient, POE does not do it reasonably efficient, so your design is broken, which is an outright absurd logic). In fact, it seems his problem is indeed the AnyEvent API and not the interface module to POE, i.e. the "broken" means I should not provide events in the form AnyEvent does, which is of course counterproductive to the goal of AnyEvent of being compatible to multiple event loops (I can't provide different APIs to different event loops...). So I conclude that even the POE author is unable to provide a (strongly) more efficient approach, which, according to his own words, would make him worse then the average first-time POE user. This is absurd, so I conclude that the original claim has been disproven. (And yes, I did ask multiple times to come up with how to better design the interface to POE, or how to solve the lifetime-issues with POE). (*) the specific problems are (taken directly from my mail to rocco): - the session must not go away, or there must be an easy way to recreate it when the kernel kills it. - the session itself must not keep the kernel "alive"/running (preferably without going away). -- The choice of a Deliantra, the free code+content MORPG -----==- _GNU_ http://www.deliantra.net ----==-- _ generation ---==---(_)__ __ ____ __ Marc Lehmann --==---/ / _ \/ // /\ \/ / pcg@goof.com -=====/_/_//_/\_,_/ /_/\_\ http://www.nntp.perl.org/group/perl.loop/2008/04/msg1114.html Mon, 28 Apr 2008 00:22:12 +0000 Re: benchmarking various event loops with and without anyevent by Marc Lehmann On Sat, Apr 26, 2008 at 08:13:27AM -0400, Rocco Caputo <rcaputo@pobox.com> wrote: > each event watcher. Anyone who knows POE can tell you this is one of > the least efficient designs possible. It is the only design that I could get working, even after consulting a few people and implementing some workarounds for the bugs in POE. In any case, you have to consider that most people on this planet don't know POE, and even if, they don't know it that well. Since the documentation for POE is in such a bad state, thats the obvious way to fix that. > In fact, this design is worse than the average for first-time POE users. If a better design is possible, it is not known to me, and you haven't suggested one either, so talk is cheap. I'd be happy to get a more efficient design for POE but nobody could come up with one that also worked reliably through multiple iterations of run and also does not keep the POE kernel from returning. If a better design *is* possible (which I don't really doubt), then it needs to be vastly more complex, or it needs some non-obvious trick. I can only imagine making some very complex on-demand instantiating and re-check wether the session still exists on each watcher creation. (It is possible that I was fooled by the docs as well, so if there is a better way, it likely isn't documented, so who could blame me). Instead of going around and accuse people of making bad designs, it would be much better to improve the documentation for POE, so that said people don't have to go around and start guessing... Oh, and while you fix the docs, could you fix the other bugs as well (the race condition with sigchld is really annoying, and the nag messages are as well). > I have specific issues with your docs. As a courtesy to you and the > list, I'll send them to you directly. Not sure why it would be a courtesy to me, do you have anything to hide or do you want to insult me even more in private so nobody sees your real self or something? (just guessing... :) I do fix my bugs and am open to suggestions and improvements. But correctness comes first, performance second. Greetings, -- The choice of a Deliantra, the free code+content MORPG -----==- _GNU_ http://www.deliantra.net ----==-- _ generation ---==---(_)__ __ ____ __ Marc Lehmann --==---/ / _ \/ // /\ \/ / pcg@goof.com -=====/_/_//_/\_,_/ /_/\_\ http://www.nntp.perl.org/group/perl.loop/2008/04/msg1113.html Sat, 26 Apr 2008 21:56:31 +0000 Re: benchmarking various event loops with and without anyevent by Rocco Caputo On Apr 25, 2008, at 22:46, Marc Lehmann wrote: > > The next release of AnyEvent contains support for a few more > "backends", > notably POE, so AnyEvent is now by definition compatible to POE > (before it > was only compatible when using an even loop used by POE, such as > Event or > EV that could be shared). > The results were mostly as expected, with EV leading and POE being > abysmal. It's no wonder that AnyEvent::Impl::POE runs so slowly. According to your documentation, you designed it to use a separate POE::Session for each event watcher. Anyone who knows POE can tell you this is one of the least efficient designs possible. In fact, this design is worse than the average for first-time POE users. On the bright side, you could still have done worse. You could have instantiated and destroyed a POE::Session for each event. Perhaps you're saving that for a future release. :) I have specific issues with your docs. As a courtesy to you and the list, I'll send them to you directly. -- Rocco Caputo - rcaputo@pobox.com http://www.nntp.perl.org/group/perl.loop/2008/04/msg1112.html Sat, 26 Apr 2008 05:13:43 +0000 Re: benchmarking various event loops with and without anyevent by Uri Guttman >>>>> "ML" == Marc Lehmann <schmorp@schmorp.de> writes: ML> The surprising one was the pure perl implementation, which was quite on ML> par with C-based event loops such as Event or Glib. I did expect the pure ML> perl implementatioon to be at least a factor of three slower than Event or ML> Glib. ML> As the pure perl loop wasn't written with high performance in mind, this ML> prompted me to optimise it for some important cases (mostly to get rid of ML> the O(n²) degenerate cases and improving the select bitmask parsing for ML> the sparse case). check out stem's pure perl event loop. there are examples in the /sessions dir on how to use that directly without the rest of the modules. it does things in a different direction and doesn't scan select's bit masks but instead it scans the interesting handles and see whether their bits are set. it should exhibit good behavior under growth as all the data are managed in hashes. ML> I then made a second benchmark, designed not to measure anyevent overhead, ML> but to measure real-world performance of a socket server. and that /sessions code also shows use of the asyncio module. if you can benchmark that i would be interested in the results. ML> The result is that the pure perl event loop used as fallback in AnyEvent ML> single-handedly beats Glib by a large margin, and even event by a factor ML> of two. ML> For small servers, the overhead introduced by running a lot of perl ML> opcodes per iteration dominates, however, reflected in the last benchmark. in a heavily loaded server most of the work in in the i/o and should overwhelm the event loop itself. that is the whole purpose of event loops as we all know here. ML> However, the net result is that the pure perl event loop performs ML> better than almost all other event loops (EV being the only exception) ML> ins erious/medium-sized cases, while I originally expected it to fail ML> completely w.r.t. performance and being only usable as a workaround when ML> no "better" event module is installed. i don't find that surprising. perl's i/o is decent and as i said above, a loaded server is doing mostly i/o. ML> All the benchmark data and explanations can be found here: ML> http://pod.tst.eu/http://cvs.schmorp.de/AnyEvent/lib/AnyEvent.pm#BENCHMARKS ML> The code is not yet released and likely still buggy (the question is ML> whether any bugs affect the benchmark results). It is only available via ML> CVS: http://software.schmorp.de/pkg/AnyEvent i will take a gander and see if i can play with it and add stem's loop to it. if you want to work on this with me, i wouldn't mind the help. thanx, uri -- Uri Guttman ------ uri@stemsystems.com -------- http://www.sysarch.com -- ----- Perl Code Review , Architecture, Development, Training, Support ------ --------- Free Perl Training --- http://perlhunter.com/college.html --------- --------- Gourmet Hot Cocoa Mix ---- http://bestfriendscocoa.com --------- http://www.nntp.perl.org/group/perl.loop/2008/04/msg1111.html Sat, 26 Apr 2008 02:34:46 +0000 Re: benchmarking various event loops with and without anyevent by Uri Guttman >>>>> "ML" == Marc Lehmann <schmorp@schmorp.de> writes: ML> On Fri, Apr 25, 2008 at 11:40:03PM -0400, Uri Guttman <uri@stemsystems.com> wrote: >> check out stem's pure perl event loop. there are examples in the ML> Maybe I'll provide a backend for stem. actually it makes more sense to me to wrap anyevent in stem. it already has several event wrappers (pure perl, event.pm and tk) and wrapping is very easy to do. not much different than the code i see in anyevent. >> modules. it does things in a different direction and doesn't scan >> select's bit masks but instead it scans the interesting handles and see >> whether their bits are set. it should exhibit good behavior under growth >> as all the data are managed in hashes. ML> That is exactly as the anyevent perl backend did in the previous ML> version. I don't see how that should exhibit good behaviour - file ML> descristpors are small integers, so hashes only add overhead over arrays, ML> and checking every file descriptor that way is much slower than scanning ML> the bitmask. ML> Especially in the important case of many handles/few active ones, an approach ML> of "scan all handles" is very slow. my code doesn't scan all handle, but scan all writeable events to see if their handle bits are set. descriptors increment and can cause bloat of the array if you have many in use (and not all of them in the event loop). ML> (The benchmarks reflect this, you could try with an older anyevent ML> release where we just check the keys of the hash storing fd ML> information: performance is much lower). ML> I then made a second benchmark, designed not to measure anyevent overhead, ML> but to measure real-world performance of a socket server. >> >> and that /sessions code also shows use of the asyncio module. if you can >> benchmark that i would be interested in the results. ML> Hmm, tt seems to have become fashionable lately to call synchronous I/O ML> asynchronous (see IO::Async, another 100% synchronous framework). What ML> exactly is asynchronous about the I/O in that example, it seems to be ML> fully synchronous to me (just event-driven). event loops are async in that you get callbacks as they are needed. sure you can't get true async behavior from any cpu/os combo but calling it async is no different than calling it parallel processing when it is just context switching among processes on a single cpu. the illusion and api are what matters. a better term may be non-blocking i/o (and socket connections) but that needs more explanation in some ways. my view is that threads are the real sync app style and event loops are the async style. but this is not the time nor place to discuss that. >> i don't find that surprising. perl's i/o is decent and as i said above, >> a loaded server is doing mostly i/o. ML> Well, since the server in this benchmark hardly does anything, and as you ML> can see by the results, the event loop completely dominates, and bad event ML> loops are a thousand times slower than good ones. i haven't looked at the benchmark stuff yet. i was browsing the code earlier. ML> When put into requests/s, this means that with POE, you are limited ML> to <1000 requests/s, while with EV you can easily reach hundreds of ML> thousands. ML> Also, this does not explain at all why Event is so slow, and why Glib scales ML> so extremely badly. Most of the stuff that slows down the perl-based event ML> loop is clearly stuff that is much much faster in C. poor memory management in the c code? i have done pure c event loops in a framework where memory management was fairly fast due to cached queues of known block sizes. alloc/free were usually a trivial call and the event code had it own private fixed size buffer queues. it had no problem doing 2k parallel web fetches including all the url parsing all on a single sparc. we had to throttle it down to keep up with the slower indexer. ML> For a reasonable event loop implemented in C, the overhead of ML> calling select should completely dominate the rest of the ML> processing (it does so in EV). true, but bad c (and perl) code is all around us. >> i will take a gander and see if i can play with it and add stem's loop >> to it. if you want to work on this with me, i wouldn't mind the help. ML> Well, can't say thereis much demand for it, but if you cna give me a pointer ML> on these things in the docs I can probably come up with one before the next ML> release: ML> - how can I do one iteration of the event loop? This is important i don't have an api for oneevent. i still haven't seen a need for it. hence having stem wrap anyevent may be the better way. the goal for me is to have stem support more (and faster) event loops. if you are doing a stem app, you should use the stem event api. ML> for condvars: the function must not block after one or more events ML> have been handled. ?? ML> - how do I register a callback to be called when an fh or fd becomes ML> "readable" or "writable"? the sessions examples show plenty of that. very similar to event.pm overall. ML> - how can I register a relative timer? relative? is that a delay timer (relative to now)? event.pm has hard (timing from before callback to next trigger) and soft (time from after callback to next trigger) timers. ML> - are there any limitations, for example, what happens when I register ML> two read watchers for one fh (or fd)? shouldn't be a problem if select handles it. the stem perl event loop is a wrapper around select. ML> - does stem provide something to catch signals synchronously? either use event.pm as the core loop or the pure perl one can do it under 5.8 and safe signals. ML> - how about child status changes or exits? signals are signals. it handles sigchld and the stem::proc module deals with reaping as needed. ML> The EV or Event implementation modules should give a good idea of whats ML> required. ML> And for the documentation: ML> - how does stem handle time jumps, if at all? ?? ML> - are it's timers based on absolute or wallclock time or relative/monotonic time? iirc it checks wallclock (HiRes::time()) after each call to select and gets its current delta from that. you can't trust a relative clock as it will skew around. uri -- Uri Guttman ------ uri@stemsystems.com -------- http://www.sysarch.com -- ----- Perl Code Review , Architecture, Development, Training, Support ------ --------- Free Perl Training --- http://perlhunter.com/college.html --------- --------- Gourmet Hot Cocoa Mix ---- http://bestfriendscocoa.com --------- http://www.nntp.perl.org/group/perl.loop/2008/04/msg1110.html Sat, 26 Apr 2008 02:34:41 +0000 Benchmarking Stem :) by Marc Lehmann Ok, I have enough for a benchmark, just short of being able to provide a working implementation. I actually ran out of memory with the Stem implementation, it seems to use a lot of additional memory when running the event loop. I had to reduce the number of handles to 20000. Otherwise, it looks quite good (this is the synthetic benchmark that doesn't test event loop performance, just overheads): name watchers bytes create invoke EV/EV 100000 244 0.54 0.45 Perl/Any 100000 513 4.61 0.87 Stem/Any 20000 1159 31.51 9.12 Event/Any 16000 936 38.80 31.91 POE/Any 2000 6658 108.61 726.39 It is only 10 times slower than anyevents pure perl backend, but three times faster than Event. For the large server benchmark, it still looks very good: name sockets create request EV 20000 68.51 10.65 Perl 20000 73.58 112.90 Event 20000 199.29 252.45 Stem 20000 128.11 660.92 Glib 20000 632.13 1818.40 POE 20000 344.56 12154.89 Only 6 times slower than anyevents pure perl backend, but faster than Glib (ok, glib clearly has issues :) So it is actually quite good, and quite usable, except for the out of memory issue. (This is without the multiple watchers per fd/combo workaround, btw., which would only affect create speed). It hurts a bit to come so far and then to only fail because stem has no one_iteration-like functionality, though (I would go with busy waiting, if that were implementable, but thats not reasonable to implement either). -- The choice of a Deliantra, the free code+content MORPG -----==- _GNU_ http://www.deliantra.net ----==-- _ generation ---==---(_)__ __ ____ __ Marc Lehmann --==---/ / _ \/ // /\ \/ / pcg@goof.com -=====/_/_//_/\_,_/ /_/\_\ http://www.nntp.perl.org/group/perl.loop/2008/04/msg1109.html Sat, 26 Apr 2008 00:15:11 +0000 Re: benchmarking various event loops with and without anyevent by Marc Lehmann On Sat, Apr 26, 2008 at 01:14:07AM -0400, Uri Guttman <uri@stemsystems.com> wrote: > actually it makes more sense to me to wrap anyevent in stem. it already Yes, after it turned out that stopping the loop also seems to clear events it became clear that Stem cannot provide even the stripped down interface of anyevent (even when one works aorund the bugs that make it unusable). Too bad, I'd liked to benchmark it, but thats almost out of the question (maybe with a quick hack...). -- The choice of a Deliantra, the free code+content MORPG -----==- _GNU_ http://www.deliantra.net ----==-- _ generation ---==---(_)__ __ ____ __ Marc Lehmann --==---/ / _ \/ // /\ \/ / pcg@goof.com -=====/_/_//_/\_,_/ /_/\_\ http://www.nntp.perl.org/group/perl.loop/2008/04/msg1108.html Sat, 26 Apr 2008 00:00:39 +0000 Re: benchmarking various event loops with and without anyevent by Marc Lehmann Hmm, more bugs: sub stopbusywaiting { Stem::Event::stop_loop; } my $stopper = new Stem::Event::Timer object => $dummy, method => "stopbusywaiting", delay => 0.05, interval => 0.05; # bug workaround warn "a\n"; $stopper->start; Stem::Event::start_loop; $stopper->cancel; warn "b\n"; This always takes a second, here is an strace: http://ue.tst.eu/57e55320bf26d8c4ae403e722bd7e268.txt adding "use Time::HiRes 'time'" to Perl.pm makes it run fast (yes, diretcly below the code that should handle this, something is broken witht he time::hires loading code apparently, maybe its this weird core-overwriting issue). -- The choice of a Deliantra, the free code+content MORPG -----==- _GNU_ http://www.deliantra.net ----==-- _ generation ---==---(_)__ __ ____ __ Marc Lehmann --==---/ / _ \/ // /\ \/ / pcg@goof.com -=====/_/_//_/\_,_/ /_/\_\ http://www.nntp.perl.org/group/perl.loop/2008/04/msg1107.html Fri, 25 Apr 2008 23:53:49 +0000 Re: benchmarking various event loops with and without anyevent by Marc Lehmann On Sat, Apr 26, 2008 at 01:14:07AM -0400, Uri Guttman <uri@stemsystems.com> wrote: > >>>>> "ML" == Marc Lehmann <schmorp@schmorp.de> writes: > > ML> On Fri, Apr 25, 2008 at 11:40:03PM -0400, Uri Guttman <uri@stemsystems.com> wrote: > >> check out stem's pure perl event loop. there are examples in the > > ML> Maybe I'll provide a backend for stem. > > actually it makes more sense to me to wrap anyevent in stem. it already > has several event wrappers (pure perl, event.pm and tk) and wrapping is > very easy to do. not much different than the code i see in anyevent. FYI: I think I found a bug, one-shot timer events never get reported: Pelr.pm: - calls timer_triggered, - which cancels, - which sets active to 0, - then calls trigger - which returns becasue active is 0 and callback never gets invoked. Maybe my reading of the sourcecode is wrong, but I can't for the life of it get any timer event out of stem. -- The choice of a Deliantra, the free code+content MORPG -----==- _GNU_ http://www.deliantra.net ----==-- _ generation ---==---(_)__ __ ____ __ Marc Lehmann --==---/ / _ \/ // /\ \/ / pcg@goof.com -=====/_/_//_/\_,_/ /_/\_\ http://www.nntp.perl.org/group/perl.loop/2008/04/msg1106.html Fri, 25 Apr 2008 23:43:57 +0000 Re: benchmarking various event loops with and without anyevent by Marc Lehmann On Sat, Apr 26, 2008 at 01:14:07AM -0400, Uri Guttman <uri@stemsystems.com> wrote: > ML> Maybe I'll provide a backend for stem. > > actually it makes more sense to me to wrap anyevent in stem. it already of course, using anyevent always makes sense. however, using anyevent doesn't solve the interoeprability problem: you still cannot use modules using anyevent when you don't use the anyevent but the wx interface, for example. > has several event wrappers (pure perl, event.pm and tk) and wrapping is > very easy to do. not much different than the code i see in anyevent. Yes, but that doesn't give you much advantage - as long as your module isn't good citizen and plays nicely with other modules by monopolising the process it is not interoperable. Making anyevent compatible to stem enables anyevent modules to be used in stem. making stem use anyevent doesn't achieve that aslong as it doesn't use it's anyevent backend, and it cnanot be used in other programs as well, as it isn't event-loop agnostic because it forces the user to use "stem" as the event loop. Also, this would make it impossible to benchmark the pure perl event loop - I would *prepdict* (but i am bad at that) that it will be the slowest one, ignoring POE, which I expect to be much slower still. > ML> Especially in the important case of many handles/few active ones, an approach > ML> of "scan all handles" is very slow. > > my code doesn't scan all handle, but scan all writeable events to see if > their handle bits are set. That's even worse, as there can be many more events (watchers?) then file descriptors (in the first benchmark you would scan, say, 10000 event watchers for the same fd). even if not, it's slow O(n), compared to fast O(n) that you can achieve with scanning the bitmasks. It is only fast if you have a lot of handles and very few active watchers, which isn't too typical (who uses all those extra fd's if not your program?). In any case, the "scan mask" approach is about many times faster in the actual benchmark with 10000 handles and 100 active ones, simply because scanning a mask is so much faster. Besides, select returns the number of active handles, so one could use both approachs and select between them, e.g. when more than 80% of the handles are active, use the scan-all-handles method, otherwise the bitmask method. > descriptors increment and can cause bloat of the array if you have many > in use (and not all of them in the event loop). That is not a common case, besides, arrays are very compact, unlike hashes, so it's not a clear win (note how the pure perl backend in anyevent comes out as one of the backends using the leats amount of memory). In any case, a lot of technology in the kernel goes into providing "small integers" as fd's, not taking advantage of that gives away optimisaiton opportunities. In this case, unix guarantees that the memory use is bounded (there is even a resource limit for it, and reserving 4 or 8 bytes/file descriptor is nothing really). Trying to avoid bloat on that side is the wrogn side to optimise for. > event loops are async in that you get callbacks as they are needed. sure Yes, but the I/O isn't async, which was my point. asynchronous I/O is quite a different beast, but few people really use it. (which is a pity, but only pelr ahs a IMnsHO decent module for it). > api are what matters. a better term may be non-blocking i/o (and socket It's actually the only correct term, as no I/O is done in the event loop. > ML> Also, this does not explain at all why Event is so slow, and why Glib scales > ML> so extremely badly. Most of the stuff that slows down the perl-based event > ML> loop is clearly stuff that is much much faster in C. > > poor memory management in the c code? Perl's memory management is quite good, yeah. I do suspect that it has somethign to do with glib scanning its watcher list (ALL of them) rep