develooper Front page | perl.libwww | Postings from April 2003

Re: Connection: close -- and LWP::RobotUA

Thread Previous
From:
Bill Moseley
Date:
April 13, 2003 18:59
Subject:
Re: Connection: close -- and LWP::RobotUA
Message ID:
Pine.LNX.4.10.10304131826180.18840-100000@mardy.hank.org
On Sun, 13 Apr 2003, Sean M. Burke wrote:

> At 11:01 PM 2003-04-12 -0700, Bill Moseley wrote:
> >I use LWP::RobotUA and use a delay.  But I'm thinking about setting the 
> >delay to zero for the *next* request when using keep-alives and not seeing 
> >the Connection: close response header.  Why delay if the server is waiting 
> >for your next request?
> 
> I'm not intimately familiar with server internals, but I can imagine that 
> the server waiting doesn't necessarily consume a lot of resources, compared 
> to the case where you just hammer it.

On the other hand, if you are delaying while holding a connection open
then you are keeping a server process busy (doing nothing) when it could
be off serving some other requests.  That might mean the server needs to
fork other processes to make up for the idle process.

Surely with a keep alive connection it's probably more load to hammer it
than to delay, but it's also likely much worse to delay longer than the
server's keep alive timeout (15 seconds is the default with Apache).  A
server can handle a lot of other static requests in 15 seconds.
LWP::RobotUA's 60 second default is probably a bit long when using
keep-alive connections.

Since it's hard to know if delaying a request is helpful or not with
keep-alives, my thinking would be if the server has a connection open
ready for a request, well, might as well send the request.

I guess for tuning a spider it would be good to have a two delays, one for
between Connection: close requests, and one to set between keep-alive
requests (using Time::HiRes for < second times) -- and maybe even a max
number of keep-alive requests to send.


-- 
Bill Moseley moseley@hank.org


Thread Previous


nntp.perl.org: Perl Programming lists via nntp and http.
Comments to Ask Bjørn Hansen at ask@perl.org | Group listing | About