develooper Front page | perl.libwww | Postings from December 2000

Re: MomSpider & Libwww-Perl libs

Thread Previous
Bill Moseley
December 23, 2000 13:12
Re: MomSpider & Libwww-Perl libs
Message ID:
At 03:07 PM 12/22/00 -0800, Steve Magee wrote:
>I inherited MomSpider several months ago.  I am trying to understand
>how my predecessor installed MomSpider.

I'm not able to offer much help, but I have to comment that about a year
ago I was working on a site that used MomSpider and it was causing big
trouble.  It would hang and take hours (sometimes days) to complete a run.
Maybe they were running an old version, I don't know, but it was not a
program that I would recommend.

So I wrote my own spider called Detect all Dead links Spider -- DadSpider
(sorry for the bad name).  It's been a year since I looked at it and I'm
sure the code could be cleand up.  It's nothing fantastic, but seems fast
enough for a small site:

   Processed 7730 URLs from 6585 files in 26 minutes, 52 seconds.
             Found 333 (4%) URL errors in 320 files.

Not lightning fast, but good enough for our use.

I might be able to make it available, but it wouldn't be for a week or so.

My real point is that Momspider code looked so bad and ran so poorly that I
decided to write my own.  It was relatively trivial to write and didn't
take that long, so I'd recommend writing your own over using the Momspider
program -- if it's the same version of momspider that I was looking at.
LWP makes it all possible.

Bill Moseley

Thread Previous Perl Programming lists via nntp and http.
Comments to Ask Bjørn Hansen at | Group listing | About