develooper Front page | perl.libwww | Postings from April 2003

Re: $ua->parse_head and gzip encoding

Thread Previous | Thread Next
Bill Moseley
April 16, 2003 08:16
Re: $ua->parse_head and gzip encoding
Message ID:
On Wed, 16 Apr 2003, Mike Simons wrote:

> - Why don't you want the header parse done on a compressed document?

LWP::Protocol only checks for text/html but not encoding so it attempts to
parse with HTML::HeadParser encoded content.  I think that's what I saw...

>   I posted a patch to transparently request and decompress (block by
> block) gzip style documents a little while ago.  The user who creates
> a UserAgent can request the transparent decompression with a option
> like "WantCompression => 1".  That code also supports block by block 
> decompression of the data as it comes in, instead of handling the
> whole decompression in memory.  I have code that handles deflate style
> data streams as well but have not made a patch for it.

That's good.  My quick code doesn't really work for me, though, because I
do use a UserAgent callback function.  It would be good to have the chunk
of content passed to the callback uncompressed as well.

Do you remember the title or URL for the archived thread?  Maybe I could
use that to improve my code -- I'd rather decode in my UserAgent callback
function in chunks so the content is available.

I agree that it should be configurable.  Sometimes you will want the
encoded form returned (but knowing the the head was not parsed).  I'd also
like a way to know if the running version of LWP can decode, although I
suppose seeing a Content-Encoding header would tell my code that it needed
to do the decoding myself.

Bill Moseley

Thread Previous | Thread Next Perl Programming lists via nntp and http.
Comments to Ask Bjørn Hansen at | Group listing | About