develooper Front page | perl.libwww | Postings from October 2019

Re: retrieving date+time stamp of remote object

Thread Previous
From:
Bradley Dean
Date:
October 23, 2019 23:26
Subject:
Re: retrieving date+time stamp of remote object
Message ID:
ad661de1-c3c7-42d8-b24d-ff95db34e20f@www.fastmail.com
On Thu, 24 Oct 2019 07:30 +11:00, L A Walsh <astara@tlinx.org> wrote:
> On 2019/10/21 17:39, Bradley Dean wrote:
> >
> > Using if-modified-since relies on the server you're querying supporting that
> ----
> It "seems" it must support it separately, "I guess"...because (see below)...

From perldoc LWP::UserAgent (see https://metacpan.org/pod/LWP::UserAgent#mirror)

    $ua->mirror( $url, $filename )
        This method will get the document identified by $url and store it in
        file called $filename. If the file already exists, then the request
        will contain an "If-Modified-Since" header matching the modification
        time of the file. If the document on the server has not changed
        since this time, then nothing happens. If the document has been
        updated, it will be downloaded again. The modification time of the
        file will be forced to match that of the server.

        The return value is the the response object.

Which is using the If-Modified-Since header (when the local file exists) and the server makes the decision (whether or not the Last-Modified header is available or making it through a proxy to you). mirror accepts a HTTP 304 Not Modified and makes no changes.

> >> Anyone know the call/header value to use for getting at the remote
> >> object's last-modified time directly?

It will be in the response if it's supported by the server (and making it through proxies between you and the server).

>     One thing I was curious about -- when I call mirror (vs. get), I get
> back
> an HTTP::Response object.  One of the fields of that object is the
> 'content' value.  I'm wondering if that is read in dynamically if
> 'accessed', since I'm
> pulling down RPM packages which can be rather large (the 10 largest packages
> are over 1GB each!).  It certainly wouldn't just store them in the response
> object, by default (would it?), since if I had references to several objects
> at the same time, in memory, that could be a huge memory load.
> 
>     I can see I'm gonna want to find a way to filter some of these packages
> before I mirror them since I don't really need the debuginfo for
> multiple kernel
> versions I'm not running nor various cross-compiler packages (Oi!)
> 
>    Whether dynamic or not, guess I'll find out -- can't believe how huge
> some of those packages are!

I've not checked specifically - but I believe mirror uses a content_file so the HTTP::Response will not have the content in memory. See content file details in https://metacpan.org/pod/LWP::UserAgent#get , https://metacpan.org/pod/LWP::UserAgent#request

Cheerio,

 Brad

-- 
Bradley Dean
Email: bjdean@bjdean.id.au
WWW: http://bjdean.id.au/

Thread Previous


nntp.perl.org: Perl Programming lists via nntp and http.
Comments to Ask Bjørn Hansen at ask@perl.org | Group listing | About