develooper Front page | perl.perl5.porters | Postings from July 2012

Re: [PATCH] Module::CoreList delta support

Thread Previous | Thread Next
From:
David Leadbeater
Date:
July 29, 2012 13:07
Subject:
Re: [PATCH] Module::CoreList delta support
Message ID:
CAP9KPhCo4FHCOvX39Vvv-1y2tbV=8bVn7A30wzkE1Lxaj81BFA@mail.gmail.com
Ah, I didn't come across that repo, interesting.

On 29 July 2012 18:07, David Golden <xdaveg@gmail.com> wrote:

> On Sun, Jul 29, 2012 at 11:08 AM, Chris 'BinGOs' Williams
> <chris@bingosnet.co.uk> wrote:
> > All the data is compressed and uuencoded.
> >
> > This does have an affect on performance.
>
> The real win might be to get to a format that can be searched on
> demand so we don't load the whole thing into memory just to check one
> module.
>

Yes, that would be nice. However I do wonder how useful it is, that seems
like a lot of work and also needs something to generate that format. I was
trying to keep my changes fairly minimal. (I agree with Leon that it would
be nice to deprecate the access as a hash style, as then it would be easier
for a clean API to just load the data it needs.)

How important is performance though?

I notice that my patched version is actually a tad faster for simple
lookups (because presumably perl has less lines to parse despite doing more
work after that), although I was mostly aiming for some size reduction. I
also notice that gzip compressing the delta file results in a slightly
smaller file than the compressed one from Module-Snorelist (even after
compressing the uuencoded data for another level of compression).

As an experiment I just tried generating something approximating the JSON
format you mentioned (without deltas) and that alone is nearly 700K of
data. The other interesting thing is as the data decreases in size it
reaches a point where it won't compress as well, and many things where size
matter are already using some form of compression.

David

Thread Previous | Thread Next


nntp.perl.org: Perl Programming lists via nntp and http.
Comments to Ask Bjørn Hansen at ask@perl.org | Group listing | About