develooper Front page | perl.module-authors | Postings from November 2010

Reducing rsync cost (was: Re: Using a better compression than .gzfor one's CPAN modules)

Thread Previous | Thread Next
From:
David Landgren
Date:
November 22, 2010 02:37
Subject:
Reducing rsync cost (was: Re: Using a better compression than .gzfor one's CPAN modules)
Message ID:
4CEA47E0.2000705@landgren.net
On 19/11/2010 20:57, dhudes@hudes.org wrote:
>>> source code, even 100KLOC? Once you go to .gz you're already at better
>>> than 2:1. What are you going to save by going to even 3:1, 10Kbytes?
>>> compared to the nuisance inflicted, it's nothing.
>>
>> Over the entire CPAN archive, it'd be significant...
>>
>> I agree on the individual case it's probably not worth worrying about too
>> much.  But if it's easy to use .bz2 or something better it wouldn't hurt
>> to get that word out.  (And it may be worth making it easy, though I'm not
>> sure about that.)
>>
>> Daniel T. Staal
>>
> Disk space is cheap. Bandwidth is cheap. What's rough is the rsync between
> mirrors. Compressing to .bz2 won't help that: the stress is doing a stat
> on every single file in CPAN not the transfer. Work toward optimizing the
> mirror distribution instead of worrying about bz2 vs gz.  Remember not

Yeah, this is the killer. In an ideal world, we would kill the symlinks 
such as authors/id/*, modules/by-category/*, modules/by-module/* and so 
on. These could be recreated via shell scripts locally on mirrors for 
people who wish to maintain these legacies. Cutting that out would 
diminish the rsync burden considerably.

David

-- 
There's bum trash in my hall and my place is ripped
I've totaled another amp, I'm calling in sick

Thread Previous | Thread Next


nntp.perl.org: Perl Programming lists via nntp and http.
Comments to Ask Bjørn Hansen at ask@perl.org | Group listing | About