develooper Front page | perl.perl5.porters | Postings from July 2019

Re: "Damaged tar archive" for perl-5.32.2 on ftp callsfromCPANmirrors

Thread Previous | Thread Next
From:
Deven T. Corzine
Date:
July 26, 2019 01:12
Subject:
Re: "Damaged tar archive" for perl-5.32.2 on ftp callsfromCPANmirrors
Message ID:
CAFVdu0S3jVjURx51GAhSjSzExXg3Hi9=vvp0bzhhwT8cpn7oWQ@mail.gmail.com
On Thu, Jul 25, 2019 at 9:18 AM sisyphus <sisyphus359@gmail.com> wrote:

>
>
> On Thu, Jul 25, 2019 at 4:22 PM Deven T. Corzine <deven@ties.org> wrote:
> > Tony's advice is exactly the right answer here.  Enabling binary mode is
> more than a good suggestion -- it's an essential bug fix, with virtually
> zero chance of breaking anything.  This change should fix the original
> issue.
>
> But it doesn't.
> Using Net::FTP to download a perl source tarball from mirrors.rit.edu
> delivers a corrupt file:
>
> $ perl ftp.pl mirrors.rit.edu 5.31.1 1
> Net::FTP>>> Net::FTP(3.11)
> Net::FTP>>>   Exporter(5.74)
> Net::FTP>>>   Net::Cmd(3.11)
> Net::FTP>>>   IO::Socket::IP(0.39)
> Net::FTP>>>     IO::Socket(1.41)
> Net::FTP>>>       IO::Handle(1.41)
> Net::FTP=GLOB(0x801524150)<<< 220 Welcome to mirrors.rit.edu.
> Net::FTP=GLOB(0x801524150)>>> TYPE I
> Net::FTP=GLOB(0x801524150)<<< 530 Please login with USER and PASS.
> Net::FTP=GLOB(0x801524150)>>> USER anonymous
> Net::FTP=GLOB(0x801524150)<<< 331 Please specify the password.
> Net::FTP=GLOB(0x801524150)>>> PASS ....
> Net::FTP=GLOB(0x801524150)<<< 230 Login successful.
> Net::FTP=GLOB(0x801524150)>>> CWD pub
> Net::FTP=GLOB(0x801524150)<<< 250 Directory successfully changed.
> Net::FTP=GLOB(0x801524150)>>> CWD CPAN/src/5.0
> Net::FTP=GLOB(0x801524150)<<< 250 Directory successfully changed.
> Net::FTP=GLOB(0x801524150)>>> PASV
> Net::FTP=GLOB(0x801524150)<<< 227 Entering Passive Mode
> (129,21,171,72,78,5).
> Net::FTP=GLOB(0x801524150)>>> RETR perl-5.31.1.tar.gz
> Net::FTP=GLOB(0x801524150)<<< 150 Opening BINARY mode data connection for
> perl-5.31.1.tar.gz (17593158 bytes).
> Net::FTP=GLOB(0x801524150)<<< 226 Transfer complete.
> Net::FTP=GLOB(0x801524150)>>> QUIT
> Net::FTP=GLOB(0x801524150)<<< 221 Goodbye.
> $ sha1 perl-5.31.1.tar.gz
> SHA1 (perl-5.31.1.tar.gz) = f8edd1080d69bf9be656d384ec772c7a398fb502
>
> Using the same script to download the same perl source from ftp.funet.fi
> works fine:
>
> $ perl ftp.pl ftp.funet.fi 5.31.1 1
> Net::FTP>>> Net::FTP(3.11)
> Net::FTP>>>   Exporter(5.74)
> Net::FTP>>>   Net::Cmd(3.11)
> Net::FTP>>>   IO::Socket::IP(0.39)
> Net::FTP>>>     IO::Socket(1.41)
> Net::FTP>>>       IO::Handle(1.41)
> Net::FTP=GLOB(0x80151d150)<<< 220---------- Welcome to Pure-FTPd [privsep]
> ----------
> Net::FTP=GLOB(0x80151d150)<<< 220-You are user number 15 of 1000 allowed.
> Net::FTP=GLOB(0x80151d150)<<< 220-Local time is now 16:03. Server port: 21.
> Net::FTP=GLOB(0x80151d150)<<< 220-Only anonymous FTP is allowed here
> Net::FTP=GLOB(0x80151d150)<<< 220-IPv6 connections are also welcome on
> this server.
> Net::FTP=GLOB(0x80151d150)<<< 220 You will be disconnected after 30
> minutes of inactivity.
> Net::FTP=GLOB(0x80151d150)>>> TYPE I
> Net::FTP=GLOB(0x80151d150)<<< 200 TYPE is now 8-bit binary
> Net::FTP=GLOB(0x80151d150)>>> USER anonymous
> Net::FTP=GLOB(0x80151d150)<<< 331 Any password will work
> Net::FTP=GLOB(0x80151d150)>>> PASS ....
> Net::FTP=GLOB(0x80151d150)<<< 230 Any password will work
> Net::FTP=GLOB(0x80151d150)>>> CWD pub
> Net::FTP=GLOB(0x80151d150)<<< 250 OK. Current directory is /pub
> Net::FTP=GLOB(0x80151d150)>>> CWD languages/perl/CPAN/src/5.0
> Net::FTP=GLOB(0x80151d150)<<< 250 OK. Current directory is /.m/mirrors/
> ftp.cpan.org/pub/CPAN/src/5.0
> Net::FTP=GLOB(0x80151d150)>>> PASV
> Net::FTP=GLOB(0x80151d150)<<< 227 Entering Passive Mode
> (193,166,3,21,142,168)
> Net::FTP=GLOB(0x80151d150)>>> RETR perl-5.31.1.tar.gz
> Net::FTP=GLOB(0x80151d150)<<< 150-Accepted data connection
> Net::FTP=GLOB(0x80151d150)<<< 150 17180.8 kbytes to download
> Net::FTP=GLOB(0x80151d150)<<< 226-File successfully transferred
> Net::FTP=GLOB(0x80151d150)<<< 226 172.328 seconds (measured here), 99.70
> Kbytes per second
> Net::FTP=GLOB(0x80151d150)>>> QUIT
> Net::FTP=GLOB(0x80151d150)<<< 221-Goodbye. You uploaded 0 and downloaded
> 17181 kbytes.
> Net::FTP=GLOB(0x80151d150)<<< 221 Logout.
> $ sha1 perl-5.31.1.tar.gz
> SHA1 (perl-5.31.1.tar.gz) = c68cbdaf18a73765421aa1bf1883691551c1aa81
>
> The fact that the system ftp retrieves a valid perl-5.31.1.tar.gz from
> mirrors.rit.edu would suggest that the problem lies with Net::FTP.
> However, I don't regard this as proven. (Perhaps the mirrors.rit.edu
> server is doing something it shouldn't ?)
>
> Cheers,
> Rob
>

Net::FTP isn't the problem.  ASCII mode is the problem, as I explained in
great detail in my previous email.

Using binary mode, I have no problem downloading a good copy of
perl-5.31.1-tar.gz from mirrors.rit.edu with Net::FTP:

$ rm -f perl-5.31.1.tar.gz; perl -e 'use Net::FTP; $ftp = Net::FTP->new("
> mirrors.rit.edu", Debug => 1); $ftp->login("anonymous"); $ftp->binary;
> $ftp->cwd("/CPAN/authors/id/E/ET/ETHER"); $ftp->get("perl-5.31.1.tar.gz");
> $ftp->quit;'; ls -l perl-5.31.1.tar.gz; sha -1 perl-5.31.1.tar.gz; gzip -tv
> perl-5.31.1.tar.gz
> Net::FTP>>> Net::FTP(3.11)
> Net::FTP>>>   Exporter(5.70)
> Net::FTP>>>   Net::Cmd(3.11)
> Net::FTP>>>   IO::Socket::IP(0.30)
> Net::FTP>>>     IO::Socket(1.36)
> Net::FTP>>>       IO::Handle(1.34)
> Net::FTP=GLOB(0x114cfd0)<<< 220 Welcome to mirrors.rit.edu.
> Net::FTP=GLOB(0x114cfd0)>>> USER anonymous
> Net::FTP=GLOB(0x114cfd0)<<< 331 Please specify the password.
> Net::FTP=GLOB(0x114cfd0)>>> PASS ....
> Net::FTP=GLOB(0x114cfd0)<<< 230 Login successful.
> Net::FTP=GLOB(0x114cfd0)>>> TYPE I
> Net::FTP=GLOB(0x114cfd0)<<< 200 Switching to Binary mode.
> Net::FTP=GLOB(0x114cfd0)>>> CWD /CPAN/authors/id/E/ET/ETHER
> Net::FTP=GLOB(0x114cfd0)<<< 250 Directory successfully changed.
> Net::FTP=GLOB(0x114cfd0)>>> PASV
> Net::FTP=GLOB(0x114cfd0)<<< 227 Entering Passive Mode (129,21,171,72,78,6).
> Net::FTP=GLOB(0x114cfd0)>>> RETR perl-5.31.1.tar.gz
> Net::FTP=GLOB(0x114cfd0)<<< 150 Opening BINARY mode data connection for
> perl-5.31.1.tar.gz (17593158 bytes).
> Net::FTP=GLOB(0x114cfd0)<<< 226 Transfer complete.
> Net::FTP=GLOB(0x114cfd0)>>> QUIT
> Net::FTP=GLOB(0x114cfd0)<<< 221 Goodbye.
> -rw-r--r--. 1 deven deven 17593158 Jul 25 20:38 perl-5.31.1.tar.gz
> c68cbdaf18a73765421aa1bf1883691551c1aa81 perl-5.31.1.tar.gz
> perl-5.31.1.tar.gz:      OK
> $


On the other hand, if I let it use the default ASCII mode, it FAILS.  This
time I get the same corrupted file that you got:

$ rm -f perl-5.31.1.tar.gz; perl -e 'use Net::FTP; $ftp = Net::FTP->new("
> mirrors.rit.edu", Debug => 1); $ftp->login("anonymous");
> $ftp->cwd("/CPAN/authors/id/E/ET/ETHER"); $ftp->get("perl-5.31.1.tar.gz");
> $ftp->quit;'; ls -l perl-5.31.1.tar.gz; sha -1 perl-5.31.1.tar.gz; gzip -tv
> perl-5.31.1.tar.gz
> Net::FTP>>> Net::FTP(3.11)
> Net::FTP>>>   Exporter(5.70)
> Net::FTP>>>   Net::Cmd(3.11)
> Net::FTP>>>   IO::Socket::IP(0.30)
> Net::FTP>>>     IO::Socket(1.36)
> Net::FTP>>>       IO::Handle(1.34)
> Net::FTP=GLOB(0x1251ed0)<<< 220 Welcome to mirrors.rit.edu.
> Net::FTP=GLOB(0x1251ed0)>>> USER anonymous
> Net::FTP=GLOB(0x1251ed0)<<< 331 Please specify the password.
> Net::FTP=GLOB(0x1251ed0)>>> PASS ....
> Net::FTP=GLOB(0x1251ed0)<<< 230 Login successful.
> Net::FTP=GLOB(0x1251ed0)>>> CWD /CPAN/authors/id/E/ET/ETHER
> Net::FTP=GLOB(0x1251ed0)<<< 250 Directory successfully changed.
> Net::FTP=GLOB(0x1251ed0)>>> PASV
> Net::FTP=GLOB(0x1251ed0)<<< 227 Entering Passive Mode
> (129,21,171,72,78,16).
> Net::FTP=GLOB(0x1251ed0)>>> RETR perl-5.31.1.tar.gz
> Net::FTP=GLOB(0x1251ed0)<<< 150 Opening BINARY mode data connection for
> perl-5.31.1.tar.gz (17593158 bytes).
> Net::FTP=GLOB(0x1251ed0)<<< 226 Transfer complete.
> Net::FTP=GLOB(0x1251ed0)>>> QUIT
> Net::FTP=GLOB(0x1251ed0)<<< 221 Goodbye.
> -rw-r--r--. 1 deven deven 17592906 Jul 25 20:53 perl-5.31.1.tar.gz
> f8edd1080d69bf9be656d384ec772c7a398fb502 perl-5.31.1.tar.gz
> perl-5.31.1.tar.gz:
> gzip: perl-5.31.1.tar.gz: invalid compressed data--format violated
> $


So, why did I get a good download from mirrors.rit.edu while you got a
corrupted one?  Based on your Net::FTP debugging output, the problem seems
obvious to me.  While your script does appear to be calling
Net::FTP::binary() to enable binary mode, it is apparently doing so BEFORE
logging into the FTP server, which is very unusual.  (I've never seen any
FTP client do that.)  Did you notice the response from the server to that
command?  "530 Please login with USER and PASS."  That's an error response,
which means the "TYPE I" command FAILED.  As a result, your FTP connection
remained in ASCII mode despite the attempt to switch to binary mode.
When I did it AFTER logging into the FTP server, it worked fine and
responded with "200 Switching to Binary mode."

Apparently, the Pure-FTPd software on ftp.funet.fi is more forgiving,
accepting the premature "TYPE I" without being logged in yet.  It returned
"200 TYPE is now 8-bit binary", so binary mode was successfully enabled.
This is a good example of how implementation-specific details can cause
unexpected results.

Regardless, the proper thing to do is to login to the FTP server FIRST.  If
you change your "ftp.pl" script to call $ftp->binary() AFTER logging in,
you should be able to use it with mirrors.rit.edu without getting corrupted
downloads.

In all of these examples, transfers done in binary mode always worked --
EVERY failing transfer was done in ASCII mode.  That isn't a coincidence;
it's the root cause of these corrupted downloads.

Deven


On Thu, Jul 25, 2019 at 9:18 AM sisyphus <sisyphus359@gmail.com> wrote:

>
>
> On Thu, Jul 25, 2019 at 4:22 PM Deven T. Corzine <deven@ties.org> wrote:
> > Tony's advice is exactly the right answer here.  Enabling binary mode is
> more than a good suggestion -- it's an essential bug fix, with virtually
> zero chance of breaking anything.  This change should fix the original
> issue.
>
> But it doesn't.
> Using Net::FTP to download a perl source tarball from mirrors.rit.edu
> delivers a corrupt file:
>
> $ perl ftp.pl mirrors.rit.edu 5.31.1 1
> Net::FTP>>> Net::FTP(3.11)
> Net::FTP>>>   Exporter(5.74)
> Net::FTP>>>   Net::Cmd(3.11)
> Net::FTP>>>   IO::Socket::IP(0.39)
> Net::FTP>>>     IO::Socket(1.41)
> Net::FTP>>>       IO::Handle(1.41)
> Net::FTP=GLOB(0x801524150)<<< 220 Welcome to mirrors.rit.edu.
> Net::FTP=GLOB(0x801524150)>>> TYPE I
> Net::FTP=GLOB(0x801524150)<<< 530 Please login with USER and PASS.
> Net::FTP=GLOB(0x801524150)>>> USER anonymous
> Net::FTP=GLOB(0x801524150)<<< 331 Please specify the password.
> Net::FTP=GLOB(0x801524150)>>> PASS ....
> Net::FTP=GLOB(0x801524150)<<< 230 Login successful.
> Net::FTP=GLOB(0x801524150)>>> CWD pub
> Net::FTP=GLOB(0x801524150)<<< 250 Directory successfully changed.
> Net::FTP=GLOB(0x801524150)>>> CWD CPAN/src/5.0
> Net::FTP=GLOB(0x801524150)<<< 250 Directory successfully changed.
> Net::FTP=GLOB(0x801524150)>>> PASV
> Net::FTP=GLOB(0x801524150)<<< 227 Entering Passive Mode
> (129,21,171,72,78,5).
> Net::FTP=GLOB(0x801524150)>>> RETR perl-5.31.1.tar.gz
> Net::FTP=GLOB(0x801524150)<<< 150 Opening BINARY mode data connection for
> perl-5.31.1.tar.gz (17593158 bytes).
> Net::FTP=GLOB(0x801524150)<<< 226 Transfer complete.
> Net::FTP=GLOB(0x801524150)>>> QUIT
> Net::FTP=GLOB(0x801524150)<<< 221 Goodbye.
> $ sha1 perl-5.31.1.tar.gz
> SHA1 (perl-5.31.1.tar.gz) = f8edd1080d69bf9be656d384ec772c7a398fb502
>
> Using the same script to download the same perl source from ftp.funet.fi
> works fine:
>
> $ perl ftp.pl ftp.funet.fi 5.31.1 1
> Net::FTP>>> Net::FTP(3.11)
> Net::FTP>>>   Exporter(5.74)
> Net::FTP>>>   Net::Cmd(3.11)
> Net::FTP>>>   IO::Socket::IP(0.39)
> Net::FTP>>>     IO::Socket(1.41)
> Net::FTP>>>       IO::Handle(1.41)
> Net::FTP=GLOB(0x80151d150)<<< 220---------- Welcome to Pure-FTPd [privsep]
> ----------
> Net::FTP=GLOB(0x80151d150)<<< 220-You are user number 15 of 1000 allowed.
> Net::FTP=GLOB(0x80151d150)<<< 220-Local time is now 16:03. Server port: 21.
> Net::FTP=GLOB(0x80151d150)<<< 220-Only anonymous FTP is allowed here
> Net::FTP=GLOB(0x80151d150)<<< 220-IPv6 connections are also welcome on
> this server.
> Net::FTP=GLOB(0x80151d150)<<< 220 You will be disconnected after 30
> minutes of inactivity.
> Net::FTP=GLOB(0x80151d150)>>> TYPE I
> Net::FTP=GLOB(0x80151d150)<<< 200 TYPE is now 8-bit binary
> Net::FTP=GLOB(0x80151d150)>>> USER anonymous
> Net::FTP=GLOB(0x80151d150)<<< 331 Any password will work
> Net::FTP=GLOB(0x80151d150)>>> PASS ....
> Net::FTP=GLOB(0x80151d150)<<< 230 Any password will work
> Net::FTP=GLOB(0x80151d150)>>> CWD pub
> Net::FTP=GLOB(0x80151d150)<<< 250 OK. Current directory is /pub
> Net::FTP=GLOB(0x80151d150)>>> CWD languages/perl/CPAN/src/5.0
> Net::FTP=GLOB(0x80151d150)<<< 250 OK. Current directory is /.m/mirrors/
> ftp.cpan.org/pub/CPAN/src/5.0
> Net::FTP=GLOB(0x80151d150)>>> PASV
> Net::FTP=GLOB(0x80151d150)<<< 227 Entering Passive Mode
> (193,166,3,21,142,168)
> Net::FTP=GLOB(0x80151d150)>>> RETR perl-5.31.1.tar.gz
> Net::FTP=GLOB(0x80151d150)<<< 150-Accepted data connection
> Net::FTP=GLOB(0x80151d150)<<< 150 17180.8 kbytes to download
> Net::FTP=GLOB(0x80151d150)<<< 226-File successfully transferred
> Net::FTP=GLOB(0x80151d150)<<< 226 172.328 seconds (measured here), 99.70
> Kbytes per second
> Net::FTP=GLOB(0x80151d150)>>> QUIT
> Net::FTP=GLOB(0x80151d150)<<< 221-Goodbye. You uploaded 0 and downloaded
> 17181 kbytes.
> Net::FTP=GLOB(0x80151d150)<<< 221 Logout.
> $ sha1 perl-5.31.1.tar.gz
> SHA1 (perl-5.31.1.tar.gz) = c68cbdaf18a73765421aa1bf1883691551c1aa81
>
> The fact that the system ftp retrieves a valid perl-5.31.1.tar.gz from
> mirrors.rit.edu would suggest that the problem lies with Net::FTP.
> However, I don't regard this as proven. (Perhaps the mirrors.rit.edu
> server is doing something it shouldn't ?)
>
> Cheers,
> Rob
>
>
>

Thread Previous | Thread Next


nntp.perl.org: Perl Programming lists via nntp and http.
Comments to Ask Bjørn Hansen at ask@perl.org | Group listing | About