develooper Front page | perl.perl5.porters | Postings from July 2019

Re: "Damaged tar archive" for perl-5.32.2 on ftp callsfromCPANmirrors

Thread Previous | Thread Next
From:
sisyphus
Date:
July 26, 2019 01:56
Subject:
Re: "Damaged tar archive" for perl-5.32.2 on ftp callsfromCPANmirrors
Message ID:
CADZSBj0Od22L1678_1HfpEyeG5stF8vs-t1mFn9PX29n7QRAmg@mail.gmail.com
Hi Deven,

Yes, I think I would have arrived more quickly at a very similar view if:
a) the mirrors.rit.edu server had not claimed to be sending the file in
binary mode;
or
b) I had explicitly checked $ftp->binary() for failure.
or
c) I had recognized "530 Please login with USER and PASS" as being an error
message.

I came to wonder whether the mirrors.rit.edu server might have been falling
short on its obligations by insisting that the switch to binary mode be
done after logging in.
But you're saying that, conversely, it's the ftp.funet.fi
server that's going above and beyond its call of duty by accepting a switch
to binary mode prior to the login.
I'll happily take your word on that.

Cheers,
Rob

On Fri, Jul 26, 2019 at 11:12 AM Deven T. Corzine <deven@ties.org> wrote:

> On Thu, Jul 25, 2019 at 9:18 AM sisyphus <sisyphus359@gmail.com> wrote:
>
>>
>>
>> On Thu, Jul 25, 2019 at 4:22 PM Deven T. Corzine <deven@ties.org> wrote:
>> > Tony's advice is exactly the right answer here.  Enabling binary mode
>> is more than a good suggestion -- it's an essential bug fix, with virtually
>> zero chance of breaking anything.  This change should fix the original
>> issue.
>>
>> But it doesn't.
>> Using Net::FTP to download a perl source tarball from mirrors.rit.edu
>> delivers a corrupt file:
>>
>> $ perl ftp.pl mirrors.rit.edu 5.31.1 1
>> Net::FTP>>> Net::FTP(3.11)
>> Net::FTP>>>   Exporter(5.74)
>> Net::FTP>>>   Net::Cmd(3.11)
>> Net::FTP>>>   IO::Socket::IP(0.39)
>> Net::FTP>>>     IO::Socket(1.41)
>> Net::FTP>>>       IO::Handle(1.41)
>> Net::FTP=GLOB(0x801524150)<<< 220 Welcome to mirrors.rit.edu.
>> Net::FTP=GLOB(0x801524150)>>> TYPE I
>> Net::FTP=GLOB(0x801524150)<<< 530 Please login with USER and PASS.
>> Net::FTP=GLOB(0x801524150)>>> USER anonymous
>> Net::FTP=GLOB(0x801524150)<<< 331 Please specify the password.
>> Net::FTP=GLOB(0x801524150)>>> PASS ....
>> Net::FTP=GLOB(0x801524150)<<< 230 Login successful.
>> Net::FTP=GLOB(0x801524150)>>> CWD pub
>> Net::FTP=GLOB(0x801524150)<<< 250 Directory successfully changed.
>> Net::FTP=GLOB(0x801524150)>>> CWD CPAN/src/5.0
>> Net::FTP=GLOB(0x801524150)<<< 250 Directory successfully changed.
>> Net::FTP=GLOB(0x801524150)>>> PASV
>> Net::FTP=GLOB(0x801524150)<<< 227 Entering Passive Mode
>> (129,21,171,72,78,5).
>> Net::FTP=GLOB(0x801524150)>>> RETR perl-5.31.1.tar.gz
>> Net::FTP=GLOB(0x801524150)<<< 150 Opening BINARY mode data connection for
>> perl-5.31.1.tar.gz (17593158 bytes).
>> Net::FTP=GLOB(0x801524150)<<< 226 Transfer complete.
>> Net::FTP=GLOB(0x801524150)>>> QUIT
>> Net::FTP=GLOB(0x801524150)<<< 221 Goodbye.
>> $ sha1 perl-5.31.1.tar.gz
>> SHA1 (perl-5.31.1.tar.gz) = f8edd1080d69bf9be656d384ec772c7a398fb502
>>
>> Using the same script to download the same perl source from ftp.funet.fi
>> works fine:
>>
>> $ perl ftp.pl ftp.funet.fi 5.31.1 1
>> Net::FTP>>> Net::FTP(3.11)
>> Net::FTP>>>   Exporter(5.74)
>> Net::FTP>>>   Net::Cmd(3.11)
>> Net::FTP>>>   IO::Socket::IP(0.39)
>> Net::FTP>>>     IO::Socket(1.41)
>> Net::FTP>>>       IO::Handle(1.41)
>> Net::FTP=GLOB(0x80151d150)<<< 220---------- Welcome to Pure-FTPd
>> [privsep] ----------
>> Net::FTP=GLOB(0x80151d150)<<< 220-You are user number 15 of 1000 allowed.
>> Net::FTP=GLOB(0x80151d150)<<< 220-Local time is now 16:03. Server port:
>> 21.
>> Net::FTP=GLOB(0x80151d150)<<< 220-Only anonymous FTP is allowed here
>> Net::FTP=GLOB(0x80151d150)<<< 220-IPv6 connections are also welcome on
>> this server.
>> Net::FTP=GLOB(0x80151d150)<<< 220 You will be disconnected after 30
>> minutes of inactivity.
>> Net::FTP=GLOB(0x80151d150)>>> TYPE I
>> Net::FTP=GLOB(0x80151d150)<<< 200 TYPE is now 8-bit binary
>> Net::FTP=GLOB(0x80151d150)>>> USER anonymous
>> Net::FTP=GLOB(0x80151d150)<<< 331 Any password will work
>> Net::FTP=GLOB(0x80151d150)>>> PASS ....
>> Net::FTP=GLOB(0x80151d150)<<< 230 Any password will work
>> Net::FTP=GLOB(0x80151d150)>>> CWD pub
>> Net::FTP=GLOB(0x80151d150)<<< 250 OK. Current directory is /pub
>> Net::FTP=GLOB(0x80151d150)>>> CWD languages/perl/CPAN/src/5.0
>> Net::FTP=GLOB(0x80151d150)<<< 250 OK. Current directory is /.m/mirrors/
>> ftp.cpan.org/pub/CPAN/src/5.0
>> Net::FTP=GLOB(0x80151d150)>>> PASV
>> Net::FTP=GLOB(0x80151d150)<<< 227 Entering Passive Mode
>> (193,166,3,21,142,168)
>> Net::FTP=GLOB(0x80151d150)>>> RETR perl-5.31.1.tar.gz
>> Net::FTP=GLOB(0x80151d150)<<< 150-Accepted data connection
>> Net::FTP=GLOB(0x80151d150)<<< 150 17180.8 kbytes to download
>> Net::FTP=GLOB(0x80151d150)<<< 226-File successfully transferred
>> Net::FTP=GLOB(0x80151d150)<<< 226 172.328 seconds (measured here), 99.70
>> Kbytes per second
>> Net::FTP=GLOB(0x80151d150)>>> QUIT
>> Net::FTP=GLOB(0x80151d150)<<< 221-Goodbye. You uploaded 0 and downloaded
>> 17181 kbytes.
>> Net::FTP=GLOB(0x80151d150)<<< 221 Logout.
>> $ sha1 perl-5.31.1.tar.gz
>> SHA1 (perl-5.31.1.tar.gz) = c68cbdaf18a73765421aa1bf1883691551c1aa81
>>
>> The fact that the system ftp retrieves a valid perl-5.31.1.tar.gz from
>> mirrors.rit.edu would suggest that the problem lies with Net::FTP.
>> However, I don't regard this as proven. (Perhaps the mirrors.rit.edu
>> server is doing something it shouldn't ?)
>>
>> Cheers,
>> Rob
>>
>
> Net::FTP isn't the problem.  ASCII mode is the problem, as I explained in
> great detail in my previous email.
>
> Using binary mode, I have no problem downloading a good copy of
> perl-5.31.1-tar.gz from mirrors.rit.edu with Net::FTP:
>
> $ rm -f perl-5.31.1.tar.gz; perl -e 'use Net::FTP; $ftp = Net::FTP->new("
>> mirrors.rit.edu", Debug => 1); $ftp->login("anonymous"); $ftp->binary;
>> $ftp->cwd("/CPAN/authors/id/E/ET/ETHER"); $ftp->get("perl-5.31.1.tar.gz");
>> $ftp->quit;'; ls -l perl-5.31.1.tar.gz; sha -1 perl-5.31.1.tar.gz; gzip -tv
>> perl-5.31.1.tar.gz
>> Net::FTP>>> Net::FTP(3.11)
>> Net::FTP>>>   Exporter(5.70)
>> Net::FTP>>>   Net::Cmd(3.11)
>> Net::FTP>>>   IO::Socket::IP(0.30)
>> Net::FTP>>>     IO::Socket(1.36)
>> Net::FTP>>>       IO::Handle(1.34)
>> Net::FTP=GLOB(0x114cfd0)<<< 220 Welcome to mirrors.rit.edu.
>> Net::FTP=GLOB(0x114cfd0)>>> USER anonymous
>> Net::FTP=GLOB(0x114cfd0)<<< 331 Please specify the password.
>> Net::FTP=GLOB(0x114cfd0)>>> PASS ....
>> Net::FTP=GLOB(0x114cfd0)<<< 230 Login successful.
>> Net::FTP=GLOB(0x114cfd0)>>> TYPE I
>> Net::FTP=GLOB(0x114cfd0)<<< 200 Switching to Binary mode.
>> Net::FTP=GLOB(0x114cfd0)>>> CWD /CPAN/authors/id/E/ET/ETHER
>> Net::FTP=GLOB(0x114cfd0)<<< 250 Directory successfully changed.
>> Net::FTP=GLOB(0x114cfd0)>>> PASV
>> Net::FTP=GLOB(0x114cfd0)<<< 227 Entering Passive Mode
>> (129,21,171,72,78,6).
>> Net::FTP=GLOB(0x114cfd0)>>> RETR perl-5.31.1.tar.gz
>> Net::FTP=GLOB(0x114cfd0)<<< 150 Opening BINARY mode data connection for
>> perl-5.31.1.tar.gz (17593158 bytes).
>> Net::FTP=GLOB(0x114cfd0)<<< 226 Transfer complete.
>> Net::FTP=GLOB(0x114cfd0)>>> QUIT
>> Net::FTP=GLOB(0x114cfd0)<<< 221 Goodbye.
>> -rw-r--r--. 1 deven deven 17593158 Jul 25 20:38 perl-5.31.1.tar.gz
>> c68cbdaf18a73765421aa1bf1883691551c1aa81 perl-5.31.1.tar.gz
>> perl-5.31.1.tar.gz:      OK
>> $
>
>
> On the other hand, if I let it use the default ASCII mode, it FAILS.  This
> time I get the same corrupted file that you got:
>
> $ rm -f perl-5.31.1.tar.gz; perl -e 'use Net::FTP; $ftp = Net::FTP->new("
>> mirrors.rit.edu", Debug => 1); $ftp->login("anonymous");
>> $ftp->cwd("/CPAN/authors/id/E/ET/ETHER"); $ftp->get("perl-5.31.1.tar.gz");
>> $ftp->quit;'; ls -l perl-5.31.1.tar.gz; sha -1 perl-5.31.1.tar.gz; gzip -tv
>> perl-5.31.1.tar.gz
>> Net::FTP>>> Net::FTP(3.11)
>> Net::FTP>>>   Exporter(5.70)
>> Net::FTP>>>   Net::Cmd(3.11)
>> Net::FTP>>>   IO::Socket::IP(0.30)
>> Net::FTP>>>     IO::Socket(1.36)
>> Net::FTP>>>       IO::Handle(1.34)
>> Net::FTP=GLOB(0x1251ed0)<<< 220 Welcome to mirrors.rit.edu.
>> Net::FTP=GLOB(0x1251ed0)>>> USER anonymous
>> Net::FTP=GLOB(0x1251ed0)<<< 331 Please specify the password.
>> Net::FTP=GLOB(0x1251ed0)>>> PASS ....
>> Net::FTP=GLOB(0x1251ed0)<<< 230 Login successful.
>> Net::FTP=GLOB(0x1251ed0)>>> CWD /CPAN/authors/id/E/ET/ETHER
>> Net::FTP=GLOB(0x1251ed0)<<< 250 Directory successfully changed.
>> Net::FTP=GLOB(0x1251ed0)>>> PASV
>> Net::FTP=GLOB(0x1251ed0)<<< 227 Entering Passive Mode
>> (129,21,171,72,78,16).
>> Net::FTP=GLOB(0x1251ed0)>>> RETR perl-5.31.1.tar.gz
>> Net::FTP=GLOB(0x1251ed0)<<< 150 Opening BINARY mode data connection for
>> perl-5.31.1.tar.gz (17593158 bytes).
>> Net::FTP=GLOB(0x1251ed0)<<< 226 Transfer complete.
>> Net::FTP=GLOB(0x1251ed0)>>> QUIT
>> Net::FTP=GLOB(0x1251ed0)<<< 221 Goodbye.
>> -rw-r--r--. 1 deven deven 17592906 Jul 25 20:53 perl-5.31.1.tar.gz
>> f8edd1080d69bf9be656d384ec772c7a398fb502 perl-5.31.1.tar.gz
>> perl-5.31.1.tar.gz:
>> gzip: perl-5.31.1.tar.gz: invalid compressed data--format violated
>> $
>
>
> So, why did I get a good download from mirrors.rit.edu while you got a
> corrupted one?  Based on your Net::FTP debugging output, the problem seems
> obvious to me.  While your script does appear to be calling
> Net::FTP::binary() to enable binary mode, it is apparently doing so BEFORE
> logging into the FTP server, which is very unusual.  (I've never seen any
> FTP client do that.)  Did you notice the response from the server to that
> command?  "530 Please login with USER and PASS."  That's an error response,
> which means the "TYPE I" command FAILED.  As a result, your FTP connection
> remained in ASCII mode despite the attempt to switch to binary mode.
> When I did it AFTER logging into the FTP server, it worked fine and
> responded with "200 Switching to Binary mode."
>
> Apparently, the Pure-FTPd software on ftp.funet.fi is more forgiving,
> accepting the premature "TYPE I" without being logged in yet.  It returned
> "200 TYPE is now 8-bit binary", so binary mode was successfully enabled.
> This is a good example of how implementation-specific details can cause
> unexpected results.
>
> Regardless, the proper thing to do is to login to the FTP server FIRST.
> If you change your "ftp.pl" script to call $ftp->binary() AFTER logging
> in, you should be able to use it with mirrors.rit.edu without getting
> corrupted downloads.
>
> In all of these examples, transfers done in binary mode always worked --
> EVERY failing transfer was done in ASCII mode.  That isn't a coincidence;
> it's the root cause of these corrupted downloads.
>
> Deven
>
>
> On Thu, Jul 25, 2019 at 9:18 AM sisyphus <sisyphus359@gmail.com> wrote:
>
>>
>>
>> On Thu, Jul 25, 2019 at 4:22 PM Deven T. Corzine <deven@ties.org> wrote:
>> > Tony's advice is exactly the right answer here.  Enabling binary mode
>> is more than a good suggestion -- it's an essential bug fix, with virtually
>> zero chance of breaking anything.  This change should fix the original
>> issue.
>>
>> But it doesn't.
>> Using Net::FTP to download a perl source tarball from mirrors.rit.edu
>> delivers a corrupt file:
>>
>> $ perl ftp.pl mirrors.rit.edu 5.31.1 1
>> Net::FTP>>> Net::FTP(3.11)
>> Net::FTP>>>   Exporter(5.74)
>> Net::FTP>>>   Net::Cmd(3.11)
>> Net::FTP>>>   IO::Socket::IP(0.39)
>> Net::FTP>>>     IO::Socket(1.41)
>> Net::FTP>>>       IO::Handle(1.41)
>> Net::FTP=GLOB(0x801524150)<<< 220 Welcome to mirrors.rit.edu.
>> Net::FTP=GLOB(0x801524150)>>> TYPE I
>> Net::FTP=GLOB(0x801524150)<<< 530 Please login with USER and PASS.
>> Net::FTP=GLOB(0x801524150)>>> USER anonymous
>> Net::FTP=GLOB(0x801524150)<<< 331 Please specify the password.
>> Net::FTP=GLOB(0x801524150)>>> PASS ....
>> Net::FTP=GLOB(0x801524150)<<< 230 Login successful.
>> Net::FTP=GLOB(0x801524150)>>> CWD pub
>> Net::FTP=GLOB(0x801524150)<<< 250 Directory successfully changed.
>> Net::FTP=GLOB(0x801524150)>>> CWD CPAN/src/5.0
>> Net::FTP=GLOB(0x801524150)<<< 250 Directory successfully changed.
>> Net::FTP=GLOB(0x801524150)>>> PASV
>> Net::FTP=GLOB(0x801524150)<<< 227 Entering Passive Mode
>> (129,21,171,72,78,5).
>> Net::FTP=GLOB(0x801524150)>>> RETR perl-5.31.1.tar.gz
>> Net::FTP=GLOB(0x801524150)<<< 150 Opening BINARY mode data connection for
>> perl-5.31.1.tar.gz (17593158 bytes).
>> Net::FTP=GLOB(0x801524150)<<< 226 Transfer complete.
>> Net::FTP=GLOB(0x801524150)>>> QUIT
>> Net::FTP=GLOB(0x801524150)<<< 221 Goodbye.
>> $ sha1 perl-5.31.1.tar.gz
>> SHA1 (perl-5.31.1.tar.gz) = f8edd1080d69bf9be656d384ec772c7a398fb502
>>
>> Using the same script to download the same perl source from ftp.funet.fi
>> works fine:
>>
>> $ perl ftp.pl ftp.funet.fi 5.31.1 1
>> Net::FTP>>> Net::FTP(3.11)
>> Net::FTP>>>   Exporter(5.74)
>> Net::FTP>>>   Net::Cmd(3.11)
>> Net::FTP>>>   IO::Socket::IP(0.39)
>> Net::FTP>>>     IO::Socket(1.41)
>> Net::FTP>>>       IO::Handle(1.41)
>> Net::FTP=GLOB(0x80151d150)<<< 220---------- Welcome to Pure-FTPd
>> [privsep] ----------
>> Net::FTP=GLOB(0x80151d150)<<< 220-You are user number 15 of 1000 allowed.
>> Net::FTP=GLOB(0x80151d150)<<< 220-Local time is now 16:03. Server port:
>> 21.
>> Net::FTP=GLOB(0x80151d150)<<< 220-Only anonymous FTP is allowed here
>> Net::FTP=GLOB(0x80151d150)<<< 220-IPv6 connections are also welcome on
>> this server.
>> Net::FTP=GLOB(0x80151d150)<<< 220 You will be disconnected after 30
>> minutes of inactivity.
>> Net::FTP=GLOB(0x80151d150)>>> TYPE I
>> Net::FTP=GLOB(0x80151d150)<<< 200 TYPE is now 8-bit binary
>> Net::FTP=GLOB(0x80151d150)>>> USER anonymous
>> Net::FTP=GLOB(0x80151d150)<<< 331 Any password will work
>> Net::FTP=GLOB(0x80151d150)>>> PASS ....
>> Net::FTP=GLOB(0x80151d150)<<< 230 Any password will work
>> Net::FTP=GLOB(0x80151d150)>>> CWD pub
>> Net::FTP=GLOB(0x80151d150)<<< 250 OK. Current directory is /pub
>> Net::FTP=GLOB(0x80151d150)>>> CWD languages/perl/CPAN/src/5.0
>> Net::FTP=GLOB(0x80151d150)<<< 250 OK. Current directory is /.m/mirrors/
>> ftp.cpan.org/pub/CPAN/src/5.0
>> Net::FTP=GLOB(0x80151d150)>>> PASV
>> Net::FTP=GLOB(0x80151d150)<<< 227 Entering Passive Mode
>> (193,166,3,21,142,168)
>> Net::FTP=GLOB(0x80151d150)>>> RETR perl-5.31.1.tar.gz
>> Net::FTP=GLOB(0x80151d150)<<< 150-Accepted data connection
>> Net::FTP=GLOB(0x80151d150)<<< 150 17180.8 kbytes to download
>> Net::FTP=GLOB(0x80151d150)<<< 226-File successfully transferred
>> Net::FTP=GLOB(0x80151d150)<<< 226 172.328 seconds (measured here), 99.70
>> Kbytes per second
>> Net::FTP=GLOB(0x80151d150)>>> QUIT
>> Net::FTP=GLOB(0x80151d150)<<< 221-Goodbye. You uploaded 0 and downloaded
>> 17181 kbytes.
>> Net::FTP=GLOB(0x80151d150)<<< 221 Logout.
>> $ sha1 perl-5.31.1.tar.gz
>> SHA1 (perl-5.31.1.tar.gz) = c68cbdaf18a73765421aa1bf1883691551c1aa81
>>
>> The fact that the system ftp retrieves a valid perl-5.31.1.tar.gz from
>> mirrors.rit.edu would suggest that the problem lies with Net::FTP.
>> However, I don't regard this as proven. (Perhaps the mirrors.rit.edu
>> server is doing something it shouldn't ?)
>>
>> Cheers,
>> Rob
>>
>>
>>

Thread Previous | Thread Next


nntp.perl.org: Perl Programming lists via nntp and http.
Comments to Ask Bjørn Hansen at ask@perl.org | Group listing | About