develooper Front page | perl.perl5.porters | Postings from September 2018

CPAN river 3000: status as of perl-5.29.3 release

From:
James E Keenan
Date:
September 22, 2018 19:01
Subject:
CPAN river 3000: status as of perl-5.29.3 release
Message ID:
20180922190126.25379.qmail@lists-nntp.develooper.com
You can find data from a test run of the "CPAN-river-3000" against
all releases in the current (perl-5.29) development cycle here:

#####
http://thenceforward.net/perl/misc/cpan-river-3000.perl-5.29.master.psv.gz
#####

In addition, please find attached a file in CSV format holding
data about those CPAN distributions which received a grade other
than 'PASS' during the most recent monthly run or whose grade
changed between the two most recent runs.

#####
changes-5.29.2-to-5.29.3.csv
#####

I. General Comments

The data in these files are intended to be used in the assessment
of the impact which Perl 5 development is having on the
installability of an important set of CPAN distributions.  These
are not the only data which can be used or which ought to be used
in that assessment.

The following items should be noted:

1. The number of CPAN modules selected for attempted build against
development releases has been increased from 1000 (in the 5.27
cycle) to 3000 (in the 5.29 cycle).  We hope that this will
provide a more thorough basis for an assessment of the impact of
blead on CPAN.

2. The analysis is being conducted on FreeBSD rather than Linux.

#####
$ uname -mrs
FreeBSD 11.1-RELEASE amd64
$ clang --version
FreeBSD clang version 4.0.0 (tags/RELEASE_400/final 297347) (based on 
LLVM 4.0.0)
Target: x86_64-unknown-freebsd11.1
Thread model: posix
InstalledDir: /usr/bin
######

By doing so, we hope to avoid a monoculture of operating systems
and call attention to the fact that the impact of blead on CPAN
can differ across platforms.

3. The list of modules selected is curated so that the automated
test-against-dev program can run smoothly over its
more-than-six-hours length.  That means excluding modules that
require interactive configuration, modules that make excessive
demands with respect to external prerequisites, modules that
require too much network access, modules that are subject to
timeouts, and so forth.

4. On the other hand, the list of modules selected has not been
filtered to exclude modules which, nothwithstanding the fact that
other CPAN distributions depend on them, have been failing tests
for years.

5. The test-against-dev process is built on top of Miyagawa's
'cpanm' and Breno G. de Oliveira's CPAN library
App::cpanminus::reporter.  It therefore benefits from the
strengths, and suffers from the flaws, of those two excellent
libraries.  In particular, if distribution XYZ has a dependency on
distribution ABC and if, earlier in the process, ABC has failed,
the test-against-dev process will steer clear of attempting to
fully test XYZ and will fail to write a report for it.  So some
modules will "not be reached" and will not show up in the output
data until such time as their upstream dependencies are fixed.

6. There are certain CPAN distributions whose prerequisites I have
not yet figured out how to install on the platform being used.
Those distributions are likely to have a grade of UNKNOWN and
their reverse dependencies will not be reached.  The most
important case of this: Tk.

If you would like further information about this project or to
help it along, please contact me off-list.

II. Specific Comments

1. If a distribution PASSed when run against perl-5.29.2 last
month and then received a grade other than PASS when run against
perl-5.29.3 today, then none of its reverse dependencies would
have been reached in the perl-5.29.3 process run today.  Those
reverse dependencies have therefore been graded 'x' for
'perl-5.29.3-grade' even if they got a 'PASS' for
'perl-5.29.2-grade'.

The most significant instance of this in this month's run is test
failures in Catalyst-Runtime.  These failures are not
"blead-breaks-CPAN" failures.  Rather, they are failures in one
test file which has already been reported as
https://rt.cpan.org/Ticket/Display.html?id=127050, to which I have
added additional data based on what I found during
test-against-dev.

2. Conversely, there are distributions which were not reached in
last month's run for perl-5.29.2 and which were therefore graded
'x' last month which were reached in yesterday's run for
perl-5.29.3 and which have therefore been given a true grade
(whether that be 'PASS', 'FAIL', etc.).  In many cases this is due
to the fact that CPAN distribution Archive-Zip, which was failing
in that manner last month, has been fixed, which has the effect of
permitting many of its reverse dependencies within the
CPAN-River-3000 of being reached and graded during
test-against-dev.

III.  Acknowledgements

This project benefited from discussion at the Perl 5 Core
Hackathon held in Amsterdam in October 2017, so thanks to
participants in that event and to the organizations which
sponsored it.

This project also benefits from the generous donation of a server
by the New York City BSD Users Group and the system administration
expertise of Mark Saad and George Rosamond.

This project also benefitted from assistance in reviewing test
failure reports by three members of Philadelphia Perlmongers:
John Karr, Walt Mankowski and Thomas McKernan.

This project also benefits from the system administration
expertise of Andrew Villano of New York Perlmongers.

IV. References

See my presentation at the North American Perl Conference held in
Salt Lake City in June 2018:
http://thenceforward.net/perl/tpc/TPC-NA-2018/test-against-dev.pdf

Thank you very much.
Jim Keenan



nntp.perl.org: Perl Programming lists via nntp and http.
Comments to Ask Bjørn Hansen at ask@perl.org | Group listing | About