develooper Front page | perl.perl5.porters | Postings from October 2018

CPAN river 3000: status as of perl-5.29.4 release

From:
James E Keenan
Date:
October 21, 2018 15:55
Subject:
CPAN river 3000: status as of perl-5.29.4 release
Message ID:
379cce46-a385-3e09-2ed5-404186235ef4@pobox.com
You can find data from a test run of the "CPAN-river-3000" against
all releases in the current (perl-5.29) development cycle here:

#####
http://thenceforward.net/perl/misc/cpan-river-3000.perl-5.29.master.psv.gz
#####

In addition, please find attached a file in CSV format holding
data about those CPAN distributions which received a grade other
than 'PASS' during the most recent monthly run or whose grade
changed between the two most recent runs.

#####
changes-5.29.3-to-5.29.4.csv
#####

I. General Comments

The data in these files are intended to be used in the assessment
of the impact which Perl 5 development is having on the
installability of an important set of CPAN distributions.  These
are not the only data which can be used or which ought to be used
in that assessment.

The following items should be noted:

1. The number of CPAN modules selected for attempted build against
development releases has been increased from 1000 (in the 5.27
cycle) to 3000 (in the 5.29 cycle).  We hope that this will
provide a more thorough basis for an assessment of the impact of
blead on CPAN.

2. The analysis is being conducted on FreeBSD rather than Linux.

#####
$ uname -mrs
FreeBSD 11.1-RELEASE amd64
$ clang --version
FreeBSD clang version 4.0.0 (tags/RELEASE_400/final 297347) (based on 
LLVM 4.0.0)
Target: x86_64-unknown-freebsd11.1
Thread model: posix
InstalledDir: /usr/bin
######

By doing so, we hope to avoid a monoculture of operating systems
and call attention to the fact that the impact of blead on CPAN
can differ across platforms.

3. The list of modules selected is curated so that the automated
test-against-dev program can run smoothly over its
more-than-six-hours length.  That means excluding modules that
require interactive configuration, modules that make excessive
demands with respect to external prerequisites, modules that
require too much network access, modules that are subject to
timeouts, and so forth.

4. On the other hand, the list of modules selected has not been
filtered to exclude modules which, nothwithstanding the fact that
other CPAN distributions depend on them, have been failing tests
for years.

5. The test-against-dev process is built on top of Miyagawa's
'cpanm' and Breno G. de Oliveira's CPAN library
App::cpanminus::reporter.  It therefore benefits from the
strengths, and suffers from the flaws, of those two excellent
libraries.  In particular, if distribution XYZ has a dependency on
distribution ABC and if, earlier in the process, ABC has failed,
the test-against-dev process will steer clear of attempting to
fully test XYZ and will fail to write a report for it.  So some
modules will "not be reached" and will not show up in the output
data until such time as their upstream dependencies are fixed.

6. There are certain CPAN distributions whose prerequisites I have
not yet figured out how to install on the platform being used.
Those distributions are likely to have a grade of UNKNOWN and
their reverse dependencies will not be reached.  The most
important case of this: Tk.

If you would like further information about this project or to
help it along, please contact me off-list.

II. Specific Comments

1. If a distribution PASSed when run against perl-5.29.3 last
month and then received a grade other than PASS when run against
perl-5.29.4 today, then none of its reverse dependencies would
have been reached in the perl-5.29.4 process run today.  Those
reverse dependencies have therefore been graded 'x' for
'perl-5.29.4-grade' even if they got a 'PASS' for
'perl-5.29.3-grade'.

a. This month, most such distributions are those that are
dependent upon File-Slurp.  File-Slurp is now being graded FAIL
due to its use of sysread and syswrite in a manner which has been
fatalized for perl-5.30
(https://rt.cpan.org/Ticket/Display.html?id=117005).  I have
submitted a pull request which addresses these problems
(https://github.com/genio/slurp/pull/2#issuecomment-430677317) but
it does not yet work on Windows.  Please take a look at the work
which Chase Whitener is doing on this distribution
(https://github.com/genio/slurp).

b. In the VM in which test-against-dev is run, there was a timeout
failure in the Expect distribution's test suite.  That test suite
cautions that some of its tests can fail due to timing issues, so
the current result is not surprising.  Expect did not install,
which prevented some of its reverse dependencies within the CPAN
River 3000 from being reached.  I re-ran Expect's test suite on
the VM but outside from the test-against-dev process and got a
PASS.  Hence, Expect's reverse dependencies are likely to be
reached in next month's testing.

c. There were similar timing-related failures this month in
Device-SerialPort.

2. Conversely, there are distributions which were not reached in
last month's run for perl-5.29.3 and which were therefore graded
'x' last month which were reached in yesterday's run for
perl-5.29.4 and which have therefore been given a true grade
(whether that be 'PASS', 'FAIL', etc.).  In many cases this is due
to the fact that CPAN distribution Catalyst-Runtime, which was failing
in that manner last month, has been fixed.  This had the effect of
permitting 40 of its reverse dependencies within the
CPAN-River-3000 of being reached and graded during
test-against-dev.

3. There were new test failures in 3 other distributions in the
CPAN River 3000 which still need investigation:

#####
HTML-DOM,PASS,FAIL
JE,PASS,FAIL
XML-Easy,PASS,FAIL
#####

III.  Acknowledgements

This project benefited from discussion at the Perl 5 Core
Hackathon held in Amsterdam in October 2017, so thanks to
participants in that event and to the organizations which
sponsored it.

This project also benefits from the generous donation of a server
by the New York City BSD Users Group and the system administration
expertise of Mark Saad and George Rosamond.

This project also benefitted from assistance in reviewing test
failure reports by three members of Philadelphia Perlmongers:
John Karr, Walt Mankowski and Thomas McKernan.

This project also benefits from the system administration
expertise of Andrew Villano of New York Perlmongers.

IV. References

See my presentation at the North American Perl Conference held in
Salt Lake City in June 2018:
http://thenceforward.net/perl/tpc/TPC-NA-2018/test-against-dev.pdf

Thank you very much.
Jim Keenan



nntp.perl.org: Perl Programming lists via nntp and http.
Comments to Ask Bjørn Hansen at ask@perl.org | Group listing | About