develooper Front page | perl.perl5.porters | Postings from March 2017

Re: Shortage of Red / Yellow on CPAN Testers Matrix

Thread Previous | Thread Next
Dan Collins
March 29, 2017 18:38
Re: Shortage of Red / Yellow on CPAN Testers Matrix
Message ID:
Huh, I just found another interesting thing. CPAN::Reporter makes no effort
to report on whether the `install` phase succeeded, but some particularly
creative dists manage to fail in the `install` phase under
PERL_USE_UNSAFE_INC=0. Look at this example
from GUIDO/libintl-perl-1.26.tar.gz - does anyone know how common this sort
of thing is? There aren't many using this exact name (

All tests successful.
Files=164, Tests=3750,  5 wallclock secs ( 0.87 usr  0.18 sys +  8.23 cusr
 0.30 csys =  9.58 CPU)
Result: PASS
(/usr/bin/make test exited with 0)
CPAN::Reporter: Test result is 'pass', 'make test' no errors.
CPAN::Reporter: preparing a CPAN Testers report for libintl-perl-1.26

CPAN::Reporter: this appears to be a duplicate report for the test phase:
PASS libintl-perl-1.26 x86_64-linux-thread-multi 3.16.0-4-amd64

Test report will not be sent.

  /usr/bin/make test -- OK
Running make install
make[1]: Entering directory
"/home/cpan2/install/bin/perl" -MExtUtils::Command::MM -e 'cp_nonempty' -- ../blib/arch/auto/Locale/gettext_xs/ 644
make[1]: Leaving directory
Manifying 30 pod documents
Manifying 29 pod documents
Manifying 31 pod documents
Manifying 29 pod documents
Manifying 30 pod documents
Can't locate in @INC (you may need to install the MyInstall
module) (@INC contains:
BEGIN failed--compilation aborted.
Makefile:1389: recipe for target 'pure_site_install' failed
make: *** [pure_site_install] Error 2
  /usr/bin/make install  -- NOT OK
Stopping: 'install' failed for 'Locale::TextDomain'.

On Wed, Mar 29, 2017 at 2:25 PM, Dan Collins <> wrote:

> Porters, I heard you wanted some red and yellow...
> I have tooling set up that allows me to test/install most of CPAN, saving
> the CPAN::Reporter results to file, and compare those stored files between
> two different versions of perl. I installed two instances of perl - cpan1
> and cpan2 - which are completely identical. For one of the environments, I
> set PERL_USE_UNSAFE_INC=0, in the other, I left it undefined.
> I'm not done yet, but I have over 5,000 dists that have been tested on
> both configurations. Of those dists, I've collected the reports that PASSED
> with PERL_USE_UNSAFE_INC=1 and FAILED or were UNKNOWN with
> PERL_USE_UNSAFE_INC=0. Those reports can be downloaded in tarball form at:
> Of the discrepancies, 242 FAILED, and 237 were UNKNOWN. The former means
> they failed in tests, the latter means they failed in Build.PL or
> Makefile.PL. Of course, some particularly special dists had multiple
> problems. There are a few cases where a dist FAILed in the control: 5 where
> it PASSed in the experiment, which are probably signs of an intermittent
> test failure and which should be ignored, and 6 where it got UNKNOWN in the
> experiment, which are probably signs of a no-dot-in-inc error *in addition
> to* a test failure.
> I haven't uploaded these test reports yet, because I want to include a
> message in the "Tester Comments" explaining the failures that module
> authors might be able to use to fix their stuff. I'll be watching the other
> thread that Kent started to see when that resource is available
> Happy to answer any questions or take any suggestions...
> --
> Dan
> On Mon, Mar 27, 2017 at 8:42 PM, Ryan Voots <>
> wrote:
>> I've gone and built a naive version of this for my own uses.  It doesn't
>> yet hook into cpanm-reporter but i don't think it'd be hard to do.  It's
>> intending to take a --module or --cpanfile to start from and works its way
>> through the dependency tree (fetches it all with cpanm --showdeps) and then
>> tests them with perlbrew exec --with ... so that the script can run in a
>> stable perl install while testing current blead/git easily.  I'll toss it
>> up on github if there's any big call for it (otherwise it'll just live
>> publicly on my server).
>> On Sun, Mar 26, 2017 at 8:39 PM, Kent Fredric <>
>> wrote:
>>> On 27 March 2017 at 16:10, Dan Collins <> wrote:
>>> > I've had a chance to make sure that this test environment isn't
>>> horribly
>>> > broken...
>>> Yeah, there's enough broken you sort of need a double-testing system.
>>> Start with =0 , and then upon hitting a failure, retry with =1 ...
>>> while paying care
>>> not to let the =1 bleed into children in the recursive step.
>>> eg:
>>> 1. Install X with UNSAFE_INC=0
>>> 2. Configure fails?
>>>     1. Report N/A Fail
>>>     2. Generate META.JSON with UNSAFE_INC=1
>>>     3. Installdeps with UNSAFE_INC=0
>>>     4. Run tests with UNSAFE_INC=0
>>>     5. Tests Fail?
>>>         1. Report FAIL
>>>         2. Re-run  tests with UNSAFE_INC=1
>>>         3. Report Test result
>>> I don't know how to make my tools do this automatically so I've been
>>> doing it manually and filing bugs, to fill the gaps in automation.
>>> --
>>> Kent
>>> KENTNL -

Thread Previous | Thread Next Perl Programming lists via nntp and http.
Comments to Ask Bjørn Hansen at | Group listing | About