develooper Front page | perl.perl5.porters | Postings from March 2017

Re: Shortage of Red / Yellow on CPAN Testers Matrix

Thread Previous | Thread Next
Dan Collins
March 29, 2017 18:25
Re: Shortage of Red / Yellow on CPAN Testers Matrix
Message ID:
Porters, I heard you wanted some red and yellow...

I have tooling set up that allows me to test/install most of CPAN, saving
the CPAN::Reporter results to file, and compare those stored files between
two different versions of perl. I installed two instances of perl - cpan1
and cpan2 - which are completely identical. For one of the environments, I
set PERL_USE_UNSAFE_INC=0, in the other, I left it undefined.

I'm not done yet, but I have over 5,000 dists that have been tested on both
configurations. Of those dists, I've collected the reports that PASSED with
PERL_USE_UNSAFE_INC=0. Those reports can be downloaded in tarball form at:

Of the discrepancies, 242 FAILED, and 237 were UNKNOWN. The former means
they failed in tests, the latter means they failed in Build.PL or
Makefile.PL. Of course, some particularly special dists had multiple
problems. There are a few cases where a dist FAILed in the control: 5 where
it PASSed in the experiment, which are probably signs of an intermittent
test failure and which should be ignored, and 6 where it got UNKNOWN in the
experiment, which are probably signs of a no-dot-in-inc error *in addition
to* a test failure.

I haven't uploaded these test reports yet, because I want to include a
message in the "Tester Comments" explaining the failures that module
authors might be able to use to fix their stuff. I'll be watching the other
thread that Kent started to see when that resource is available

Happy to answer any questions or take any suggestions...


On Mon, Mar 27, 2017 at 8:42 PM, Ryan Voots <>

> I've gone and built a naive version of this for my own uses.  It doesn't
> yet hook into cpanm-reporter but i don't think it'd be hard to do.  It's
> intending to take a --module or --cpanfile to start from and works its way
> through the dependency tree (fetches it all with cpanm --showdeps) and then
> tests them with perlbrew exec --with ... so that the script can run in a
> stable perl install while testing current blead/git easily.  I'll toss it
> up on github if there's any big call for it (otherwise it'll just live
> publicly on my server).
> On Sun, Mar 26, 2017 at 8:39 PM, Kent Fredric <>
> wrote:
>> On 27 March 2017 at 16:10, Dan Collins <> wrote:
>> > I've had a chance to make sure that this test environment isn't horribly
>> > broken...
>> Yeah, there's enough broken you sort of need a double-testing system.
>> Start with =0 , and then upon hitting a failure, retry with =1 ...
>> while paying care
>> not to let the =1 bleed into children in the recursive step.
>> eg:
>> 1. Install X with UNSAFE_INC=0
>> 2. Configure fails?
>>     1. Report N/A Fail
>>     2. Generate META.JSON with UNSAFE_INC=1
>>     3. Installdeps with UNSAFE_INC=0
>>     4. Run tests with UNSAFE_INC=0
>>     5. Tests Fail?
>>         1. Report FAIL
>>         2. Re-run  tests with UNSAFE_INC=1
>>         3. Report Test result
>> I don't know how to make my tools do this automatically so I've been
>> doing it manually and filing bugs, to fill the gaps in automation.
>> --
>> Kent

Thread Previous | Thread Next Perl Programming lists via nntp and http.
Comments to Ask Bjørn Hansen at | Group listing | About