[PATCH?] $ENV{TEST_JOBS} vs $ENV{HARNESS_OPTIONS}

I'm confused by how Test::Parser and how Tap::Parser works so I don't  
know if this patch should or should not be applied...


$ENV{TEST_JOBS} = 5 :

A list of rules is build which specify what to run in parallel and  
what to run in sequence.

Then Tap::Harness->new and runtest are called with the number of jobs,  
the rules and the test.


$ENV{HARNESS_OPTIONS} = j5 :

The list of rules is not build.

Test::Harness::runtest is called which parses $ENV{HARNESS_OPTIONS}  
and sets the number of jobs to 5.

Then it start aggregating the tests and before doing that it does a   
local $ENV{HARNESS_OPTIONS}.

Severeal tests fail in this situation because all tests are being run  
in parallel even those that aren't parallel-safe (yet).

Tests that fail:

Test Summary Report
-------------------
../ext/DB_File/t/db-btree                                    (Wstat:  
512 Tests: 17 Failed: 1)
   Failed test:  17
   Non-zero exit status: 2
   Parse errors: Bad plan.  You planned 197 tests but ran 17.
../ext/IO_Compress_Zlib/t/005defhdr                          (Wstat:  
3328 Tests: 595 Failed: 13)
   Failed tests:  537-538, 540, 550-551, 553, 563-564, 566
                 576-577, 579, 594
   Non-zero exit status: 13
../ext/IO_Compress_Zlib/t/004gziphdr                         (Wstat:  
65280 Tests: 905 Failed: 16)
   Failed tests:  835-836, 838, 849-850, 852, 863-864, 866
                 877-878, 881, 892-893, 896, 905
   Non-zero exit status: 255
   Parse errors: Bad plan.  You planned 910 tests but ran 905.
../ext/IO_Compress_Zlib/t/100generic-deflate                 (Wstat:  
65280 Tests: 58 Failed: 1)
   Failed test:  58
   Non-zero exit status: 255
   Parse errors: Bad plan.  You planned 670 tests but ran 58.
../ext/IO_Compress_Zlib/t/100generic-gzip                    (Wstat:  
65280 Tests: 330 Failed: 0)
   Non-zero exit status: 255
   Parse errors: Bad plan.  You planned 670 tests but ran 330.
../ext/IO_Compress_Zlib/t/101truncate-deflate                (Wstat:  
65280 Tests: 109 Failed: 1)
   Failed test:  109
   Non-zero exit status: 255
   Parse errors: Bad plan.  You planned 734 tests but ran 109.
../ext/IO_Compress_Zlib/t/101truncate-gzip                   (Wstat:  
65280 Tests: 362 Failed: 1)
   Failed test:  362
   Non-zero exit status: 255
   Parse errors: Bad plan.  You planned 978 tests but ran 362.
../ext/IO_Compress_Zlib/t/105oneshot-gzip                    (Wstat:  
65280 Tests: 400 Failed: 0)
   Non-zero exit status: 255
   Parse errors: Bad plan.  You planned 970 tests but ran 400.
../ext/IO_Compress_Zlib/t/105oneshot-deflate                 (Wstat:  
65280 Tests: 574 Failed: 5)
   Failed tests:  565-567, 570, 574
   Non-zero exit status: 255
   Parse errors: Bad plan.  You planned 970 tests but ran 574.
../ext/IO_Compress_Zlib/t/105oneshot-rawdeflate              (Wstat:  
65280 Tests: 524 Failed: 0)
   Non-zero exit status: 255
   Parse errors: Bad plan.  You planned 970 tests but ran 524.
../ext/IO_Compress_Zlib/t/105oneshot-gzip-only               (Wstat:  
2048 Tests: 70 Failed: 8)
   Failed tests:  4, 6, 8-10, 12-13, 24
   Non-zero exit status: 8
../ext/IO_Compress_Zlib/t/105oneshot-zip-only                (Wstat:  
65280 Tests: 74 Failed: 10)
   Failed tests:  4, 6, 8-10, 12-13, 24, 73-74
   Non-zero exit status: 255
   Parse errors: Bad plan.  You planned 146 tests but ran 74.
../ext/IO_Compress_Zlib/t/109merge-rawdeflate                (Wstat:  
65280 Tests: 73 Failed: 0)
   Non-zero exit status: 255
   Parse errors: Bad plan.  You planned 165 tests but ran 73.
../lib/CPANPLUS/t/03_CPANPLUS-Internals-Source               (Wstat:  
1536 Tests: 39 Failed: 6)
   Failed tests:  30, 32-33, 35-36, 38
   Non-zero exit status: 6
../lib/CPANPLUS/t/19_CPANPLUS-Dist                           (Wstat:  
256 Tests: 94 Failed: 1)
   Failed test:  2
   Non-zero exit status: 1
../lib/CPANPLUS/t/20_CPANPLUS-Dist-MM                        (Wstat:  
65280 Tests: 41 Failed: 4)
   Failed tests:  15-16, 40-41
   Non-zero exit status: 255
../lib/CPANPLUS/t/21_CPANPLUS-Dist-No-Build                  (Wstat:  
65280 Tests: 2 Failed: 1)
   Failed test:  1
   Non-zero exit status: 255
../lib/CPANPLUS/t/40_CPANPLUS-Internals-Report               (Wstat:  
65280 Tests: 10 Failed: 1)
   Failed test:  10
   Non-zero exit status: 255
../lib/CPANPLUS/t/30_CPANPLUS-Internals-Selfupdate           (Wstat:  
256 Tests: 47 Failed: 1)
   Failed test:  18
   Non-zero exit status: 1
../lib/DBM_Filter/t/int32                                    (Wstat:  
256 Tests: 22 Failed: 1)
   Failed test:  21
   Non-zero exit status: 1
../lib/DBM_Filter/t/encode                                   (Wstat:  
256 Tests: 26 Failed: 1)
   Failed test:  25
   Non-zero exit status: 1
../lib/ExtUtils/t/basic                                      (Wstat:  
18432 Tests: 83 Failed: 72)
   Failed tests:  3-4, 6-55, 57-59, 61-67, 69, 71-74, 76-77
                 79-81
   Non-zero exit status: 72
../lib/IO/Zlib/t/getc                                        (Wstat:  
512 Tests: 4 Failed: 1)
   Failed test:  4
   Non-zero exit status: 2
   Parse errors: Bad plan.  You planned 10 tests but ran 4.
../lib/Pod/t/basic                                           (Wstat:  
512 Tests: 2 Failed: 0)
   Non-zero exit status: 2
   Parse errors: Bad plan.  You planned 11 tests but ran 2.
../lib/Pod/t/man                                             (Wstat:  
512 Tests: 12 Failed: 0)
   Non-zero exit status: 2
   Parse errors: Bad plan.  You planned 24 tests but ran 12.
../lib/Pod/t/pod-parser                                      (Wstat: 0  
Tests: 3 Failed: 2)
   Failed tests:  2-3




So far this makes sense.

Now the confusing part:

$ENV{TEST_JOBS} = 5
$ENV{HARNESS_OPTIONS} = j5

Now the list of rules is build, Tap::Harness->new and runtest are called.
But Tap::Harness never look (as far as I can tell) at $ENV{HARNESS_OPTIONS}.

When the aggregating of tests starts it doesn't do a  local  
$ENV{HARNESS_OPTIONS}.

The result of this is that some tests fail, but not as much as when  
$ENV{HARNESS_OPTIONS} is set and $ENV{TEST_JOBS} is unset.

Tests that fail:

Test Summary Report
-------------------
../lib/ExtUtils/t/basic                                      (Wstat:  
256 Tests: 83 Failed: 1)
   Failed test:  25
   Non-zero exit status: 1
../lib/Module/Build/t/basic                                  (Wstat:  
512 Tests: 52 Failed: 2)
   Failed tests:  24-25
   Non-zero exit status: 2
../lib/Module/Build/t/compat                                 (Wstat:  
512 Tests: 122 Failed: 2)
   Failed tests:  113, 115
   Non-zero exit status: 2
../lib/Test/Harness/t/compat/env                             (Wstat: 0  
Tests: 0 Failed: 0)
   Parse errors: Bad plan.  You planned 1 tests but ran 0.
../lib/Test/Harness/t/compat/inc-propagation                 (Wstat: 0  
Tests: 0 Failed: 0)
   Parse errors: Bad plan.  You planned 2 tests but ran 0.
../lib/Module/Build/t/test_type                              (Wstat:  
768 Tests: 8 Failed: 3)
   Failed tests:  4-5, 7
   Non-zero exit status: 3
../lib/Module/Build/t/test_types                             (Wstat:  
5120 Tests: 26 Failed: 20)
   Failed tests:  3-4, 6-13, 16-17, 19-26
   Non-zero exit status: 20



So I assume something is happening with $ENV{HARNESS_OPTIONS} but I  
can't figure out what or where...

The attached patch parsers HARNESS_OPTION in t/harness and copies the  
options to Tap::Harness but I do not know if this is a good idea or  
not...

With the patch no tests fail when:
- $ENV{TEST_JOBS} is set,
- $ENV{HARNESS_OPTIONS} is set,
- $ENV{TEST_JOBS} and $ENV{HARNESS_OPTIONS} are set.

• [PATCH?] $ENV{TEST_JOBS} vs $ENV{HARNESS_OPTIONS} by Bram
• Re: [PATCH?] $ENV{TEST_JOBS} vs $ENV{HARNESS_OPTIONS} by Eric Wilhelm
• Re: [PATCH?] $ENV{TEST_JOBS} vs $ENV{HARNESS_OPTIONS} by Bram
• Re: [PATCH?] $ENV{TEST_JOBS} vs $ENV{HARNESS_OPTIONS} by Rafael Garcia-Suarez