develooper Front page | perl.perl5.porters | Postings from January 2014

Re: [perl #120936] Not OK: perl 5.18.2 +RC4 on cygwin-thread-multi-64int1.7.25(0.27053) report 5.18.2-RC4 on cygwin (32bits)

Thread Next
From:
st├ęphane g. titard
Date:
January 14, 2014 06:53
Subject:
Re: [perl #120936] Not OK: perl 5.18.2 +RC4 on cygwin-thread-multi-64int1.7.25(0.27053) report 5.18.2-RC4 on cygwin (32bits)
Message ID:
CAL8915ZPwJeBRg63fgkYfTxFN3tQSpa3Cvvz3twJ-NFsEuOUPg@mail.gmail.com
On Tue, Jan 7, 2014 at 12:38 AM, James E Keenan via RT
<perlbug-followup@perl.org> wrote:
>
> On 1/5/14 8:21 PM, (via RT) wrote:
> > # New Ticket Created by
> > # Please include the string:  [perl #120936]
> > # in the subject line of all future correspondence about this issue.
> > #<URL: https://rt.perl.org/Ticket/Display.html?id=120936>
> >
> >
> > This is a build failure report for perl from sgtmadpm@gmail.com,
> > generated with the help of perlbug 1.39 running under perl 5.18.2.
> >
> >
> > This is the result of building perl 5.18.2-RC4 on cygwin 32 bits.
>
> Were you able to build perl on this configuration with 5.18.0 or 5.18.1?
>

Not at 100%.

> If not, do you know when you were last able to build perl successfully
> on this platform?

see my answer below.
>
> >
> > Note that Devel::CallChecker on CPAN does work but does not pass tests
> > on cygwin.
> >
> > Failed 4 tests out of 2340, 99.83% okay.
> >          ../ext/XS-APItest/t/call_checker.t
> >          ../lib/File/stat.t
> >          io/shm.t
> >          op/taint.t
>
> Thank you very much.
> Jim Keenan
>
>



Hi Jim,
Thanks for the quick feedback.
Happy new year and many many thanks to the porters and folks that made
5.18.2 possible.

IIRC BinGOS mentioned a bit before 5.18.2-RC1 a 100% build of the 5.18.x
on cygwin, and that surprised me as it has been a long time I haven't seen a
fully clean build.

I have seen Reini's message but I don't agree completely ;). In my dists I got
failures for shm.t since at least 5.14.1 (which I still keep around and use
for minicpan).

Anyway I did run prove -v on the failing tests.

File/stat.t   passes all tests but overall returns a non-0 code
              seems to be an issue with File::Temp. Maybe a race? when the
              END{} is run the dir is (about to be) gone.
              I straced this but did not see anything obvious.
              Running like Administrator gives 80 failed tests.

shm.t         SIGSYS right at the start. Weird considering configure probes
              for shm support and finds it. Maybe need a windows privilege.
              I did try running as Admin. but same thing.

taint.t       Bails out at 2/3 of the tests. And there is a (ess. empty)
              stackdump. This needs more research and an exe compiled with -g.
              Strictly speaking it is a security problem (but one could
              probably argue that about cygwin itself, still it would be nice
              not to segfault).

callchecker   No opinion. I suppose it's kind of related to Devel::CallChecker.
              (unrelated plug) I hope Zefram will fix the tests for cygwin
              but it does work.


---
quick summary of my cyg config (see D])

* ld2 g++ wrapper
* quick rebase after make before make test
* use HOME not USERPROFILE and co. it's un*x!
* use un*x paths the only ones with POSIX semantics (and options via mount)
* stat tests are in /tmp and are not affected by non-POSIX semantics (I hope)
  (my dist goes in /opt symlink to C:/xopt/opt which has not full POSIX
  semantics)

---

The question is:
if some tests are expected to fail why not SKIP or TODO them (for cygwin)?

The good news if that 5.18.2 fails less tests than the two previous versions.
5.16.3 which has been very stable for me had even more failures.

I did test 5.18.0 with a lot of extra modules but I found it a bit unstable:
* some locking issues
* some socket issues (sockets closing one end, hopefully fixed by Tony C.).
* some dists like Mojolicious and IO::Async had intermittent failures mostly
  timeout opers I think.
* the core had a lot re*.t failures (and this did not change in 5.18.1)

hth
p.s lots of details below (for ref).
There is IMHO a nasty bug affecting xsubpp described in D].



----------------------------- DETAILS -------------------------------------
There are four parts:
A] gives the test failures for some perl dists.
B] gives details of the failures of 5.18.2-RC4..
C] some comments about perlbug for cygwin.
D] recaps some particularities of the cygwin platform (snd my config).



A] failures for dists I have (before 5.18.2)

I have quite a few dists but only could find the tests results for recent
ones. I have used perl-5.16.2 extensively (with hundreds of CPAN modules)
and I know it's quite stable. The same is true for perl-5.16.3

* perl-5.16.1

    Failed 8 tests out of 2268, 99.65% okay.
        ../cpan/Win32/t/Unicode.t
        ../cpan/Win32API-File/t/file.t
        ../dist/Net-Ping/t/110_icmp_inst.t
        ../dist/Net-Ping/t/500_ping_icmp.t
        ../dist/Net-Ping/t/510_ping_udp.t
        ../ext/XS-APItest/t/call_checker.t
        io/shm.t
        op/taint.t

* perl-5.16.3

    Failed 9 tests out of 2266, 99.60% okay.
        ../cpan/Win32/t/Unicode.t
        ../cpan/Win32API-File/t/file.t
        ../dist/Net-Ping/t/110_icmp_inst.t
        ../dist/Net-Ping/t/500_ping_icmp.t
        ../dist/Net-Ping/t/510_ping_udp.t
        ../ext/XS-APItest/t/call_checker.t
        io/shm.t
        op/taint.t
        op/threads.t

note that my perl-5.16.3 now has the latest threads modules and does not fail
test op/threads.t

    % stephan@armen (/opt/perl_inst/perl5163_inst/perl-5.16.3/t) %
    % /opt/perl5163/bin/prove -v op/threads.t 2>&1 | tail -n 9
    ok 22 - Just special casing lexicals in ?{ ... }
    ok 23 - 0 refcnt during CLONE
    ok 24 - avoid peephole recursion
    ok 25 - Pipes shared between threads do not block when closed
    ok 26 - globs cloned and joined are not recloned
    ok
    All tests successful.
    Files=1, Tests=26,  9 wallclock secs ( 0.05 usr  0.14 sys +  5.94
cusr  2.64 csys =  8.77 CPU)
    Result: PASS

my cygwin1.dll was updated on

    % stephan@armen (/home/stephan) %
    % ll /bin/cygwin1.dll
    -rwxr-xr-x 1 stephan None 3115385 Aug 31 20:40 /bin/cygwin1.dll

so no use to shadow sitelib to try recover the tests ;) would not be the same
environment.

* finally for 5.18.0 I posted something on perlmonks. lots of test failures.

    http://www.perlmonks.org/?node_id=1034252

    Failed 26 tests out of 2336, 98.89% okay.
        ../cpan/Encode/t/CJKT.t
        ../cpan/Encode/t/at-cn.t
        ../cpan/Encode/t/perlio.t
        ../cpan/Pod-Simple/t/corpus.t
        ../cpan/Pod-Simple/t/pulltitl.t
        ../cpan/Text-Tabs/t/Tabs-ElCid.t
        ../cpan/Text-Tabs/t/Wrap-JLB.t
        ../cpan/podlators/t/man.t
        ../ext/XS-APItest/t/call_checker.t
        ../lib/File/stat.t
        io/shm.t
        op/bop.t
        op/taint.t
        op/threads.t
        re/fold_grind.t
        re/pat_rt_report.t
        re/pat_rt_report_thr.t
        re/regex_sets_compat.t
        re/regexp.t
        re/regexp_noamp.t
        re/regexp_notrie.t
        re/regexp_qr.t
        re/regexp_qr_embed.t
        re/regexp_qr_embed_thr.t
        re/regexp_trielist.t
        uni/overload.t

* 5.18.1 had IIRC 3or4 less failures (of the re*t group), but similar.
  Cannot find the data...

C] prove -v on failed tests

I ran this script:

    % stephan@armen (/home/stephan) %
    % cat probe_failing_tests_perl_5_18_2_RC4

    #!/bin/mksh

    IFS_DEF=$IFS NL='
    '
    trap clean EXIT
    temp="/tmp/${0##*/}00"
    function clean { [[ -f $temp._0 ]] && rm -f $temp._0; }
    function p_ { print -u2 "$@"; }
    function run_echo { print ".run-cmd [$@]"; "$@"; }

    cd /opt/perl_inst/perl5182_RC4_inst/perl-5.18.2-RC4/t || {
        p_ "cannot find instdir for perl 5.18.2-RC4";
        exit 2;
    }

    perlbin=/opt/perl5182/bin
    list='
    ../ext/XS-APItest/t/call_checker.t
    ../lib/File/stat.t                  25
    io/shm.t
    op/taint.t                          25
    '

    IFS=$NL; for testline in $list
    do
        # 1: testfile   2:lines to keep
        IFS=$IFS_DEF; set -- $testline

        echo "\n---\\/----------------------------------------------------------"
        echo ".testing $1..."
        run_echo "$perlbin/prove" -v "$1" 2>&1 | tail -n ${2:-15} #
quote paths;)
        # catch stderr :(
        err_msg=$("$perlbin/prove" -v "$1" 2>&1 1>&-)
        [[ -n $err_msg ]] && print ".err_msg: [$err_msg]"
    done

    #
    ## eof
    #



---\/----------------------------------------------------------
.testing ../ext/XS-APItest/t/call_checker.t...
Can't locate XS/APItest.pm in @INC (you may need to install the
XS::APItest module) (@INC contains:
/opt/perl5182/lib/site_perl/5.18.2/cygwin-thread-multi-64int
/opt/perl5182/lib/site_perl/5.18.2
/opt/perl5182/lib/5.18.2/cygwin-thread-multi-64int
/opt/perl5182/lib/5.18.2 .) at ../ext/XS-APItest/t/call_checker.t line
5.
BEGIN failed--compilation aborted at ../ext/XS-APItest/t/call_checker.t line 5.
# Looks like your test exited with 2 before it could output anything.
../ext/XS-APItest/t/call_checker.t ..
1..76
Dubious, test returned 2 (wstat 512, 0x200)
Failed 76/76 subtests

Test Summary Report
-------------------
../ext/XS-APItest/t/call_checker.t (Wstat: 512 Tests: 0 Failed: 0)
  Non-zero exit status: 2
  Parse errors: Bad plan.  You planned 76 tests but ran 0.
Files=1, Tests=0,  1 wallclock secs ( 0.05 usr  0.14 sys +  0.01 cusr
0.11 csys =  0.31 CPU)
Result: FAIL
.err_msg: [Can't locate XS/APItest.pm in @INC (you may need to install
the XS::APItest module) (@INC contains:
/opt/perl5182/lib/site_perl/5.18.2/cygwin-thread-multi-64int
/opt/perl5182/lib/site_perl/5.18.2
/opt/perl5182/lib/5.18.2/cygwin-thread-multi-64int
/opt/perl5182/lib/5.18.2 .) at ../ext/XS-APItest/t/call_checker.t line
5.
BEGIN failed--compilation aborted at ../ext/XS-APItest/t/call_checker.t line 5.
# Looks like your test exited with 2 before it could output anything.]

---\/----------------------------------------------------------
.testing ../lib/File/stat.t...
ok 4995 - no other warnings seen for -C for
/cygdrive/c/xopt/opt/perl5182/bin/perl
ok 4996 - Overload succeeds for -C under use filetest 'access' for
/cygdrive/c/xopt/opt/perl5182/bin/perl
ok 4997 - no warnings about VMS ACLs for -C under use filetest
'access' for /cygdrive/c/xopt/opt/perl5182/bin/perl
ok 4998 - should be no warning for -C under use filetest 'access' for
/cygdrive/c/xopt/opt/perl5182/bin/perl
ok 4999 - no other warnings seen for -C under use filetest 'access'
for /cygdrive/c/xopt/opt/perl5182/bin/perl
# Not testing -A for /cygdrive/c/xopt/opt/perl5182/bin/perl
ok 5000 - should build a stat object isa File::stat
ok 5001 - -t overload fails
ok 5002 - -T overload fails
ok 5003 - -B overload fails
ok 5004 - ... should be able to find filehandle isa File::stat
ok 5005 - ... and filehandle in another package isa File::stat
ok 5006 - ... and must match normal stat
ok 5007 - -p should be false on a file
ok 5008 - check -p detects a pipe
1..5008
Dubious, test returned 29 (wstat 7424, 0x1d00)
All 5008 subtests passed

Test Summary Report
-------------------
../lib/File/stat.t (Wstat: 7424 Tests: 5008 Failed: 0)
  Non-zero exit status: 29
Files=1, Tests=5008,  1 wallclock secs ( 0.42 usr  0.12 sys +  1.11
cusr  0.12 csys =  1.78 CPU)
Result: FAIL
.err_msg: [Cannot chdir to /tmp/rniYXdCaKx: Permission denied at
../lib/File/Temp.pm line 915.
END failed--call queue aborted.
# Looks like your test exited with 29 just after 5008.]

---\/----------------------------------------------------------
.testing io/shm.t...
.run-cmd [/opt/perl5182/bin/prove -v io/shm.t]
Caught a SIGSYS at ./test.pl line 247.
io/shm.t ..
1..
1..
Dubious, test returned 88 (wstat 22528, 0x5800)
No subtests run

Test Summary Report
-------------------
io/shm.t (Wstat: 22528 Tests: 0 Failed: 0)
  Non-zero exit status: 88
  Parse errors: No plan found in TAP output
Files=1, Tests=0,  0 wallclock secs ( 0.05 usr  0.14 sys +  0.01 cusr
0.09 csys =  0.30 CPU)
Result: FAIL
.err_msg: [Caught a SIGSYS at ./test.pl line 247.]

---\/----------------------------------------------------------
.testing op/taint.t...
ok 523
ok 524
ok 525
ok 526
ok 527
ok 528 - ge?cos
ok 529
ok 530 - shell
ok 531
ok 532
ok 533
ok 534
ok 535
ok 536
ok 537
Failed 260/797 subtests
        (less 8 skipped subtests: 529 okay)

Test Summary Report
-------------------
op/taint.t (Wstat: 140 Tests: 537 Failed: 0)
  Non-zero wait status: 140
  Parse errors: Bad plan.  You planned 797 tests but ran 537.
Files=1, Tests=537,  1 wallclock secs ( 0.09 usr  0.16 sys +  0.17
cusr  0.14 csys =  0.55 CPU)
Result: FAIL



C] comments on perlbug and cygwin

1] perlbug tries to use sendmail if it finds it. On cygwin the problem
is that it can be a link to something else...

    % stephan@armen (/home/stephan) %
    % ll /usr/sbin/sendmail
    lrwxrwxrwx 1 stephan None 16 Sep 18  2010 /usr/sbin/sendmail ->
/usr/bin/cronlog

2] To make a prepared message the incantation is not simple.
   but when you send (using whatever means you have) you need to extract
   the mail headers X-Message-id and the subject or it does not reach RT.

D] cygwin and cygperl

* Cygwin is a un*x(linux) platform on top of windows. cyg32 is the stable
platform but hard work is being done on cyg64.
There are may edge cases coming from the two directions: the implementation
of the un*x itself can be non-conforming, and the fact that it uses
windows libs underneath think e.g of network libs.

A version of cygwin has two components: the version of cygwin1.dll and the
version of windows.

    % stephan@armen (/home/stephan) %
    % uname -a
    CYGWIN_NT-6.1-WOW64 armen 1.7.25(0.270/5/3) 2013-08-31 20:39 i686 Cygwin

It is important to remember that! as soon as you update the cygwin1.dll then
if you rerun the tests of a previous install you might get different results.

* cygwin's released perl is lagging behind at 5.14.2 and there is not as much
testing of CPAN modules as it used to be. It is true that there are many options
 to run a unixy perl on windows (colinux, virtualized linux, linux on a stick).
Still for people who care about portability un*x/linux/windows cygperl
is a good bet.

* Fork is *weak* on cygwin for one main reason. For it to work a child process
must be able to load any dll at the same address at the parent. Rebasing is the
process of giving an adequate base address to an executable or dll.
The rebase process used by the setup cygwin program is much better now
as it builds a rebase database that is available for updates (done
through setup).

Still it is __only a stop-gap measure__ and the dynamic loader will load
a dll at another address if there is a base address clash; dll injection
made by certain windows sw can ruin the day.
The more cyg libs you use, the more likely to get clashes.
So the size of distribution can matter (both cygwin and cygperl).

Note that if you have many perl dists the rebase database is not useful.
It is much better to rebase at the same address all the equiv. dlls across all
your perls. That's why I use as LD ld2 a simple wrapper around g++
(LD=g++ is what
Reini U. uses in the stock cyg perl 5.14.2 config) giving me the
opportunity to rebase
using a prebuilt table each time a new dll is created (rebasing only that dll!).

Another good news is that cyg64 will be much safer as it will be easier to find
a range of addresses not used by anybody. An incremental rebase strategy as
described above could then be added to perlcore I think.

the bare ld2 I used is:

    % stephan@armen (/opt/perl_inst/perl5182_inst) %
    % cat /bin/ld2
    #!/bin/ksh
    # used to have a chance of rebasing after each shared library creation
    # should be put in perl bin later! (meanning after perl construction)
    function p_ { print -u2 "$@"; }
    p_ ".linker: [/usr/bin/g++ $@]"
    /usr/bin/g++ "$@"

    #
    ## eof
    #

* After running make on a fresh perl, people? normally do a *quick rebase*
before make test (where you choose a start address in an address window where
you don't expect too many clashes).

I use a script with:

    find . -type f -name '*.dll' | rebase -v -n -b 0x58000000 -T -

Reini's perlrebase is similar and does a bit more. What I really use in my
toolchain (above the toolchain) to install my favourite (hundreds of) CPAN
modules has been sketched above.

* I have seen code that checks Windows envvars before HOME.
This is wrong cygwin is a un*x!. I always wrap cpanm with a script that
unsets USERPROFILE and the such. It is easy to catch on a recent dist that uses
Build.pm say PocketIO, running cpanm --test-only ... I see this:



* A small note Win32 non-core addons. I usually build them just after core perl.
The "site_lib first in INC" move was a good move as it keeps the core intact
in case you need it.
But I found a gotcha, there is cargo-cult makemaker code for XS that uses
"perl -I... /path/to/xsubpp" where -I loads libraries in the lib/5*, bringing
(latent possibly silent) havoc after you update xsubpp. I mentioned this
here as this affected some of the Win32::* modules that Jan D maintains.

I had some hard time with this. I could not rebuild something I had built
before! (cygwin1.dll did not change).
This is a who-comes-first-in-path kind of problem but xsubpp should make sure
it loads its own libs not the ones of a previous version.
I caught it only because there was a recent move to OO-code and the module
did not compile (new could not be found).

* Filesystem weirdos. un*x slashy paths have un*x/POSIX semantics and follow
whatever option is used in the mount command. DOS-like paths don't follow
any POSIX (mount) semantics and should not be used.
There is some place in the guts that does not follow this as I often see the
nodosfilewarning warning in cpanminus logs (one day I tracked this and came to
the conclusion it was from perl's C code somewhere).

Finally a few comments on my own cygwin install...

* Since cygwin 1.7.x my cygwin dist goes into C:\cyg. At some point I
had two cygwin
coexisting and C\cygwin was home of the 1.5.x series.

    % mount
    C:/cyg/bin on /usr/bin type ntfs (binary,auto)
    C:/cyg/lib on /usr/lib type ntfs (binary,auto)
    C:/cyg on / type ntfs (binary,auto)
    C: on /cygdrive/c type ntfs (binary,posix=0,user,noumount,auto)
    Z: on /cygdrive/z type smbfs (binary,posix=0,user,noumount,auto)


* As I bundle the full cygwin dist for use in other machines, I try to separate
cygwin proper and my perl dists. So /opt is a link to C:\xopt\opt. I
wonder if this
can make a difference wrt tainting i.e  /opt/perl5182/bin/perl.exe
versus the value
held in $^X. For the stat stuff it does not matter as /tmp has the POSIX attrs.

    % stephan@armen (/home/stephan) %
    % /pb/perl -we 'print $^X'
    % /cygdrive/c/xopt/opt/perl5163/bin/perl

Note that I use /pXb /pXd for all dists. There are simply aliases to
/opt/perlXXX/bin and /opt/perlXXX resp.

--------------------------- END DETAILS --------------------------------------

Thread Next


nntp.perl.org: Perl Programming lists via nntp and http.
Comments to Ask Bjørn Hansen at ask@perl.org | Group listing | About