perl.qa

perl.qa http://www.nntp.perl.org/group/perl.qa/ ... Copyright 1998-2012 perl.org Wed, 16 May 2012 19:34:06 +0000 ask@perl.org Re: Tagging tests by Daniel Perrett Thanks, I'll look into both! Daniel On 1 May 2012 19:17, Lars Dɪᴇᴄᴋᴏᴡ 迪拉斯 <daxim@cpan.org> wrote: >> Is there any way to 'tag' tests in Perl? > <http://p3rl.org/Test::Less> > >> - If there are common problems (computer can't access the net, unicode >> handling is dodgy), this makes it more straightforward to diagnose >> than reading through logs of very long test scripts with lots of >> failure diagnostics > <http://analysis.cpantesters.org/> http://www.nntp.perl.org/group/perl.qa/2012/05/msg13183.html Tue, 01 May 2012 11:58:04 +0000 Re: Tagging tests by Lars Dɪᴇᴄᴋᴏᴡ 迪拉斯 > Is there any way to 'tag' tests in Perl? <http://p3rl.org/Test::Less> > - If there are common problems (computer can't access the net, unicode > handling is dodgy), this makes it more straightforward to diagnose > than reading through logs of very long test scripts with lots of > failure diagnostics <http://analysis.cpantesters.org/> http://www.nntp.perl.org/group/perl.qa/2012/05/msg13182.html Tue, 01 May 2012 11:17:33 +0000 Re: Devel::Cover problem: getting analysis of .xs file but not of.pmfile by James E Keenan On 4/25/12 9:08 PM, James E Keenan wrote: > Friends, > > > Can anyone advise as to why I am getting coverage analysis of Util.xs > but not of lib/Hash/Util.pm? > Locking %ENV frustrates Devel::Cover. Starting at approximately line 172 of t/Util.t, we have a test which, after I've reformatted a few lines, looks like this: lock_keys(%ENV); eval { () = $ENV{I_DONT_EXIST} }; like( $@, qr/^Attempt to access disallowed key 'I_DONT_EXIST' in a restricted hash/, 'locked %ENV' ); I commented out all the tests in Util.t, then worked my way through un-commenting them and making sure that all tests still passed. Until I got to the test above, I got favorable results calling: prove -vb t/Util.t cover -delete; harness; cover -summary ... where 'harness' simply calls: HARNESS_PERL_SWITCHES=-MDevel::Cover make test ... and "favorable results" means something like this: ---------------------- ------ ------ ------ ------ ------ ------ ------ File stmt bran cond sub pod time total ---------------------- ------ ------ ------ ------ ------ ------ ------ Util.xs 88.2 58.3 n/a n/a n/a n/a 70.7 blib/lib/Hash/Util.pm 82.9 59.1 11.1 87.5 92.0 100.0 78.8 Total 83.6 58.7 11.1 87.5 92.0 100.0 77.4 ---------------------- ------ ------ ------ ------ ------ ------ ------ But when I extended the uncommented lines to include the 'like' after lock_keys(%ENV), 'prove' still PASSed all tests but the line for 'blib/lib/Hash/Util.pm' vanished. When I kept the lock_keys lines commented out, but uncommented the balance of the file, both prove and cover DWIMmed. Hence, these few lines were the cause of the problem. So now this becomes an obscure bug in Devel::Cover: Why does lock_keys(%ENV) cause reporting on a package to vanish -- but not affect the reporting on the XS file? I'll leave that for others (read: pjcj) to ponder. In the meantime, I hope to add tests to shore up the surprisingly poor coverage of Hash::Util by its test file. Thank you very much. Jim Keenan http://www.nntp.perl.org/group/perl.qa/2012/04/msg13181.html Thu, 26 Apr 2012 17:54:09 +0000 Test::Builder 1.5.0 alpha 5 released by Michael G Schwern The fifth alpha for Test::Builder 1.5 has been released. It contains work from Matthew Horsfall, Peter Rabbitson, geistteufel, Karen Etheridge, Michael Schwern and others. It primarily addresses threading issues and regressions discovered via testing CPAN modules. https://metacpan.org/release/MSCHWERN/Test-Simple-1.005000_005/ https://github.com/schwern/test-more/ Please report any bugs or regressions in your CPAN modules at https://github.com/schwern/test-more/issues We very much want to hear about broken CPAN modules, CPAN is a very good test suite. Smokers, please load your alpha smokers with this new version. 1.005000_005 Thu Apr 26 15:23:25 PDT 2012 New Features * cmp_ok() will error when used with something which is not a comparison operator, including =, += and the like. [github 141] Bug Fixes * Using a reference as a test name works again. [github #264] * Protected against Test::More accidentally overwriting $! [github #268] [github #266] * Restored the behavior of a fork, it will not issue its own plan, making testing with fork easier. * Test::Builder->name() has been restored. [github #220] (Matthew Horsfall) * TB2::BlackHole no longer tries to AUTOLOAD its destructor. [github #280] (Peter Rabbitson) * Threads work on 5.12 and 5.10 again. (Peter Rabbitson) Doc Fixes * use_ok() has been discouraged and de-emphasized as a general replacement for `use` in tests. [github #288] Incompatible Changes with previous alphas * TB2::Counter has been removed [github 119] * The test counter has been moved from the TAP formatter into TB2::History [github 190] * TB2::TestState is no longer a complete subclass of TB2::EventCoordinator, but delegates most EC methods. The isa/can trickery caused threading issues. [github 291] * TB2::TestState->current_coordinator is now TB2::TestState->ec. * use_ok() will no longer apply lexical pragams. The incompatibilities and extra complexity is not worth the marginal use. [github #287] 1.005000_004 Sun Mar 25 15:01:49 BST 2012 Bug Fixes * Fix the thread crashes occurring with 5.12 and down. [github 261] Test Fixes * t/History/child_process.t was failing, the forked processes were improperly coordinated. Incompatible Changes with previous alphas * coordinate_threads() has been removed. It had no use case and was complicating threads which are quite complicated enough thank you. -- 31. Not allowed to let sock puppets take responsibility for any of my actions. -- The 213 Things Skippy Is No Longer Allowed To Do In The U.S. Army http://skippyslist.com/list/ http://www.nntp.perl.org/group/perl.qa/2012/04/msg13180.html Thu, 26 Apr 2012 15:46:28 +0000 Devel::Cover problem: getting analysis of .xs file but not of .pmfile by James E Keenan Friends, As an extension of my work on https://rt.perl.org/rt3/Ticket/Display.html?id=112126, I am trying to measure the test coverage for Perl 5 core distribution Hash-Util provided by the Perl 5 test suite. For ease of development, I have copied ext/Hash-Util/lib/Hash/Util.pm and ext/Hash-Util/t/Util.t into a typical CPAN distribution setup. I have applied the patches submitted in RT #112126 and modified t/Util.t to remove all the setup needed to run it as part of the Perl 5 test suite and instead run it as if it were a part of a regular CPAN distribution. So I'm now at a point where I want to perform coverage analysis. However, try as I might, I cannot get a report of the test coverage of lib/Hash/Util.pm I want -- and this having used Devel::Cover for nine years! Please see the attachment in which I show the output of: perl Makefile.PL && make && cover -test Note that I *am* getting coverage output for Util.xs, but I'm *not* getting coverage for lib/Hash/Util.pm. This is most puzzling. I have Devel::Cover version 0.80 installed and am running this with Perl 5.14.2. To facilitate getting assistance with this, I have push my quasi-distribution to Github: https://github.com/jkeenan/Hash-Util Can anyone advise as to why I am getting coverage analysis of Util.xs but not of lib/Hash/Util.pm? Thank you very much. Jim Keenan http://www.nntp.perl.org/group/perl.qa/2012/04/msg13179.html Wed, 25 Apr 2012 18:08:18 +0000 Re: Tagging tests by Greg Sabino Mullane On Tue, Apr 24, 2012 at 07:09:19PM +0100, Daniel Perrett wrote: > Is there any way to 'tag' tests in Perl? Not that I can think of, in the way that you want. > It looks likely that even though all the search tests fail, they are > failing because there is no working connection, as tested by the first > http request. Although five of the unicode tests are failing, three > aren't (throwing unicode characters at the syntax). Seems to me the easiest way to handle that is to have simple tests at the start of your chain that BAIL OUT if they see a major problem. While the unicode wouldn't be in that solution, I can't see how the complexity of adding tags would prevent a user from having to manually look at the failing tests anyway. > (I guess one answer could be 'write them in separate test scripts' but > what I want is tags (many-to-many) rather than categories > (many-to-one), and more files is a bit cumbersome.) Perhaps you can give a more convincing/real-world test case where this would be needed? I would think that a single early test catching many known major failures would be the way to go. If there was something, such as unicode, which warranted tags, I'm not sure why it wouldn't warrant it's own group of tests. Tests are the one are where I never worry about overlap, efficiency, or having too many files. :) -- Greg Sabino Mullane greg@endpoint.com End Point Corporation PGP Key: 0x14964AC8 http://www.nntp.perl.org/group/perl.qa/2012/04/msg13178.html Wed, 25 Apr 2012 11:45:17 +0000 Tagging tests by Daniel Perrett Is there any way to 'tag' tests in Perl? What I mean is, that ideally, if you tagged all your tests according to the functionality they depended on, you could use the tags to more easily work out what was going wrong. # Looks like you failed 21 tests of 75. # Failures by tag: # syntax: 0/50 # lwp: 21/25 # http: 21/21 # search: 20/20 # unicode: 5/8 This report is for some hypothetical module which imports some syntax and allows the user to run some searches over the web via an LWP object. Here, we can see at a glance that there is nothing wrong with the syntactic code, and the code which requires LWP itself is probably fine because the problem only comes when the user tries to make http requests. It looks likely that even though all the search tests fail, they are failing because there is no working connection, as tested by the first http request. Although five of the unicode tests are failing, three aren't (throwing unicode characters at the syntax). Assuming there isn't any way to 'tag' tests, like this, should there be? The problems I can see are: - How to make it easy to do syntactially, avoiding lots of repetition - Whether this requires a complete rewrite of the assumptions of the core test handling modules - The effort involved for test authors in tagging tests in this detail - Some test failures happen something unexpected went wrong, and tagging is only useful if we know what categories are useful. - It could encourage authors to be lazy But the advantages are - If there are common problems (computer can't access the net, unicode handling is dodgy), this makes it more straightforward to diagnose than reading through logs of very long test scripts with lots of failure diagnostics - Might be useful for coverage checking (you could write something asking if you have a test which has a particular combination of tags) - You can write more clever algorithms to try to pinpoint where the problem is (e.g. which combinations of tags always fail) - Makes TDD easier because you can write lots of tests which will fail (and you know you can ignore because they depend on code that isn't written yet), and you can still focus on the features you're writing right now. (I guess one answer could be 'write them in separate test scripts' but what I want is tags (many-to-many) rather than categories (many-to-one), and more files is a bit cumbersome.) Daniel http://www.nntp.perl.org/group/perl.qa/2012/04/msg13177.html Tue, 24 Apr 2012 11:09:31 +0000 Re: Testing Output and Functionality by Jed Lund Eirik, Thank you for the suggestion. I had not seen Test::Trap in my searches so I will investigate. Best Regards, Jed On Mon, Apr 23, 2012 at 11:44 PM, Eirik Berg Hanssen < Eirik-Berg.Hanssen@allverden.no> wrote: > On Tue, Apr 24, 2012 at 6:35 AM, Jed Lund <jandrewlund@gmail.com> wrote: > >> Hello, >> >> I was wondering if I could get some pointers towards a module or package >> that would allow me to test the functionality of various elements of code >> that I was developing while I also tested the STDERR and STDOUT output >> from >> that element of code? I have been hunting around CPAN for the right fit >> but so far I have only found STDERR/STDOUT tests or functionality tests >> not >> a way to test both results from the same method call. >> > > This sounds like a job for ... Test::Trap! > > > The most basic case would be to test the output of a method that returned >> some defined result while also capturing any warnings prompted by >> malformed >> input which did not cause catastrophic failure. The goal would be to >> certify the output and the warning message from the same method call. >> > > > my $result = trap { $obj->method(@args) }; # or my @results, for list > context > # Possibly: > $trap->warn_is_deeply( \@expected_warnings, "Got exactly the expected > warnings?" ); > # ... but usually more usefully: > $trap->stderr_like( $pattern, "Trapped stderr matches expectations" ); > # ... and you may still test $trap->stdout, $result/@result (or > $trap->return), and whatever else, from the same call. > > > Eirik > http://www.nntp.perl.org/group/perl.qa/2012/04/msg13176.html Tue, 24 Apr 2012 09:28:05 +0000 Re: Testing Output and Functionality by Jed Lund David, Thank you for your suggestion. I haven't gone down that road yet so I will try it. Best Regards, Jed On Tue, Apr 24, 2012 at 4:07 AM, David Golden <xdaveg@gmail.com> wrote: > On Tue, Apr 24, 2012 at 12:35 AM, Jed Lund <jandrewlund@gmail.com> wrote: > > I understand that this is a fairly basic question but I am stuck trying > to > > find the right path forward. > > It's not entirely basic, because you're testing an array of three > "outputs" for each functionality test instead of just one. > > I'd probably use Capture::Tiny: > > use Capture::Tiny qw/capture/; > > my ($out, $err, $result) = capture { your_code_here }; > # then test $out, $err, $result however you want > > If that's repetitive, stick it in a loop or write a function to > abstract testing the three results against your expectations. > > -- David > http://www.nntp.perl.org/group/perl.qa/2012/04/msg13175.html Tue, 24 Apr 2012 09:28:03 +0000 Re: Testing Output and Functionality by David Golden On Tue, Apr 24, 2012 at 12:35 AM, Jed Lund <jandrewlund@gmail.com> wrote: > I understand that this is a fairly basic question but I am stuck trying to > find the right path forward. It's not entirely basic, because you're testing an array of three "outputs" for each functionality test instead of just one. I'd probably use Capture::Tiny: use Capture::Tiny qw/capture/; my ($out, $err, $result) = capture { your_code_here }; # then test $out, $err, $result however you want If that's repetitive, stick it in a loop or write a function to abstract testing the three results against your expectations. -- David http://www.nntp.perl.org/group/perl.qa/2012/04/msg13174.html Tue, 24 Apr 2012 04:08:26 +0000 Re: Testing Output and Functionality by Eirik Berg Hanssen On Tue, Apr 24, 2012 at 6:35 AM, Jed Lund <jandrewlund@gmail.com> wrote: > Hello, > > I was wondering if I could get some pointers towards a module or package > that would allow me to test the functionality of various elements of code > that I was developing while I also tested the STDERR and STDOUT output from > that element of code? I have been hunting around CPAN for the right fit > but so far I have only found STDERR/STDOUT tests or functionality tests not > a way to test both results from the same method call. > This sounds like a job for ... Test::Trap! The most basic case would be to test the output of a method that returned > some defined result while also capturing any warnings prompted by malformed > input which did not cause catastrophic failure. The goal would be to > certify the output and the warning message from the same method call. > my $result = trap { $obj->method(@args) }; # or my @results, for list context # Possibly: $trap->warn_is_deeply( \@expected_warnings, "Got exactly the expected warnings?" ); # ... but usually more usefully: $trap->stderr_like( $pattern, "Trapped stderr matches expectations" ); # ... and you may still test $trap->stdout, $result/@result (or $trap->return), and whatever else, from the same call. Eirik http://www.nntp.perl.org/group/perl.qa/2012/04/msg13173.html Tue, 24 Apr 2012 01:01:47 +0000 Re: Testing Output and Functionality by Jed Lund Andy, Thank you for the reply. I did try Test::Output but I ran into a Smart::Comments incompatibility. This is a sort of corner case to the issue I described where I am trying to print the Smart::Comments but capture the STDOUT output for testing. When I run the following; use Test::Most; use Test::Output; use Smart::Comments '###'; stdout_is( sub{ print "Hello World\n"; ### Goodby World ... }, "Hello World\n", 'Test Hello World' ); done_testing(); I get the following error; Can't locate object method "TELL" via package "Test::Output::Tie" at C:/Perl/site/lib/Smart/Comments.pm line 438. While a warn or carp statement in the same place seems to work just fine the Smart::Comments don't. I kind of walked away at that point and started looking somewhere else. Should I just be more persistent to fix this issue or is there another solution? Best Regards, Jed On Mon, Apr 23, 2012 at 9:53 PM, Andy Lester <andy@petdance.com> wrote: > > On Apr 23, 2012, at 11:35 PM, Jed Lund wrote: > > that element of code? I have been hunting around CPAN for the right fit > but so far I have only found STDERR/STDOUT tests or functionality tests not > a way to test both results from the same method call. > > > Sounds like you're looking for Test::Output. > > http://search.cpan.org/dist/Test-Output/ > > xoxo, > Andy > > -- > Andy Lester => andy@petdance.com => www.petdance.com => AIM:petdance > > http://www.nntp.perl.org/group/perl.qa/2012/04/msg13172.html Tue, 24 Apr 2012 01:01:46 +0000 Re: Testing Output and Functionality by Andy Lester On Apr 23, 2012, at 11:35 PM, Jed Lund wrote: > that element of code? I have been hunting around CPAN for the right fit > but so far I have only found STDERR/STDOUT tests or functionality tests not > a way to test both results from the same method call. Sounds like you're looking for Test::Output. http://search.cpan.org/dist/Test-Output/ xoxo, Andy -- Andy Lester => andy@petdance.com => www.petdance.com => AIM:petdance http://www.nntp.perl.org/group/perl.qa/2012/04/msg13171.html Mon, 23 Apr 2012 21:53:19 +0000 Testing Output and Functionality by Jed Lund Hello, I was wondering if I could get some pointers towards a module or package that would allow me to test the functionality of various elements of code that I was developing while I also tested the STDERR and STDOUT output from that element of code? I have been hunting around CPAN for the right fit but so far I have only found STDERR/STDOUT tests or functionality tests not a way to test both results from the same method call. The most basic case would be to test the output of a method that returned some defined result while also capturing any warnings prompted by malformed input which did not cause catastrophic failure. The goal would be to certify the output and the warning message from the same method call. I understand that this is a fairly basic question but I am stuck trying to find the right path forward. Best Regards, Jed http://www.nntp.perl.org/group/perl.qa/2012/04/msg13170.html Mon, 23 Apr 2012 21:43:59 +0000 use_ok tasks by Michael G Schwern I've cut three tasks related to the use_ok() discussion, should people be interested in doing some small work to that end. Revert use_ok() back to its 0.98 features https://github.com/schwern/test-more/issues/287 Discourage use_ok() in the docs https://github.com/schwern/test-more/issues/288 Add test_history() to Test::More to access the History object (this makes it much easier to bail on fail) https://github.com/schwern/test-more/issues/286 -- 191. Our Humvees cannot be assembled into a giant battle-robot. -- The 213 Things Skippy Is No Longer Allowed To Do In The U.S. Army http://skippyslist.com/list/ http://www.nntp.perl.org/group/perl.qa/2012/04/msg13169.html Mon, 23 Apr 2012 12:34:50 +0000 Re: Revert use_ok() change to allow lexical effects? by Michael G Schwern On 2012.4.14 4:03 AM, Aristotle Pagaltzis wrote: >> Which is a bit of a mouthful. Providing a function to get at the >> history object would make a bunch of test state introspection easier. >> >> # That looks pretty good. >> END { BAIL_OUT() unless test_history->test_was_successful } > > I like that. Go at it. https://github.com/schwern/test-more/issues/286 -- package Outer::Space; use Test::More tests => 9; http://www.nntp.perl.org/group/perl.qa/2012/04/msg13168.html Mon, 23 Apr 2012 11:33:37 +0000 Re: Thanks for supporting the Perl QA Hackathon 2012 by L Boivin Hello. I am ashamed. I forgot some sponsors. I've no excuse. As Community sponsor, I want also thank : Stiching Perl Promotie Nederland - http://www.sppn.nl/ We had also in-kind donations: some companies and organizations sponsored directly some of our attendees, paying for their travel fees and accomodation: Procura - http://procura.nl/ Booking.com - http://www.booking.com/ Dyn - http://www.dyn.com/ Dijkmat - http://www.dijkmat.nl/ Shadowcat Systems - http://www.shadow.cat/ The Perl Foundation - http://www.perlfoundation.org/ Many thanks for your help. Many thanks to all the attendees too, for the job done! Kind Regards Laurent, ashamed On 15/04/2012 18:13, L Boivin wrote: > Hello, > > The Perl QA Hackathon 2012 is now over. The event took place 2 weeks > ago, for three days, and was well received by the attendees both on > the technical side (having all of them in the same location) and on > the social and logistical side (thanks to your support). > > In the end, 40 people from 11 countries attended the hackathon. These > people are well-known in the Perl community for their interest in and > their knowledge of Perl, Quality Assurance and the CPAN toolchain. > They spent three days together in Paris, in the "Carrefour Numérique" > of "La Cité des Sciences et de l'Industrie-Universcience", debugging, > improving, creating processes, tools, test environments for the good > of the Perl community. > > The results of the week-end are listed on the hackathon website: > http://2012.qa-hackathon.org/qa2012/wiki?node=Results > > Such results would never have been possible without the logistical > support provided to our hackers. We would never have been able to > provide it without you, our sponsors for the Perl QA Hackathon 2012. > > On behalf of the Perl community, I would like to thank you for your support. > > The money collected has been spent on travel (32%) and hotel (20%) > refunds for some attendees, catering (34%) and bank fees. The > surplus (10 %) will be donated to the organizers of the Perl > QA Hackathon 2013, which will be held in London. The final budget > will be available in a few days on the hackathon website. > > Among the wiki pages added at the end of the week-end, a page > collecting blog posts has been started: > http://2012.qa-hackathon.org/qa2012/wiki?node=Blogs > It collects links to blog posts that talk about the event. > > Once again, the Perl QA Hackathon 2012 participants and organizers, > as well as the Perl community as a whole, would like to thank for > the support you have provided to this event. > > Our partner:: > Le Carrefour Numérique - http://www.cite-sciences.fr/ > > Our "corporate" sponsors: > diabolocom - http://www.diabolocom.com/ > Dijkmat - http://www.dijkmat.nl/ > DuckDuckGo - http://www.duckduckgo.com/ > Dyn - http://www.dyn.com/ > Feeside - http://freeside.biz/ > Hedera Technology - http://www.hederatech.com/ > Jaguar Network - http://www.jaguar-network.com/ > Shadowcat Systems - http://www.shadow.cat > Splio - http://www.splio.com/ > TECLIB' - http://www.teclib.com/ > weborama - http://weborama.com/ > > Nos "Community" sponsors: > EPO - http://www.enlightenedperl.org/ > $foo Magazin - http://www.perl-magazin.de/ > Les Mongueurs de Perl - http://www.mongueurs.net/ > > Our donators: > Martin Evans > Mark Keating > Prakash Kailasa > Neil Bowers > 加藤 敦 (Ktat) > Karen Pauley > Chad Davis > Franck Cuny > 近藤嘉雪 > Tomohiro Hosaka > Syohei Yoshida > 牧 大輔 (lestrrat) > > > A big thank to all of you! > > Kind Regards > Laurent Boivin > Philippe Bruhat > for the Perl QA Hackathon 2012 > > -- Laurent Boivin -- mailto:l.boivin@free.fr Member of Grenoble.pm, Paris.pm http://www.nntp.perl.org/group/perl.qa/2012/04/msg13167.html Sat, 21 Apr 2012 10:56:46 +0000 [PQAH2012] Financial Report -- final by L Boivin Hello, This is the financial report of the Perl QA Hackathon 2012. It was a nice exercise of accountancy :-) I'm happy to announce that the Perl QA Hackathon 2013 will start with € 1495.66, let me know when/how you want it. If you're interested in the details, here's the spreadsheet I used (LibreOffice), with details and explantions: http://dl.dropbox.com/u/44168013/PQAH2012_financial_report.ods I've also updated the Budget page on the web site. It'll appear at the next synchro (at midnight, CEST). Preview on the test website (http://test.mongueurs.net/qa2012/budget.html) Kind Regards Laurent -- Laurent Boivin -- mailto:l.boivin@free.fr Member of Grenoble.pm, Paris.pm http://www.nntp.perl.org/group/perl.qa/2012/04/msg13166.html Sat, 21 Apr 2012 10:49:17 +0000 Re: Revert use_ok() change to allow lexical effects? by Aristotle Pagaltzis Hi Michael, * Michael G Schwern <schwern@pobox.com> [2012-04-13 04:00]: > On 2012.4.11 1:01 PM, Aristotle Pagaltzis wrote: > >>Unless I'm mistaken, Test::AutoBailOut is doing to need a global > >>$SIG{__DIE__} handler or override CORE::require or add something to > >>@INC or try to parse exception messages or something like that. Any > >>of those have global side effects which can potentially interfere with > >>or be overwritten by the code being tested. > > > >… what for? Is Ovid’s solution of just using an END block insufficient? > >Why? > > You're referring to this? > http://www.nntp.perl.org/group/perl.qa/2012/04/msg13128.html yes. > If all you want is to bail out if the test fails to complete, that's > easy(er). That’s all AutoBailout does, it is just Ovid’s code packaged up as a tiny module. That is why I suggested shipping it with Test::More. I cannot imagine it ever causing a burden (though I’m fine with shipping it separately too). Just because it does so little, bordering on nothing. > I thought it was going to do something specifically when a `use` > fails, thus all the complexity. Well, does it need to? Test reports will contain STDERR anyway. All you lose is that the error message is a bit more raw. > A general "bail out if this test file fails" function might be handy. > An even more general "do X if the test finishes in Y state" would be > even more handy. > > # There is no test_passed() > END { BAIL_OUT() unless test_passed(); } > > I don't want to add a huge bunch of functions to Test::More to cover > all the possible test states, in 1.5 you can get them from the > TB2::History object. > > # This works in TB 1.5 > END { BAIL_OUT() unless Test::More->builder->history->test_was_successful } > > Which is a bit of a mouthful. Providing a function to get at the > history object would make a bunch of test state introspection easier. > > # That looks pretty good. > END { BAIL_OUT() unless test_history->test_was_successful } I like that. Regards, -- Aristotle Pagaltzis // <http://plasmasturm.org/> http://www.nntp.perl.org/group/perl.qa/2012/04/msg13165.html Sat, 14 Apr 2012 01:03:12 +0000 Re: Revert use_ok() change to allow lexical effects? by Michael G Schwern On 2012.4.11 1:01 PM, Aristotle Pagaltzis wrote: >> Unless I'm mistaken, Test::AutoBailOut is doing to need a global >> $SIG{__DIE__} handler or override CORE::require or add something to >> @INC or try to parse exception messages or something like that. Any >> of those have global side effects which can potentially interfere with >> or be overwritten by the code being tested. > > … what for? Is Ovid’s solution of just using an END block insufficient? > Why? You're referring to this? http://www.nntp.perl.org/group/perl.qa/2012/04/msg13128.html If all you want is to bail out if the test fails to complete, that's easy(er). I thought it was going to do something specifically when a `use` fails, thus all the complexity. A general "bail out if this test file fails" function might be handy. An even more general "do X if the test finishes in Y state" would be even more handy. # There is no test_passed() END { BAIL_OUT() unless test_passed(); } I don't want to add a huge bunch of functions to Test::More to cover all the possible test states, in 1.5 you can get them from the TB2::History object. # This works in TB 1.5 END { BAIL_OUT() unless Test::More->builder->history->test_was_successful } Which is a bit of a mouthful. Providing a function to get at the history object would make a bunch of test state introspection easier. # That looks pretty good. END { BAIL_OUT() unless test_history->test_was_successful } -- Look at me talking when there's science to do. When I look out there it makes me glad I'm not you. I've experiments to be run. There is research to be done On the people who are still alive. -- Jonathan Coulton, "Still Alive" http://www.nntp.perl.org/group/perl.qa/2012/04/msg13164.html Thu, 12 Apr 2012 18:55:02 +0000 Re: Revert use_ok() change to allow lexical effects? by Michael G Schwern On 2012.4.11 1:45 PM, Andy Lester wrote: > test_requires is Module::Build only, right? I don't use Module::Build. It was just added to ExtUtils::MakeMaker but I don't know when a stable will be released. -- There will be snacks. http://www.nntp.perl.org/group/perl.qa/2012/04/msg13163.html Thu, 12 Apr 2012 18:38:06 +0000 Re: Revert use_ok() change to allow lexical effects? by Eirik Berg Hanssen On Wed, Apr 11, 2012 at 6:32 PM, Michael G Schwern <schwern@pobox.com>wrote: > On 2012.4.10 6:21 PM, The Sidhekin wrote: > > * How would you rewrite a test script such as my own > > http://cpansearch.perl.org/src/EBHANSSEN/Test-Trap-v0.2.2/t/00-load.t so > > that it does not use use_ok()? > > use Test::More tests => 1; > > use Test::Trap::Builder::TempFile; > *cut* If this fails, the test script will terminate immediately: * I won't get to know if any of the other modules loaded correctly, or how they fail. Less of the interesting output. * And there will be no BAIL_OUT, so the rest of the tests will run, burying the interesting output. More uninteresting output. Which are two very annoying changes in your rewrite. And of course, there's more of the same in the part of the rewrite I cut. > * Why would you? :-\ > > Because it reads like normal Perl and doesn't rely on more code to do > something rather simple: load a module. > But it fails to DWIW: report clearly on failures. Perhaps what it is doing is not so simple, after all? > But quite honestly, if use_ok() works for you, if all you need is require + > import + test, continue to use it. I don't plan on eliminating it. > Thank you. Eirik http://www.nntp.perl.org/group/perl.qa/2012/04/msg13162.html Thu, 12 Apr 2012 01:36:45 +0000 Re: 00-load.t tool by Buddy Burden Guys, > Looking at this a different way, instead of a library, make a distzilla > extension (or whatever) which generates (and regenerates) a 00-load.t > as per Ovid's earlier example. > : > : This sounds like the best idea to me.             -- Buddy http://www.nntp.perl.org/group/perl.qa/2012/04/msg13161.html Wed, 11 Apr 2012 18:15:52 +0000 Re: Revert use_ok() change to allow lexical effects? by James E Keenan On 4/11/12 12:33 PM, Michael G Schwern wrote: > > Let me reiterate, I have no plans to *deprecate* `use_ok`. Even if I wanted > to there are simply too many users to make deprecation worth while. > > It works fine if what you want is a runtime require + import + assert, and > sometimes you want that. The problem is it's been overused and has come to > replace a simple `use` in test scripts. To that end the question is whether > to *discourage* its use in the documentation: to scale it back from "use this > when you load a module" to "use this for some special cases". > > +1 to discouraging use_ok http://www.nntp.perl.org/group/perl.qa/2012/04/msg13160.html Wed, 11 Apr 2012 17:01:21 +0000 Re: Revert use_ok() change to allow lexical effects? by Andy Lester On Apr 11, 2012, at 5:48 PM, Aristotle Pagaltzis wrote: > The stop energy > he is throwing at it has no substantive reason so far, only “I don’t > care for this”. No, no, no, I'm sorry, I didn't mean to make it sound like "You shouldn't do this module." No stop energy intended. I just wanted to provide a counter viewpoint from a potential customer. I don't want to have another module prereq. I'd probably just do the cut & paste into my t/00-load.t. Carry on! xoa -- Andy Lester => andy@petdance.com => www.petdance.com => AIM:petdance http://www.nntp.perl.org/group/perl.qa/2012/04/msg13159.html Wed, 11 Apr 2012 16:24:31 +0000 Re: Revert use_ok() change to allow lexical effects? by Aristotle Pagaltzis * Michael Peters <mpeters@plusthree.com> [2012-04-11 23:15]: > On 04/11/2012 04:45 PM, Andy Lester wrote: > >test_requires is Module::Build only, right? I don't use > >Module::Build. > > No, I'm pretty sure it works with Module::Install and > ExtUtils::MakeMaker now too. Although that might just be something > that was worked on at the qa-hackathon. If the issue is really the extra download (for whatever reason that isn’t obvious to me), Andy can also just copy it into t/lib. It’s one dozen-line pure-perl file that isn’t likely to ever change, so… Might as well have it on CPAN instead of pasting a recipe into the Test::More docs and then having everyone paste that into their test scripts in turn. > >Even if I did, I don't think I'd require the user to go through > >a download and temporary build of AutoBailout.pm just to remove > >boilerplate. > > That's definitely a reasonable position to take. Sure. Same can be said of many Test modules. Test::Exception or just make do with just an `eval` and some logic and a few extra tests? Test::Deep or just stick with the worse output of `is_deeply` when it suffices? Etc. Every author can make each of these choices whichever way they like, and so can Andy. Likewise with AutoBailout. The stop energy he is throwing at it has no substantive reason so far, only “I don’t care for this”. Regards, -- Aristotle Pagaltzis // <http://plasmasturm.org/> http://www.nntp.perl.org/group/perl.qa/2012/04/msg13158.html Wed, 11 Apr 2012 15:48:14 +0000 Re: Revert use_ok() change to allow lexical effects? by Michael Peters On 04/11/2012 04:45 PM, Andy Lester wrote: > test_requires is Module::Build only, right? I don't use Module::Build. No, I'm pretty sure it works with Module::Install and ExtUtils::MakeMaker now too. Although that might just be something that was worked on at the qa-hackathon. > Even if I did, I don't think I'd require the user to go through a > download and temporary build of AutoBailout.pm just to remove boilerplate. That's definitely a reasonable position to take. -- Michael Peters Plus Three, LP http://www.nntp.perl.org/group/perl.qa/2012/04/msg13157.html Wed, 11 Apr 2012 14:11:01 +0000 Re: Revert use_ok() change to allow lexical effects? by Andy Lester On Apr 11, 2012, at 3:38 PM, Michael Peters wrote: >> As a module author, I would not require a user to install AutoBailout.pm just to remove boilerplate in my t/00-load.t > > With the test_requires stuff they won't have to. It will only be used for testing and not installed permanently on their system. test_requires is Module::Build only, right? I don't use Module::Build. Even if I did, I don't think I'd require the user to go through a download and temporary build of AutoBailout.pm just to remove boilerplate. xoa -- Andy Lester => andy@petdance.com => www.petdance.com => AIM:petdance http://www.nntp.perl.org/group/perl.qa/2012/04/msg13156.html Wed, 11 Apr 2012 13:45:14 +0000 Re: Revert use_ok() change to allow lexical effects? by Aristotle Pagaltzis * Andy Lester <andy@petdance.com> [2012-04-11 22:30]: > As a module author, I would not require a user to install > AutoBailout.pm just to remove boilerplate in my t/00-load.t If it’s only a test_requires the user won’t have to. (It would be pretty silly to keep discarding it though if it gets widespread use, esp seeing as it’s only a dozen lines of pure Perl.) > But that's just me. Yup. And it’s a free country. :-) Regards, -- Aristotle Pagaltzis // <http://plasmasturm.org/> http://www.nntp.perl.org/group/perl.qa/2012/04/msg13155.html Wed, 11 Apr 2012 13:43:43 +0000 Re: Revert use_ok() change to allow lexical effects? by Michael Peters On 04/11/2012 04:29 PM, Andy Lester wrote: > As a module author, I would not require a user to install AutoBailout.pm just to remove boilerplate in my t/00-load.t With the test_requires stuff they won't have to. It will only be used for testing and not installed permanently on their system. -- Michael Peters Plus Three, LP http://www.nntp.perl.org/group/perl.qa/2012/04/msg13154.html Wed, 11 Apr 2012 13:38:09 +0000 Re: Revert use_ok() change to allow lexical effects? by Andy Lester On Apr 11, 2012, at 3:01 PM, Aristotle Pagaltzis wrote: > OK. Then how about I stick AutoBailout on CPAN with a SYNOPSIS that > covers `t/00-load.t`, and then you change the `use_ok` docs to a) > discourage its use and b) point to AutoBailout for `t/00-load.t` uses. > Sound good? As a module author, I would not require a user to install AutoBailout.pm just to remove boilerplate in my t/00-load.t But that's just me. xoa -- Andy Lester => andy@petdance.com => www.petdance.com => AIM:petdance http://www.nntp.perl.org/group/perl.qa/2012/04/msg13153.html Wed, 11 Apr 2012 13:29:36 +0000 Re: Revert use_ok() change to allow lexical effects? by Aristotle Pagaltzis Btw Ovid, * Ovid <publiustemp-perlqa3@yahoo.com> [2012-04-11 19:10]: > done_testing is applicable to every test module and solves the far > more common issue of hating maintain a plan. ^^^^^^ a little Freudian slip there? :-) http://www.nntp.perl.org/group/perl.qa/2012/04/msg13152.html Wed, 11 Apr 2012 13:13:10 +0000 Re: Revert use_ok() change to allow lexical effects? by Aristotle Pagaltzis * Ovid <publiustemp-perlqa3@yahoo.com> [2012-04-11 21:55]: > >From: Michael G Schwern <schwern@pobox.com> > >Personally I'm a fan of "scroll up and read the first failure".  It > >always works! > > Try it on a Test::Class test suite running thousands of test in > a single process, whizzing past on your terminal :)  That is not relevant to t/00-load.t though, nor applicable when BAIL_OUT is involved, and here we are considering both of these. Regards, -- Aristotle Pagaltzis // <http://plasmasturm.org/> http://www.nntp.perl.org/group/perl.qa/2012/04/msg13151.html Wed, 11 Apr 2012 13:03:49 +0000 Re: Revert use_ok() change to allow lexical effects? by Aristotle Pagaltzis * Michael G Schwern <schwern@pobox.com> [2012-04-11 20:10]: > On 2012.4.11 9:53 AM, Aristotle Pagaltzis wrote: > >I don’t see how it is any more magic than `done_testing`. > > done_testing() has no global side effects, it's just a function. > > Unless I'm mistaken, Test::AutoBailOut is doing to need a global > $SIG{__DIE__} handler or override CORE::require or add something to > @INC or try to parse exception messages or something like that. Any > of those have global side effects which can potentially interfere with > or be overwritten by the code being tested. … what for? Is Ovid’s solution of just using an END block insufficient? Why? > > My suggestion to ship AutoBailOut was so you would be able to > > suggest it as a replacement in the docs, as it covers the one case > > where `use_ok` is even of interest (though still not the right > > solution). > > > > But I guess you could do that even if it ships outside of > > Test::More. > > Precisely. For example, it already suggests Test::Differences and > Test::Deep. OK. Then how about I stick AutoBailout on CPAN with a SYNOPSIS that covers `t/00-load.t`, and then you change the `use_ok` docs to a) discourage its use and b) point to AutoBailout for `t/00-load.t` uses. Sound good? Regards, -- Aristotle Pagaltzis // <http://plasmasturm.org/> http://www.nntp.perl.org/group/perl.qa/2012/04/msg13150.html Wed, 11 Apr 2012 13:01:38 +0000 Re: Revert use_ok() change to allow lexical effects? by Ovid ----- Original Message ----- > From: Michael G Schwern <schwern@pobox.com> >>   But it fails to DWIW: report clearly on failures.  Perhaps what it is > doing >> is not so simple, after all? > > Personally I'm a fan of "scroll up and read the first failure".  > It always works! Try it on a Test::Class test suite running thousands of test in a single process, whizzing past on your terminal :)  Cheers, Ovid -- Live and work overseas - http://www.overseas-exile.com/ Buy the book - http://www.oreilly.com/catalog/perlhks/ Tech blog - http://blogs.perl.org/users/ovid/ Twitter - http://twitter.com/OvidPerl/ http://www.nntp.perl.org/group/perl.qa/2012/04/msg13149.html Wed, 11 Apr 2012 12:51:19 +0000 Re: Revert use_ok() change to allow lexical effects? by Michael G Schwern On 2012.4.11 11:43 AM, Eirik Berg Hanssen wrote: > If this fails, the test script will terminate immediately: > > * I won't get to know if any of the other modules loaded correctly, or how > they fail. Less of the interesting output. > * And there will be no BAIL_OUT, so the rest of the tests will run, burying > the interesting output. More uninteresting output. Like I said, if use_ok() is working for you keep using it. Your example where you want to load a bunch of modules just to make sure they compile is one of those cases where use_ok() is probably the right thing. > > * Why would you? :-\ > > Because it reads like normal Perl and doesn't rely on more code to do > something rather simple: load a module. > > But it fails to DWIW: report clearly on failures. Perhaps what it is doing > is not so simple, after all? Personally I'm a fan of "scroll up and read the first failure". It always works! -- 7. Not allowed to add "In accordance with the prophesy" to the end of answers I give to a question an officer asks me. -- The 213 Things Skippy Is No Longer Allowed To Do In The U.S. Army http://skippyslist.com/list/ http://www.nntp.perl.org/group/perl.qa/2012/04/msg13148.html Wed, 11 Apr 2012 12:27:18 +0000 Re: 00-load.t tool by Eric Wilhelm # from Michael G Schwern # on Wednesday 11 April 2012 11:06: >> What exactly makes you uneasy? Maybe there is a way to address that >> if you can be more specific. > >Mostly that it's cramming yet more complexity into core test library > for a fairly narrow use case and functionality that lives quite > happily in its own module. Â It's already difficult enough to keep > stable. > >To reverse it, why should it be in Test::More? Â "People don't like to > have extra test dependencies" is not accepted. Looking at this a different way, instead of a library, make a distzilla extension (or whatever) which generates (and regenerates) a 00-load.t as per Ovid's earlier example. The trouble with having it in a library is not just that you have a test dependency, but also that you have to add more complexity to the library (and then you have a test dependency on a newer version) whenever someone wants to do a complex thing. Many modules in a distribution may not load unless $optional_dependency is satisfied. But, it is good to precheck all of the modules in a distribution before starting tests (aside: did someone say prefork?) and helps to wrap this in some useful diagnostic + bail out. I think something which embeds code in a 00-load.t, plus helps you maintain a list of modules by scanning lib/ would be a handy way to do that. (Perhaps a list + hash to address the optional dependencies and each optional module has a subref to determine if it is enabled.) One could argue that the functionality should be included in each test file to keep things orthogonal. Such a one may find that there may be a way to require("./t/00-load.t") from each test file, but might also end up with a library and/or a hybrid with regenerative breaking (prefork.) --Eric -- But you can never get 3n from n, ever, and if you think you can, please email me the stock ticker of your company so I can short it. --Joel Spolsky --------------------------------------------------- http://scratchcomputing.com --------------------------------------------------- http://www.nntp.perl.org/group/perl.qa/2012/04/msg13147.html Wed, 11 Apr 2012 11:27:05 +0000 Re: Revert use_ok() change to allow lexical effects? by Michael G Schwern On 2012.4.11 9:53 AM, Aristotle Pagaltzis wrote: > * Michael G Schwern <schwern@pobox.com> [2012-04-11 18:35]: >> Nope, too much magic for too small a use case. > > And faithfully duplicating `use` would be less so? :-) This discussion is about backing away from that. The most magical bits of use_ok() have not yet gone out in a stable and can be reverted. > I don’t see how it is any more magic than `done_testing`. done_testing() has no global side effects, it's just a function. Unless I'm mistaken, Test::AutoBailOut is doing to need a global $SIG{__DIE__} handler or override CORE::require or add something to @INC or try to parse exception messages or something like that. Any of those have global side effects which can potentially interfere with or be overwritten by the code being tested. > What exactly makes you uneasy? Maybe there is a way to address that > if you can be more specific. Mostly that it's cramming yet more complexity into core test library for a fairly narrow use case and functionality that lives quite happily in its own module. It's already difficult enough to keep stable. To reverse it, why should it be in Test::More? "People don't like to have extra test dependencies" is not accepted. If you want to bundle it all together into a sort of universal set of testing extensions, Test::Most does that. While it could be argued that every distribution should be testing that their modules load and bailing out if they don't, the bailing out part we can live without. But it is handy. >> It works fine if what you want is a runtime require + import + assert, >> and sometimes you want that. The problem is it's been overused and >> has come to replace a simple `use` in test scripts. To that end the >> question is whether to *discourage* its use in the documentation: to >> scale it back from "use this when you load a module" to "use this for >> some special cases". > > *Which* special cases? I would rather not recommend it in any case ever. In my reply to Andy I outlined some cases where you need a runtime require + import + assert and its better to have a function which does that than write it all out by hand. > My suggestion to ship AutoBailOut was so you would be able to suggest it > as a replacement in the docs, as it covers the one case where `use_ok` > is even of interest (though still not the right solution). > > But I guess you could do that even if it ships outside of Test::More. Precisely. For example, it already suggests Test::Differences and Test::Deep. -- 185. My name is not a killing word. -- The 213 Things Skippy Is No Longer Allowed To Do In The U.S. Army http://skippyslist.com/list/ http://www.nntp.perl.org/group/perl.qa/2012/04/msg13146.html Wed, 11 Apr 2012 11:06:13 +0000 Re: Revert use_ok() change to allow lexical effects? by Aristotle Pagaltzis * Smylers <Smylers@stripey.com> [2012-04-11 18:20]: > Aristotle Pagaltzis writes: > > my $reason = 'Tests must succeeded'; > > Grammar correction if anybody is going to publish this in a module: > "succeed", rather than "succeeded". Woops. Thanks. > > Ah d’uh! Now I feel stupid. > > Please don't! You're still ahead many of us ... See above. ;-) Regards, -- Aristotle Pagaltzis // <http://plasmasturm.org/> http://www.nntp.perl.org/group/perl.qa/2012/04/msg13145.html Wed, 11 Apr 2012 10:49:06 +0000 Re: Revert use_ok() change to allow lexical effects? by Michael G Schwern On 2012.4.11 9:39 AM, Andy Lester wrote: > In this example: > > BEGIN { > use_ok( 'App::Ack' ); > use_ok( 'App::Ack::Repository' ); > use_ok( 'App::Ack::Resource' ); > use_ok( 'File::Next' ); > } > diag( "Testing App::Ack $App::Ack::VERSION, File::Next $File::Next::VERSION, Perl $], $^X" ); > > it sounds like we're saying that the use_ok() doesn't help at all, and I might as well write > > use App::Ack; > use App::Ack::Repository; > use App::Ack::Resource; > use File::Next; > diag( "Testing App::Ack $App::Ack::VERSION, File::Next $File::Next::VERSION, Perl $], $^X" ); > > Agreed? Yes, exactly. -- I do have a cause though. It's obscenity. I'm for it. - Tom Lehrer http://www.nntp.perl.org/group/perl.qa/2012/04/msg13144.html Wed, 11 Apr 2012 10:43:26 +0000