From: Buddy Burden David,

> * first, when a bug gets reported in live, I like to create a test case
> from it, using data that at least replicates the structure of that
> that is live. This will, necessarily, be an end-to-end test of the
> whole application, from user interaction through all the layers that
> make up the application down to the database, and back up through all
> those layers again to produce broken output. As I debug and fix I may
> also create unit tests for individual bits of the application.

See, I would approach that differently. I would start with creating a test
for just the bit of data that I suspected was causing the problem. If that
worked (or didn't work, technically--if that reproduced the bug, I mean),
fine, I'm done. If that _doesn't_ work, though, I have to add more and
more bits of data specific to the user's exact situation ... which then
shows me how much that extra data is contributing to the problem vs how
much is irrelevant. By the time I'm done, I have exactly the data I need
to reproduce, which makes it _much_ easier to figure out what's wrong.

But then, on our team, we only produce unit tests. Integration tests are
produced by QA, and they might do it more as you suggest.

Of course, they don't get to start with an empty database either. It might
be _nice_ to start with a blank slate every time, I'm not sure--I'd worry
that I'd constantly miss scalability issues that are currently outed fairly
early with our current process--but the point is, I don't have that luxury,
whether I want to or not. ;->

(OTOH, I'll soon be moving to a new job where they don't even _have_ a QA
department, so I'll have to rethink the whole process. :-D )


-- Buddy

From: David Cantrell On Sun, May 05, 2013 at 11:03:49PM -0700, Ovid wrote:
> From: Buddy Burden <barefootcoder@gmail.com>
> >I lean more towards Mark's approach on this one, albeit with a slight twist.
> Given the very interesting discussion and the fact that I hate dogma (such as what I was throwing down), I have to say that I'm going to rethink my position on this. Thanks for all of the informative comments, everyone!

I lean very much towards starting from a known state. I don't care what
that state is, just that it's known. This is because I see the tests I
write as being primarily a debugging tool, not merely a way of proving
that what I've done is correct - we all know that tests can't do that,
right? Debugging is easiest if you don't have to sift through loads of
irrelevant noise to find the problem.

It's a good point that the live environment is unknown, and potentially
full of broken old rubbish from previous versions of the software. That
is why we do two things:

* first, when a bug gets reported in live, I like to create a test case
from it, using data that at least replicates the structure of that
that is live. This will, necessarily, be an end-to-end test of the
whole application, from user interaction through all the layers that
make up the application down to the database, and back up through all
those layers again to produce broken output. As I debug and fix I may
also create unit tests for individual bits of the application.
* second, we don't just deploy from dev to live. We have a staging
environment in which real people hammer on the application. Ideally,
they'll record what they do and create repeatable tests completely
independently of the developers.

--
David Cantrell | semi-evolved ape-thing

You don't need to spam good porn

From: Buddy Burden Lasse,

> Interesting... Developers in our project have a local copy of the
production database for working with but our unit test runs always create a
database from scratch and run all schema migrations on it before running
the tests. Creating and migrating the unit test DB usually takes between 10
and 30 seconds so setup time is not really an issue... We're currently on
MySQL but will be migrating to Oracle in the near future. Could you
elaborate on why this approach might not be viable on Oracle?

First of all: my condolences. :-D

Creating a "database" on MySQL is essentially the same as creating a
directory. Tables are files, so a collection of tables (a database) is
just a collection of files (i.e. a directory). However, in other RDBMSes
(such as Sybase or Informix--I lack enough experience with Postgres to
comment there) a database is a more complex creature, and takes a bit more
effort to create.

The situation in Oracle is much worse. What Oracle calls a "database" is
actually a server. That is, on Sybase or Informix (also I believe DB2
works this way), you can have several different databases on a single
server. Not so in Oracle. You need a whole new server for a new database,
which means a new port, and new entry in oranames.tns, etc etc etc. It's
quite a PITA. I've never seen a dev create a database in Oracle--it's
_always_ been something that the DBAs do. Maybe there's a way to do it,
particularly if you have Oracle's "personal" version, but I've just never
seen it done.

Other remarkably painful things to look forward to: Say goodbye to
autoincrement columns, and hello to "sequences." You know how you think
that NULL and the empty string are two different things? HA! No more temp
tables--Oracle has "global" (i.e. permanent) temp tables instead, which are
often useless for how you normally use temp tables. Probably there are
more things, but those are the most egregious that spring to mind ATM.


-- Buddy

From: Lasse Makholm On Sat, May 4, 2013 at 10:37 PM, Buddy Burden <barefootcoder@gmail.com>wrote:

>
> We have several databases, but unit tests definitely don't have their
> own. Typically unit tests run either against the dev database, or the QA
> database. Primarily, they run against whichever database the current
> developer has their config pointed to. This has to be the case, since
> sometimes we make modifications to the schema. If the unit tests all ran
> against their own database, then my unit tests for my new feature involving
> the schema change would necessarily fail. Or, contrariwise, if I make the
> schema modification on the unit test database, then every other dev's unit
> tests would fail. I suppose if we were using MySQL, it might be feasible
> to create a new database on the fly for every unit test run. When you're
> stuck with Oracle though ... not so much. :-/
>

Interesting... Developers in our project have a local copy of the
production database for working with but our unit test runs always create a
database from scratch and run all schema migrations on it before running
the tests. Creating and migrating the unit test DB usually takes between 10
and 30 seconds so setup time is not really an issue... We're currently on
MySQL but will be migrating to Oracle in the near future. Could you
elaborate on why this approach might not be viable on Oracle?

As to why we do this - I guess it's mainly history... We've only recently
cleaned up our tests to not rely on each other so we're only now getting to
a point where we can start running them in random order - let alone in
parallel... I guess the upsides of starting from a clean database are
mainly matters of convenience; single-digit IDs are easier to read
ten-digits ones and debugging failures is easier on a table with 10 rows
instead of 10 million. The flip-side is of course, as previously mentioned
is that production code is expected to work in a "dirty" rather than
"clean" environment...

Your points about parallelization and using it to flush out
locking/contention issues are interesting and something that we haven't
really explored in our test setup but something we could certainly benefit
from... (Having had our fair share of those issues in the past...)

/L


>
> So all our unit tests just connect to whatever database you're currently
> pointed at, and they all create their own data, and they all roll it back
> at the end. In fact, our common test module (which is based on Test::Most)
> does the rollback for you. In fact in fact, it won't allow you to commit.
> So there's never anything to clean up.
>
> AFA leaving the data around for debugging purposes, we've never needed
> that. The common test module exports a "DBdump" function that will dump
> out whatever records you need. If you run into a problem with the data and
> you need to see what the data is, you stick a DBdump in there. When you're
> finished debugging, you either comment it out, or (better yet) just change
> it from `diag DBdump` to `note DBdump` and that way you can get the dump
> back any time just by adding -v to your prove.
>
> AFAIK the only time anyone's ever asked me to make it possible for the
> data to hang around afterwards was when the QA department was toying with
> the idea of using the common test module to create test data for their
> manual testing scenarios, but they eventually found another way around
> that. Certainly no one's ever asked me to do so for a unit test. If they
> did, there's a way to commit if you really really want to--I just don't
> tell anyone what it is. ;->
>
> Our data generation routines generate randomized data for things that have
> to be unique (e.g. email addresses) using modules such as String::Random.
> In the unlikely event that it gets a collision, it just retries a few
> times. If a completely randomly generated string isn't unique after, say,
> 10 tries, you've probably got a bigger problem anyway. Once it's inserted,
> we pull it back out again using whatever unique key we generated, so we
> don't ever have a need to count records or anything like that. Perhaps
> count the number of records _attached_ to a record we inserted previously
> in the test, but that obviously isn't impacted by having extra data in the
> table.
>
> Unlike Mark, I won't say we _count_ on the random data being in the DB; we
> just don't mind it. We only ever look at the data we just inserted. And,
> since all unit test data is in a transaction (whether ours or someone
> else's who happens to be running a unit test at the same time), the unit
> tests can't conflict with each other, or with themselves (i.e. we do use
> parallelization for all our unit tests). The only problems we ever see
> with this approach are:
>
> * The performance on the unit tests can be bad if lots and lots of things
> are hitting the same tables at the same time.
> * If the inserts or updates aren't judicious with their locking, some
> tests can lock other tests out from accessing the table they want.
>
> And the cool thing there is, both of those issues expose problems in the
> implementation that need fixing anyway: scalability problems and potential
> DB contention issues. So forcing people to fix those in order to make
> their unit tests run smoothly is a net gain.
>
> Anyways, just wanted to throw in yet another perspective.
>
>
> -- Buddy

From: yary I've done some heavy DB work/testing and like your idea of simply
turning off autocommit, rolling back for all the database tests. It's
not what we did- we just truncated all the test tables to start from a
good state, and the only parallel testing we did were specifically
designed concurrency tests. Not saying that's what I'd do again, just
for a data point.

Anyway, while rolling back after DB unit tests seems like a good
solution in general, and is something I would try in the future, it
can't work in all situations- specifically when you want to test
things that rely on transactions. Part of our
designed-to-be-concurrent tests had to do with subroutines that
required a lock so changes to 2-3 tables would be either all done, or
all rolled back on failure. Another needed an exclusive-select-row
lock. Those needed to commit so we could both see the data succeed and
so we could see the concurrent threads continue (or report "could not
get lock" properly, etc)

-y


On Sat, May 4, 2013 at 4:37 PM, Buddy Burden <barefootcoder@gmail.com> wrote:
> Ovid,
>
> I lean more towards Mark's approach on this one, albeit with a slight twist.
...

From: Ricardo Signes * Paul Johnson <paul@pjcj.net> [2013-05-01T16:38:26]
> In accordance with the terms of my grant from TPF this is the monthly
> report for my work on improving Devel::Cover covering March 2013.

+1, thanks, Paul!

--
rjbs

From: Buddy Burden Ovid,

I lean more towards Mark's approach on this one, albeit with a slight twist.

> For many of the test suites I've worked on, the business rules are
complex enough that this is a complete non-starter. I *must* have a
database in a known-good state at the start of every test run.
>
> is $customer_table->count, 2, "We should find the correct number of
records";

I've just never written a test like that. I can't think that I've ever
needed to, really ... although I think I vaguely remember writing something
which used $^T to guarantee I was only pulling records that I'd added, so
perhaps that's close. But that was only once.

We have several databases, but unit tests definitely don't have their own.
Typically unit tests run either against the dev database, or the QA
database. Primarily, they run against whichever database the current
developer has their config pointed to. This has to be the case, since
sometimes we make modifications to the schema. If the unit tests all ran
against their own database, then my unit tests for my new feature involving
the schema change would necessarily fail. Or, contrariwise, if I make the
schema modification on the unit test database, then every other dev's unit
tests would fail. I suppose if we were using MySQL, it might be feasible
to create a new database on the fly for every unit test run. When you're
stuck with Oracle though ... not so much. :-/

So all our unit tests just connect to whatever database you're currently
pointed at, and they all create their own data, and they all roll it back
at the end. In fact, our common test module (which is based on Test::Most)
does the rollback for you. In fact in fact, it won't allow you to commit.
So there's never anything to clean up.

AFA leaving the data around for debugging purposes, we've never needed
that. The common test module exports a "DBdump" function that will dump
out whatever records you need. If you run into a problem with the data and
you need to see what the data is, you stick a DBdump in there. When you're
finished debugging, you either comment it out, or (better yet) just change
it from `diag DBdump` to `note DBdump` and that way you can get the dump
back any time just by adding -v to your prove.

AFAIK the only time anyone's ever asked me to make it possible for the data
to hang around afterwards was when the QA department was toying with the
idea of using the common test module to create test data for their manual
testing scenarios, but they eventually found another way around that.
Certainly no one's ever asked me to do so for a unit test. If they did,
there's a way to commit if you really really want to--I just don't tell
anyone what it is. ;->

Our data generation routines generate randomized data for things that have
to be unique (e.g. email addresses) using modules such as String::Random.
In the unlikely event that it gets a collision, it just retries a few
times. If a completely randomly generated string isn't unique after, say,
10 tries, you've probably got a bigger problem anyway. Once it's inserted,
we pull it back out again using whatever unique key we generated, so we
don't ever have a need to count records or anything like that. Perhaps
count the number of records _attached_ to a record we inserted previously
in the test, but that obviously isn't impacted by having extra data in the
table.

Unlike Mark, I won't say we _count_ on the random data being in the DB; we
just don't mind it. We only ever look at the data we just inserted. And,
since all unit test data is in a transaction (whether ours or someone
else's who happens to be running a unit test at the same time), the unit
tests can't conflict with each other, or with themselves (i.e. we do use
parallelization for all our unit tests). The only problems we ever see
with this approach are:

* The performance on the unit tests can be bad if lots and lots of things
are hitting the same tables at the same time.
* If the inserts or updates aren't judicious with their locking, some tests
can lock other tests out from accessing the table they want.

And the cool thing there is, both of those issues expose problems in the
implementation that need fixing anyway: scalability problems and potential
DB contention issues. So forcing people to fix those in order to make
their unit tests run smoothly is a net gain.

Anyways, just wanted to throw in yet another perspective.


-- Buddy

From: chromatic On Friday, May 03, 2013 01:34:35 PM Ovid wrote:

> ... you'll have to do a lot of work to convince people that starting out
> with an effectively random environment is a good way to test code.

Before you dismiss the idea entirely, consider that our real live code running
for real live clients runs in an effectively random environment.

This reminds me of the (false) debate over using mock objects to enforce some
standard of unit testing purity. It's interesting if individual pieces of code
adhere to expected behavior in isolation, but when the code has to work, it's
not doing so in isolation.

Isolation of individual test cases is, of course, essential to
parallelization--but I'm trying to maximize the return on confidence for
testing investment in what's inherently a stochastic process, so pursuing
isolation between individual test units is rarely worth it, in my experience.
Writing the right tests which test the right things is.

-- c

From: Mark Stosberg
> OK, but you still have to clean out your database before you start each
> independent chunk of your test suite, otherwise you start from an
> unknown state.

In a lot of cases, this isn't true. This pattern is quite common:

1. Insert entity.
2. Test with entity just inserted.

Since all that my test cares about is the unique entity or entities, the
state of the rest of database doesn't matter. The state the matters is
in a "known state".

> What about when you're not running under Jenkins. Like when you're
> writing and testing your code. You still need to start testing from a
> known state each time, which means you must clean out the database at
> startup.

We have a cron job that runs overnight to clean up anything that was
missed in Jenkin's runs.

We expect our tests to generally work in the face of a "dirty" database.
If they don't, that's considered a flaw in the test. This is important
to run several tests against the same database at the same time. Even if
we did wipe the database for we tested, all the other tests running in
parallel would be considered to making the database "dirty". Thus, if a
pristine database is a requirement, only one test could run against the
database at the time.

We run our tests 4x parallel against the same database, matching the
cores available in the machine.

We also share the same database between developers and the test suite.
This "dirty" environment can work like a feature, as it can sometimes
produce unexpected and "interesting" states that were missed by a
clean-room testing approach that so carefully controlled the environment
that some real-world possibilities.

For example, perhaps a column allows null values, but the test suite
never tests that case because it "shouldn't happen". A developer might
manually create a value, which could expose a problem spot-- perhaps the
field should be "not null", or the app should handle that case gracefully.

A perfect clean-room approach would cover all these cases, but I don't
assume our tests are perfect.

Mark

From: David Cantrell On Fri, May 03, 2013 at 11:03:28AM -0400, Mark Stosberg wrote:
> > No, you can't have your tests clean up after themselves. For two
> > reasons. First, that means you have to scatter house-keeping crap all
> > over your tests. Second, if you have a test failure you want the
> > results to be sitting there in the database to help with debugging.
> There is another way to have tests clean up after themselves, which
> addresses both the shortcoming you address.
>
> First, I have functions like "insert_test_user()". At the end of these
> functions, there is another call like this:
>
> schedule_test_user_for_deletion(...)
>
> That inserts a row into a table "test_ids_to_delete" which includes
> column for the table name and primary key of the entity to delete.
> Another column has the insertion timestamp.
>
> So, there's no "clean-up" code in all of our test scripts, and we have
> the data there for debugging when the tests are done.

OK, but you still have to clean out your database before you start each
independent chunk of your test suite, otherwise you start from an
unknown state. You might, for example, start with broken data still
hanging around from a previous test failure, which then causes your
correct code to fail, which is a massive pain in the arse if you are, as
you should be, using your tests as a debugging aid.

Consider, for example, your test for "retrieve a list of all customers
from the database and inflate them to objects". If some previous test
left a broken customer record in the database, then your code for
inflating customers to objects will (correctly but unexpectedly) fail.

If you want to paralellise tests that use a shared resource, such as a
database, then you will normally need to mock that shared resource.

> In Jenkins, after the test suite runs, a job to "Delete Test Data" is
> kicked off, which deletes all test data older than hour. (Longer than it
> takes the test suite to run).

What about when you're not running under Jenkins. Like when you're
writing and testing your code. You still need to start testing from a
known state each time, which means you must clean out the database at
startup.

--
David Cantrell | Cake Smuggler Extraordinaire

Nuke a disabled unborn gay baby whale for JESUS!

From: Michael G. Schwern Huzzah!


On 5/1/13 3:38 PM, Paul Johnson wrote:
> In accordance with the terms of my grant from TPF this is the monthly
> report for my work on improving Devel::Cover covering March 2013.
>
> Sorry for the delay in this report. Most of March and all of April has been a
> very busy time for me outside of grant work on Devel::Cover.
>
> This month I released Devel::Cover 1.01.
>
> March's work started by continuing there February's had left off. David
> Golden had politely observed that the coverage reporting for ||= operators in
> his Path::Iterator::Rule module was somewhat less than optimal. So I got hold
> of the module's source, pared the module down to the minimum required to
> reproduce the problem, and fixed it, along with another problem that showed up
> too. The problem boiled down to C< $x ||= $y > being in void context, but we
> want to show the coverage as if were not. This was one of those problems
> where the majority of the effort was spent in locating and defining the
> problem, and the solution was relatively simple from that point. In any case,
> the coverage looks much healthier now:
>
> http://cpancover.com/latest/Path-Iterator-Rule-1.005/blib-lib-Path-Iterator-Rule-pm--condition.html
>
> This month, perls 5.14.4, 5.16.3, 5.17.10 were released. I tested against
> these, along with 5.17.9 from February. It's obviously important that
> Devel::Cover works with newly released stable versions of perl, so I always
> try to test with the release candidates too. I also try to keep up with the
> development releases (5.17.x at the moment), which are far more of a moving
> target as far as Devel::Cover is concerned. Of course, this should make it
> more likely that there won't be any problems when 5.18.0 is released, and it
> also means that the cpantesters failure reports I get from those people
> running development releases use generally useful.
>
> Devel::Cover now also reports coverage for more of the files that gcov can
> exercise.
>
> I investigated removing the dependency on B::Deparse. Unfortunately, that
> won't be simple, and may not be worth the effort, but I did at least reduce
> the number of calls to B::Deparse.
>
> Closed Github tickets:
>
> 52 cover ignores .cc file
>
> Merged pull requests:
>
> 47 Fix for mod_perl on Debian setting $^X to apache2
> 49 fix: respect to $Devel::Cover::Silent
>
> You can see the commits at https://github.com/pjcj/Devel--Cover/commits/master
>
> Hours worked:
>
> 01.03 7:30
> 29.03 2:15
>
> Total 9:45
>
> Total hours worked on grant: 257:35
>

From: Mark Stosberg
> No, you can't have your tests clean up after themselves. For two
> reasons. First, that means you have to scatter house-keeping crap all
> over your tests. Second, if you have a test failure you want the
> results to be sitting there in the database to help with debugging.

There is another way to have tests clean up after themselves, which
addresses both the shortcoming you address.

First, I have functions like "insert_test_user()". At the end of these
functions, there is another call like this:

schedule_test_user_for_deletion(...)

That inserts a row into a table "test_ids_to_delete" which includes
column for the table name and primary key of the entity to delete.
Another column has the insertion timestamp.

So, there's no "clean-up" code in all of our test scripts, and we have
the data there for debugging when the tests are done.

In Jenkins, after the test suite runs, a job to "Delete Test Data" is
kicked off, which deletes all test data older than hour. (Longer than it
takes the test suite to run).

There's a third reason not to do in-line test clean-up, which is that a
SQL "DELETE" operation can be relatively slow, especially when complex
RI is involved. Doing this "offline" as we do speeds that up.

There are still a few places where we have clean-up code in tests, but
it is exception. Those are the cases in which we can't use functions
like "insert_test_user()".

For example, if we are creating an entity by filling out a form on the
web and submitting it, then we may need to manually register that for
clean up later.

Mark

From: David Cantrell On Thu, May 02, 2013 at 12:51:06PM -0700, Karen Etheridge wrote:
> On Thu, May 02, 2013 at 02:39:13PM -0500, brian d foy wrote:
> > In HARNESS_OPTIONS we can set -jN to note we want parallel tests
> > running, but how can a particular module, which might be buried in the
> > dependency chain, tell the harness it can't do that?
> When can a test not be parallelizable?

When you use Test::Class and all your tests inherit this:

sub start_from_a_known_state_because_doing_anything_else_is_stupid :Test(startup) {
shift()->truncate_all_database_tables();
}

No, you can't have your tests clean up after themselves. For two
reasons. First, that means you have to scatter house-keeping crap all
over your tests. Second, if you have a test failure you want the
results to be sitting there in the database to help with debugging.

--
David Cantrell | even more awesome than a panda-fur coat

All children should be aptitude-tested at an early age and,
if their main or only aptitude is for marketing, drowned.

From: Christian Walde On Wed, 01 May 2013 22:40:17 +0200, Paul Johnson <paul@pjcj.net> wrote:

> This month I released Devel::Cover 1.02 but, apart from that, almost all
> of my work on Devel::Cover was either at or in association with the QA
> Hackathon and, as such, I will not be charging it to the grant.
>
> Hours worked:
>
> Total 0:00
>
> Total hours worked on grant: 257:35

+1, more good stuff and quite gracious of you. :D

--
With regards,
Christian Walde

From: Christian Walde On Wed, 01 May 2013 22:38:26 +0200, Paul Johnson <paul@pjcj.net> wrote:

> Hours worked:
>
> 01.03 7:30
> 29.03 2:15
>
> Total 9:45
>
> Total hours worked on grant: 257:35

+1, good stuff. :)

--
With regards,
Christian Walde

From: Olivier Mengué 2013/5/2 brian d foy <brian.d.foy@gmail.com>

> In HARNESS_OPTIONS we can set -jN to note we want parallel tests
> running, but how can a particular module, which might be buried in the
> dependency chain, tell the harness it can't do that?
>
> It seems to me that by the time the tests are running, it's too late
> because they are already in parallel and the best we can do is issue a
> warning or decide to fail.


Note that Test::Synchronized may be a solution to your problem.

For a more general solution a simple way to do it would be to let harness
know that some tests must not be run in parallel by their file path.
Something like storing those tests in a particular place such as
"t/non-parallel/" that harness would exclude from parralel testing and
would run sequentially after all the parallel tests.

A related question is: do we want the tests to be explicitely aware at
runtime that some others tests may be running in parallel (or in the
opposite, that they are running in an exclusive mode)? That would give more
information to Test::Is/Test::DescribeMe, but I don't think that would be a
good idea. Non-parallel tests must stay the exception.

Olivier.

From: Paul Johnson In accordance with the terms of my grant from TPF this is the monthly
report for my work on improving Devel::Cover covering March 2013.

Sorry for the delay in this report. Most of March and all of April has been a
very busy time for me outside of grant work on Devel::Cover.

This month I released Devel::Cover 1.01.

March's work started by continuing there February's had left off. David
Golden had politely observed that the coverage reporting for ||= operators in
his Path::Iterator::Rule module was somewhat less than optimal. So I got hold
of the module's source, pared the module down to the minimum required to
reproduce the problem, and fixed it, along with another problem that showed up
too. The problem boiled down to C< $x ||= $y > being in void context, but we
want to show the coverage as if were not. This was one of those problems
where the majority of the effort was spent in locating and defining the
problem, and the solution was relatively simple from that point. In any case,
the coverage looks much healthier now:

http://cpancover.com/latest/Path-Iterator-Rule-1.005/blib-lib-Path-Iterator-Rule-pm--condition.html

This month, perls 5.14.4, 5.16.3, 5.17.10 were released. I tested against
these, along with 5.17.9 from February. It's obviously important that
Devel::Cover works with newly released stable versions of perl, so I always
try to test with the release candidates too. I also try to keep up with the
development releases (5.17.x at the moment), which are far more of a moving
target as far as Devel::Cover is concerned. Of course, this should make it
more likely that there won't be any problems when 5.18.0 is released, and it
also means that the cpantesters failure reports I get from those people
running development releases use generally useful.

Devel::Cover now also reports coverage for more of the files that gcov can
exercise.

I investigated removing the dependency on B::Deparse. Unfortunately, that
won't be simple, and may not be worth the effort, but I did at least reduce
the number of calls to B::Deparse.

Closed Github tickets:

52 cover ignores .cc file

Merged pull requests:

47 Fix for mod_perl on Debian setting $^X to apache2
49 fix: respect to $Devel::Cover::Silent

You can see the commits at https://github.com/pjcj/Devel--Cover/commits/master

Hours worked:

01.03 7:30
29.03 2:15

Total 9:45

Total hours worked on grant: 257:35

--
Paul Johnson - paul@pjcj.net
http://www.pjcj.net

From: Paul Johnson In accordance with the terms of my grant from TPF this is the monthly
report for my work on improving Devel::Cover covering April 2013.

April has been a very busy time for me outside of grant work on Devel::Cover.
Apart from other matters, I was fortunate enough to be able to attend the Perl
QA Hackathon in Lancaster. This is an annual event in which many of the
people involved in QA and toolchain projects are able to get together for
three days of discussion, decision making and hacking. See
http://qa-hackathon.org/ for more information.

The event generally results in important decisions being made, direction being
determined, and much interesting and useful code being written. This year was
no exception.

With regard to Devel::Cover the main areas of focus were providing coverage
for the perl core for C, XS and Perl code, and integrating cpancover into
metacpan. Neither of these was able to be completely finished, but much
progress was made and I am very grateful to Jim Keenan, Abe Timmerman and
Dinis Rebolo who all put in lots of effort in these areas. More information
is available at http://2013.qa-hackathon.org/qa2013/wiki?node=Achievements

In addition Thomas Klausner got dragged in from sunny Vienna and Kan Fushihara
wrote Devel::Cover::Report::Coveralls during the satellite QA hackathon in
Tokyo organised by Miyagawa. (And you really should check that out:
https://metacpan.org/module/Devel::Cover::Report::Coveralls)

I would like to specifically thank Mark Keating, Ian Norton and Claire Jackson
for all the work they put into the excellent organisation. And many thanks
too to the wonderful sponsors, without whom the event could not take place:
cPanel, Dijkmat, Dyn, Eligo, Evozon, $foo, Shadowcat Systems Limited,
Enlightened Perl Organisation and Mongueurs de Perl.

This month I released Devel::Cover 1.02 but, apart from that, almost all of my
work on Devel::Cover was either at or in association with the QA Hackathon
and, as such, I will not be charging it to the grant.

Closed Github tickets:

59 Devel::Cover::DB::Structure: uninitialized value in write()

Merged pull requests:

52 new function "write_csv" to create a csv file to be use from metacpan
57 cover -test exit code is always success, when test fails

Closed RT tickets:

#31733 Devel::Cover 0.63 fails two tests on Darwin with Perl 5.10.02
#58996 IPC::Run in a BEGIN{} block crashes Devel::Cover
#72027 Devel::Cover does not detect branch coverage with DBM::Deep
#68225 Devel::Cover crashes on overloaded method that creates closure
#69209 Fatal error (Can't locate object method "NAME" via package "B::SPECIAL") under 5.14.0 on Darwin
#38978 report on source of coverage
#27526 "Bizarre Copy of Hash" error
#16379 Html_minimal gets confused by refactoring
#49916 Devel::Cover doesn't like code that changes EUID

You can see the commits at https://github.com/pjcj/Devel--Cover/commits/master

Hours worked:

Total 0:00

Total hours worked on grant: 257:35

--
Paul Johnson - paul@pjcj.net
http://www.pjcj.net

From: Mark Stosberg
> When can a test not be parallelizable? Most of the examples that I can
> think of (depending on a file resource, etc) smell like a design failure.
> Tests should usually be able to use a unique temp file, etc.

Here's an example:

Say you are testing a web application that does a bulk export of a
database table.

The test works by doing a slow "count(*)" on the table, then having the
app what should generate 2 rows, and then running another slow
"count(*)" on the table, then checking that if the new value is the
original value plus 2.

This isn't parallel safe, because other tests running in parallel could
insert rows into the table in between the before and after counts.

One solution is to use a unit test instead. If you do that, the test and
the application can be made to share the same database handle. In
PostgreSQL, you can create a temporary table of the same name which
masks the original. Thus, your test has exclusive access to the table,
and the test can be made to run parallel-safe. It may also run much
faster, as the temporary table may have 10 rows in it instead of 100,000.

Other kinds of mocking could be used at this point as well.

Alternatively, you could still use a functional-style test by using
Test::WWW::Mechanize::PSGI instead of testing through your web server.
In this arrangement, the app and the test also run in the same process,
and can be made to share the same database handle, allowing the same
kinds of solutions as above.

I'll be talking more about related topics at YAPC::NA in my talk on
improving the performance of large test suites.

Mark

From: Mark Stosberg On 05/02/2013 03:39 PM, brian d foy wrote:
> In HARNESS_OPTIONS we can set -jN to note we want parallel tests
> running, but how can a particular module, which might be buried in the
> dependency chain, tell the harness it can't do that?
>
> It seems to me that by the time the tests are running, it's too late
> because they are already in parallel and the best we can do is issue a
> warning or decide to fail.

I spent considerable time researching the topic of partially-parallel
test suites in Perl. Some of research is published here:

http://mark.stosberg.com/blog/2012/08/running-perl-tests-in-parallel-with-prove-with-some-exceptions-2.html
http://stackoverflow.com/questions/11977015/how-to-run-some-but-not-all-tests-in-a-perl-test-suite-in-parallel

In the end, the most efficient path forward for my particular case was
to look harder at the exceptional cases, and find ways to make them
parallel-safe.

Also, higher level automation like Jenkins or a wrapper script could be
used to first run all the tests that can be run in parallel, and then
run serial-only tests.

I have hope for better support within prove in the future, but
pragmatically for now I've find it easier to address through other parts
of the toolchain.

Mark

From: Karen Etheridge On Thu, May 02, 2013 at 02:39:13PM -0500, brian d foy wrote:
> In HARNESS_OPTIONS we can set -jN to note we want parallel tests
> running, but how can a particular module, which might be buried in the
> dependency chain, tell the harness it can't do that?

When can a test not be parallelizable? Most of the examples that I can
think of (depending on a file resource, etc) smell like a design failure.
Tests should usually be able to use a unique temp file, etc.

> It seems to me that by the time the tests are running, it's too late
> because they are already in parallel and the best we can do is issue a
> warning or decide to fail.

Is it too late if the test itself declares "I cannot be parallelized"? If
not, then this declaration could be worked into Test::DescribeMe and
friends.

From: brian d foy In HARNESS_OPTIONS we can set -jN to note we want parallel tests
running, but how can a particular module, which might be buried in the
dependency chain, tell the harness it can't do that?

It seems to me that by the time the tests are running, it's too late
because they are already in parallel and the best we can do is issue a
warning or decide to fail.

From: Ricardo Signes * Adrian Howard <adrianh@quietstars.com> [2013-04-27T20:09:09]
> On 27 April 2013 15:19, Ricardo Signes <perl.qa@rjbs.manxome.org> wrote:
> > Test::Exception fails, which breaks a lot of my basic toolchain from
> > installing.
>
> T::E 0.32 just uploaded that kills the false failure.

Thanks! With that, the next problem is Test::Moose::More, which appears to be
testing exact isa_ok output.

--
rjbs

From: Adrian Howard On 27 April 2013 15:19, Ricardo Signes <perl.qa@rjbs.manxome.org> wrote:
> Test::Exception fails, which breaks a lot of my basic toolchain from
> installing.

T::E 0.32 just uploaded that kills the false failure.

Adrian

--
http://quietstars.com adrianh@quietstars.com twitter.com/adrianh
t. +44 (0)7752 419080 skype adrianjohnhoward pinboard.in/u:adrianh

From: Michael G. Schwern On 4/27/13 3:41 PM, Karen Etheridge wrote:
> On Sat, Apr 27, 2013 at 03:22:31PM -0700, Michael G. Schwern wrote:
>> No, it's not Test::Exception's fault.
>
> No, I understand -- I just prefer Test::Fatal, so I convert dists when
> seeing them and I also have a tuit to spare.

Oh, ok then.

From: Karen Etheridge
On Sat, Apr 27, 2013 at 03:22:31PM -0700, Michael G. Schwern wrote:
> No, it's not Test::Exception's fault.

No, I understand -- I just prefer Test::Fatal, so I convert dists when
seeing them and I also have a tuit to spare.

From: Michael G. Schwern On 4/27/13 2:33 PM, Karen Etheridge wrote:
> On Sat, Apr 27, 2013 at 10:19:44AM -0400, Ricardo Signes wrote:
>> Test::Exception fails, which breaks a lot of my basic toolchain from
>> installing.
>
> Which dists? I'll convert them to Test::Fatal.

No, it's not Test::Exception's fault. They have a test which looks for
the particular output of either isa_ok() or new_ok which changed in
0.98_05. Could have happened to any module.

I have a test for dependents which includes Test::Exception, but I
forgot I never backported it to the 0.98 branch so it didn't get run.
Test::Exception and Test::Class are failing due to the format change.

From: Karen Etheridge On Sat, Apr 27, 2013 at 10:19:44AM -0400, Ricardo Signes wrote:
> Test::Exception fails, which breaks a lot of my basic toolchain from
> installing.

Which dists? I'll convert them to Test::Fatal.


--
Why do programmers always get Halloween and
Christmas confused? Because oct(31) == dec(25).
. . . . .
Karen Etheridge, karen@etheridge.ca GCS C+++$ USL+++$ P+++$ w--- M++

From: Ricardo Signes * "Michael G. Schwern" <schwern@pobox.com> [2013-04-25T13:35:49]
> 0.98_05 is a release candidate for 0.99. I would say this is the last
> stable release of Test::More before 1.5.0 but hahahaha I won't say that.
> https://metacpan.org/release/MSCHWERN/Test-Simple-0.98_05/

Test::Exception fails, which breaks a lot of my basic toolchain from
installing.

--
rjbs

From: Michael G. Schwern 0.98_05 is a release candidate for 0.99. I would say this is the last
stable release of Test::More before 1.5.0 but hahahaha I won't say that.
https://metacpan.org/release/MSCHWERN/Test-Simple-0.98_05/

There hasn't been a stable release in two years, so there's likely to be
code which has wrapped itself around this version. If smokers and
Test::* module authors would give it a go and report and problems I'd
appreciate that.

Here's the changes since 0.98 which might conceivably have an impact.

* cmp_ok() will error when it gets a non-comparison operator
* isa_ok() and new_ok() default names have changed
* like() and unlike() no longer warn about undef
* subtests will output their name at the start of the subtest

From: Michael G. Schwern On 4/18/13 6:42 AM, Karen Etheridge wrote:
> On Wed, Apr 17, 2013 at 11:58:55AM +0200, Philippe Bruhat (BooK) wrote:
>> I would like to collect your impressions about what worked and what
>> didn't, what was missing, what you want for next year
>
> For those of us participating remotely, some small amount of video
> conferencing, perhaps to allow eavesdropping on/participating with some
> discussions, would be helpful. Perhaps just throwing a webcam on in the
> corner? But being able to conduct multi-way conversations would be even
> nicer.

+1 to that. A sacrificial laptop or three doing video conferencing
which can be carried around the room. Or encourage one on-site person
from each project to be in a Google Hangout (or similar technology).

From: Jens Rehsack On 17.04.13 20:43, Michael G. Schwern wrote:
> On 4/15/13 7:06 PM, Jens Rehsack wrote:
>> On 15.04.13 18:56, Michael G. Schwern wrote:
>>> TL;DR version...
>>> IMO we only need to clarify what "conflicts" means and what actions CPAN
>>> tools should take.
>>>
>>> We talked in the Consensus Dome about the need for and meaning of the
>>> "conflicts" relationship in the CPAN::Meta::Spec.
>>> https://metacpan.org/module/CPAN::Meta::Spec#Prereq-Spec
>>>
>>> IIRC the conclusion was...
>>> * We need it.
>>> * It's not clear what it means. (I don't recall what the confusion was)
>>
>> The difference between "A conflicts with B op VER" and "A breaks B op
>> VER" and "A superflous B".
>
> IIRC those meant...
>
> "A conflict B" is A and B cannot be on the machine at the same time.
> For example, they're both http servers and need the same port. I can't
> think of an example where CPAN has needed that for modules.

Or - maybe - an ancient distribution (META spec is per dist)
is mis-using a module name, and my shiny-new-dist uses the same.
This might be a candidate for superfluous, but doesn't need to...

> "A breaks B" is if A is installed B will break. This is what we need on
> CPAN for things like Test::Builder breaks Test::Class < 0.39.

Yes.

> "A superfluous B" is if A is installed B is no longer necessary.
> Usually called "obsoletes". This is mostly useful if you rename a
> package, though I'd rather that was made more explicit in the meta data.
> Most of the use cases I can think of are better handled by the
> installed modules database and CPAN distribution meta data.

Looking to XML-LibXML, that extra knowlegde of consequences of
splitting and remerging dists could had helped to avoid a lot
of confusion around XML::LibXML::XPathContext.

>>> * Let's see what Debian does.
>>
>> Or MacPorts, HomeBrew or pkgsrc :P
>
> Whatever packaging system I may actually use, I go to Debian first
> because they extensively document this sort of thing, both the meanings
> and the rationales.

I don't know how Debian docs are - I see only the results ;)

> Let's have a look at what the others do...

I suggest we're asking some of the developers there ...

> pkgsrc has CONFLICTS=Some-Name-[0-9]* essentially a regex.
> http://www.netbsd.org/docs/pkgsrc/fixes.html#conflicts
> All they say is the "package may conflict with other packages a user
> might already have installed on his system"

We also allow "patterns" like "scrnsaverproto<1.2.0".
That's why I recommended to look what "the others do" - not
what is somewhere described ;)

...

Cheers,
Jens

From: Karen Etheridge On Wed, Apr 17, 2013 at 11:58:55AM +0200, Philippe Bruhat (BooK) wrote:
> I would like to collect your impressions about what worked and what
> didn't, what was missing, what you want for next year

For those of us participating remotely, some small amount of video
conferencing, perhaps to allow eavesdropping on/participating with some
discussions, would be helpful. Perhaps just throwing a webcam on in the
corner? But being able to conduct multi-way conversations would be even
nicer.

From: Leon Timmermans On Wed, Apr 17, 2013 at 11:58 AM, Philippe Bruhat (BooK) <
philippe.bruhat@free.fr> wrote:

> Hi,
>
> As the main organizer of the next Perl QA Hackathon in Lyon (the venue
> is here: http://goo.gl/maps/mXo15), I would like to collect some feedback
> on the hackathon that we just came back from, in Lancaster.
>
> I would like to collect your impressions about what worked and what
> didn't, what was missing, what you want for next year (including dates,
> but know it's going to be a Friday-Sunday hackathon). You can reply in
> public or in private. I already wrote down all the things I liked from
> this year, as the team has set the bar quite high already. :-)
>
> The goal is to add a bunch of bullet points to our work document, so that
> we can start early on making the next QA hackathon even more productive
> and useful.
>
> Thanks in advance,
>
>
> Post-scriptum: The main communication channels about the Lyon Perl QA
> Hackathon will be the perl-qa mailing list and the #perl-qa IRC channel.
> I don't intend to spam them too much until we get closer to the event. :-)
>
>
I agree with most of the things Salve and Schwern already said. Two
comments though:

1) Make sure to involve the perl 6 people (this might involve some
harassing), this year Liz was the only one. They're doing somewhat similar
things in a vacuum, it would be helpful for them to not repeat some of our
mistakes.
2) Have some kind of post-dinner plan. It seems people ended up in
different pubs without knowing where the rest was. This decreases chances
for useful evening-time discussion. Something as simple as a designated pub
for the evening could be helpful.

Leon