develooper Front page | perl.qa | Postings from August 2000

RFC 182 (v1) JART - Just Another Regression Test

From:
Perl6 RFC Librarian
Date:
August 31, 2000 16:52
Subject:
RFC 182 (v1) JART - Just Another Regression Test
Message ID:
20000831235208.6968.qmail@tmtowtdi.perl.org
This and other RFCs are available on the web at
  http://dev.perl.org/rfc/

=head1 TITLE

JART - Just Another Regression Test

=head1 VERSION

    Maintainer: Michael G Schwern <schwern@pobox.com>
    Date: Aug 30 2000
    Mailing List: perl-qa@perl.org
    Version: 1
    Number: 182
    Status: Developing

=head1 ABSTRACT

Regression tests are often contrived and do not represent real world
usage nor test features in combination.  JAPHs represent a repository
of real world code which is simple to turn into regression tests.  Use
JAPHs to test Perl!

=head2 DESCRIPTION

Testing code is often contrived and incomplete.  Its often optimistic,
simplistic and doesn't stress the program as much as real world code
will.  It would be best to use real world code in our regression
tests, but unfortunately it is difficult to collect code which is
itself simple to test.  Complex tests which force interactions between
features could be written with effort, but traditionally few have
wanted to put in the effort on writing tests.

So it would be wonderful to have a set of real world code which stresses all
the features of Perl, but is also easy to test.  Fortunately we do.  JAPHs! 
Hundreds and hundreds and hundreds of freely available small bits of test code
using almost the entire range of Perl and all with the same output!

The main purpose of the JART is to test backwards compatibility and
also to test existing perl features in combination with each other.

=head2 Process

The process of converting a JAPH to a test is straight forward.

=over 4

=item 1 - Harvesting

JAPHs can be actively collected from various sources: clp.misc
archives, p5p archives, CPAN/misc/japhs, Abigail's hard drive,
etc...  It shouldn't be terribly hard to write simple clients to pull
JAPHs from the signatures of known authors.

Also, it will be hoped that JAPH authors will submit their work
directly to JART.

JART need not be limited to JAPHs.  Any other eclectic code with
predictable output could be used.  Obfuscated Perl Contest entries
come immediately to mind.


=item 2 - Auditing

Only certain JAPHs can be used for the JART and they will have to be
audited.  There are three considerations: security, features
and portability.

Since the JART will be running as trusted code, it must be made sure
that the JAPH doesn't do anything nasty to the system.  This includes
writing files, opening sockets, spawning processes, etc...  Future
versions of JART may run in a safe sand box making the security audit
less important.

Many JAPHs make use of undocumented features and subtle bugs in Perl.
It must be made sure that the JAPHs used in JART rely only on
documented features.  As a derivative benefit I'm sure we will
discover many undocumented features which should be documented.  These
will be reported back as patches to perl.

Finally, JAPHs may rely on files, programs and features which are not
portable.  These should be checked for and if found either the JAPH is
rejected or it is only run on certain systems.

This auditing is the most manpower and clue intensive part of the
JART, and is aggravated by the inherently obfuscated nature of the
code.  This is somewhat alleviated by the small size of the code.
JAPHs which are collected will have to be audited.  JAPHs which are
donated should come with an audit by the author.

=item 3 - Add it

Once a JAPH passes the audits it is made a part of the JART.  This
entails adding it to the list of JAPHs and noting its expected output.

=back


=head2 JAPH Archive

A pleasant consequence of gathering lots of JAPHs is that we're
gathering lots of JAPHs!  This can be spun off into a nice, annotated
JAPH archive for people who's lives aren't already weird enough.


=head1 IMPLEMENTATION

JART will act like any other of Perl's regression tests, probably
added to perl as t/JART.t.  It will run each JAPH and note its output and
print the usual "ok/not ok".

The JAPHs will be organized one per file.  This avoids the issues of
having to escape out characters in the JAPHs and getting spaces and
the number of characters exactly right.  As the output of each JAPH is
subtly different, each JAPH file will have an associated output file
which contains the expected output of each JAPH.  This will all be
tossed into one directory, probably t/JART/JAPHs/.

A prototype implementation exists.

Auditing will probably be done by simple way of a mailing list.  Newly
arriving JAPHs will be posted and auditors (subscribers to the list)
will perform audits at their leisure.  An incoming queue will also be
maintained, accessible by auditors, to make sure things don't get lost
in the noise.

JAPH authors submitting their work will be given a questionnaire to
fill out to allow them to conduct their own audit of their work.


=head1 REFERENCES

None.




nntp.perl.org: Perl Programming lists via nntp and http.
Comments to Ask Bjørn Hansen at ask@perl.org | Group listing | About