develooper Front page | perl.perl6.language | Postings from August 2006

Integrating the Pugs test suite into the Synopses

Agent Zhang
August 22, 2006 06:56
Integrating the Pugs test suite into the Synopses
Message ID:
Hi, there~

When you see the title, you may wonder what that means. Here is the answer:

Search for links named like "Show the snippet from ..." and click on
them to find out what will happen. :)

Well, in short, we have divided the .t files in the Pugs test suite
into pieces and inserted every resulting snippet after the
corresponding paragraph of the Synopses.

The job was done by the Perl 5 script util/ living in the
Pugs source tree. And all the *.html under are updated every hour with the
latest Synopses and Pugs test suite. Thanks to the cron program on
feather. ;-)

At this point, there is a question. How does manage to
determine which snippet of the tests corresponds to which paragraph of
a certain synopsis?

Hmm, I'm sure you may have already found the answer from the web
pages. There's hundreds of "smartlinks" spreading over the whole test
suite in the form of comments. Here're two examples:

  # L<S04/The do-once loop/"can't" put while or until modifier>

  # L<S04/"Switch statements" /"is exactly equivalent to">

You see, the "smartlink" has its deep root in our POD, but contains an
extra keyword list after the second slash. This list of keywords
specifies which paragraph in the given section is relevant to the
tests right below the comment.

Multiple adjacent smartlinks can share the same group of tests:

  # L<S02/Context/boolean "?">
  # L<S03/Changes to Perl 5 operators/"?" imposes boolean context>
  { ... }

Please see t/README in Pugs repository for more detailed information
on smartlinks:

Embedding test snippets into the Synopses has several benefits:

*  Providing more sample code to the Synopses

Synopses are known for their rich samples. Unfortunately, the sample
code our Synopses are not enough at all. Many sections go without
sample code.

On the other hand, the Pugs test suite contains tens of thousands of
tests, which can certainly serve as very good examples as long as they
are correct and up-to-date.

*  Intuitive estimation for the quality of the tests

When we browse the web pages with test snippet embedded, it's quite
easy to figure out the validity and also the completeness of the test

For instance, if a whole section has no snippet links at all, then
that probably means we need to add more tests as well as smartlinks.
These HTML pages are good visual coverage reports per se.

Also, the correctness of a group of tests is easiest to determine if
it appears right in the context of the synopses.

Currently we also have TODO and SKIP marks in the test suite, hence we
can get a good estimation for how much of the Synopses are implemented
by Pugs at a quick glance.

*  Easier to detect mismatch between tests and docs

Everyone who has subscribed p6l can see the Synopses are changing
pretty fast. In contrast, many tests in the Pugs test suite are
terribly out of date even though Larry has the habit of updating some
tests after updating the documents.

But if we have enough smartlinks distributed in the test suite, the
mismatch between tests and docs can be detected in the form of
"broken" links. To get a list of broken links, we only need a single

  $ util/ --check t/*/*.t t/*/*/*.t

Unmatched smartlinks (if any) will be reported in great detail:

  ERROR: t/operators/quoting.t: line 173: pattern ``/Split result on
         words (no quote protection)/'' failed to match any paragraph
         in L<S02/Literals>.

  ERROR: t/data_types/arglist.t: line 6: section ``Operator renaming''
         not found in S03.

As I told Juerd on #perl6, the initial motivation of developing is simply to make myself more motivated to contribute
tests to Pugs.

Thanks to other Pugs members who worked on the smartlinks long before
I joined the Pugs team. Without their work, won't
produce anything meaningful today. :)

This mail is also posted to Audrey's blog in the form of HTML:


Agent Perl Programming lists via nntp and http.
Comments to Ask Bjørn Hansen at | Group listing | About