| Note: these documents may be out of date. Do not use as reference! |
To see what is currently happening visit http://www.perl6.org/
JART - Just Another Regression Test
Maintainer: Michael G Schwern <schwern@pobox.com> Date: 30 Aug 2000 Mailing List: perl-qa@perl.org Number: 182 Version: 1 Status: Developing
Regression tests are often contrived and do not represent real world usage nor test features in combination. JAPHs represent a repository of real world code which is simple to turn into regression tests. Use JAPHs to test Perl!
Testing code is often contrived and incomplete. Its often optimistic, simplistic and doesn't stress the program as much as real world code will. It would be best to use real world code in our regression tests, but unfortunately it is difficult to collect code which is itself simple to test. Complex tests which force interactions between features could be written with effort, but traditionally few have wanted to put in the effort on writing tests.
So it would be wonderful to have a set of real world code which stresses all the features of Perl, but is also easy to test. Fortunately we do. JAPHs! Hundreds and hundreds and hundreds of freely available small bits of test code using almost the entire range of Perl and all with the same output!
The main purpose of the JART is to test backwards compatibility and also to test existing perl features in combination with each other.
The process of converting a JAPH to a test is straight forward.
JAPHs can be actively collected from various sources: clp.misc archives, p5p archives, CPAN/misc/japhs, Abigail's hard drive, etc... It shouldn't be terribly hard to write simple clients to pull JAPHs from the signatures of known authors.
Also, it will be hoped that JAPH authors will submit their work directly to JART.
JART need not be limited to JAPHs. Any other eclectic code with predictable output could be used. Obfuscated Perl Contest entries come immediately to mind.
Only certain JAPHs can be used for the JART and they will have to be audited. There are three considerations: security, features and portability.
Since the JART will be running as trusted code, it must be made sure that the JAPH doesn't do anything nasty to the system. This includes writing files, opening sockets, spawning processes, etc... Future versions of JART may run in a safe sand box making the security audit less important.
Many JAPHs make use of undocumented features and subtle bugs in Perl. It must be made sure that the JAPHs used in JART rely only on documented features. As a derivative benefit I'm sure we will discover many undocumented features which should be documented. These will be reported back as patches to perl.
Finally, JAPHs may rely on files, programs and features which are not portable. These should be checked for and if found either the JAPH is rejected or it is only run on certain systems.
This auditing is the most manpower and clue intensive part of the JART, and is aggravated by the inherently obfuscated nature of the code. This is somewhat alleviated by the small size of the code. JAPHs which are collected will have to be audited. JAPHs which are donated should come with an audit by the author.
Once a JAPH passes the audits it is made a part of the JART. This entails adding it to the list of JAPHs and noting its expected output.
A pleasant consequence of gathering lots of JAPHs is that we're gathering lots of JAPHs! This can be spun off into a nice, annotated JAPH archive for people who's lives aren't already weird enough.
JART will act like any other of Perl's regression tests, probably added to perl as t/JART.t. It will run each JAPH and note its output and print the usual "ok/not ok".
The JAPHs will be organized one per file. This avoids the issues of having to escape out characters in the JAPHs and getting spaces and the number of characters exactly right. As the output of each JAPH is subtly different, each JAPH file will have an associated output file which contains the expected output of each JAPH. This will all be tossed into one directory, probably t/JART/JAPHs/.
A prototype implementation exists.
Auditing will probably be done by simple way of a mailing list. Newly arriving JAPHs will be posted and auditors (subscribers to the list) will perform audits at their leisure. An incoming queue will also be maintained, accessible by auditors, to make sure things don't get lost in the noise.
JAPH authors submitting their work will be given a questionnaire to fill out to allow them to conduct their own audit of their work.
None.