Implementing Apache-Test Support into Simple Test

tap-reporter.php

All of the current Apache-Test support resides in this one file. The clas in this file is TAPReporter. It is a subclass of SimpleReporter and is really quite simple in nature. I'll post the stub and documentation of the class below:

<?php
    /**
     *	Tap Compliant Reporter
     *	@package	SimpleTest
     *	@subpackage	UnitTester
     *    @author     Mike Lively <http://www.digitalsandwich.com>
     *	@version	$Id$
     */

    /**#@+
     *	include other SimpleTest class files
     */
    require_once(dirname(__FILE__) . '/scorer.php');
    /**#@-*/
    
    /**
     *    Tap Compliant Reporter. Displays test results that SHOULD be 
     *    compatible with Perl's Test::Harness
	 *	  @package SimpleTest
	 *	  @subpackage UnitTester
     */
    class TAPReporter extends SimpleReporter {
        
        /**
         *    Does nothing yet. The first output will
         *    be sent on the first test start. For use
         *    by a testing framework such as Apache-Test.
         *    @access public
         */
        function TAPReporter($character_set = 'ISO-8859-1') { }
        
        /**
         *    Paints the content type header for the test. (text/plain)
         *    @param string $test_name      Name class of test.
         *    @access public
         */
        function paintHeader($test_name) { }

        /**
         *     Provides a total count of how many tests have passed, 
         *     failed, or been exceptioned in the test case thus far.
         *     @access protected
         */
        function _countTests() { }
        
        /**
         *    Paints the end of the test with a test plan.
         *    @param string $test_name        Name class of test.
         *    @access public
         */
        function paintFooter($test_name) { }
        
        /**
         *    Paints the test failure with a test number and description
         *    @param string $message    Failure message displayed.
         *    @access public
         */
        function paintFail($message) { }
        
        /**
         *    Paints a PHP error or exception.
         *    @param string $message       Failure message displayed.
         *    @access public
         *    @abstract
         */
        function paintException($message) { }
        
        /**
         *    Increments the pass count.
         *    @param string $message        Passing message displayed.
         *    @access public
         */
        function paintPass($message) { }

        /**
         *    Paints formatted text such as dumped variables.
         *    @param string $message        Text to show.
         *    @access public
         */
        function paintFormattedMessage($message) { }
        
        /**
         *    Paints a simple supplementary message.
         *    @param string $message        Text to display.
         *    @access public
         */
        function paintMessage($message) { }
        
    }
    

?>

If you were to see the code inside you would realize how simple it really was to add the basic support. All praise for that goes to Marcus Baker and the other original developers of the project. It's been a while since I have come accross such an easily extensible open source project.

What it Does

Like I have said before this new file adds just the basic syntax required by Apache-Test. It was the best I could do without having to modify some of the already existant classes. In my opinion it is a great starting point to implement full TAP functionality and in the process complete Apache-Test support. Here is a list of what the frame work does:

Test Plan

Test plans had to be held off for display at the end of each test. This isn't necessarily the best way to do it but it is somewhat mandated to be done that way barring some changes in the main SimpleTest API and underlying code.

Test Lines

Almost all of the test line support is there. Test numbers are calculated and displayed and test descriptions are given. The only thing missing here is the ability to add directives to tests. This again would take some heavier additions and modifications to SimpleTest.

Diagnostics

Diagnostic lines are implemented via the TAPReporter::paintFormattedMessage() and TAPReporter::paintMessage() functions. Any information passed to these two functions will be printed out as diagnostics. Some automatic diagnostics can probably be considered, things such as printing out the test name and what not. I however did not implement anything of that sort this time around.

Bail out!

There is no support for Bail out! I am not currently familiar if there are any functions in the Simple Test API that support this feature of TAP. If there is I do not know what they are, so if you do, tell me .

Possible Changes for Simple Test

Now following are some ways that I could see changing SimpleTest such that it would be able to more fully support all of the features of TAP:

Better Plan Support

As I have already mentioned, there is no way in SimpleTest currently to forecast the number of tests that will be made. While there are no real ill side effects of this, I do think it would be nice to offer the ability. The best route I could see in this is to add a function to SimpleTestCase called plan. This function could be passed a number that should be representative of the number of tests that will be performed. This would by no means be a function that users would be required to use. However if they would like to have better management over the number of subtests they would like to run in the script they would have that ability. The GroupTest class can handle these plans by simply adding up the number of plans each added test has. If any of the added tests do not have a set plan, then the GroupTest's plan would be automatically unset and the script would revert to the current behaviour of establishing and displaying the plan after the test has ended.

Directive Support

The TAP format has two different directives allowed for each test. TODO and SKIP. The TODO directive is used to denote portions of the application being tested that are incomplete. The SKIP directive is used if there is no need to complete the given test. There isn't a really easy way to integrate directives into the tests unfortunately. The best way that I can think of is by adding two more parameters to the various assert*() functions found in the SimpleTestCase class and its descendants. The first parameter would be something like $directive and would contain a constant (either SIMPLE_TEST_DIRECTIVE_TODO or SIMPLE_TEST_DIRECTIVE_SKIP) indicating which directive to use. The second parameter to be added would be $directivemsg and would be used to pass an optional description with the directive. If both parameters are ommitted then the behaviour would be as it is now and no directives would be printed out. These two parameters would also have to be existant in the following functions (and any overwrites of the functions):

• SimpleTestCase::pass()
• SimpleTestCase::fail()
• SimpleScorer::paintPass()
• SimpleScorer::paintFail()

The last change that would have to be made would be in the TAPReporter class's version of paintPass() and paintFail(). The only change there would be to simply add the two parameters and display them in the output (in the appropriate place.

bail out!

As I mentioned before, I do not know of any functions in the SimpleTest API that allow you to bail out of a test. My thought currently is to overide the SimpleScorer::paintError() function in TAPReporter to display the error as a bailout. The only thing I have against that is whether or not there will be instances when a bail out is not necessary. The workaround is to of course introduce a new function into SimpleTestCase called bailout($msg). This function would then call a method that would be added to SimpleRunner called paintBailout($msg) which would in turn call the same method in SimpleScorer. The default implementation of SimpleScorer::paintBailout() could be to just simply exit; out of the script. TAPReporter will of course extend it to print out the bail out! <msg> line before exiting.

Download the Work So Far

I am sure you would all like to see this all in action so I have created a new Tarball of simple test containing apache test. Before attempting to run the apache-tests I recommend you read through simpletest/apache-test/INSTALL-APACHE-TEST. After you have ensured that Apache-Test is installed properly just use the following commands in the simpletest/apache-test directory.

1. perl Makefile.pl - This will ask you the location of your httpd and apxs file. The location's mentioned should be right, if they are not you can probably just find them using which apxs or find / -name apxs
2. make test

and that's all there is to it.

*EDIT* I have updated tap-reporter.php since this post, for a brief overview of the changes go here. The link below has been updated so download away .

simpletest_apache-test.tar.gz