With thanks to Mike Ruggiero, who put me on the path of unit testing.

One of the big problems with unit testing is that it attempts to do the impossible: resetting your environment for each test, in a context (LAMP) where a true reset is impossible. The resulting compromise means that you must do a fairly rigorous assessment of your environment before embracing unit testing as a strategy.

Just to note: I have only attempted unit testing under PHPUnit; there may be implementations where this is not true, but I suspect most of the below is generally true in any Unit Testing environment in (and possibly beyond) LAMP.

The PHP Environment

PHP establishes several layers of information that may or may not be reversable.* This is not a "Flaw" as such, but a reflection of the fact that every PHP execution is the response to an HTTP Request; all these contexts are scrubbed at the end of the request when the result is returned to the browser. (and the CLI is just another form of a request for this purpose.)

This works in the real world -- but a unit test recirculates the same codebase several times, looping thorugh code that may not have been designed to execute more than once.

These are several contexts of information that PHP relies on. The "Reversable" column is the real key to whether or not a context is problematic in unit testing.

define('FOO', 3)

is_defined('FOO')

$GLOBALS['foo'] = 3

isset($foo) (in global context)  
or isset($GLOBALS['foo'])

class Fooer { ... }

class_exists('Fooer');

class Fooer{
static $bar;
... }

or Fooer::$bar = 3

isset(Fooer::$bar)
or Fooer::$bar > 0

MySql Environment

You cannot by definition "reset" a database entry. Even Transactions have to commit at some point to be meaningful. The only way to "reset a database" is to erase and reload it. By definition, "Undo" is not built into MySQL. INSERT, UPDATE, and DELETE do not have commesurate "UNINSERT", "UNUPDATE" or "UNDELETE" commands, nor does MySQL have any sort of SVN-like "revision" number

The unit test system suggests using "mock databases" to submit data into your object structure. This is all well in good for a fully OOP environment; however I'm currently in the process of unit testing a massive code library whose predecessor used low level mysql library calls that are impossible to fully encapsulate due to the sheer size of the codebase. The option that leaves me with is doing a full database reset for every test -- which will probably create a pretty huge overhead for the testing environment.

The Database Paradigm

Unit testing (nearly) demands that you use mock data access. If you are using a database abstraction, this is not a huge issue -- you simply create a mechanism to localize your data access -- either by having a "mode" in your DBA that pulls information in and out of a self-contained object structure, or a duplicate "Mock" DBA that has all the properties of your real abstraction layer.

However if your code has low level SQL access (no abstraction) you are somewhat screwed. The best you can do is keep a SQL file that is a "starting snapshot" and to load it before each test. Note -- I did not say test case. As this is often a large load, its a good idea to have two dumps: one that loads all the static data (that a test is not likely to chang) and one that has a transient dump (that resets all the tables that a unit test might alter). figuring out which rows go in which dump requires differential analysis (say a before/after diff on your database). The smaller differential dump can load on each test, while the larger initial dump loads for each test case (or suite).

Also: by definition, unit tests that rely on direct database access cannot run concurrently! If concurrent unit test execution is a much try experimenting with temporary tables. Temp tables have a lifespan equal to a request and don't interact across sessions. I have not actually walked all the way down this route but I made some experiments that seemed successful at first blush.

The Unit Test Paradox

PHPUnit attempts to "Reset the environment" for every test. However by definition even if it could reset the entire environment, this would be a problem. Say, for example, it were possible to reset (erase) class definitions. That would mean the unit test suite were capable of resetting its own class definitions, because the unit test suite is itself a PHP Class!

Its a bit like using rodents for cancer research. I'm certain that given the amount of research money being put to the subject, we will eventually cure mouse cancer.

Getting Along with Unit Testing

The only real way for unit testing to work perfectly is for PHP /Zend to hard code it into the language; we would need a meta-php and a meta-sql, which stores virtual versions of its contexts that reset magically upon command and coordinate with each other. I.E., I don't think this is going to happen any time soon.

In the meanwhile, go into testing with the expectation that you are entering into a compromise in order to get your tests working.

• To ensure that your unit tests behave, you need to introspect into their elements in the areas that Unit Testing does not do automatically.
• Avoid using a combination of defines and globals -- or disable global housekeeping in PHPUnit ***
• Avoid using statics to store transient information -- or cleanse them yourself.
• Understand and manage your data adaptive strategies either with a mock data class system or a test database you can safely re-scrub.

There is a mantra against statics in general; this is one of the instances that proves that mantra accurate.You will have to either eliminate transient statically stored data from your code, or scrub it in the setup methods of your tests, to keep your code unit testable.

One example of a "survivable breach" of this pattern is non-transient static fields. For instance, because constants cannot be array typed I have a lot of statics that are simply predefined arrays of options in my code base. This violates the rule, but not the spirit, of the ban on static variables.

Also I use table definitions within my Zend_DB manager as singleton static objects (Zend_Db_Table child classes) that lazy load. While I could simply stop using statics, retuning a new table object every time one was needed, this would be wasteful. I know for a fact that the table objects only contain table metadata and the adapter itself, so I prefer to let them stay as they are, manually managing their adapters within the unit testing context.

If I were starting this project from scratch I wouldn't have to worry about database context, because my database adapters could be redirected towards mock data. However my project doesn't give me that luxury.  For now I am scrubbing and reloading the data at the SQL level for each test, in order to keep my Zend DB data and the heritage low level SQL calls in sync.

Other Forms of Testing

When I first started working on this project I took an abortave stab at unit testing and gave it up because of the abovementioned contextual problems. Having returned to Unit Testing, I wanted to at least give nod to the other methods of testing that I used in the meantime.

Reports

I created a series of reports that used the object components I developed to give data that the customer facing application did not. Each report gave background data on a customer job that was previously unavailable. What you lose here is true coverage; the report was only examined to analyze a problem, and was therefore not predicting or preventing problems, which is the goal of unit test coverage. But it was giving us better insight into the hidden mechanics of our code that were producing behavior that we could not otherwise easily analyze.

Log Dumps

I established a series of dumps for each request cycle; this is essentially another form of report, but one that was less specific than my targeted web based views. I started out with simple error log dumps but these were porblematic for several reasons: 

1. The error log was clotted with preexisting (and new) error messages
2. It was troublesome to view which messages were generated by which requests
3. It adds a bit of overhead to the application, and involves itself with the file permission system (which is solvable with database based logging).
4. It provides systematic information on the whole system -- but that much data is difficult to parse when you have a particular problem whose scope hasn't been identified.

Where Unit Testing Excels

Now that I am reinserting unit testing, I wanted to emphasize the reason and virtue of the unit testing strategy, and what I am looking forward to.

1. Unit testing allows you to systematically test for many combinations of situations before they occur in the customer facing application.
2. Unit testing allows you to examine specific execution paths to more quickly diagnose where the problem in your code is occuring.
3. Unit testing forces you to write deterministic tests: "If A and B THEN C". It is quite possible to write massive codebases empirically without clearly stating the dependant logic behind it. Without this as a pre-stated premise, you end up backwards-engineering your codebases logic only when it breaks and customers are upset.
4. It locks in your wins. That is, if you write a patch that solves one problem but creates another, unit testing alerts you to problems in areas that you might not otherwise involve youself in response to a customer complaint.

 Code Coverage

The problem in some cases with code coverage is that you often have to manually pre-determine the expected results from many different combinations of data. When each combination must be manually instantiated within the application, this is a bit tedious. If you can automatically create M x N x O x P test cases, then you are set, but generally unit testing revolves around a finite combination of circumstances that represent the whole (Synecdoche).

Some intelligence is required for representational data. If you are testing a sales tax application, you are not as likely to get meaningful tests by testing fifty ranges of costs (say $10 ... 500) as you are testing different states. Most code does not vary by quantity -- it varies by quality. If, however, you have an incentive program that kicks in at a certain cost level (say, 5% off for purchases over $1,000), then quantity (cost) IS a quality. You would want to test against a reasonable subset ($0, $100, $999, $1000, $10001, $2000) of data. More granularity than that is probably a waste. However if you have several different incentives at different cost breaks, you might want to determine not only what the final price is for a wider granularity of tests (say $100 to $5000) but to test which incentives are applied in each context. But here you would have to manually determine which incentives you expect (probably with a spreadsheet) to set the expected results.

The point is, in order for unit testing to work, you have to have a mechanism that is NOT your codebase to generate the expected results until you have validated your codebase. From then on, you can dump your codebases results and use them as the basis of further unit tests.

The important distinction here is what is the "meta question" your unit tests are asking: is it "Does my codebase work" or "Does my code base still work" ? The answer to this question determines what strategy you need to take in developing the baseline data for your unit tests. If you presume your codebase works and use it to generate baseline data, your unit tests will at best validate system consistency. They will not validate your systems underlying logic unless you generate baseline data outside of the system (with a spreadsheet, calculator, paper and pencil).

Seperate Paths

Is Schodinger's cat taking a dump in your shoes? Its easy to assume that the order in which you query your system is irrelevant to the answer you get in many cases, but you really don't know until you ask each question in an isolated context. This is because in order to "improve efficiency" (note -- efficiency has nothing to do with accuracy) programmers often save results the first time a question is asked, which makes you ignorant to the possible side effects of changes in dependant data.

Say, for instance, you have an OOP shopping cart. Your cart has a method sales_tax(); you decide to "speed up" the cart by caching it in a private $_sales_tax parameter. This works fine if you:

1. add items to the cart
2. display their costs
3. save the data

but what if you display a running total of sales tax? Your tax will always reflect the sales tax on the first item of the cart.

A unit test doesn't depnd on reflecting your code's UI so you can write tests in many combinations of scenarios, even ones that don't currently execute in your real codebase. You can add items, call sales tax, add more items, and call sales tax again, testing it against the value you predict from your spreadsheet.

Determining the truth

The underlying premise of your system can easily get lost in the execution. Unit test writing forces you to remember the underlying causal rules of your system and often forces you to read low level code that has long since been left dormant, as it is accessed several layers below the current focus of the company.

When code is written, the expected inputs are usually quite finite. Circumstances that change these finitie combinations usually are developed at much higher levels, and much later. While your unit tests will not "Magically" create more baselines for themselves, its much easier to remember to expand your baselines than it is to manually examine your entier codebase against the new circumstances.

Unit tests are a great lever to reflect the effects an expanding dataset has against your codebase, and find the areas that fail. While expanding your test suite is a drag, especially if you maually create baselines, revisiting the testing suite every time you expand your dataset to build a different feature in your system is revelative. It may even allow you to remember a "Hack" that will cause problems before you even begin unit testing.

"You're all bad programmers."

I went to a Zend conference in which the Zend keynote speaker asked, "You're all doing unit tests on your systems, right?" After a few seconds of awkward silence, he followed up with "Well, maybe you are just not very good programmers."**** While it was meant (I think) as a joke, there is something to that. Unit testing is one of the benchmarks of development both for a company and a developer. If you are unwilling or unable to put the underlying premise of your system to a test -- unit or otherwise -- then you do not leave any artifact or benchmark for future development to work against.

Without a cybercanary that goes off when the system fails at its underlying tasks, you are relying on your powers of observation every time you alter the system. And often its simply not realistic to expect that the symptoms your change creates will be immediately visibile to you, your testers (you do have testers, don't you?) or even your final customer base.

The fact is, the type of person who chooses PHP over, say, Java or .NET perfers freedom and pragmatism over structure and convention. You don't needs a templating "System" in PHP -- just close your script (?>) and type some markup. You can even do this in the body of a function. Better yet, you can buffer the results and spit it back as a return value. Don't like OOP? Don't use it. You can get a huge amount of work done with anonymous hashes and arrays. These and other options are why PHP developers bury Java developers in the dust. However, PHP is essencially an extension of PEAR, a legacy UNIX scripting language notorious for giving programmers enough rope to hang themselves and anyone who'd follow in their footsteps. PHP may have a more uniform syntax, but it does give programmers the same capability of generating unmaintainable code; as such, while structure may not be an obligation of the frist pass of a project, adding structure to successive development is a moral imperitave, and at lest trying and understanding the midddleware frameworks that do this for us is fundamental to sustainable development. Unit testing is the glue towards building systems that are the foundation of sustainable codebase, and sustainable companies. Because no client will ever force you to develop unit tests, it is up to you to insist to your clients that they need tests in order to have any chance at long term survival -- and in many cases, even short term acceptance.

The guy who broke me into application development was a Microsoft veteran. His philosophy on testing was "We don't need tests -- we have customers." If you don't have a testing system in place, then you are in effect saying, "I'm fine with waiting for my customers to complain before I begin examining my codebase for flaws." Even if there were a shred of ethical behavior behind this outlook, it still doesn't address the fact that even after a bug is found, fixing it is not always enough. What about the many transactions that have been logged with the bug in place: do you expect your customers to go back and re-bill their customers? Do you plan on giving them a net profit/loss statement to let you know how much each bug costs them?

Every business survives in an imperfect state; nobody is comparing their results against an inhumanly perfect system. However putting test protocols in place is the best insurance policy you can have against your own limitations, and gives you reason to have confidence that feature development builds upon a solid foundation. It also allows you to improve foundational code to expand its abilities and scope, and a mechanism to measure the blowback fo doing so.

class CostModelTest extends PHPUnit_Framework_TestCase
{
        protected $backupGlobals = FALSE; // you can keep your globals