One of the most anxiety filled moment in web development is the transition from a staging version of a release to the live publication of a new codebase.While in a perfect world all the kinks will have been worked out in development and testing, its pretty common for even the most rigorously prepared release to develop flaws when released to the public that were either unnoticed in development or unnoticeable without a true client load on the system.

All of this means the timing of a release can have critical effects on the organizations' ability to do post release repairs.

Unfortunately releases are often set for the wrong reasons:

• To sync up with a specific event or opportunity
• To satisfy a benchmark established in a strategic project schedule
• "When we're done."
• When the load is light, and problems are exposed to the smallest possible audience

Failing to recognize the fact that a release date is the beginning of a time-critical stage of monitoring and cleanup, not just the end of a development stage, can leave the company ill prepared to receive and respond to the emergence of post release flaws and system stress management.

You're never too big for a crappy release, as this screen shot shows...

That Magic Moment

Releases are the most important moment in a company's evolution: they are where the internal divisive crap that it took to get something done (or fixed) ends and the public gets its first glimpse at what the company wants to present this cycle. In olden days, it is the equivalent of going to print. While digital content can be more easliy revised and repaired, this moment in time creates lasting impressions long after the reasons for those impressions fade, and define what your most active and vocal clients think of you.

For this reason it is imperitave that a push not be screwed up. This takes time and discipline, and a heathy dose of pessimism. Presume flaws, and allot time for human error and response. 

Pushed Releases

The event driven deadline is more of a grey area. If there is a specific opportunity attached to a release, make sure that there are two full business days (morning to night) before the opportunity to manage the release and adjust for it. Also, the release and release management is kept distinct from event management. Pushing a release too close to a specific event can make things worse, not better, especially if key personnel get diverted to event related activity and the communications chain is diverted from development to event and client management; as well, emotions run high and panic factors are elevated during key events, making post release management demoralizing and sometimes sloppy, creating further cascading effects

Morning Noon and Night

The best way to release a codebase upgrade is like the release of a new service: with a fully staffed, alert and rested team of engineers, CSR phone staff and any other resources needed to do revision and reallocation of resources. This is, from a tactical point of view, around 10:00, Monday through Thursday. That gives you at least two full days to do any work needed to keep the release on course or at worst case revert it to its previous state. I've heard arguments for evening releases exposing the company to fewer people. But for perspective, this is an argument coming out of PST. A 6:00 PM might take three or four hours to show sparks, putting the critical staff into party mode, or sleep. Worst of all is the Friday night release. In my view, it is better to expose a new release in the light of day and risk a brief but public embarrasment than to release at night. They are likely to amount to the same thing if a bug slips past you when you're too tired to think straight.

By the time they show up on the next day, rested from their last day of development, at 10:00 AM, it is 1:00 PM EST and the whole east coast has been suffering any side effects of an imperfect release for hours.

So the last criteria for a healthy release, minimizing audience and load, is an extremely ephemeral target, and it generally coincides with having key personnel to fix problems out of reach. Even if they can be reached in off hours, they are likely to be fatigued, uncoordinated and in off peak condition.

Release Stages

These stages extend backwards from the push. The net span of this activity should span at least a day -- longer for more eventful pushes.

1. The first stage of a release is a code freeze. This means new features, development, etc. ends and the code is put into a releasable state. It has always been my firm belief that code should freeze a day before the release. If you are working on code, by definition it is not ready to test and you have zero margin for failure. If for instance,
2. Following (and hopefully before) the freeze, any testing and patching is done. Some items may drop off the release schedule if they are deemed too buggy. Generally, testing and patching is done on staging servers. It is imperitve that staging servers be made as identical to the true servers as possible.
3. The updates are pushed (via SVN, SFTP, or whatever other mechanism works) to update the live site.
4. The live site is then tested to ensure the push went well.While, ideally, step 2 should give you 100% confidence that your live code is solid, there is ALWAYS the chance that some difference or error in procedure has jammed up the live version of the code. Leave nothing to chance: never just "throw the switch and check out."

Tiny Bite-sized Chunks

The worst release I ever oversaw was four months in the making. The biggest lesson learned was that from the clients' point of view, four one month releases would have been so, so much better. It would have focused our testing in concrete areas.

While there may be marketing reasons for a "big release" there are always ways of introducing and testing elements of the big release in anticipation of the public unveiling. Ideally, limited audience pre-release exposure and testing of elements will insulate you from shocking post-release discovery.

Even if your internal testers are the only ones who know of tiny pre-release cycles, do whatever you can to avoid large complex unveiling of new code systems. The potential "Big Win" for a new system is rarely worth the risk that a quantum upgrade involves.

That being said, if you are forced to launch a custom quantum change or new product. test the hell out of it. Bring in cold users who have not been in the dev team; graph out all possible permutations. Unit test, report, and devote whatever resources you can to skeptical analysis of the new feature or system.

A Healthy Release

In short, to make releases stable and healthy, they need to be timed with the availability of the people behind it in mind, including a healthy post-release schedule that is not overlapped into another critical event, which is common for an event driven release, or a weekend or evening, both of which diffuse a companies' ability to respond to emergent situations from the release.