Software Testing

(Alias: acceptance testing, system testing, integration testing, unit testing, module testing )

Testing is the process of executing a program with the intent of finding errors.
- Glenford Myers, The Art of Software Testing

Software testing is best defined in terms of testing objectives. Any definition should be suitable for inclusion in the introduction to your Test Plan. Testing objectives need to be stated in a form that testers can understand and achieve. Testing objectives also need to speak to the attitudes of the tester. Fundamentally testers need to expect to find errors. Glenford Myers' definition provides one very important objective but there are several others.

A healthy attitude towards testing is an aggregate of the following perspectives listed in increasing order of maturity. Each attitude is translated into a hard objective you can include in your test planning documentation. Note that the most advanced and effective attitude towards testing is summed up at maturity level 4 Testing is a state of mind.

Attitudes to Testing

	0 Testing is debugging There is no difference between testing and debugging. Other than support of debugging, testing has no purpose.
	Objective: identify any defects injected by programmers and not detected in normal debugging
	1 Testing proves the software works The purpose of testing is to show that the software works.
	Objective: validate and verify software against its specifications and design descriptions
	2 Testing proves the software doesn't work The purpose of testing is to make the software fail.
	Objective: place the software under extreme stress to identify all plausible failure modes
	3 Testing is risk reduction Testing increases our level of confidence that the software will not fail when delivered to the customer; more testing means less risk.
	Objective: reduce the risk of delivering defective software to a customer by rigorous testing conducted by an independent group not responsible for building the software
	4 Testing is a state of mind Testing is not a single act. It is a mental discipline that results in software that doesn't need much testing. Testable code has fewer bugs.
	Objectives: engage with the software development processes during all phases of the product's life cycle to ensure that: Requirements specifications reflect the customer's needs and are testable Design descriptions address all customer's requirements and describe software that will be observable and testable Code implements design descriptions and is free from defects through code inspections, module testing and unit testing Identify defective emergent properties in the integrated system through integration testing Identify any failure modes that may occur when the system is placed under stress of high volumes and transaction rates and hostile environments through comprehensive system testing Identify any failure modes that may emerge when the system is integrated on-site through site integration testing Identify any noncompliances with customer's requirements prior to placing the system in common use through acceptance testing Identify any faults induced by maintenance activities through comprehensive regression testing of all changes.

Case Study: Chernobyl - Catastrophic Failure of a Test Program

The image above is a photograph taken from space by an American space shuttle mission. At the top of the photograph is the Chernobyl nuclear reactor adjacent to it's cooling pond. The Chernobyl nuclear accident (April, 1986) came about because of an ill conceived and poorly planned and executed test on an unstable reactor design. The result was a disaster. An explosion and fire released large quantities of radioactive contamination into the atmosphere, which spread over much of Western USSR and Europe. It is considered the worst nuclear power plant accident in history.

The reactor design. A graphite moderated and water-cooled reactor controlled by 211 control rods. The reactor was operated with a complex computer control system.

The test objective. The test was to determine whether, after steam had been shut off from the turbo generator, the inertia of the still rotating generator would be sufficient to generate enough electricity to operate auxiliary motors that were part of the reactor emergency cooling system. Note that the nuclear power plant did not rely on its own electricity for operation. Offsite electricity was provided. The objective of the experiment was to see how long the locally generated electricity would power the main pumps that keep the cooling water flowing over the fuel.

What went wrong. The test procedure was not prepared or approved in the proper way and was of low quality. In addition:

The operators deviated from the plan which created emergency conditions
The reactor was to be run at low power so the automatic control system had to be switched off
At low power this reactor design is unstable
The operators inserted the graphite rods but the power dropped off to quickly. They then tried to increase power by pulling out the control rods. They eventually withdrew all the control rods to get the power back up. Pulling all rods out is dangerous as a minimum of 30 rods are needed for emergency shutdown. At this point only six to eight rods were in the core
The operators were manually controlling the flow of water into the reactor. This is very difficult as the reactor is unstable at low power and would normally be shut down automatically
To stop an automatic shutdown the operators short circuited the emergency protection signals
In its unstable state the reactor suddenly increased its power output
The operators manually initiated the emergency SCRAM button which inserts all rods into the reactor. It was too late. The great amount of heat produced by the increased rate of fissioning caused an explosion
Enormous pressure built up in the vertical pressure pipes, which lifted the concrete slab above the reactor, took off the roof of the building above the concrete slab, and blew the fuel out.

A bad attitude to testing. Chernobyl 4 was a model plant. It ran very well. Its operators viewed themselves as an elite crew; they had become overconfident. The operators did not think carefully enough about the impacts of their test on the reactor. They were not risk aware. In other words they were running a test but they didn't expect to find an error. This is generally a bad attitude in a tester. The operators had only seconds to react to the emergency - a bad situation in any safety-critical plant.

Plant management blamed the operators. But scientists from other countries agreed that the unstable nature of the reactor design made it difficult to control. You can not make an unsafe design safe by having clever operators.

Lessons learned. The operators at Chernobyl had become complacent. They slipped into the dangerous attitude that an accident could never happen. The testing process was not capable of demonstrating enhanced reactor functionality or improving reactor safety; it only indicated how unsafe it really was. A few simple steps could have avoided a global disaster: