IT Quality Control

(Contrast with: quality management, quality assurance)

If we deliver on time, but the product has defects, we have not delivered on time.
- Philip Crosby, quality control manager of the Pershing missile program

IT Quality control is the process of testing software intensive systems to uncover defects and hence measuring actual quality. In the software development context test candidates can be specifications, design descriptions, code listings, executable software modules, units, subsystems or complete systems.

Quality Control Activities

Quality control activities should occur throughout the software development life cycle. The earlier defects are discovered the less expensive they are to fix.

Quality control activities include:

Defining and classifying the severity of defects
Inspecting documents
Inspecting code (either manually or via automatic static code analysis)
Testing executable software. For example: module, unit, integration, system and acceptance testing
Recording of defects
Setting quality control limits outside of which corrective action must be taken (refer figure: Run Chart, upper and lower control limits)
Tracking corrective action on defects
Defect data analysis - e.g. tracking defect trends over time (refer figure: Run Chart).

Note that quality control metrics are used as an indicator of software development process effectiveness and act as a trigger for process improvement.

Measures of Software Quality

The most common measure of software quality is defect density expressed as defects per thousand lines of code. Defect density is measured at various points in the software product's life cycle. ISO/IEC 9126 Software engineering - Product quality¹ provides many additional measures.

The defect density of delivered software is a simple number by which a software development organisation can be judged. All software development managers should know how their group rates.

Metric	Formula	Significance
Code defect density (in system test)	NDT/KLOC	Indicates the quality of the code presented to the system test group
Code defect density (in operation)	NDO/KLOC	Indicates the quality of the code delivered to the customer
Defect removal efficiency	NDT/(NDT + NDO)	Indicates the effectiveness of the test function in removing defects from the delivered software product

Where:
KLOC = Thousand Lines of Code
NDT = Number of Defects Detected in Testing
NDO = Number of Defects Detected in Operation

Software Quality Benchmarks

When defining software quality control limits, what are reasonable values for defect densities? When should we decide that our software development process is out of control and needs serious improvement? At what point should a customer reject a custom developed software product if failures are experienced in operation?

This is a difficult question as tolerable defect densities in software vary widely as a function of application type and criticality. For example we routinely tolerate high defect densities in word processing software whereas the avionics software that controls flight surfaces in an aircraft must have extremely low (or zero) defect densities in delivered software. Valid comparisons of defect densities across application domains and organisations are also hampered by variations in the process of defect classification and data analysis methods. Organisations are also understandably reticent to air their dirty linen and share defect density metrics.

In his book Code Complete² Steve McConnell goes out on a limb and provides the following guidelines:

Context	Defect Density (defects/KLOC)
Context	In-house Test	Delivered Software
Industry average		15 - 50
Microsoft Applications Division	10 - 20	0.5
Clean room development (Harlan Mills³)	3	0.1
Space shuttle software		0 defects in 500,000 lines of code