Enhancing Defect Removal Efficiency (DRE) using Adoption Rate for Released Products
Introduction:
The classic Defect Removal Efficiency (DRE) calculation has been successfully used to determine the efficiency of the defect removal process at various stages of the Product Development Lifecycle.
DRE(1) is calculated as follows
The value of
the DRE is a percentage and the higher the percentage the better
because it
represents the timely identification and removal of defects at any
particular
phase.
This calculation can be extended for released products as a measure of the number of defects in the product that were not caught during the product development or testing phase.
Hence the DRE calculation ( 1 ) can be modified for released products to calculate the Testing Defect Removal Efficiency (TDREReleased)
TDRE
for released products can be calculated as
T
Missing Parameters:
A couple of factors that are not accounted for by this metric is the number of customers using this product after it has been released and the time for this which this product has been available for customers to use. Both of these factors significantly affect the determination of the defect removal efficiency.
Number of Customers using the product: The greater the number of customers using a particular product, the greater the chance of finding a defect in the product after completion. Some inherent assumptions made here are that all customers are exercising many different components of the product. (This is true in the case of webMethods where the products are configurable and customizable in many ways based on the needs of the customer.)
Time after Product Completion: The time period for which the TDREReleased value is calculated is significant to understand and compare the value. The longer the duration the higher the chance of customers using the product and hence the higher the chance of issues being found. The assumption made is that there are a few customers starting to use the product with every passing period of time after the product is released. (This fact is also true in the case of webMethods.) Another assumption made is that the customer adoption pattern follows the classic theory of the “Innovators”, “Early Adopters”, “Early Majority”, “Late Majority” and “Laggards”.
Factoring Missing Parameters:
It might be worth combining these two parameters, Number of Customers and Time After Product Completion into a new parameter, the “Product Adoption Rate” (PAR), calculated by:
PAR can be used as a basis for comparing the TDREReleased values of two versions of the same product or for two products. A consistently high PAR value over a period of time would indicate a high adoption rate. Just by looking at the trend of the PAR value over time, one can guess the adoption pattern of that product. Also a high PAR value product means that the product is in demand and might be a likely candidate for a strategically significant initiative going forward for the organization. A diminishing PAR value or a sudden decrease in PAR value of a product could be used as an alarm for investigation. The PAR value is a good indicator of an organization’s product strategy. For a new product, a consistently low and diminishing PAR value clearly indicates a product that is not required in the market and denotes poor product strategy if adoption is any indication of good product strategy (excluding the effect of other parameters like sales/pricing/marketing etc).
Incorporating Missing Parameters:
Incorporating Product Adoption Rate with Testing Defect Removal Efficiency helps compare like products with each other. It eliminates the comparison of a lesser adopted product with that of a higher adopted product because a higher adopted product is likely to have a greater number of fixes created for it. Integrating PAR with TDRE also involves the two significant factors “Customers” and “Time” into the DRE calculation.
Testing Defect Removal Number (TDRN) can be calculated as
TDRN = PAR * TDREReleased ( 4 )
(Since this is not a percentage, we cannot call it efficiency and hence the modified DRE is just a mere number)
There is an inherent advantage to using PAR as a multiplier to calculate the TDRN. To compare TDRN values across multiple products for which the PAR is not comparable, multiply the lower TDRN value with the multiplication factor between the PARs and the TDRNs are now comparable. Though this is not a recommended practice from a business perspective, it is doable.
Analysis:
There are three parameters involved and hence based on the values of the three parameters, different conclusions can be drawn. Below is a summary.
TDREReleased = Low
PAR = Diminishing
TDRN = Low
Clearly indicates Low Adoption Rate and Low Defect Removal Efficiency resulting in low over all TDRN. This can be a result of poor product strategy and poor implementation.
TDREReleased = High
PAR = Increasing
TDRN = High
Clearly indicates High Adoption Rate and High Defect Removal Efficiency resulting in high over all TDRN. This is where we want to be. This is a classic case of creating a product that is in tune with the market’s requirements and good implementation.
TDREReleased = High
PAR = Diminishing
TDRN = Close to PAR
Clearly indicates Low Adoption Rate and High Defect Removal Efficiency resulting in TDRN close to PAR. This is a case of poor strategy but excellent implantation.
TDREReleased = Low
PAR = Increasing
TDRN = Away from PAR
Clearly indicates High Adoption Rate and Low Defect Removal Efficiency resulting in a TDRN well below PAR. This is a case of good strategy and poor implementation.
Enhancements:
The TDRNReleased is currently not a percent value. This metric needs to be modified or adjusted to either convert it to a percent value or calculate a benchmark for comparison.
The TDRNReleased needs some validation based on real data and that will be incorporated shortly.