Effective Software Development Metrics and Measurement Practices

Metrics That Matter Real Measures to Improve Software Development Steven Borg, Principal ALM Consultant Steven.Borg@nwcadence.com

Every best in class company measures software quality. There are no exceptions. If your company does not do this it is not an industry leader and there is a good chance that your software quality levels are marginal at best. Capers Jones, Applied Software Measurement

You get what you measure. If you don t measure it, you can t manage it. You cannot improve what you can t measure. Garbage in, garbage out. If you don t measure it, it s just a hobby. "You can't manage what you can't control, and you can't control what you don't measure. " Tom DeMarco

Metrics Matter Without metrics you can t predict you can t judge quality you can t accurately estimate you can t measure impacts you can t improve consistently

Characteristics of Effective Metrics Comparable Honest Simple Actionable

True Metrics vs Proxy Metrics True Metric What you SHOULD measure Proxy Metric What you CAN measure Tip #1 Focus on the true metric, not proxy metrics

The companies that wish to improve but do not measure are at the mercy of fads and chance. Progress may not be impossible, but it is certainly unlikely. Capers Jones Tip #2 Only introduce one or two new metrics at a time

Deming Cycle 1st 2nd 4th 3rd

ITIL Seven Step Improvement Process 1. Define what you should measure. 2. Define what you can measure. 3. Gather the data. Who? How? When? Integrity of the data. 4. Process the data. Frequency, format, system, accuracy 5. Analyze the data. Relationships, trends, according to plan, targets met, corrective action 6. Present and use the information. 7. Implement the corrective action. ITIL: Information Technology Infrastructure Library

Tip #3 Standardize by setting triggers to alert you to slips in prior metrics A few effective metrics: Quality Metrics Sizing Metrics Productivity Metrics User Metrics Business Metrics

Quality Metrics Defect Removal Efficiency Warning: Not always simple Code Coverage Warning: Not always honest Performance Profiling Code Coverage Test Cases per feature Bugs per feature (bug density metrics)

Sizing Metrics Lines of Code Warning: Not always honest Function Points Warning: Not simple Effort (in hours, days, weeks, etc) Warning: Not always honest or comparable Story Points Warning: Rarely comparable

Productivity Metrics Velocity Warning: Not always simple Throughput Lines of Code / Developer / Day (???) Quality* Note: Defect Removal Efficiency is highly correlated with Productivity So highly correlated it can often act as proxy metric for productivity

User Metrics User satisfaction Warning: Not always comparable Post-release bug count Number of Help Desk calls Warning: Not always honest

Business Metrics Schedule Estimation Accuracy Warning: Not always honest Cost Estimation Accuracy Warning: Not always honest Scope Estimation Accuracy Warning: Rarely honest ROI Estimation Accuracy Warning: Rarely simple , Not always honest

Tip #4 First identify your problem, THEN identify the metric The problem He broke the Build!!! Tip #5 Balance introduction of new metrics across different categories

Tip #5 Balance introduction of new metrics across different categories The problem today is not a deficiency in software measurement technology itself; rather, it is cultural resistance on the part of software management and staff. The resistance is due to the natural human belief that measures might be used against them. This feeling is the greatest barrier to applied software measurement. Capers Jones, Applied Software Measurement

Tip #6 NEVER use metrics to evaluate individuals Tip #7 Watch your metrics closely; adjust when necessary

Estimation and Planning There is a perfect correlation between measurement accuracy and estimation accuracy: Companies that measure well can estimate well; companies that do not measure accurately cannot estimate accurately either. Capers Jones "Optimism is an occupational hazard of programming: feedback is the treatment." Kent Beck Planning and estimation are the mirror images of measurement. Capers Jones

ALM (Application Lifecycle Management Tooling) Next Slide

Tip #8 Count first, Calculate next, use Judgment last Why ALM (Application Lifecycle Management Tooling)? "Tools are very important element of defining a path of least resistance. If I can set up a tool so that it s easier for a developer to do something the way that I want the developer to do it, and harder for the developer to do it some other way, then I think it s very likely the developer is going to do it the way I want them to, because it s easier. It s the path of least resistance." Steve McConnell

Application Lifecycle Management Tooling Tip #9 When possible, use an integrated ALM Tool to gather metrics Tip #10 Start using the built-in reports right away!

Cultural issues with Metrics The problem today is not a deficiency in software measurement technology itself; rather, it is cultural resistance on the part of software management and staff. The resistance is due to the natural human belief that measures might be used against them. This feeling is the greatest barrier to applied software measurement. Capers Jones, Applied Software Measurement

"The problem of software process change are often complicated by the fact that no one is responsible to make it happen. If software process improvement isn't anybody's job, it is not surprising that is doesn't get done! If it is important enough to do, however, someone must be assigned the responsibility and given the necessary resources. Until this is done, software process development will remain a nice thing to do someday, but never today." Watts Humpthey

Tip #11 Put someone in charge of process improvement use metrics to show change "If it ain't broke, don't fix it," the saying goes. Common software development practices are seriously broken, and the cost of not fixing them has become extreme. Traditional thinking would have it that the change represents the greatest risk. In software's case, the greatest risk lies with not changing - staying mired in unhealthy, profligate development practices instead of switching to practices that were proven more effective many years ago. Steve McConnell

Tip #12 Start collecting metrics now. Next year, you shouldn t be saying I wish I had some metrics Steven Borg