Finding Content in a Capacity-Based World: Three Case Studies

UNCLASSIFIED Department of the Air Force I n t e g r i t y - S e r v i c e - E x c e l l e n c e Finding Content in a Capacity-Based World Three Case Studies from the World of SW Cost Estimating Babak Damadi Meghan Kennedy Jonathan Lister 10 December 2024 UNCLASSIFIED

UNCLASSIFIED Intro Over the last 10 years, content-based Equivalent Source Lines Of Code (ESLOC) estimates have become less prevalent and ESLOC is used less and less Initial replacement estimates were capacity-based (FTE * Labor Rate), however, little effort was put into defining what content that capacity could deliver Over the last few years, AFCAA has looked at different ways to incorporate content back into these capacity-based estimates through data collection and metric derivation The names are made up, but the problems are real. The following examples are inspired by actual programs These are not definitive solutions to software challenges, but instead offer innovative perspectives and fresh approaches to understanding and tackling software issues I n t e g r i t y - S e r v i c e - E x c e l l e n c e UNCLASSIFIED 2

UNCLASSIFIED The Need for Content-Driven Estimates Multiple customers and stakeholders of the cost estimate {program organization (program management, contracting, finance, etc.), acquisition authorities, higher headquarters, users, and developers} across the life cycle Three basic estimate purposes: Strategic long term broad look; early planning/affordability estimates across multiple programs or specified program Operational mid term planning look; varied degree of estimate detail targeted at service/portfolio/program levels Tactical near term management look; detailed program estimates and monitoring for specific programs Variety of Quality Data Required to Support Quality Estimates for All Users and Purposes I n t e g r i t y - S e r v i c e - E x c e l l e n c e UNCLASSIFIED 3

UNCLASSIFIED Users and Decisions Behind the Demand Decisions Supported Early Planning Estimates Competing Requirements & Priorities Broad Alternative Analysis (e.g. AoAs) Establish Program Baseline Users AQ, OSD Equivalent Strategic Long Term CAPE, Services Milestone Decisions POM/PB Budget Allocation Operational Mid Term PEO, MAJCOMs Contract Award, Execution & Performance To Complete Estimating Tactical Near Term Developers, PMO Many levels of communication Information and data needs vary across the levels I n t e g r i t y - S e r v i c e - E x c e l l e n c e UNCLASSIFIED 4

UNCLASSIFIED Software Estimating: The Agile Movement Waterfall Development Agile Development Source: University of St. Louis Missouri Metrics: SLOC, Function Points Metrics: Story Points, Features, Velocity Estimate Type: Content-Driven Estimate Type: Capacity-Based I n t e g r i t y - S e r v i c e - E x c e l l e n c e UNCLASSIFIED 5

UNCLASSIFIED Department of the Air Force I n t e g r i t y - S e r v i c e - E x c e l l e n c e Utilizing SW Management Tools and Subjective Metrics Presented by: Jonathan Lister UNCLASSIFIED

UNCLASSIFIED Terminology Example High Level Requirements Length: Years Measurement: Measured at Lower Levels Initiative 1 Initiatives broken down into agile efforts that can be accomplished within a 3-month Program Increment Length: Months Measurement: T-Shirt Sizing early on, Measured at Lower Level later on Epic 1 Epic 2 Epic 3 Epics broken down into agile efforts that can be accomplished in a 3-week sprint Length: Days-Weeks Measurement: Story Points Issue 1 Issue 2 Issue 3 I n t e g r i t y - S e r v i c e - E x c e l l e n c e UNCLASSIFIED 7

UNCLASSIFIED Using Subjective Metrics Ideally, metrics have the flexibility to be used outside of the program it was collected on The argument for SLOC or Function Points is that they are a universal metric The more subjective agile data points tend to be specific not only to that program, but the individual team that they were collected on. Velocity (SPs/month) 111.3 90.6 80.6 76.7 94.1 33.6 Is the Team 6 30% as productive as Team 1? Not necessarily. Team 1 Team 2 Team 3 Team 4 Team 5 Team 6 Not only are story points inherently subjective, decomposition of requirements into agile parameters (features, user stories, etc.) is also subjective However, that does not mean these parameters are not useful in creating an estimate I n t e g r i t y - S e r v i c e - E x c e l l e n c e UNCLASSIFIED 8

UNCLASSIFIED Step 1: Get the Structure in Place Work to set ground rules with program office and development team Traceability must exist between initiatives, epics, and issues 1. Epic Issue Initiative Issue Key UBER-543 Issue Type Summary Story Parent Link UBER-485 Issue Key UBER-245 Issue Type Summary Initiative Issue Key UBER-485 Issue Type Summary Epic Parent Link UBER-245 Provide real-time demand tracking Implement dynamic pricing Develop surge pricing for peak times All current and future epics must be sized 2. Issue Key Issue Key UBER-485 UBER-487 UBER-488 UBER-489 UBER-490 Issue Type Issue Type Summary Epic Epic Epic Epic Epic Summary Develop surge pricing algorithm for peak times Perform demand-supply analysis Develop time-based pricing adjustments Develop location-based pricing adjustments Incorporate local event schedule into pricing Parent Link Parent Link UBER-245 UBER-245 UBER-245 UBER-245 UBER-245 T-Shirt Size T-Shirt Size Large X-Large Medium Medium Small All issues must be sized 3. Issue Key UBER-543 UBER-544 UBER-545 UBER-546 UBER-547 Issue Type Summary Story Story Story Story Story Parent Link UBER-485 UBER-485 UBER-485 UBER-485 UBER-485 Story Points Provide real-time demand tracking Incorporate driver incentive mechanism Develop geofenced pricing zones Develop real-time price multipliers Provide surge transparency for users 8 1 5 5 13 I n t e g r i t y - S e r v i c e - E x c e l l e n c e UNCLASSIFIED 9

UNCLASSIFIED Step 1: Get the Structure in Place (Cont.) All epics must display team and current state (To do, In Progress, Test) 4. Issue Key Issue Key UBER-485 UBER-544 UBER-545 UBER-546 UBER-547 Issue Type Issue Type Summary Epic Epic Epic Epic Epic Summary Develop surge pricing algorithm for peak times Perform demand-supply analysis Develop time-based pricing adjustments Develop location-based pricing adjustments Incorporate local event schedule into pricing Parent Link Parent Link UBER-245 UBER-245 UBER-245 UBER-245 UBER-245 T-Shirt Size T-Shirt Size Status Large X-Large Medium Medium Small Status In Progress Done To Do To Do Done Roadmap must lay out future epics using same sizing 5. Roadmap Roadmap Initiatve/Epic Initiatve/Epic Implement dynamic pricing Develop surge pricing algorithm for peak times Implement predictive surge pricing for users Develop real-time multi-platform demand coordination Implement personalized dynamic pricing Issue Key Issue Key UBER-245 UBER-485 Team 2 Team Team FY25 FY25 FY26 FY26 FY27 FY27 FY28 FY28 Large Team 2 Team 2 Team 2 Small X-Large Medium I n t e g r i t y - S e r v i c e - E x c e l l e n c e UNCLASSIFIED 10

UNCLASSIFIED Step 1: How Did We Estimate We had no useable data on our current program Ideally, we would have objective metrics that we could use from a similar program Have not figured this part out yet To start, dev teams gave initial estimates on how long a single team would take to complete each size Size Size Initial Guess Initial Guess Small 1-2 Sprints Medium 3-4 Sprints Large 5-6 Sprints X-Large 7+ Sprints Using the traceability to high level requirements and laying out the work to go in the roadmap using the developer s own estimate, more fidelity was laid into the single slide IMS I n t e g r i t y - S e r v i c e - E x c e l l e n c e UNCLASSIFIED 11

UNCLASSIFIED Step 2: Collect Data, Build Metrics As opposed to using developer s guess at how long something would take, we wanted to collect data on completed epics to have a more objective look. Needed to answer two questions: When Team X gives a T-Shirt size, how many story points does that epic usually contain (you figure this out by summing up all the stories mapped to that epic)? 1. Epics Completed Total SPs SPs per Epic Small Medium Large X-Large Small Medium Large X-Large Small Medium Large X-Large Team 1 Team 2 Team 3 Team 4 Team 5 Team 6 14 4 22 6 27 5 21 9 7 9 14 14 4 2 8 7 5 4 1 3 2 1 3 2 363 196 1779 981 864 419 637 647 500 300 890 456 350 290 145 450 345 95 310 185 25.9 49.0 26.4 50.5 51.6 26.8 84.7 109.0 123.4 46.6 45.5 46.2 125.0 150.0 111.3 65.1 70.0 72.5 145.0 150.0 172.5 95.0 103.3 92.5 581.5 303 1393 134 How fast can Team X complete stories? 2. Velocity (SPs/month) 111.3 90.6 80.6 76.7 94.1 33.6 Team 1 Team 2 Team 3 Team 4 Team 5 Team 6 I n t e g r i t y - S e r v i c e - E x c e l l e n c e UNCLASSIFIED 12

UNCLASSIFIED Step 3: Growth and Tech Debt Growth Initiatives Defined high level requirements not expected to change without major change to program Epics Efforts to break down initiatives into agile chunks not always 100% correct Original epic may be too large so is further broken down into additional epics Original decomposition does not include all necessary epics leading to additional epics Examined the number of new epics seen over time and created a story point quantity for a generic epic based on a weighted average of completed epics to-date SPs per Generic Epic New Epics New Epics 9 8 Team 1 Team 2 Team 3 Team 4 Team 5 Team 6 66.1 82.1 59.9 51.2 48.6 50.2 7 6 5 4 3 2 1 0 Nov-23 Dec-23 Jan-24 Feb-24 Mar-24 Apr-24 May-24 Jun-24 Jul-24 Aug-24 I n t e g r i t y - S e r v i c e - E x c e l l e n c e UNCLASSIFIED 13

UNCLASSIFIED Step 3: Growth and Tech Debt (cont.) Tech Debt Critical to understand how developer plans on capturing tech debt If an epic goes to test and has issues, are those issues added to the pre-existing epic or is a new epic created? In this example, tech debt is captured within the existing epic Tech Debt Open Story Points Q3 2023 Q4 2023 Q1 2024 Q2 2024 Develop surge pricing algorithm for peak times Perform demand-supply analysis Develop time-based pricing adjustments Develop location-based pricing adjustments Incorporate local event schedule into pricing 5 45 67 24 80 104 67 105 50 88 102 40 98 58 67 87 16 8 22 43 In addition to already accumulated tech debt, also important to account for future tech debt as more epics are brought to test I n t e g r i t y - S e r v i c e - E x c e l l e n c e UNCLASSIFIED 14

UNCLASSIFIED Steps 2 and 3: How Did We Estimate Overall, the annual cost did not change significantly as the program s plan did not involve adding or removing teams to the existing list Effort was more on how much the team could accomplish over a given timeframe Traceability to top-level requirements remains critical in order to have tangible progress to measure Team 2 Burndown 800 700 600 Velocity = 90.6 SPs/Month 500 400 300 200 100 0 Month 0 Month 2 Month 4 Month 6 Month 8 Defined SPs Undefined SPs Defined Tech Debt Undefined Tech Debt I n t e g r i t y - S e r v i c e - E x c e l l e n c e UNCLASSIFIED 15

UNCLASSIFIED Future Goals Get Better Cost Data! Many modern software programs inherited contracts from legacy programs which may not have much/any cost reporting Reliant on org charts and invoices to paint a picture of current FTE levels Not possible to get actual hours so forced to prorate (generally using story points) Measuring Progress For efforts that have started but not completed, it is likely that there are still additional user stories to be defined SPs SPs Team 2 Team 2 Historical Historical Large SPs Large SPs 150 Defined Defined on Day 1 on Day 1 78 Issue Key Issue Key UBER-485 Issue Type Issue Type Summary Epic Summary Develop surge pricing algorithm for peak times Parent Link Parent Link UBER-245 T-Shirt Size T-Shirt Size Large Final SPs Final SPs 135 When do you overwrite the historical amount with the actual amount? I n t e g r i t y - S e r v i c e - E x c e l l e n c e UNCLASSIFIED 16

UNCLASSIFIED Department of the Air Force I n t e g r i t y - S e r v i c e - E x c e l l e n c e From Analysis to Insights: Software Development Schedule Trends & Forecasts Presented by: Babak Damadi UNCLASSIFIED

UNCLASSIFIED Introduction and Objective Purpose: To predict deployment date by analyzing quarterly program increment (PI) data; calculating key software metrics like velocity and backlog by manually tracking completed tasks Objectives: Demonstrate Methodologies: Showcase the methodology used to calculate key metrics from quarterly program increment data Predict Future Program Increments: Illustrate how data can be used to predict potential outcomes for upcoming program increments Share Insights: Present key insights and lessons learned from the analysis Highlight Practices: Identify successful strategies and areas for improvement in software cost estimation I n t e g r i t y - S e r v i c e - E x c e l l e n c e UNCLASSIFIED 18

UNCLASSIFIED Methodologies and Data Analysis Overview Quarterly program increment data, completed tasks Manually tracked the remining backlog at the end of each PI Data Collection Data Used: Metrics on velocity and backlog from the last two quarters Velocity: Calculated by counting the number of tasks completed Backlog: Quantified the remaining tasks after each PI Metric Calculation Analyzed trend in velocity and backlog across the PI Analysis Period: 3 PIs (P0,P1, and P2) Data Analysis Used calculated metrics to predict outcomes for future PIs Scenario Forecasting I n t e g r i t y - S e r v i c e - E x c e l l e n c e UNCLASSIFIED 19

UNCLASSIFIED Scenario Overview Scenario A Conservative: Average velocity is used as a baseline, with temporary extra teams added during specific periods to handle the highest backlog growth. Scenario B Aggressive: Average velocity is used as a baseline, with a significant efficiency boost applied at each time period. Temporary extra teams are added during specific periods, anticipating smaller backlog growth. Scenario C Realistic: The most recent velocity is used as a baseline, with temporary extra teams added during specific periods to handle moderate backlog growth. Scenario D Realistic+: The most recent velocity is used as a baseline, with extra teams remaining for the entire program duration to accommodate moderate backlog growth. I n t e g r i t y - S e r v i c e - E x c e l l e n c e UNCLASSIFIED 20

UNCLASSIFIED Scenario Feature Completion Velocity I n t e g r i t y - S e r v i c e - E x c e l l e n c e UNCLASSIFIED 21

UNCLASSIFIED Feature Burndown Based on Actuals I n t e g r i t y - S e r v i c e - E x c e l l e n c e UNCLASSIFIED 22

UNCLASSIFIED Lessons Learned and Key Takeaways Positive Outcomes: Insightful Data Analysis: team s performance, burn rate, and remaining workload Normalized Features Sizes: relative measures like percentage Challenges: Insufficient data on individual feature sizes Features were frequently moved between PI with different names, complicating status tracking Takeaways: Adapt to Data Quality: extract meaningful insights Employ What-If Analysis: impact of various variables Maintain Consistency: in naming, tracking, and documentation of feature changes Foster Collaboration and Collective Ownership: of analysis process I n t e g r i t y - S e r v i c e - E x c e l l e n c e UNCLASSIFIED 23

UNCLASSIFIED Department of the Air Force I n t e g r i t y - S e r v i c e - E x c e l l e n c e Feature (and Subsystem Feature) -Based Metrics Presented by: Meghan Kennedy UNCLASSIFIED

UNCLASSIFIED Terminology IFC Example: Capabilty1 Capability 3 Capability 2 Capability: Aerial Refueling (AR) SF: AR Door Signal Updates SSF: Provide AR Door Control SSF: Provide Signal Amp SSF: Provide AR Receptacle SF: Display AR Statuses SSF: Process AR Status Outputs System Feature 3 System Feature 2 System Feature 1 Subsystem Feature 3 Subsystem Feature 1 Subsystem Feature 2 Current metrics collected beginning 2019: Labor Hours System Features (SFs) Subsystem Features (SSFs) Story Points (SPs) Program Action Request System (PARS) I n t e g r i t y - S e r v i c e - E x c e l l e n c e UNCLASSIFIED 25

UNCLASSIFIED Tracking SSFs and Hours I n t e g r i t y - S e r v i c e - E x c e l l e n c e UNCLASSIFIED 26

UNCLASSIFIED Software Models CER Model Cumulative Hours versus Cumulative Closed Subsystem Features Hrs per SSF per month Expected to increase as more SW completed (more to test) Resulted in two CER models to forecast certification dates: Power Model Step-Down Linear Model (large number of SSFs closed right before cert) SP Analysis Historical completion per HW/SW Component (HSC) testing effort (1 HSC = 3 months) WTG = Planned SPs + carryover SPs + historical growth WTG divided by Historical completion = Projected Time to Finish Used as a crosscheck / high level indicator PAR Analysis Only track Mission Critical and Flight Safety PARS Simply status +/- by IFC and Severity For awareness / context I n t e g r i t y - S e r v i c e - E x c e l l e n c e UNCLASSIFIED 27

UNCLASSIFIED Grading Our Homework I n t e g r i t y - S e r v i c e - E x c e l l e n c e UNCLASSIFIED 28

UNCLASSIFIED Results Able to forecast schedule of various SW-dependent milestones Able to lay out future contractor SW capacity crunch as Development and Modernization phases overlap SW Developers SW Int and Test, SILs Able to show contractor forecasted schedule is months to years out of bed with what current performance shows Results are possible given cost team has access to the contractor s software management tool, streamlining the process and allowing for real-time data collection I n t e g r i t y - S e r v i c e - E x c e l l e n c e UNCLASSIFIED 29

UNCLASSIFIED Final Thoughts All three examples focused on working with the data already being received on the program Being proactive in the early stages of a program could offer a more structured approach to receiving the data we need especially through the linkage to cost reporting Already receiving this type of data more formally through CSDR submissions (traditional and custom) Data is housed both formally on CADE and other classified networks as well as pulled from informal sources like software management tools (e.g. Jira) and PI reports I n t e g r i t y - S e r v i c e - E x c e l l e n c e UNCLASSIFIED 30