Effective Time-Phasing Strategies in Air Force Acquisitions

AIR FORCE LIFE CYCLE MANAGEMENT CENTER AIR FORCE LIFE CYCLE MANAGEMENT CENTER TIME-PHASING RDT&E IN AIR FORCE ACQUISITIONS Capt Trevor Bills 10 Dec 2024

Overview Background Research Questions Literature Review Data Source Methodology Results Conclusions Future Research Questions

Desired Takeaways What this brief is: A continuation of analysis on the best distributions for time-phasing cost profiles An exploration of a new data source An after-action on how to transform and utilize real (read: messy) data What this brief is not: A revelation of new information that will require a change in your future processes

Background: Role of Analyst Role of Cost Analyst: Produce accurate and precise estimates Provide decision support to acquisition leadership Important aspect of that accuracy and precision? Time Time indicates complexity, requires proper distribution of funds across FYs That distribution of funds across years is known as Time-Phasing

Why Should We Care? BLUF: If you don t know when you will need money, you won t spend it properly Air Force Cost Analysis Handbook (AFCAH) Chap 15: [An] issue is whether a new program is putting adequate monies in its initial years to fund technical personnel and analysts when the program most needs them. A program can inadvertently weaken itself by stretching this requirement out over a number of years, just to have a smooth expenditure profile Better Buying Power 1.0: The leisurely schedule of even the simplest and least ambitious Department programs not only delays the delivery of needed capability to the warfighter, but directly affects program cost

How to Time-Phase Three primary ways to time-phase: Milestone Approach Utilize detailed schedule to ID program milestones Time-phase each milestone, then compile Considered the most accurate, but a detailed schedule is not always available Analogy Approach Least strenuous ID prior analogous program, apply that time-phased distribution Have to hope current program follows analogy S-curve Approach Uses any from a family of Cumulative Distribution Functions (CDFs) Utilizes multiple analogous programs, more robust than Analogy More schedule agnostic than Milestone

S-curves Beta Two shape parameters ( and ) Immense flexibility Weibull one shape parameter ( ) one scale parameter ( ) Rayleigh Subset of Weibull held constant at 2 ? =1 ? ? 1 ? ? 1 (? + ?) ? (?) ? ? ? ? = 1 ? 2 ? ? ? = 1 ?

Research Questions What program and/or schedule characteristics best predict distribution parameters? How do contract expenditures vary from non-contract expenditures in relation to program schedule and what are the implications for program time-phasing? Which distribution (Rayleigh, Weibull, or Beta) provides the best S-curve model for time-phasing RDT&E programs?

Literature Review: The O.G.s P.V. Norden (1970) Contractor data Attempts to model development expenditures using the Rayleigh Relate monthly manpower to elapsed project time Finds that large projects (high importance) reach peak manpower sooner (large ), small (low importance) projects take longer to peak (smaller ) Putnam (1978) Contractor data Continues Norden s theory, applies Rayleigh to model manpower-hours in Army software development Similar findings, early peak with long tail (skewed right) Those programs that didn t fit Rayleigh showed stepwise increases Postulates this is a result of poor management Abernethy (1984) NPS thesis, models Rayleigh distribution for 21 Navy contracts Contractor data Fits model to individual contracts, can t get any 1 model to fit all contracts to within 5% of actual final cost

Literature Review: The 2000s Unger (2001) / Unger et al. (2004) Selected Acquisition Reports (SAR) data Created Weibull and Rayleigh models to project final program cost and schedule Limited success, only 7/37 models provided a final cost closer than the initial outlay Encouraged future research to find better ways to estimate Weibull parameters Brown et al. (2002) SAR data Ran with Unger s recommendation, used multi-stage regression with least squares to develop models Weibull was 60% better at forecasting than Rayleigh

Literature Review: The 2000s Burgess (2006) Contractor data Tested Beta, Weibull, and Rayleigh Beta beat Rayleigh, but Weibull was the best (far from perfect) Davis (2008) Contractor data Utilized Rayleigh to predict final program cost Much greater success than Abernethy or Unger >90% explanation of variation in 93 contracts

Literature Review: Latest Brown et al. (2015) SAR data Explores Beta, Weibull, and Rayleigh Compares against the 60/40 rule Weibull is the best, followed by Rayleigh, then Beta, then 60/40 Schiavoni (2019) Adaptive curve fitting Running model that utilizing current program data to provide cost and schedule forecast Benefits S-curves don t offer

Data Comprehensive Cost and Requirement (CCaR) Enterprise Analytics (EA) located in Program Management Resource Tools (PMRT) Reports real-time expenditures against a program s LOA Ensures complete expenditure profile No need for outlay rates Any program on the IML, not just ACAT I programs Provides exceeding detail Obligating document Execution date Sort to US programs (filter out Foreign Military Sales (FMS)) Sort to RDT&E funding (filter out procurement) CCaR only contains data from 2014 forward Find programs that started n.l.t. 1 October 2013 Those same programs must have ended by FY23 This proved to be the biggest LIMFAC for data selection

Data Exclusion Criteria Criteria Start US-Funds Only RDT&E Only Complete Profiles Final Dataset Programs Remaining 6472 1825 776 89 89 Programs

Methodology Normalized data for Escalation (TY$ exp to CY$ exp) USAF RDT&E Raw Inflation Index (Base Year FY22) Fiscal Year (FY) Index Value 2014 0.850 2015 0.860 2016 0.870 2017 0.885 2018 0.904 2019 0.922 2020 0.934 2021 0.962 2022 1.000 Convert transaction dates and costs into % Schedule and % Expenditures

Methodology Real expenditure data is inherently messy Fixed expenditures/reimbursements that led to values less than 0 or greater than 1

Methodology Singular expenditures occur months or even years after >97% execution This leads to asymptotic S-curves Truncating data at 97% expenditures drastically reduces effect of these final expenses

Methodology Estimating Distribution Parameters Minimize Sum of Squared Error utilizing Excel Solver Squared Error (Estimated - Actual)2 2.03E-09 0.00022 0.00022 0.00042 0.00060 0.00060 0.00060 0.00095 0.00123 0.00167 0.00150 0.00150 0.00146 1.8431 Actual % Expenditures 0.00005 0.00015 0.00009 0.00025 0.00037 0.00032 0.00029 0.00043 0.99354 0.99966 0.99998 0.99996 1 Estimated % Expenditures 0 0.0149 0.0149 0.0209 0.0248 0.0248 0.0248 0.0313 0.9584 0.9589 0.9613 0.9613 0.9618 Sum-Squared Error (SSE) % Schedule 0 Parameters 0.0162 0.0162 0.0210 0.0240 0.0240 0.0240 0.0287 0.9802 0.9826 0.9970 0.9970 1 Alpha 1.3045 Beta 0.4038

Methodology Stepwise Regression of 19 potentially explanatory variables: Program Duration Aircraft, Ground Hardware, Software, Support, Munitions Joint Upgrade/Mod Contains Contract Contract Percentage, Contract > 25%, Contract > 50% ACAT I, ACAT II, ACAT III First Half > 80%, First Half > 70% Second Half > 80%, Second Half > 70% Weibull/Beta and were also tested for their corresponding parameter

Methodology Programs by Category Category Aircraft Ground Hardware Software Munitions Support Experimental Contract Non-Contract Contract > 25% Contract > 50% Upgrade/Mod First Half > 80% First Half > 70% Second Half > 80% Second Half > 70% Number of Programs 45 11 24 5 3 1 54 35 10 9 30 11 17 8 16

Methodology Imputation of Schedule Deciles Too many data points to allow for correlational analysis Comparison of real and predicted percent expenditures occurs at each decile Most programs don t have exact decile expenditures ? ??=?? ?? 1 ? ??+1 ? ?? 1+ ? ?? 1 ??+1 ?? 1 Where: ? ??= ??????? % ??????????????? ??= % ?? ????? (?????????????) ?? 1= ???????? % ?? ????????????? ??+1= ????????? % ?? ???????????? ? ?? 1= ???????? % ??????????????????? ? ??+1= ????????? % ??????????????????

Results Weibull alpha Important notes: PRESS R2 suggests overfitting Cook s D and Breusch-Pagan failure

Results Weibull beta Important notes: Passes Cook s D, Anderson-Darling, Breusch-Pagan

Results Rayleigh beta Important notes: Passes Cook s D, Anderson-Darling, Breusch-Pagan

Results Beta alpha Important notes: PRESS R2 suggests overfitting Fails Cook s D, Anderson-Darling, Breusch-Pagan

Results Beta beta Important notes: PRESS R2 suggests overfitting Fails Cook s D, Anderson-Darling, Breusch-Pagan

Results Weibull Performance

Results Rayleigh Performance

Results Beta Performance

Results Percentile Deviation Major (>20%) Minor (> 10%) Major (>20%) Minor (> 10%) Major (>20%) Minor (> 10%) Major (>20%) Minor (> 10%) Major (>20%) Minor (> 10%) Major (>20%) Minor (> 10%) Major (>20%) Minor (> 10%) Major (>20%) Minor (> 10%) Major (>20%) Minor (> 10%) Weibull Rayleigh 3 7 7 21 10 35 8 35 2 38 5 43 5 30 5 15 1 4 Beta 3 8 7 23 10 41 7 37 2 28 8 33 10 21 6 18 3 6 2 7 5 22 10 37 7 37 9 38 7 37 7 26 4 22 4 15 10% 20% 30% 40% 50% 60% 70% 80% 90%

Results Two-way ANOVA testing

Results Two-way ANOVA testing

Conclusions What program and/or schedule characteristics best predict distribution parameters? Nine variables were found to be significant in predicting at least one parameter: Second Half > 70% (4) First Half > 70% (4) Second Half > 80% (2) ACAT II (2) First Half > 80% (1) Weibull beta (1) Beta alpha (1) ACAT III (1) Second Half > 70% * Contract > 25% (1)

Conclusions How do contract expenditures vary from non-contract expenditures in relation to program schedule and what are the implications for program time-phasing? A contract variable only appears significant one time, and only as an interaction term No indication presence or percentage significantly changes expenditure profile Implication: analysts needn t worry about manner of expenditure in time-phasing Which distribution (Rayleigh, Weibull, or Beta) provides the best S-curve model for time-phasing RDT&E programs? No significant difference detected via two-way ANOVA Implication: all distributions are beneficial to time-phasing, determination might depend on other factors: risk analysis, individual model performance, OLS assumptions, etc.

Future Research Future Research Replicate in 5 to 10 years with a much larger dataset Re-explore contract vs non-contract expenditures What causes First Half > 70% (80%)? What causes Second Half > 70% (80%)?

AIR FORCE LIFE CYCLE MANAGEMENT CENTER AIR FORCE LIFE CYCLE MANAGEMENT CENTER Questions?