Understanding Operational Testing with Cyber Threats in 2025

DATAWorks 2025 Motivating Effects to Missions from Cyber Threats During Operational Testing Dr. Jordon R. Adams, IDA 21 March 2025 Institute for Defense Analyses 730 East Glebe Road Alexandria, Virginia 22305

Operational test and evaluation with cyber threats informs decision-making by providing examples of adversarial cyber effects to systems and the translated effects those exploits cause to supported missions. 1

Outline Motivation Current evaluation strategy for OT&E with Cyber Threats Challenges with current test strategies Ways to improve T&E with cyber threats Conclusions 2

Motivation 3

DOT&E guidance is not consistently followed, inhibiting the value of operational testing with cyber threats. Testing needs to mirror the way the U.S. will fight a near-peer adversary. 4

DOT&E Released New Guidance Documents DoDI 5000.98 OT&E and Live Fire Test and Evaluation Overview DoDM 5000.96 Operational and Live Fire Test and Evaluation of Software DoDM 5000.99 Realistic Full Spectrum Survivability and Lethality Testing DoDM 5000.101 OT&E and Live Fire Test and Evaluation of Artificial Intelligence- Enabled and Autonomous Systems DoDM 5000.100 Test and Evaluation Master Plans and Test and Evaluation Strategies DoDM 5000.102 Cyber OT&E Guidebook February 2025 M&S VV&A for OT&E and Live Fire Test and Evaluation M&S: Modeling and simulation; OT&E: Operational Test and Evaluation; VV&A: Verification, Validation, and Accreditation 5

Current Evaluation Strategy for OT&E with Cyber Threats 6

First, what do we mean by mission? First, what do we mean by mission ? Systems are employed by users to serve a broader purpose. Cyber is just part of the environment, regardless of the mission. Blue Forces and Red Forces have intent. We must understand each mission and the operational environment. Cyber affects both Cyber affects both. 7

The Real Point: Effects to Missions The Real Point: Effects to Missions 3 2 1 Cyber domain effect is just as important as other domains Examples of cyber & translated effects Cyber effects vs. translated effects Intercept a video feed from an Unmanned Aerial Vehicle Enemy repositions forces to avoid destruction or create a lure for an ambush Cyber is the only man-made domain Cyber effects are what happens to the system in the cyber domain. Cyber is a pervasive domain Cyber effects can translate to other domains, but not all of these effects are mission effects. Inject false information on fuel levels Vehicle unexpectedly runs out of fuel, halting an advance Deny access to email and chat servers The unit cannot exchange plans and orders, delaying or misleading an action 8

How Well Can the Unit Accomplish Its Mission While How Well Can the Unit Accomplish Its Mission While Under Cyber Under Cyber- -Attack? Attack? Can the unit overcome effects caused by cyber Can the unit overcome effects caused by cyber- -attacks? The ability of the blue team to complete its mission, but cyber also needs to be part of the red team plan Evaluation framework element Evaluation framework element The cyber evaluation should include aspects of cyber defense and aggression Include cyber as a factor in existing mission designs Include cyber as a factor in existing mission designs Cyber is part of the environment, like everything else Tie cyber evaluation to effectiveness/suitability Tie cyber evaluation to effectiveness/suitability Cyber evaluation should tie back to the non-cyber metrics of the evaluation attacks? Effectiveness Suitability Survivability .. .. .. .. .. .. .. .. .. Cyber threats can affect all of these! 9

Defensive Tasks in Cyberspace Blue Force Prevent Identify the environment and Protect mission functions from cyber threats Recover Recover Recover mission function versus conduct a cyber Restore Prevent Restore Identify React Protect Detect Mitigate Mitigate Detect and React to cyber-attacks to enable completion of mission functions 10 Joint Chiefs of Staff J6, Cyber Survivability Endorsement Implementation Guide, Version 1.01, January 2017 National Institute of Standards and Technology, Framework for Improving Critical Infrastructure Cybersecurity, Version 1.1, April 2018

DOT&E Guidance Defines Two Kinds Of Assessments DOT&E Guidance Defines Two Kinds Of Assessments Cooperative Vulnerability Penetration Assessment Cooperative Vulnerability Penetration Assessment Cooperative investigation of comprehensive attack space Can we get in? Can we move around? What can we do when we get in? What do the users see? What would the defenders do? Adversarial Assessment Adversarial Assessment Can red team affect blue missions? Can defenders operate against cyber aggression? How does that affect missions? Each has different goals. Both are owned by the Operational Test Agency. Both are operational assessments in operational environments, with operational configurations, and operational users. Scoping of assessments for acquisition pathway and system maturity 11

Current Requirements in Operational Testing with Cyber Configured as it will be in the field, including all interfacing systems of systems, joint networks, and representative amount (and type) of network traffic. Considerations for acquisition pathway, and system maturity Include all defenders, operators, users, maintainers, and administrators. Ensure that training for all personnel represents field training. Include realistic and intel-informed cyber threats in order to generate empirical mission effects. Need to represent near-peer adversaries. Include systems and supporting units conducting operational missions. Ensure data collectors are situated in a way that allows them to directly measure effects to missions following cyber-attacks. Adversaries can also cause integrity effects from cyber compromise nefarious attacks. The test community does not always see these requirements play out. 12

Challenges with Current Test Strategies 13

Current Gaps in Operational Testing with Cyber Lack Lack of of operational operational realism realism System configuration, realistic data flows Operational defenders, users, training Lack Lack of of cyber cyber integration integration with with functional functional testing testing Cyber assessments often segregated, stove-piped (CVPAs, AAs) Testing isolated systems, instead of the unit operating interfacing systems of systems Lack Lack of of structured structured cyber cyber- -attacks attacks in in operational operational testing testing Lack of coordination between cyber teams, using CVPAs to inform and shape AAs Lack Lack of of dedicated dedicated data data collection collection from from operational operational test test agencies agencies Effects to missions require data collection from mission experts, not cyber test teams Lack Lack of of advanced advanced cyber cyber- -attacks attacks such such as as supply supply chain chain compromises compromises or or cyber cyber- -RF RF attacks attacks This also includes nefarious actions with lower likelihood for detection Representative of near-peer threats None of these are by design, and all result from failures in test planning AA: Adversarial Assessment; CVPA: Cooperative Vulnerability and Penetration Assessment; RF: Radio Frequency 14

Ways to Improve T&E with Cyber Threats 15

Ways to Improve Operational Testing with Cyber Threats Improve Improve operational Integrate systems under test with their interfacing systems of systems Requires inter-program cooperation, large-scale exercises, training events Ensure all personnel and configurations are operational (as possible) Engage in realistic and proficient training prior to testing I Integrate ntegrate cyber cyber aggression aggression into into functional functional operational Synergistic effects of cyber threats with other threats Combine testing of systems and threats to better evaluate second- and third- order effects Full-Spectrum Survivability and Lethality Leverage Leverage CVPA CVPA results results to to inform inform AA AA planning Reduce redundant testing, focus on cyber-attacks during missions operational realism realism operational testing testing planning AA: Adversarial Assessment; CVPA: Cooperative Vulnerability and Penetration Assessment 16

Ways to Improve Operational Testing with Cyber Threats More More methodical methodical data data collection collection for for cyber cyber assessments assessments Operational Test Agencies should place data collectors at cyber tests, too Carry Carry out out more more advanced advanced cyber cyber- -attacks attacks Better represent near-peer threats Coordination with RF-cyber, EMSO, close access teams, etc. Full-Spectrum Survivability and Lethality Supply chain attacks can be simulated or stimulated actions Assume compromise, always Cyber Cyber tests tests should should be be more more efficiently efficiently scoped scoped Intelligence can inform, but should not determine scope Engage in better test planning, MBCRA exercises 17 EMSO: Electromagnetic Spectrum Operations; MBCRA: Mission-Based Cyber Risk Assessment; RF: Radio Frequency

Conclusions 18

Conclusions We Can Draw from Observations Cyber is just another component of the environment, a more complex threat type. The Operational Testing community needs to include cyber as a threat, integrated with red and blue missions, interfacing systems, users, and other threat types. The goal should be to assess how cyber threats affect those missions. Cyber effects are not mission effects. Data collection is key. Operational Operational Cyber Cyber testing testing is is just just Operational is is cyber cyber- -threatened threatened. . Operational Testing Testing while while a a system system 19

Questions? 20