Identifying Outside Influences as Latent Factors to Risk in Human Performance

Human Systems Integrations (HSI) Identifying Outside Influences as Latent Factors to Risk in Human Performance Presented by: Tony Thomas NASA LaRC HSI April 24, 2024

HSI Implementation HSI In a Not So Ideal World HSI is the FIRST to be minimized and marginalized when shortfalls exist in program funds, leading to shortcuts in design and a heavy reliance on the resilience of human operators and maintainers to keep the system running Presentation at DoD HFE TAG 75 (Dahlgren) Shifting the View of HSI for Better Implementation Two Main Themes 1. HSI Late Starts Additional Training Needed for PMs/Leaders 2. Outside influences not being accounted for early Policy, Process, Organization (PPO) 2

Consistent Issues Two Main Themes 1. HSI Late Starts The misconceptions contribute to late starts and limited HSI support Activities not completed or supported by HSI Reviewed assessments HSI Continues to be an after thought until something happens 2. Outside influences not being accounted for early In addition to activities that do not get accomplished with HSI participation: Outside Influences affect Human Performance, often not accounted for with HSI Domains or early analysis Changes in technology and organizational processes/regulations happen across the Product Life Cycle, causing operational performance to drift 3

Misconceptions about HSI It s the same as Human Factors Engineering HSI only applies to programs with human operators It s HMI or Human Machine Interface I m a human, so I must be an expert HSI is just for displays HS-I Must not start until Implementation or Production Phase Misconceptions about HSI leads to: Misunderstanding of what HSI can do for a program HSI SMEs included too late in the Product Life Cycle Limited HSI activities or limited scope 4

HSI in the Product Life Cycle HSI must be considered early and often in the conceptual design and requirements development phases of system design and acquisition Most HSI Activities occur Pre-Material Solution Analysis though Engineering & Manufacturing Development Very few HSI activities at Production & Deployment Phase and on When HSI introduced late, missed opportunities In risk identification/mitigation Design tradespace analysis 5

Assessment Findings for HSI Source Needs/Goals General Causes Long Term Affects Value of HSI known with Leadership Common Message to Stakeholders/Leadership Human Performance Total System Performance Total Ownership Cost Program/Project Schedule Confidence in System Operation Workarounds become common Training Time Maintenance and Support Time Unexpected Outcomes Potential Mishaps CBA 2018 HSI Not Early enough in Acq Quantitative Tools and Metrics needed Better predict System and Mission performance HSI Framework, Architecture, Domain broadening Requirements, Manpower, OPTEMPO, Fatigue, Training Organizational Issues and Operational Processes NCMAS 6 Sources: Capabilities Based Assessment Report 2018 and National Commission for Military Aviation Safety (NCMAS) Report 2020

Outside Influences Not Considered Outside Influences include (outside normal HSI domains activities): Organizational Policies, Processes, and Culture (HQ to Deploying Unit) Post design new technology, new tools, workarounds from the resilient Ops/Maint personnel, Unwritten Rules Regulations and design standards may change, but not on Contract Outside Influences affecting Human Performance are addressed Early DOTmLPF-P Analysis; determining material solution Socio-Technical Systems Framework assesses System Performance to predict Human Error Human Factors Analysis and Classification System (HFACS) identifies Latent Factors and the Unsafe Acts that leads to a mishap 7

DOTmLPF-P Doctrine, Organization, Training, material, Leadership and Education, Personnel, Facilities, Policy DOTmLPF-P Analysis recommends a material or non-material approach to fill a capability; Considers all HSI Domains [AcqNotes] Cross-walk Analysis to HSI Domains DO-P only considered in early phases of life cycle DO-P does not account for HCD; no cross-walk 8 AcqNotes: JCIDS Process, DOTMLPF-P Analysis - https://acqnotes.com/acqnote/acquisitions/dotmlpf-analysis

Socio-Technical Systems Framework1 Application of Organizational Psychology to predict Human Error Framework comprised of both socio elements (left side) and technical elements (right side) Assesses system performance with other outside influences, including processes, technology, and culture Socio Elements Technical Elements Technology People Culture Infrastructure Processes Goals Only assessed during Production phase 1Hexagonal socio-technical systems framework (adapted from Clegg 1979; Challenger et al. 2010; Davis et al. 2014) 9

HFACS Framework HFACS used in Mishap Investigations Covers Organizational Influences, Supervisory Factors, and Preconditions as Latent Factors Mishaps usually occur during Production & Operations and Support phases Latent Factors identified late; Too late to affect design; Changes in training and policies only https://www.hfacs.com/hfacs-framework.html 10

Operational Performance Drifts Practical Drift2 Baseline performance of a system drifts away from its original design A consequence of real-life operations and changes in the operational and regulatory environment the slow steady uncoupling of practice from written procedure [Snook 2000] allowing the practice to become legitimized through unremarkable repetition Changes in Technology, Training, Processes, Policies and Regulations drive the drift Practical Drift is to be expected over long-term operation The system is not being used the way it was originally designed; The larger the Delta, the higher the risks 2Snook, Scott A. 2000. Friendly fire: The accidental shootdown of US Black Hawks over northern Iraq. Princeton, USA. Princeton University Press, 11

Examples of Affected Designs & Ops Accidents requiring Policy shifts Space Shuttle Columbia (Processes affected operations) Reliance on past success as a substitute for sound engineering practices; Normalization of deviance NASA created the Technical Authority process to ensure that the engineering, health, and safety communities have an independent, influential role in providing alternate perspectives during the decision-making process. https://appel.nasa.gov/technical-authority/ 12

Examples of Affected Designs & Ops Accidents requiring Policy shifts USS John McCain (Processes affected operations) Some safety issues identified included: Navy policy for the automatic identification system data transmission for Navy vessels The procedures for the transfers of steering and thrust control on board the John S McCain The training of Navy bridge watchstanders The design of the destroyer s Integrated Bridge and Navigation System Some changes made by the USN after comprehensive review Adjusted manning policies to ensure Japan-based ships operating at higher tempo in congested water, are appropriately manned Surface Warfare Officer (SWO) career paths were restructured to ensure time to improve maritime skills 13

Latent Factors Latent Factors Feed the Drift Why do Latent Factors exist? Numerous reasons, for many we may have no controls in place Changes to processes, regulations, standards, and technology occurred between program inception and operation phase Latent Factors likely always exist How bad are they? Best case Complex system with HSI support, there are no reviews for Latent Factors leading to practical drift Worst case - NO early HSI, creating a Deficit in Human-Centered Design and many more Latent Factors, most likely causing additional Preconditions and Unsafe Acts Organizational reviews are not assessed by programs throughout the lifecycle Latent Factors laying and waiting is the definition of an Accident waiting to happen 7-9 Years 14

Human Performance Human Performance is affected during operations when: Gaps or changes with Policies, regulations, and internal processes with new technology, new science, new challenges, and new organizations Operators and maintainers are expected to keep systems running, even under manpower and resource constraints; Likely through Workarounds (e.g. NCMAS reported AC parts being refurbished) HSI Reviews for Latent Factors may provide data to forecast risk in Human Performance Assess how these outside changes may affect human performance during system operation, affecting system and mission performance Pre-Production reviews at each phase could keep track of potential Latent Factors as they arise, allowing a means to address them early HSI conducts and supports activities that may help reduce Latent Factors, if brought in early CONOPS development, interview legacy system users/leadership, lessons learned and mishap reports, support AoA/CBA, derive human performance metrics, review operational processes and policy changes Conduct Policy, Process, and Organization (PPO) reviews 15

Reducing Latent Factors What can we do for the identification of Latent Factors? Identify the precursors for Practical Drift Conduct PPO Reviews before each phase closeout Conduct Operational Assessments starting in Production phase Has the drift begun? Identify Relevant Activities for Reviews Current HSI activities not often conducted early New activities that support the identification and forecasting of Latent Factors HSI is the FIRST to be minimized and marginalized when shortfalls exist in program funds... Prevention for Latent Factors will be non-existent if the HSI program is not implemented early in the PLC. 16

Where to Look for Latent Factors START with the DOTmLPF-P analysis What current policies exist at the developing organization / using organization? Consider policies in operations and maintenance that are associated with the system in design Look at current activities at the organization associated with the mission Is there a history of mishaps with any legacy systems? Are there lessons learned? Are the users at the gaining organizations engaged with the new system design? This should not only include Operators and Maintainers, but Leadership as well. Is there currently a shortfall in Manpower and/or Personnel at the gaining organizations that is driving system design? Are there Funding issues at the gaining Organization driving design? Assess new technologies and processes that could affect design of the system Will these technologies or processes be in effect prior to Product and Deployment? Do they need to be considered in earlier design? Was the new technology included in the AoA? What affects will they potentially have on Human Performance? 17

Conducting Reviews and Assessments PPO Initial PPO Review OA OAn PPO Review PPO Review PPO Reviews Conduct Initial by MSA and recurring reviews before each phase closeout/milestone review Latent Factors that exist should be reported to and tracked by the Risk Management Board Operational Assessments (OA) Should be conducted once the system begins operations looking for precursors to Practical Drift OA can be conducted at any time during operations For larger systems, should be conducted at multiple organizations of varying sizes 18

Assessing for PPO PPO Initial Review Activities (Proposed) Check for current policies around the system and mission operation. Are the policies out of date? Are there expected changes? Look for new technology associated with the same mission operation. Will that technology be a part of the technical solution? Will the new technology change the processes and how missions are conducted? Check the gaining organization for current mission solutions; looking for similar/legacy systems. What do the users expect from the new system? What do leaders expect from the new system? What does the organization currently do to try and accomplish the mission? Also note any policies associated with the seven domains. Are there expected changes that may affect design or user performance? NOTE: All assessments should begin with reviewing the DOTmLPF-P analysis 19

Assessing for PPO Additional PPO Review Activities (Proposed) Review previous questions... Are the answers still the same? Note the Deltas Identify any new technologies or processes that have creeped into the system in design Identify any new policies that have been implemented that would affect the operation of the system Review the expected population of users. Does the anthropometric data support the population? Begin finding user advocates from the gaining organizations. Socialize all results with the project team Recommend changes to design based on analysis Report risks to Risk Management Board (RMB) NOTE: Remember there could be several years between reviews 20

Assessing for PPO Operational Assessment Activities (Proposed) Review results from PPO Reviews Invest Stakeholders (PM, Safety, Ops Leads) Use HFACS as a review standard to determine initial latent factors, Before the Unsafe Act NOTE: PD Phase may not have enough operations for full assessment Assess from Operator standpoint (Include maintenance/support) Health monitoring, fatigue, OpsTempo, manning, support services, Effects on Human Performance due to new tech, new policies, new processes Once in Operations & Sustainment Phase, Assess several operating sites Small/Large High Activity/Low Activity Identify the shifts in the Operational Performance Recommend solutions or engineering changes to adjust for Practical Drift Recommended to perform multiple Operational Assessments through Ops/Sust Phase 21

Policy, Process, and Organization Defined PPO Definition to be used for PM/Leadership training: The organizational policies, processes, and practices that, in concert with operational requirements, effect human performance in the operation of a system or the system design concept, and when considered throughout the product life cycle, helps create and maintain safe operational parameters free from restrictions, limitations, or the need for alternative solutions. PPO will be referenced as an HSI body of work and an unofficial domain while more research in activities is being conducted 22

Possible Changes on the Horizon Examples of Changes in Acquisition, System Design, and System Performance MBSE New activities for design, modeling tools varying from service to service or program to program, for tool usage YMMV, and new standards and policies; Same cook, new kitchen AI Working into system operations, effects total system performance; Largely research in AI effects on Human Performance (e.g. cognitive performance and Human-machine teaming) Cyber Issues in security may affect human performance; continually evolving These will cause changes to policies, standards, and how we do business in general. These changes will affect long-term system operations and Human Performance in those systems. Other Examples: Telepresence in Remote Piloted Vehicles New Visual aids Tactile systems Low Sonic Booms Electric Powered Systems verse coal/gas New engine designs 23

Why is PPO Important? Benefits: PPO provides flexibility outside the 7 domains to account for the other Influences and Latent Factors that creep into the system operation and cause Practical Drift PPO directly address the CBA 2018 Needs for: HSI Framework, Architecture, Domain broadening Better predict System and Mission performance Common Message to Acq Community PPO directly addresses the General Causes outlined by NCMAS: Organizational Issues and Operational Processes Findings on policies affecting Human Performance in system operations can be shared with the larger HSI community PPO can be the vehicle that provides preliminary data to advocate for policy changes WE MUST consider novel changes to the HSI framework to address the new ways we will conduct business within the Acquisition Program Lifecycle 24

Final Thoughts Going Forward Continuing development of training in NASA for PM, CE, LSE -> PPO activities is a part of that discussion and training Continuing research with NASA Complex Systems to leverage a previous study in Stakeholder Perspectives Taking3 (Operator verse Leader verse Maint.) Future Presentation on the use of HFACS for Preventive and Corrective Research Proposal to DoD Services PPO Activities proposed in this presentation need to be reviewed and vetted. Looking for participation in getting these activities standardized. 3Rieken E, Bond K, Best RM, Burleson G, Brubaker ER. A SPECTRUM OF STAKEHOLDER PERSPECTIVE TAKING IN EARLY-STAGE DESIGN. Proceedings of the Design Society. 2023;3:395-404. doi:10.1017/pds.2023.40 25

End Presentation Tony Thomas, HSI Lead and Training Design Anthony.thomas@nasa.gov 26

Back up Slides Sources: 1Hexagonal socio-technical systems framework (adapted from Clegg 1979; Challenger et al. 2010; Davis et al. 2014) 2Snook, Scott A. 2000. Friendly fire: The accidental shootdown of US Black Hawks over northern Iraq. Princeton, USA. Princeton University Press, 3Rieken E, Bond K, Best RM, Burleson G, Brubaker ER. A SPECTRUM OF STAKEHOLDER PERSPECTIVE TAKING IN EARLY-STAGE DESIGN. Proceedings of the Design Society. 2023;3:395-404. doi:10.1017/pds.2023.40 Additional Sources: Capabilities Based Assessment Report 2018 National Commission for Military Aviation Safety (NCMAS) Report 2020 AcqNotes: JCIDS Process, DOTMLPF-P Analysis -https://acqnotes.com/acqnote/acquisitions/dotmlpf-analysis HFACS: https://www.hfacs.com/hfacs-framework.html NASA Technical Authority: https://appel.nasa.gov/technical-authority/ 27

Recent Assessments Capabilities Based Assessment Report 2018 Identified 5 gaps areas Gap 1: Institutionalize an HSI Body of Knowledge Gap 2: Standardize Best Practices across the Services Gap 3: HSI Workforce Development Gap 4: HSI Tools and Processes to Support HSI Analyses Early in Acq Gap 5: Implement Outreach and Marketing function National Commission for Military Aviation Safety (NCMAS) Report 2020 Assessed Mishaps from 2013 to 2020 with Losses of: 224 Lives, $11.6B, and 186 Aircraft 28

Examples of Affected Designs & Ops Policy changes that were necessary OBOGS new MIL-STD-3050 (Technology affected operations) Unexplained Physiological Episodes; hypoxia-like symptoms Required Aeronautical Engineers, Safety Engineers, Crew Systems Engineers, Flight Physiologists/Doctors, and more Many hours and meetings debating the standards USAF Woman s Initiative Team (WIT) support for Anthropometric design changes (Policy affects designs) Changes how we design pilot and crew systems in the future to accommodate the population from which the USAF wants to recruit Changes to Air Force Instruction (AFI) 63-101/20-101 Physiological Episodes Reported 2013-2018 USAF 719 USN 699 From NCMAS Report 2020 29