Reliable Verification Process of AI-Based Software for Autonomous Systems

Verification of AI-based SW ESA ESTEC 27-Oct-2022 For ESA Internal Use Only 1 ESA UNCLASSIFIED Limited Distribution

Focus/Content Focus: activities initiated to prepare for defining reliable verification process of AI-based flight software and AOCS/GNC systems. Also the use of AI to improve Test Process. Specifically: VIVAS (Verification and Validation of Autonomous Systems) MALT-P (Machine Learning Automated Test Process) 2

Verification by test Verification methods: Similarity, Test Technique Scenario testing Test Strategy Inspection, Review of Design, Sanity test Recovery test Middle-out, risk driven Mutation testing Analysis, and/or Testing (supported by simulation) Top-down Bottom-up Domain testing Coverage testing Big-bang Testing is most powerful , costly and complex method Stress tests Random Applied to software can be classified in: Test Method: e.g. Static Analysis (e.g. the examination of code) or Dynamic Analysis (e.g. execution of the code) Test Type: e.g. Functional Test Strategy: e.g. risk driven Unit Test Functional Non-Functional Integration Test Static/Dynamic Development System Test Deterministic/Statistical Acceptance Test White-box/Black-box Test Types Informal Formal Test Technique: e.g. stress testing Test Scope: e.g. System level test Test Scope Closed-loop Open-loop Multiple N-version tests Independent verification Regression testing A combination of these classifiers elements can be chosen 3 Test Method

VIVAS Verification and Validation of Autonomous Systems Rationale: Artificial Intelligence (AI), and especially data driven techniques such as Machine Learning (ML) have gained tremendous attention, particular the subfields of supervised-learning and reinforcement-learning have reached an incredible maturity level with countless real world applications that just a few years ago were unthinkable Main issues (model and system level) with AI systems for properties such as: Robustness/Stability: appropriate response to (external) disturbances Reliability: demonstration of proper performing of intended functions Uncertainty: handling unforeseen variation in behaviour as a result of quick and autonomous adaptability Objective: to propose and demonstrate a generic Verification and Validation methodology based on the usage of the System-level Simulation Facilities, specifically targeted at verifying autonomous systems using AI-models Simulation allows to: create complex scenarios to train validate and test a model, create stress-test scenarios, that might be under-represented in the training data, in order to evaluate the robustness of the model or system create failure situations, like sensor failures to evaluate the response of the model and the trustworthiness of that response 4

VIVAS First project milestone results (initial framework definition): Symbolic model Abstract and concrete scenario generator (using symbolic model checking) Monitor executor System level simulator Model Lifecycle management (CI/CD and ML-Ops) 5

MALT-P: Machine Learning Automated Test Process Objective: Improve the overall test process by introducing automation and machine learning methods and technologies with focus on Automated Test Generation Continuation/follow-up on several previous studies, for example: DAFA (Distributed Agents For bringing Autonomy in space systems): objective being the demonstration that distributed multi- agent system technologies can be applied to complex space systems. Next to this, identify appropriate methodologies for the system design of agent-based systems. Finally demonstrate added value of such an application by applying a distributed multi- agent framework to a reference scenario and comparing results. MASTV( Multi-Agent System for autonomy in Testing and Verification). The objective of this activity being the definition, development and demonstration of a Multi-Agent System but in this case for bringing autonomy to the Testing and Verification phase of a software product. ATMS: objective to design infrastructure for Automated Test Generation (ATG) and Automated Test Evaluation (ATE), using test case prioritisation 6

MALT-P: Traditional Test Process Main blocks: Product Under Test Product Environment Test Generation Test Evaluation Test Test Evaluation Generation (1) Inputs (2) Outputs (3) Environment (4) Environment control 2 1 Product Under Test input (commands) output (measurements) 3 3 Environment Of Product 4 7

MALT-P: Traditional Test Process Main blocks: Product Under Test 9 Environment SW Model Product Environment Test Generation 8 8 output (measurements) input (commands) Product SW Model Test Evaluation Product Software Model Environment Software Model 6 7 Test Test Evaluation Generation (1) Inputs (2) Outputs (3) Environment (4) Environment control 2 1 Product Under Test input (commands) output (measurements) 5 5 3 3 (5) Test Feedback (6) Simulation Output Environment Of Product (7) Simulation Input (8) Simulated Environment 4 8 (9) Simulated Environment control Test Feedback

MALT-P: Traditional Test Process Model Feedback 0 Main blocks: Product Under Test 9 Environment SW Model Product Environment Automated Test Generation Automated Test Evaluation Adaptive Product Software Model Environment Software Model 8 8 output (measurements) Product SW Models input (commands) Product Model 6 7 Automated Test Evaluation Automated Test Generation (0) Model Feedback (1) Inputs (2) Outputs (3) Environment (4) Environment control 2 1 Product Under Test input (commands) output (measurements) 5 (5) Test Feedback (6) Simulation Output 5 3 3 Environment Of Product (7) Simulation Input (8) Simulated Environment 4 9 (9) Simulated Environment control Test Feedback

MALT-P State-of-the-Art process Automated Test Generation: possibility to generate tests (e.g. from AI algorithm on previous tests, Product Software Model Automated Test Evaluation: based on Product Software Model, AI algorithm on test results Product Software Model: possibility to adapt, automatically correlate, learn from PUT results Traditional Test process No automated Test Generation: mostly manual defined No automated Test Evaluation: mostly manual defined or ad-hoc Product Software Model: off-line comparison or reference, static model with no adaptation No AI or Adaptive software in Product Under Test (deterministic behaviour) AI or Adaptive software in Product Under Test (non- deterministic behaviour): Training, Validation and Test 10

AI based SW and use of AI in Verification/Testing Artificial Intelligence (AI)/Machine Learning (ML) is there and can be used, however many methods exists, tailoring for specific use case needed Supervised-learning (e.g. evaluating test results) Gaussian Process, Logistic Regression, Decision Trees, Random Forests, Support Vector Machines Neural Networks Unsupervised-learning (e.g. detection of anomalies) Clustering methods Auto-Encoders Reinforcement-learning Natural Language Processing (NLP) (e.g. test generation) Recurrent Neural Networks (RNN) Transformers Supporting methodology: Domain Specific Language (DSL) (e.g. test definition, test procedures and test evaluation) 11

Conclusion Verification of AI based SW and use of AI in Verification applied to AOCS/GNC software is challenging. Interest to apply AI in these application is driven by AI technology breakthrough (in other domains) ESA is involved in several activities, specifically: VIVAS (ongoing) to study verification of AI based SW using simulation in space domain MALT-P (under evaluation): to study AI to improve the Verification process in relevant infrastructure. The two studies shall provide: Technical report with synthesis and recommendations as well as lesson learned Prototype infrastructure to demonstrate features 12

Thank you for your attention 13