AI Data Coding Standards: MPAI-AIF.V2 Use Cases & Functional Requirements

MPAI-AIF V2 Use Cases and Functional Requirements . Andrea Basso, Leonardo Chiariglione 2022/07/11 T15:00 UTC

Standards for AI-based data coding Media coding standards have been at the root of the success of digital media Moving Picture, Audio, and Data Coding by Artificial intelligence (MPAI) targets AI-enabled data coding standards. Data coding: transforms data from a format into another more suitable for an application. MPAI standardisation targets Components (AI Modules AIM): their function, I/O data Organised in workflows (AI Workflows AIW): their function, I/O data, AIM topology Executed in MPAI specified environments (AI Framework AIF): metadata, API

MPAI-AIF: AI Framework AI module (AIM) Text MPAI standards are component-based Speech Speech analysis Emotion AI Workflow (AIW) AI AI Access Module (AIM) Module AIM Inputs Outputs User Agent AI AI AIM Storage Module AIM Module AIM Controller MPAI Store Global Storage Access Communication AI Framework (AIF) 3/19/202 3

What we have done so far what we ll do next Standard Name Acronym MPAI-AIF MPAI-CAE MPAI-MMC MPAI-CUI MPAI-GME Acronym MPAI-AIF MPAI-MMC MPAII-NNW V 1.1 1.3 1.2 1.0 1.0 V 2.0 2.0 1.0 AI Framework Context-Based Audio Enhancement Multimodal Conversation Compression and Understanding of Financial Data Governance of the MPAI Ecosystem Project name AI Framework Multimodal Conversation Neural Network Watermarking

MPAI-CAE V1: Emotion Enhanced Speech Mode Selection Speech Feature Analyser1 Speech Features1 Speech with Emotion Emotion Inserter1 Model Utterance User Agent Emotion- less Speech Emotion- less Speech Features Speech Feature Analyser2 Emotion Feature Producer Emotion Inserter2 Speech with Emotion Speech Features2 Emotion List Language Controller Communication Global Storage MPAI Store

MPAI-MMC V1: Conversation with Emotion Input Selection Input Text Meaning (Text) Output Text Language Under- standing Text (LangUnd) Dialog Proce- ssing Emotion (Text) User Agent Speech Recog- nition Input Speech Output Speech Recognised Text Speech Synthesis (Emotion) Emotion (Speech) Text with Emotion Output Speech Emotion Fusion Fused Emotion Lips Anim- ation Output Video Output Emotion Video Analysis Input Video Emotion (Video) Controller Video Of Faces KB MPAI Store Communication Global Storage

MPAI-CUI V1: Company Performance Prediction Prediction Horizon Organisational Model Index Governance Governance Assessment Governance Features Default Probability Prediction User Agent Financial Statement Financial Assessment Financial Features Default Probability Business Discontinuity Probability Perturbation Risk Matrix Generation Risk Risk Matrix Assessment Controller MPAI Store Global Storage Communication

Multimodal Conversation (MPAI-MMC) V2 Use Case 1 Personal Status Extraction Text PS-Text Descriptors PS-Text PS-Text Description PS-Text Interpretation Speech PS-Speech Descriptors PS-Speech PS-Speech Description PS-Speech Interpretation Personal Status Fusion Personal Status Face Object PS-Face Descriptors PS-Face PS-Face Description PS-Face Interpretation Human Object PS-Gesture Descriptors PS-Gesture PS-Gesture Description PS-Gesture Interpretation

Personal Status A set of internal human characteristics: 1. Emotion and Cognitive State result from human interaction with the Environment. a. Cognitive State is more rational (e.g., Confused , Dubious , Convinced ). b. Emotion is less rational (e.g., Angry , Sad , Determined ). a. Attitude is the stance a human takes when s/he has reached an Emotion and Cognitive State (e.g., Confrontational , Respectful , Soothing ). Shown in one of the following modalities: Text, Speech, Face and Gesture. 9 3/19/20 25

Multimodal Conversation (MPAI-MMC) V2 Use Case 2 Personal Status Display Text Text Speech Speech Synthesis (PS) PS-Speech Machine Speech Face PS-Face Face Synthesis (PS) Text Gesture Gesture Synthesis (PS) PS-Gesture

Connected Autonomous Text Autonomous Motion Subsystem Inside Audio Descriptions Recognition Vehicle - Human-CAV Speaker Speech Command Speaker ID Audio Scene Description Speech Descriptors Speech Environment Sensing Subsystem Recognised Text Recognition Outside Audio Speech Speech Question and Dialogue Processing Response Recognised Text Meaning Personal Status Language Understanding Extraction Speech Personal Status Text Face Object S Human Object Outside Video Personal Status Interaction Machine Face Visual Scene Description PS Interpretation Personal Status Display Description (Face) Object Object Physical Object Descriptors Physical Object ID PS Machine Gesture Inside Video Physical Object (Gesture) Lang-Und Text User Agent Output Text Machine Text Recognition Description PS Machine Speech Face Face Face Face ID Face Object (Speech) Descriptors Controller MPAI Store Communication Global Storage

Avatar-Based Videoconference - Virtual Secretary Text (xN) Recognition Recognised Text Lang-Und Text Speech Speech (xN) Question and Dialogue Processing Language Understanding Summary Recognised Text Meaning Speech (xN) Personal StatusExtraction User Agent Personal Status S Text (xN) Meaning PS Machine Face Personal Status Display Summarisation Edited Summary (Face) PS Machine Speech Machine Text Lang-Und Text Face Object xN) (Speech) Summary Output Text Human Object (xN) Personal Status PS Machine Gesture Personal Status (Gesture) Controller Global Storage MPAI Store Communication

An AI Framework (AIF) is needed An AI framework enables creation, execution, composition and update of AIM- based workflows for high-complexity AI solutions interconnecting multi-vendor AIMs trained to specific tasks, operating in the standard AI framework and exchanging data in standard formats. It will benefit various actors: Technology providers will be able to offer their conforming AI tecnologies to an open market Application developers will find on the open market for their applications need Innovation will be fueled by the demand for novel and more performing AI components Consumers will be offered a wider choice of better AI applications by a competitive market Society will be able to lift the veil of opacity from large, monolithic AI-based applications. 13 3/19/2025

MPAI-AIF: AI Framework AI AI Module (AIM) Module (AIM) Outputs Inputs User Agent AI Workflow (AIW) AI AIM Storage AI Module (AIM) Module (AIM) Controller Global Storage MPAI Store Communication Access

AIF Components Access: provides access to static/slowly changing data, e.g., domain data, data models, etc. AI Module (AIM): data processing element receiving Inputs and producing Outputs according to its Function. An AIM may be an aggregation of AIMs. AI Workflow (AIW): organised aggregation of AIMs implementing a Use Case Communication: connects the Components of an AIF. Controller: Exposes three APIs: i. AIM API: modules can register, communicate and access the rest of the AIF environment; can be started, stopped and suspended ii. User API: user or other Controllers can perform high-level tasks (e.g., switch Controller on/off, give inputs to the AIW through the Controller). iii. MPAI Store API: AIF-MPAI Store communication. May run one or more AIWs. Global Storage: stores data shared by AIMs. Internal Storage: stores data of the individual AIMs. MPAI Store: stores Implementations for users to download. User-Agent: interfaces the user with an AIF through the Controller

AI Framework Features Event-based and port and channel-based (unicast) Messages are of two types: High-Priority Messages and Normal-Priority Messages Messages may be communicated through Channels or Events. Controller may run on a different computing platform than the AIW. The AIMs of an AIW may run on different computing platforms, e.g., in the cloud or on swarms of drones. Appropriate API Profiles allow Implementation on different computing platforms and different programming languages. The Controller will always be present even if the AIF is a lightweight Implementation. AIMs may be hot-pluggable and register themselves on the fly. Support swarms of devices with multiple controllers.

MPAI-AIF and Security MPAI AIF Version 1 Requirements multiple technology providers with potentially different security requirements multiple users with potentially different security requirements MPAI AIF Version 2 Requirements MPAI-AIF V2 intends to provide a security infrastructure to the standard AI Framework 17 3/19/2025

18 3/19/2025

MPAI-AIF Version 2 Reference Model AI AI Module (AIM) Module (AIM) Outputs Inputs User Agent AI Workflow (AIW) AI AIM Storage AI Module (AIM) Module (AIM) Controller Global Storage MPAI Store Communication Access Encryption Service AIM Storage Service Communication Service Attestation Service AIM Model Service AIM Security Engine Trusted Services

MPAI AIF And Security The AIF Components shall access high-level implementation- independent Trusted Services API to handle: Encryption Service Attestation Service. Trusted Communication Service 20 3/19/2025

MPAI-AIF specific services Trusted AIM Storage Service including the following functionalities: Initialisation (secure and non-secure flash and RAM) Read/Write. De-initialisation. Trusted AIM Model Services including the following functionalities: Secure and non-secure Model Storage. Model Update. Model Validation. AIM Security Engine including the following functionalities: Model Encryption. Model Signature. Model Watermarking. 21 3/19/2025

Different security regimes Application developers shall be able to select the application s security either or both by: Level of security that includes a defined set of security features for each level. Developer-defined security, i.e., a level that includes a developer-defined set of security features. 22 3/19/2025

Trusted Service Integration The AIF Components shall be easily integrated with the above Services. The AIF Trusted Services shall be able to use hardware and OS security features already existing in the hardware and software of the environment in which the AIF is implemented. The specification of the AIF V2 Metadata shall be an extension of the AIF V1 Metadata supporting security with either or both standardised and developer-defined levels. 23 3/19/2025

Next Steps MPAI intends to develop three standards MPAI-AIF V2 MPAI-MMC V2 MPAI-NNW V1 with the following process Year 2022 Month June July July September 13 October October Spring Day 22 18 19 Who Does Approves Deliver Publishes CfT, UCFR, FWL Notify Intention to submit proposal Submit proposals Kicks off Evaluations Approves Technical Specification What MPAI Principal Members MPAI Respondents Respondents MPAI MPAI UCFR and CfT Framework Licence 10 12 2023 24 3/19/2025

Thank you! Leonardo Chiariglione leonardo@chiariglione.orgnardo@chiariglione.org Andrea Basso speedrun2006@gmail.com Join the fun Build the future More about this call at http://aif.mpai.community/ 25 3/19/2025