Orchestrating Tasks with Mistral Workflow Service at Austin OpenStack Summit 2016
Explore Mistral workflows, their significance in orchestrating tasks, and the core concepts behind workflow management systems. Learn about Mistral's evolution, features, and its role in the OpenStack community, illustrated with insightful examples and definitions.
Download Presentation
Please find below an Image/Link to download the presentation.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author. If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.
You are allowed to download the files provided on this website for personal or commercial use, subject to the condition that they are used lawfully. All files are the property of their respective owners.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author.
E N D
Presentation Transcript
Austin OpenStack Summit April 2016 Orchestrating Tasks with Mistral Workflow Service Renat Akhmerov @Nokia
What we will cover General info about Mistral Workflows and why they are needed Mistral in a nutshell Mistral workflow language & features Briefly on Mistral status and plans Q & A 2
General Info on Mistral Started in Nov, 2013 Summits behind: Hong Kong, Atlanta, Paris, Vancouver, Tokyo Part of The Big Tent since May, 2015 Stable community Significant interest growth in OpenStack community Latest stable version: 2.0.0 (Mitaka) 3
Definition #1 Wikipedia: A workflow manages and monitors the state of activities, such as the processing and approval of a loan application form, and determines which new activity to transition to according to defined processes (workflows). 6
Definition #2 Wikipedia: A workflow engine is a software application that defines a process, the rules governing process decisions, and routes information. 7
Definition #3 Wikipedia: A workflow management system (WfMS) is a software system for the set-up, performance and monitoring of a defined sequence of tasks, arranged as a workflow. 8
Example: Getting to OpenStack Summit Workflowis: Graph Tasks Connections State Result Data Tasks may be long Asynchrony Register (get a badge) Done! Fly to Summit Buy Summit Pass Arrange trip (flights, hotel) Air tickets etc. Pass Number Become an ATC Discount Code Extra budget Contribute to OpenStack Or with you boss 10
How are these concepts related to computer systems? 11
Example: Parallel Computing Orchestration Create VM1 Configure VM1 Compute Can be done by Heat Build a report Create VM2 Configure VM2 Compute Notify on completion Create VM50 Configure VM50 Compute 12
What is Mistral? Mistral is an OpenStack Service that manages workflows 13
Main properties of a workflow system Language to describe workflows Tasks are distributed across enterprise/cloud environment Support parallelism for running tasks Synchronization of parallel branches Dataflows from one task to another APIs to create/update/delete workflows APIs and active component to execute workflows APIs to monitor state of running workflows and tasks Persistent state of running workflows and tasks 14
Persistent state is a key! Progress Observability Execution history Persistent result Asynchrony Effective error handling Recovery Task 4 Task 4 Task 5 Task 5 Task 6 Task 6 Task 3 Task 3 Task 2 Error!! ! Task 1 Task 1 Task 7 Critically important at large scale! 15
Right tool for the right task! Scripts: Error handling is challenging All is OK while a script is working On an error we re left with a mess Not scalable Lack of monitoring No parallelism & synchronization Not suitable for large scale orchestration! 16
Mistral Workflow language YAML-based Relies on YAQL expression language Main features Defining workflow tasks Conditional task transitions Publishing persistent variables Fork & Join Loops (to process data collections) Engine commands (fail, succeed, pause, noop) Task policies (retry, timeout, wait-before, wait-after, pause-before) Nested workflows 18
Example: Deleting VMs by criteria version: '2.0' tenant_cleanup: description: Deletes virtual machines containing mistral in their names. tasks: get_vms: action: nova.servers_list publish: vms: <% $.get_vms['mistral' in $.name].select(dict(id=>$.id,name=>$.name)) %> on-success: - delete_vms on-error: - fail delete_vms: with-items: vm in <% $.vms %> action: nova.servers_delete server=<% $.vm.id %> 19
Pluggable task actions Defines what exactly a task should do Written in Python Synchronous and asynchronous > 500 actions out of the box Standard actions: std.http, std.ssh, std.email, std.javascript etc. Actions for OpenStack services (Nova, Heat etc.) User can plug in new actions 20
Conditional transitions report_vms: input: - email_info tasks: get_vms: action: nova.servers_list on-success: - send_report: <% $.get_vms.len() > 0 %> on-error: - send_error get VMs send_report: action: std.email send report send error send_error: action: std.email 21
Publishing persistent variables version: '2.0' tenant_cleanup: tasks: get_vms: action: nova.servers_list publish: vms: <% $.get_vms['mistral' in $.name].select(dict(id=>$.id,name=>$.name)) %> on-success: - delete_vms delete_vms: with-items: vm in <% $.vms %> action: nova.servers_delete server=<% $.vm.id %> 22
Fok & Join version: '2.0' branch 1 branch 2 deploy_app: tasks: install_db: action: install_mysql on-success: - install_app install web server install DB join point install_web_server: action: install_web_server on-success: - install_app install app install_app: join: all action: install_my_app merge of branch 1 & 2 23
Loops version: '2.0' tenant_cleanup: description: Deletes virtual machines containing mistral in their names. tasks: get_vms: action: nova.servers_list publish: vms: <% $.get_vms['mistral' in $.name].select(dict(id=>$.id,name=>$.name)) %> on-success: - delete_vms on-error: - fail delete_vms: with-items: vm in <% $.vms %> action: nova.servers_delete server=<% $.vm.id %> 24
Engine commands version: '2.0' deploy_app: tasks: get_vms: action: nova.servers_list on-success: - fail: <% $.get_vms.len() = 0 %> - install_app fail - ends entire workflow with a failure succeed - ends entire workflow with success pause - puts entire workflow on hold (can be resumed manually) install_app: 25
Task policies cinder_backup: tasks: start_backup: action: cinder.backups_create publish: backup_id: <% $.start_backup.id %> on-success: - wait_for_backup retry - repeats a task on failure wait-before - waits a number of seconds before starting a task wait-after - waits a number of seconds after task completion pause-before - puts entire workflow on hold before running a task timeout - fails a task is running longer than configured value wait_for_backup: action: fail_if_not_ready id=<% $.backup_id %> wait-before: 30 retry: count=3 delay=5 on-success: - report_backup_is_ready report_backup_is_ready: 26
Nested workflows version: '2.0' deploy_app: input: - vm_ids tasks: install_db: workflow: install_db vm_id=<% $.vm_ids[0] %> on-success: - install_app install_app: with-items: id in <% $.vm_ids %> action: std.ssh host=<% $.id %> input: cmd: sudo apt-get install my_app 27
Cron trigger Cloud alternative to crontab HA Runs workflow by configured cron pattern Can limit number of executions Example using CLI: $ mistral cron-trigger-create trig tenant_cleanup --pattern */5 * * * * 28
Back to Parallel Computing Orchestration example Create VM 1 Join Configur e VM 1 Compute Build report Create VM 2 Configur e VM 2 Compute Fork Send an email Configu re VM 50 Configur e VM 50 Compute 29
More use cases Scheduled administrative tasks Integration and orchestration of systems/services Healing Auto-scaling Orchestration of application components Cross-cloud coordination Data center automation Deployment Long-running business processes BigData analysis & reporting 30
Mistral current status Workflow language completeness Mistral works in HA Seamless integration with OpenStack APIs Command Line Interface Simple UI A number of production installations 31
Mistral plans for Newton Multi-region support Usability improvements Working with sensitive data Performance tuning Better HA Workflow visualization ~ 100 other blueprints on Launchpad 32
Q & A Official documentation: http://docs.openstack.org/developer/mistral/ Launchpad: https://launchpad.net/mistral Wiki: https://wiki.openstack.org/wiki/Mistral 34
