Machine Learning for Prediction in High Load Production Systems
Explore the challenges of simulating and monitoring high-load production systems, propose anomaly detection using machine learning models, and discuss cold and hot prediction approaches based on task execution logs and external data sources. Leveraging historical data and diverse input vectors, the feasibility of predicting production system behavior is examined.
Download Presentation
Please find below an Image/Link to download the presentation.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author. If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.
You are allowed to download the files provided on this website for personal or commercial use, subject to the condition that they are used lawfully. All files are the property of their respective owners.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author.
E N D
Presentation Transcript
Prominent ProdSys features Very high load; Changing structure; Size too large to feasibly simulate; Some simulation frameworks (SimGrid) may be of use to simulate (small) parts of ProdSys, but simulating it entirely is currently not feasible.
ProdSys logs sources Oracle: Other Current ProdSys state according to monitoring; Production requests; Date trends; Dataset access logs; Physic conference schedules; Tasks; Jobs. User input. Hadoop: Tasks; Jobs; ElasticSearch:
Anomaly detection proposal ProdSys is too big to simulate directly; ProdSys behavior is extensively logged; ProdSys often exhibits anomalies: Unusual task completion times; Balancing issues; Connectivity issues; It should be possible to model ProdSys behavior using Machine Learning-based models in conjunction with ProdSys logs and external data sources.
First step: cold prediction Based purely on ProdSys task execution logs; Does not make use of external data sources or ProdSys load; Based on historical averages. Considers all tasks of the same category (Project+Type+step) to have same duration. Implemented.
Using ML for hot prediction Hot prediction will use current ProdSys state and external data sources as well as historical data; Data for hot predictions can be viewed as input vectors for ML-based models; There is enough historical data to teach ML models for prediction of ProdSys; The data is very diverse; preliminary data analysis (strong predictor detection) is underway. Predicting ProdSys using ML-based models taught using historical data seems feasible, if data is prepared and pre-selected beforehand.
