Application Log Processing Methodologies in a Service-Oriented Architecture
This content covers methodologies for processing application logs in a service-oriented architecture, including terminology, motivation, modern approaches, log collection interfaces in Java and PHP, log destinations, log collection techniques, and the principles of separation in service-oriented architecture. It emphasizes the challenges and tools available for efficient log management in complex IT environments.
Download Presentation
Please find below an Image/Link to download the presentation.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author.If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.
You are allowed to download the files provided on this website for personal or commercial use, subject to the condition that they are used lawfully. All files are the property of their respective owners.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author.
E N D
Presentation Transcript
Application log processing methodologies in a service oriented architecture Bogdans Ozerkins
Terminology and abbreviations SOA service oriented architecture DWH data warehouse BI business intelligence Big Data usually referred to extremely large datasets, containing valuable insights
Motivation Systems write logs in different ways / formats Programming languages provide different out of the box tools SOA implies a multitude of languages Requirements for log management depend on the task DWH and BI solve similar problems for data collection / analysis
Modern approach function <severity> ( <message>, <meta data>) Logger frameworks are configurable Middleware formatting / handling of log records Transportation into external systems
Log Collection Interface (Java) http://commons.apache.org/proper/co mmons-logging/apidocs/index.html
Log Collection Interface (PHP) https://www.php-fig.org/psr/psr-3/
Log destinations File system RFC-5452 (syslog) Email Chat (Slack, Hip-Chat, Messenger, Telegram, WhatsApp) Database / Data Storage Log Collection Service (GrayLog, Sentry, Loggly) Infrastructure Log Collection (AWS CloudWatch, Stackdriver Logging, Azure Logs)
Log collection Local storage (FS, Memory, network-attached storage) Remote storage (Database, Service) Delivery protocols (UDP, TCP, HTTP) Buffering (application, local agent, sub-process) Storage control (storage overflow, hardware failure)
Separation into distinct unit Loose coupling Service autonomy Service reusability Service encapsulation Communication approach choices Direct communication Event-driven architecture Enterprise service bus Service Oriented Architecture
Centralized logging https://aws.amazon.com/s olutions/centralized- logging/
Business Data Processing
Database replication Bulk import Analytics data collection Programmatic data push / pull Enterprise service bus Data pipelines
Data Warehouse / Data Mart Data Lakes (schema on read) Analytics data storage Event Analytics (Columnar storage) Distributed FS
What are the pros & const for different log collection approaches? Which data formats should be used for log collection? Are current log generation interface and collection standards most optimal? Research questions Currently log data & Analytics data approaches are somewhat similar, can they be merged? Can application logs be collected / merged with system logs? Which of the approaches could be beneficial in SOA?
Logs collection implementation across languages Future research Structured logs Process tracking in SOA, using application logs
