4.7 TOI Custom Engineering
"The document titled "TOICustom.EngineeringPatti.LangerJuly.2013|Quantum.Confidential" contains confidential information related to engineering matters dated July 2013 and is intended for limited access. The content may involve specialized knowledge or processes within the engineering field, authored by Patti Langer."
Download Presentation
Please find below an Image/Link to download the presentation.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author.If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.
You are allowed to download the files provided on this website for personal or commercial use, subject to the condition that they are used lawfully. All files are the property of their respective owners.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author.
E N D
Presentation Transcript
4.7 TOI Custom Engineering Patti Langer July 2013 | Quantum Confidential
CE Projects Delivered in 4.7 IBM TS1130 tape drive support Debian 6.0.x OS SAN client support SM Priority Queuing Enhancements StorNext MAS Agent Feature | Quantum Confidential
Priority Queuing Enhancements TSM request queue insertion algorithm for retrieve requests Allnew or reprioritized retrieve requests are inserted in queue based on priority, media identifier, media position Requests of same priority are grouped by media identifier Requests are ordered by media position to reduce tape mounts and positioning Requests that have reached the configured pri.config max priority are not affected The feature is configurable using fs_sysparm OPTIMIZE_RETRIEVE_QUEUE with the default being true New fsretrieve -p option to indicate top priority Requests receive priority over all other retrieve requests in TSM queue fsqueue output identifies top-of-queue retrieve requests from normal retrieve requests with the letter P for current priority pri.config default age times for reprioritization have increased for retrieve and store requests Allows request to stay in current priority longer to maximize benefit of algorithm change | Quantum Confidential
StorNext MAS Agent Architecture Overview MAS - StorNext Integration Turner Configuration and Operations MAS Agent software overview | Quantum Confidential
StorNext MAS Agent Feature - Overview Licensed Product Separate RPM that is installed with DSM by install.stornext Bin files installed on server and client in /opt/quantum/mas_agent Targeted for Turner Broadcasting, Atlanta Provides the storage solution for a large and complex infrastructure that provides a reliable and high availability archive media management system that is managed by the Harmonic MAS The MAS manages all archives, restores, and pre-stage of media to playout servers Integrated with the Harmonic MAS via http with an XML specification | Quantum Confidential
The Solution Provides 24x7 Operations High Availability and Redundancy 2 StorNext HA sides (4 MDCs) Each side is comprised of one pair of HA MDCs One side can be stopped for maintenance while the other side continues Non-Shared Each side has their own databases and tape libraries | Quantum Confidential
High Availability and Redundancy HA Pairs for high availability Data archived and restored round-robin between sides Data is replicated between both sides in real-time Both sides are fully redundant Replications are queued if the other side is down Data is replicated between sides Side A HA Side B HA Active Active Standby Standby | Quantum Confidential
Data Sharing Sides do not share databases (SN or MAS Agent DB) Disks from each side are cross mounted for efficient sync ing of files between sides Side A HA Side B HA Managed Managed UnManaged UnManaged HaShared HaShared | Quantum Confidential
File Naming Files are stored to StorNext using a collection GUID and a Volume Group MAS provides the File GUIDs, Collection GUID, and Volume Group to use for the store The Volume Group is the Policy Directory in StorNext The MAS Agent derives a three-level hierarchy from the Collection GUID to avoid large directories <mnt pt>/<volume group>/<xx>/<xx>/<xx>/<collection guid>/<file guids> The mas_laf (list agent files) command helps to find the storage location of files | Quantum Confidential
StorNext MAS Agent Overview MAS - StorNext Integration Turner Configuration and Operations MAS Agent software overview | Quantum Confidential
MAS and StorNext Integration Media Grid Harmonic MAS StorNext Archive Control cmds FTP in and out Archive, pre-stage, and restore commands AIR FTP out Servers MAS sends requests to the Archive via HTTP StorNext FTPs files between various systems | Quantum Confidential 11
MAS Issues Three Requests Archive FTP files from 3rd party storage to StorNext Replicate files to other side Storage Manager archives files to tape based on policy Pre-stage Retrieve files from tape to disk (if necessary) Copy files to other side Restore FTP files from StorNext to 3rd party storage | Quantum Confidential 12
Archive Request Overview Archive request from MAS round-robin to sides A and B Side A HA pair Side B HA pair Copy to other side un-managed copy from Side B un-managed copy from Side A Side A managed Side B managed Tape Tape Files from 3rd party storage placed on Side A managed Files are replicated to Side B managed Files are on both sides for redundancy Policy eventually backs up files to tape | Quantum Confidential
Pre-stage Request Overview Pre-stage request from MAS round-robin to sides A and B Side A HA pair Side B HA pair Copy to other side un-managed copy from Side B un-managed copy from Side A Side A managed Side B managed Tape Tape Ensures files are on disk prior to a Restore request Retrieve files from tape to Side A managed Files are copied to Side B un-managed Files available to both sides if one side is down | Quantum Confidential
Restore Request Overview Restore request from MAS round-robin to sides A and B Side A HA pair Side B HA pair Restore from either side un-managed copy from Side B un-managed copy from Side A Side A managed Side B managed Restore request from MAS issued to Side A If side A down, files are available from Side B Files are FTP d to requested 3rd party storage | Quantum Confidential
StorNext MAS Agent Overview MAS - StorNext Integration Turner Configuration and Operations MAS Agent software overview | Quantum Confidential
Turner's Hardware Configuration MediaGrid Air Servers Other Servers FTP SN Client SN Client SN Client SN Client SN Client SN Client Side A Side B Tape Tape SAN Interconnect MDC MDC MDC MDC Tape Tape Side A managed Side B backup Side A backup Side B managed Tape Tape | Quantum Confidential
Operations Overview (Pre-stage) MAS sends a 7-day rolling pre-stage request to the Archive each day Make sure the next week of programming is on disk On Monday, pre-stage Tue-Mon On Tuesday, pre-stage Wed-Tue Pre-stage process Fsretrieve files from tape if necessary Copy to other side for redundancy | Quantum Confidential 18
Operations Overview (Restore) Each day a restore request comes through for the next day s programming Commercial and program changes are done on- demand as well | Quantum Confidential 19
StorNext MAS Agent Overview MAS - StorNext Integration Turner Configuration and Operations MAS Agent software overview | Quantum Confidential
Software Overview MAS uses web services (HTTP) to communicate with StorNext Requests and responses are XML StorNext web services Adds an entry to MySQL database Places a job on the job manager queue 3rd party queuing system (beanstalk) Job manager controls jobs and queues Places individual job requests on workers queues Workers pull jobs from their queue Workers do the bulk of the work | Quantum Confidential 21
Software Overview (per side) HTTP StorNext Web Service MAS MySQL XML Job Queue Worker Worker Worker Worker Worker Use MySQL DB and Beanstalk queuing for asynchronous support Use Job queues and workers for parallel distributed processing Most workers run on client machines and serve either side A or B Fsretrieve workers are on the MDC | Quantum Confidential 22
Job Queues Examples of job queues, each queue has a name FTP-in - for archive jobs FTP-out - for restore jobs Pre-stage - fsretrieve commands Copy-archive - copy archive files to other managed side Copy-pre-stage - copy restore file to other unmanaged side On-demand - alert MAS when status changes Delete - delete a job Workers Each worker listens to a particular queue name Job is guaranteed to go to one and only one worker Event model is blocking, i.e., workers do not poll | Quantum Confidential 23
Workers Start a job Worker pull job from a queue End of job Worker places a job on the next queue Use queues to chain jobs Minimal code Worker does one type of work, such as copy a file Can increase/decrease number of workers for performance Increase workers to scale work loads Limit workers to protect back-end resources Based on workload and hardware characteristics Must restart worker if make a change | Quantum Confidential 24
MAS Agent Commands mas_agent_control - starts and stops MAS Agent on MDC service mas_agent - starts and stops MAS Agent on clients mas_caf - copies MAS Agent files from one side to other mas_chk - performs consistency check mas_cleanup - removes expired-job information from DB mas_config_file - prints information from the configuration file mas_laf - lists file-path information mas_ping - verifies the MAS Agent worker connections mas_status - displays a collection of status information | Quantum Confidential 25
Logs and Debugging Logs are stored in: /opt/quantum/mas_agent/log Each daemon has its own log file Logs are collected on MDC using /usr/adic/PSA/bin/pse_snapshot Logs are collected on Clients using /usr/cvfs/bin/cvgather | Quantum Confidential
Reference Information StorNext MAS Agent Archive Administration Guide (6-67899-01) Man pages (/opt/quantum/mas_agent/man) Release Notes | Quantum Confidential
