Enhancing Application Performance: Root Causes and Solutions

www.regouniversity.com Clarity Educational Community Enhancing Application Performance Root Causes and Quick Solutions Presented by: Steve Seaney & Chris Shaffer

Agenda Survey Why Performance is Important Improving Performance Monitoring Performance Real-world Cases 2 Clarity Educational Community

Survey How many of you have users that complain about CA PPM performance? How many of you feel performance negatively impacts overall perception of CA PPM? Any great performance improvement stories? What did you do? 3 Clarity Educational Community

Why Performance is Critical Performance issues impact the usability, dependability, and scalability of a system Application performance always has the potential to burden a CA PPM instance and discourage users Usability Dependability Scalability 4 Clarity Educational Community

Where Performance Data Exists There are a variety of sources providing key information on performance, but most are only available for On Premise hosting Rego has portlets that help monitor jobs, processes, and database (Oracle) performance Log & Config Files App Logs BG Logs App Access Logs Process engine Infrastructure Diagrams Confirm Cores Confirm Memory Confirm Storage Confirm IO Confirm Network GC Performance Database Reports Oracle advantages Deadlocks Waits Physical IO Cache Memory Long sessions Top SQL by CPU Top SQL by IO Table fragmentation Table configuration 5 Clarity Educational Community

Key Levers to Improve Performance Items driving performance are typically resolved by adjustments to the base infrastructure, application deployment, and the application configuration. Base Infrastructure VM Architecture Database Cores Database Memory Shared usage Application Deployment JVM Architecture JVM Memory App Server Maintenance Database Settings Application Configuration Jobs Pages Security Settings Code Optimization Data Limits On Premise Only 6 Clarity Educational Community

Base Infrastructure With an on premise instance, one of the first things to validate is that the overall sizing and setup of the infrastructure is correct These are not settings, they are changes that require purchases and/or changes to the physical/virtual setup VM Architecture Virtual machines can be used for most of the infrastructure, but we recommend the production database NOT be virtual 1-2 Cores per APP or BG Database Memory Most common problem CA PPM LOVES memory Database Cores 1 core for every 125-150 concurrent users (minimum 2 cores) Additional cores can be needed if heavy reporting is done during business hours Shared usage Applications sharing the base infrastructure need to play nice 7 Clarity Educational Community

Application Deployment Application deployment is how the application and database settings are configured for optimal use. This is a common area to find issues with the application technical setup. JVM Architecture 1 or 2 BGs (Never More than 2 for the process engine) 50 to 150 concurrent users per app JVM Leverage a XOG / Admin / Scheduler app JVM to offload non-UI users JVM Memory 6 GB for the APP JVM is great Never less than 2 GB for the BG typically 2.25 GB is good Application Server Maintenance Weekly restarts may be necessary, but can usually be avoided if architecture is sized correctly Monitor heap usage for excessive CPU and garbage collecting Parallel garbage collection if more than 1 core Database Settings CA PPM demands a high speed SAN; Memory compensates for IOPs. Do not run standard Oracle database stats job Contact Rego for environment specific database parameters Monitor and plan for Crystal and Webi reporting 8 Clarity Educational Community

Application Configuration Application configuration issues typically require a mix of both functional and technical changes. The changes are often caused by suboptimal configurations. Jobs Pages Security Settings Code Optimization Suboptimal SQL and GEL is the number one issue we see Understand that OOTB does not always mean optimal code you may want to re-write some OOTB portlets Data Limits Minimize your set of roles Purge old investments shrinking key tables like investments, tasks, assignments, time CA PPM 13.3+ contains governors for aggregation rows, Export to Excel, and XOG usage Leverage evening downtime window but coordinate with DBA Run database and system maintenance jobs after CA PPM jobs complete Avoid large data changes (e.g. post timesheets regularly) Avoid conflicts between slicing, posting, etc. Pages load Portlet queries together when the page is loaded limit pages to 1 Portlet if possible Use global rights if possible Minimize the use of instance rights Minimize slice time ranges Close time and fiscal periods Set portlets to wait to display results until after filter Turn off display conditions in the project and other object list views 9 Clarity Educational Community

Application Configuration Best Practice Set realistic session timeouts. Set Attribute Value Protection settings. Run the Delete Process Instance job on a regular basis. Run the Remove Job Logs & Report Library Entries job on a regular basis. Ensure proper data is being audited. Do not audit every attribute. Run database heavy jobs off hours. 10 Clarity Educational Community

Performance Improvement Process Always start performance work with the facts. Without understanding exactly what you are trying to fix, you will never know when to declare success. After you have the facts, addressing performance issues is an iterative process you start with the low hanging fruit and move through additional layers over time. 1. Identify the issue. Understand the exact behavior that is causing the issue. Key questions: all users? All times of the day? Certain pages? 4. Make the Change. Test the theory by making changes based on your initial thoughts. Go after low hanging fruit or larger changes trying to remove big things early on. 3. Decide on an Action. Review the data to find your first course of action (may be a few things) a theory about what the issue may be and how to solve it. 2. Gather baseline data. Capture as much information as you can from user timings, logs, AWR reports, etc. Make sure you have something to compare to once changes are made. 5. Analyze and Regroup. Analyze results compared with the baseline, maybe collect more data, and determine next steps 11 Clarity Educational Community

Monitoring Performance Proactively monitor CA PPM utilization, user response, and performance. Remote monitoring On-Premise and On-Demand Incites BEFORE end user concerns When and how end users are using CA PPM 12 Clarity Educational Community

DISCUSSION Any questions before we continue? 13 Clarity Educational Community 13

Real-world Case Application Sizing Symptom Application heap dumps, out of memory warnings, and periodic slowness Database appears to be healthy Clues CA PPM 13 requires additional capacity (memory & cores) JVMs heaps were 1.5 GB or the app and 1.0 GB for the BG Application JVMs spending significant time garbage collecting Symptoms appear during peak usage time Database is healthy Resolution Increase memory for JVMs to 6 GB for the apps and 2 GB for the BB Increase database cores Add JVM for XOGs / Admin / Scheduler Result Stable Dev environment Pushing to production If cores are limited, try memory increase first and watch GC performance 14 Clarity Educational Community

Real-world Case Database Tuning Symptom Frequent & persistent application slowness Database CPU and application CPU are inconsistent Clues Long running sessions on the database (more than a few minutes) Blocking sessions High physical I/O Resolution Moved to the CA PPM stats job (Oracle) nightly & disabled the Oracle cron job Increased the size of the database cache (PGA and SGA) Rebuilt indexes associated with long running SQL Result Stable and consistent production environment Exposed additional queries that needed to be tuned Exposed need to establish tuning and maintenance process 15 Clarity Educational Community

Real-world Case User Portlets and Configuration Symptom Select users long waits with select portlets and pages Periodic system slowness Oddly high database CPU Clues Long running sessions on the database (more than a few minutes) Top SQL includes portlet queries with poor SQL Resolution Tuned the queries causing the issue Changed the page configurations to render on filter only Set default filter for the portlets Result Improved portlet speed Eliminated long running sessions (led to system issues) This can also apply to Webi and reports 16 Clarity Educational Community

Real-world Case Security Rights Symptom Most users experiencing long wait times throughout the application Periodic system slowness Clues Admins did not experience slowness Tested key portlets by removing the @SECURITY@ OOTB calls to see difference in timing Resolution Tuned the queries causing the issue by moving the security call within an inner join vs. at the end Removed 70% of the instance and OBS rights moved to global View of resources and projects Result Portlet response time cut in half nearly equal to admins Login and general navigation faster CA PPM using OOTB security calls throughout 17 Clarity Educational Community

Questions We hope that you found this session informative and worthwhile. Our primary goal was to increase your understanding of the topic and CA PPM in general. Phone 888.813.0444 Email info@regouniversity.com There were many concepts covered during the session, if you would like to contact any presenter with questions, please reach out to us. Website www.regouniversity.com Thank you for attending regoUniversity 2015! 18 Clarity Educational Community