Performance Testing Findings and Recommendations for SBCTC PeopleSoft Systems

SBCTC PeopleSoft Load Testing Results Review DG2, DG3, DG4 & DG5 Prepared by Kastech Software Solutions Group

Agenda v Scope Approach Assumptions Findings Results based on Recommendations Key Takeaways

Scope of the Project v Perform load testing on the technical components of ctcLink (App, Web, database and network) and ensure that Application works as expected under heavy loads Verify prior recommendations to adjust from the pre-DG4 baseline performance profile for all the SBCTC PeopleSoft systems hosted on AWS environment Perform the tests on Production-like environment with respect to architecture and capacity and configuration of the servers at recommended infrastructure scale up state Ensure the PeopleSoft Applications have adequate capacity to support anticipated User loads Identify any new bottle necks in the system (Application, Web, Database and network) and ensure that Application works as expected under predefined loads Determine application limitations under maximum number of active users performing common transactions Provide documentation on any changes recommended to improve service and delivery for students/employees using the PeopleSoft environment

Approach v Include most frequently used transactions that might have potential impact to the load on the system Test system with 30% above the current production population anticipated for DG2, DG3, DG4 & DG5 Include Fluid User Interface transactions which includes FSSF transactions Execute multiple user type scripts (Student, Faculty, Staff, Employee and Manager) Followed an incremental/Iterative approach and identify the break points for predefined load Burgundy team to monitor the servers/infrastructure on AWS and make changes according to the load Student count* Deploy Group Number of Load users by Application & User Type Student count* Deploy Group Code College Code College WA062 Seattle WA171 Spokane CC 7737 DG2 CS Student 24000 6178 DG4 Central WA172 Spokane Falls CC 4394 DG2 CS Faculty 4000 WA063 North Seattle 5535 DG4 WA220 Tacoma 6757 DG2 CS Staff 4000 WA140 Clark 8819 DG2 WA064 South Seattle 3647 DG4 HR Employee 12000 WA110 Pierce 8462 DG3 HR Manager 1500 WA090 Highline 7912 DG4 WA300 Cascadia 2252 DG3 WA120 Centralia WA150Wenatchee Valley WA230 Edmonds 2341 DG4 Procurement Finance Accountant WA010 Peninsula 1694 DG3 FIN 300 2964 DG4 WA030 Olympic College 5720 DG3 WA130 Lower Columbia 2791 DG3 7194 DG4 *Fall Term Enrolled Student Count (UGRD/CNED)

v Assumptions Testing to be performed on SBCTC AWS production like environment to meet the loads of students and staff All the tests were executed through Interaction HUB portal Recommendations were provided based on the Deployment Group 2, Deployment Group 3, Deployment Group 4 & Deployment Group 5 load. Changes to infrastructure in the future should be relative the future state anticipated loads Code College Student count* Deployment Group WA020 WA040 WA050 WA080 WA100 WA180 WA210 WA250 740 1112 3420 6405 3841 677 1433 1014 DG5 DG5 DG5 DG5 DG5 DG5 DG5 DG5 Grays Harbor Skagit Valley Everett Bellevue Green River Big Bend Whatcom Bellingham *Fall Term Enrolled Student Count in Performance Test Bed (UGRD/CNED)

CurrentProduction vs Recommended Configuration v Pillar CS CS CS User Type Count Student Faculty Staff 24000 4000 4000 Portal Campus Parameter Recommended Config Recommended Config Current Current Number of Application Server Instances 2 48 6 128 Number of Webserver Instances 2 48 6 128 Number of Application Server processes 8 48 8 128 DB instance Size m5.large m5.8xlarge m5.2xlarge db.m5.12xlarge Jolt min handlers 8 50 8 50 Jolt max handlers 12 70 12 70 4000 Recycle count 4000 2500 2500

Key Takeaways - Course Search using Elastic Search v Before implementing the recommended settings After implementing the recommended settings 28% of student transactions were executed successfully 72% of student transactions failed 99.99% of the student transactions were executed successfully Only 0.01% of the student transactions failed

Results Before and after implementing Suggestions v Below are the results for Course Search with 2 criteria transaction. This is with Elastic Search. Test script was executed through iHub for 24000 users with a ramp up of 0.1 seconds. This was failing after 1000. Turns out the bottleneck was Elastic Search. We increased the size of the ES servers from 2x to 8x. Will test again. 4000 users test passed, but ES server was at around 84% CPU. So, going to increase to a 24x and try the 24,000-user test. Elastic Search 7 node cluster on m5.12xlarge (48/192). One server at r5.24xlarge (96/768) was not enough for 24k users. Finally worked for 24 K with Elastic Search 7 node cluster on m5.12xlarge (48/192). With Recommended Config 28% students were able to perform the transaction successfully. 72% students were dropped out at multiple stages of transaction. Reasons for failure: Web server not available/not responding Gateway timeout Bad Gateway 99.99% students were able to complete the transaction successfully. 0.01% students were dropped out.

CS Results Student Scripts v Error % S.No Transaction Name Key Takeaway: Current Production environment set to handle 8K 0.21 1 Login into CS student load and positioned to scale to handle 24K student load. 0.09 2 Course Search with 2 criteria This will support DG5 college go live. NA 3 Register for Four (4) Classes Details 0.14 4 View Grades All scripts were executed through iHUB portal 0.22 5 View Financial Aid Student transactions were tested for 24000 users with a ramp NA 6 Add 4 Courses up of 0.1 seconds. NA 7 Ad hoc add course Current Production average user load is around 1500 in an NA 8 Add a Course hour NA 10 Drop Class Note: Two transactions in BOLD were successful when we NA 11 Swap Class tested it with 64 servers for 8000 students equivalent to 128 NA 12 Calendar Inquiry 0.37 13 Modify Personal Data servers for 24000 students. 128 Servers scenarios will likely 0.15 15 Account Inquiry (SF) not happen until we go live with all 34 colleges 0.14 16 View Class Schedule 0.07 17 Search for Open Class

CS Results Faculty Transactions v Error % S.No Transaction Name NA 1 Track Attendance Key Takeaway: Current Production environment sized Lookup teaching schedule Lookup advisee schedule Lookup advisee grades 0.36 2 to handle the 4K load Details 0 3 All scripts were executed through iHUB portal 0.33 4 0.38 5 Modify Bio/Demo Faculty and staff tested for 4000 users with a ramp 0.33 6 Search Classes up of 0.1 seconds. 0 7 View Student Account Lookup Class Roster 0.08 8

FSCM New Configuration Results v User Type Count 300 Procurement/Finance/Accountant FSCM (Current Prod) FSCM (Suggested Config) Parameter S. No Transaction Name Load Error % Number of Application server Instances Number of Webserver Instances 1 300 0.00% 2 48 Create PO 2 300 0.01% 2 48 Enter Change Order to PO 3 300 0.37% Create Voucher DB instance Size m5.2xlarge m5.8xlarge 4 300 0.94% Create Journal Jolt min handlers 8 50 5 300 0.00% Enter Suppliers Jolt max handlers 12 70 6 300 0.01% Update Supplier 7 No. of Application server processes 300 0.00% Delete Supplier 8 40 8 300 Journal Edit 9 Recycle count 4000 2500 300 0.00% Journal Post 10 300 0.00% Create Requisitions Key Takeaway: Current Production environment sized to handle the FSCM Load

HCM Load Testing v All the tests are done through iHUB portal Tests were performed on 12000 employees and 1500 Managers with ramp up period of 0.1 seconds User Type Counts Pillar HCM Employee 12000 HCM Manager 1500

HCM - On Recommended Configuration v Parameter HCM (Current Production) HCM (Recommended Configuration) Number of Application server Instances 24 48 Number of Webserver Instances 24 48 DB instance Size m5.xlarge m5.4xlarge Jolt min handlers 8 50 Jolt max handlers 12 70 No. of Application server processes 8 40 Recycle count 4000 2500 Employee Transactions Manager Transactions S.No Employee Transactions Error % S.No Managers Transactions Error % 0.47% View Paycheck 1 Job Data Inquiry 1 0.50% View W2 2 0.06% Promotions 2 0.55% W2 Consent 3 Terminations 3 Submit W4 Tax 4 0.03% Transfer 4 View Dependent Beneficiaries 5 Address Change 5 View Benefits summary 6 Key Takeaway: Current Production environment sized to handle the Manager Self-service Load Key Takeaway: Current Production environment sized to handle the Self-service/Employee Load

FASF Results Using New Configuration v S.No Managers Transactions Error % These are the results for FS SF module 1 Fluid UI-Payment History 0.09% Fluid UI-Account SummaryPage 0.15% 2 Test script was executed through iHub for 24000 users with a ramp up of 0.1 seconds. 0.22% 3 SS-Fluid-1098T 0.25% 5 Charges Due Page Since environment was sized already with CS tower, all the scripts were executed successfully with < 1% errors. 0.27% 6 SS-Fluid-Make a Payment Fluid UI-College Financing Plan 0.00% 8 0.00% 9 Fluid UI 0.00% 11 Fluid UI-Inquiry Data Page 0.00% 12 Fluid UI-FA Landing Page 0.00% 13 Fluid UI-Disbursements Key Takeaway: Current Production environment sized to handle the 24K load

Key Takeaways - Summary v On an average, errors went down by 95% based on our findings & recommended settings (from prior test round). It is assumed that the recommended configuration is maintained 24/7 to meet the anticipated load Load balancer server prior to recommendation was not distributing the load among multiple servers equally, and the issue has been addressed. If SBCTC thinks that the overall cost to maintain the recommended configuration is too high, data analysis can be carried out after 6 months or 1 year to identify the total load on the system and configuration can be modified accordingly Based on our findings the current infrastructure hosted on AWS need to be ramped up from fewer times of the current infrastructure The recommendations are for DG2, DG3, DG4 & DG5 Kastech recommends to turn on the autoscaling feature on AWS