Lessons Learned for Enhancing Power Grid Reliability and Security
Explore valuable lessons learned from cases of communication failures in power grid control centers, emphasizing the importance of maintaining monitoring and control, implementing corrective actions, and optimizing operational strategies to bolster reliability, resilience, and security.
Uploaded on | 0 Views
Download Presentation
Please find below an Image/Link to download the presentation.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author. If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.
You are allowed to download the files provided on this website for personal or commercial use, subject to the condition that they are used lawfully. All files are the property of their respective owners.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author.
E N D
Presentation Transcript
Lesson Learned Loss of Monitoring and Control due to a Communication Failure between Control Centers Wei Qiu ERCOT TWG Conference Call May 29, 2025 RELIABILITY | RESILIENCE | SECURITY
Limited Disclosure Remote Operation 2 RELIABILITY | RESILIENCE | SECURITY Limited Disclosure
Limited Disclosure Case 1 Planned maintenance on UPS at control B Unintended consequence A momentary loss of power Stand-by backup firewall not recover properly ARP routine Issue Data was prevented from routing correctly through the firewall Correction Actions Operators transitioned from control center A to control B Informed RC 3 RELIABILITY | RESILIENCE | SECURITY Limited Disclosure
Limited Disclosure Case 2 Problem VPN tunnel collapsed (with unknown reasons) Both sites became Primary by design due to loss of communications between control centers After connecting both sites, a split-brain scenario was formed The system struggled to determine which COM was primary, which delayed the restoration process The databases out of sync Corrective Actions A patch from EMS vendor 4 RELIABILITY | RESILIENCE | SECURITY Limited Disclosure
Limited Disclosure Case 3 Problem Periodically operating from the alternate control center An issue with Ethernet Virtual Private Line (EVPL) connection Two-direction communication downgraded to one-direction Corrective Actions the web-based read-only version of the EMS and communicated instructions 5 RELIABILITY | RESILIENCE | SECURITY Limited Disclosure
Limited Disclosure Lessons Learned Utilizing EMS servers located in the same area as the operation personnel, especially during nights, weekends, and holidays Keeping/deploying a small number of operators work from the backup center during planned maintenance activities Configuring workstations within the data center where the EMS server are hosted avoid a need for VPN tunnels Additional VPN tunnels across different geographical areas And more 6 RELIABILITY | RESILIENCE | SECURITY Limited Disclosure
Limited Disclosure Questions and Answers Contact Information: wei.qiu@nerc.net 7 RELIABILITY | RESILIENCE | SECURITY Limited Disclosure
