Dancing with Tropical Storm Irene in Vermont
Expert discussion on dealing with Tropical Storm Irene in Vermont, including lessons learned, business continuity, and weather forecasts. Explore the challenges faced and strategies employed during the storm to ensure safety and resilience.
Download Presentation
Please find below an Image/Link to download the presentation.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author. If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.
You are allowed to download the files provided on this website for personal or commercial use, subject to the condition that they are used lawfully. All files are the property of their respective owners.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author.
E N D
Presentation Transcript
EXPERT DISCUSSION: DANCING WITH TROPICAL STORM IRENE IN VERMONT Duncan Sutherland Chief Technology Officer Vermont Law School Vermont State Colleges Linda Hilton Chief Information Officer
VERMONT STATE COLLEGES http://elm.vsc.edu/PublishingImages/csclogoportal.gif http://elm.vsc.edu/PublishingImages/Lyndon-State-College-Logo2.png http://elm.vsc.edu/PublishingImages/VTC%20logo%20153%20x%2053.jpg http://elm.vsc.edu/PublishingImages/CCV_Logo-portal.jpg http://elm.vsc.edu/PublishingImages/JSC-H-rgb2.jpg
THE GOOD STUFF UP FRONT: LESSONS LEARNED Have a plan, regardless of resources! Good relationships sustain; bad ones are killer Stay on the lookout for good ideas and be flexible The clich is true: never waste a good disaster Keep track document Communications are critical & difficult
VERMONT STATE COLLEGES Shared Services Data center disaster recovery Business Continuity
AS OF 427 AM EDT FRIDAY...HURRICANE IRENE REMAINS THE MAJOR PLAYER FROM SATURDAY NIGHT INTO MONDAY MORNING AS IT LIKELY TREKS NNE FROM NEAR CAPE HATTERAS TO LONG ISLAND/CT AND ACROSS CENTRAL/NRN NEW ENGLAND. LATEST MEDIUM RANGE MODELS NOW ALL COMING TOGETHER ON THIS SOLN INCLUDING THIS MORNING`S GFS AND ECMWF WHICH ARE AMONG THE MORE RELIABLE IN HANDLING TROPICAL SYSTEMS. THESE TRACKS ARE INCIDENTLY NEARLY IDENTICAL TO THE OFFICIAL NHC FORECAST SO CONCERN IS NOW INCREASING CONSIDERABLY FOR A VERY HIGH IMPACT EVENT FOR MUCH OF OUR AREA...THE LIKES WE`VE NOT SEEN HERE IN QUITE SOME TIME. THIS WOULD INCLUDE TORRENTIAL RAINFALL WITH A HIGH RISK OF FLOODING...AND VERY STRONG AND GUSTY WINDS FOR THE EASTERN 2/3RDS OF THE FORECAST AREA. INDEED...THE WIND COULD LIKELY BE STRONGER THAN IN TROPICAL STORM FLOYD BACK IN 1999 AS MODELS CONTINUE TO SUGGEST IRENE`S INNER CORE DYNAMICS WILL NOT DETERIORATE AS QUICKLY AS NORMAL AFTER LANDFALL. LIKELY AIDING IN IRENE`S SLOWER THAN AVERAGE WEAKENING WILL BE A POWERFUL 250-300 MB JET ACROSS SRN/CENTRAL QUEBEC ALLOWING ROBUST EVACUATION AND DIFFLUENCE ATOP THE SYSTEM AS IT NEARS AND CROSSES OUR REGION. BY MONDAY...IRENE SHOULD BE EXITING QUICKLY INTO QUEBEC THROUGH THE DAY WITH WINDS SLOWLY ABATING AND OVERALL IMPROVING WEATHER CONDITIONS EXPECTED. STORM SPECIFICS FOLLOW...THOUGH PLEASE NOTE THESE ARE MY BEST ESTIMATES VIA COORDINATION WITH SURROUNDING OFFICES AND USE OF BLENDED NHC/HPC GUIDANCE. CONDITIONS COULD PAN OUT SLIGHTLY DIFFERENT. RAINFALL... TIMING...WIDESPREAD HEAVY RAINFALL EXPECTED SUNDAY INTO SUNDAY NIGHT FROM THE ADIRONDACKS EASTWARD...ENDING BY MONDAY MORNING. AMOUNTS...LIKELY TO RANGE FROM LESS THAN AN INCH IN THE ST LAWRENCE VALLEY...1-3 INCHES IN THE ADIRONDACKS...2-5 INCHES IN THE CHAMPLAIN VALLEY AND WESTERN VT...AND 3 TO AS MUCH AS 7 INCHES FROM THE GREEN MTNS INTO ERN VT WITH LOCALLY HIGHER AMOUNTS. FLOODING... AREA OF CONCERN...BIGGEST CONCERN FOR VERMONT STREAMS AND RIVERS BUT SOME THREAT FOR THE EASTERN SLOPES OF THE ADIRONDACKS AS WELL. TIMING...HIGHEST FLOOD THREAT WOULD OCCUR SUNDAY NIGHT INTO MONDAY AND COULD BE QUITE SIGNIFICANT. WINDS... IRENE
THE BAD STUFF: WHAT HAPPENED On Aug 28, 2011, at 8:13 PM, "Pollak, Dianne M" <dianne.pollak@vsc.edu> wrote: > Just got a text from michael... First floor of the hospital is under water... They are evacuating randall st. We are on generator.
THE BAD STUFF: WHAT HAPPENED By 6 am on 8/29 water had filled basement and up to 5 feet of the first floor Generator under water = power gone Entire state office complex flooded Roads to town closed
THE BAD STUFF: BUSINESS IMPACT Two colleges had started classes Two colleges were about to start classes on Tuesday One college was still in marketing mode
VERMONT STATE COLLEGES SHORT-TERM SOLUTIONS Monday - Emergency web site up by Monday Monday - Alternate e-mail addresses collected Tuesday - Wide-area network and Internet restored Monday - Colleges using internal e-mail Monday-Thursday Some paper paychecks handwritten Tuesday-Friday Air-conditioning and server issues Friday morning ERP and other critical systems restored
Data center
High Water Mark DATA CENTER GENERATOR...
VERMONT STATE COLLEGES EARLY LESSONS Take notes everywhere, in every conversation Have a conference call system set up Texting is critical (and set up those contacts!) Have an alternate e-mail address for all critical contacts IT and priority business areas Have a plan for alternate web site hosting Staff check-in: food and sleep Staff in different places on the resilience scale As everyone gets tired, mistakes will be made Danger in false sense of stability/reliability
From: The EDUCAUSE CIO Constituent Group Listserv [mailto:CIO@LISTSERV.EDUCAUSE.EDU] On Behalf Of Theresa Rowe Sent: Thursday, November 03, 2011 8:44 AM To: CIO@LISTSERV.EDUCAUSE.EDU Subject: Re: [CIO] Snowtober Thanks so much for taking the time to write out the details of your experiences. We all learn from these posts. Our experiences with electrical power failures and generators have been very difficult to manage. Every situation has been different, so you really need thinking, problem-solving people on hand, rather than a scripted action. You've given us more to think about. Theresa
Thinking, problem solving people
VERMONT STATE COLLEGES OPPORTUNITIES Reinforce the importance of both disaster recovery and business continuity planning and underscore the differences Make the case for disaster recovery resources- less difficult in the reality of business interruption Change things that were unchangeable Focus on what s important: people and mission
VERMONT STATE COLLEGES TODAY Access to building restricted to 2-3 staff IT Staff distributed across the state Moving primary data center to co-location Dealing with displaced employees Re-projecting the annual project list Getting back to normal
DANCING WITH IRENE Technology, Tropical Storm Irene, and the Best Laid Plans Duncan Sutherland Chief Technology Officer Vermont Law School
A few facts about VLS One of a handful of independent, private law schools in the U.S. Established in 1972 Located in South Royalton, VT Situated on a 13-acre campus with 19 buildings ~ 600 students in three programs (JD, LLM, MELP) ~ 70 faculty ~ 100 administrative staff Consistently top-ranked in environmental law by U.S. News & World Reports
A few facts about me Prior to VLS, Smart Grid strategy at Seattle City Light (Seattle, WA) Prior to Seattle City Light, Microsoft (Redmond, WA) CTO for a couple of large law firms Faculty member at the George Washington University School of Business and Public Management Way, way back, Wang Laboratories, Inc. Final interview at VLS on August 25, 2011 Started (early) at VLS on September 13, 2011 Decidedly notthe dance card I expected for my first 6 months!
A few facts about VLS technology prior to Irene Ten folks in IT department ~70 servers Unmanaged environment Novell and GroupWise (lots of grumbling from users, support intensive, complex to manage) Windows XP running on a wide variety of hardware Mix of Office 2003 and 2010 Jenzabar QX (long-in-the tooth) Fiber campus backbone (long-in-the tooth) Modern data center with fire suppression, adequate cooling, backup generator, etc.
A few facts about VLSs encounter with Irene One of the costliest hurricanes on record in the Northeastern United States Storm crossed the Vermont/New Hampshire border early in the morning of August 29, 2011 as an extratropical cyclone School suffered some damage from flooding, but Cooling was lost in the data center leading to a catastrophic heat event! Staff was delayed in responding to the situation due to impassable roads, destroyed bridges, etc. Immediate challenges were: (1) data protection; (2) keeping school running (start of fall classes was imminent); and (3) replacing damaged servers but, replacing them with what?
A few facts about VLS technology after Irene Decision to virtualize our data center (fortunately, our hardware was covered by insurance!) Decision to immediately move to a fully-managed environment (starting with a VLSDCC based on the FDCC ) Decision to immediately move to AD Decision to immediately move to Windows 7, Microsoft Exchange, and Office 2010 Decision to begin deploying thin clients and virtualized desktops Decision to delay the kickoff of our planned upgrade from Jenzabar QX to Jenzabar EX
A few lessons learned Assume nothing! When it comes to disaster planning, consider the skill set your team will need (both initially and going forward) if you are forced to make a technological leap even if you plan on using contractors to do the heavy lifting Identify and leverage allies within the organization who will be excited and supportive about getting rid of old technology folks who can talk knowledgeably about how great the new world will be! Make sure that your vendors are as prepared as you are Seek out and leverage opportunities even in the face of major disasters Test, test, test!
And in conclusion My first six months at VLS have been really interesting! Thank you!
Thank you for attending! Please remember to fill out this session s evaluation by going to the appropriate listing in the conference program on the EDUCAUSE website. Your feedback directly affects future planning.
