Unlocking the Potential of DevOps in Organizations

MN Department of Health Journey to DevOps Andrew Will-Holmberg | Cloud Operations Supervisor Brenda Gabriel | Director IT Projects & Applications Joseph Pugh | Development & Data Strategy Shawn Kammerud | CISO

Agenda Introduction to DevOps Creating a Roadmap MN Department of Health: a case study Questions and Answers!

Introduction to DevOps Andrew Will-Holmberg | Cloud Operations Supervisor

What is DevOps? DevOps is a set of practices and principles that integrate application development and IT operations throughout the entire application lifecycle. Under a DevOps model, multidisciplinary teams work to improve speed and reliability by automating historically manual processes, and prioritizing interoperability of their tools and systems.

Benefits of DevOps Speed Scalability Reliability Security Efficiency Improved Collaboration

DevOps Principles Collaboration Infrastructure as Code Automation Self-Service Iteration

Collaboration Break down silos Engage all stakeholders Consolidate technical documentation and logs Focus on ease of adoption for tools and procedures Think about systems holistically

Infrastructure as Code Cast a wide net if it CAN be defined as code, do so All changes versioned in source control

Automation Automate systemically If it can be defined as code, it can be automated. The greater the consequence of human error, or more frequent the task, the greater the need for automation. Don t automate waste Consider process improvements before automating.

Self-service In order to scale, heterogeneous workloads require either flexible systems and processes, or segmentation of centralized services. Prioritize tools with APIs. Templatize your work with the intent to share.

Iteration Small frequent changes are easier and more reliable than large infrequent changes. Feedback is key! Continual Integration Continual Deployment (CI/CD)

Creating a Roadmap Andrew Will-Holmberg | Cloud Operations Supervisor

Getting started Identify all stakeholders involved with organizing a DevOps initiative and establish goals. Identify all of the teams involved with your application lifecycle and determine how they will be represented in your DevOps initiative. Assess current applications, tools, and procedures to identify constraints. Use small achievable iterations. Frame your deliverables with the organization s goals in mind.

Common pitfalls to avoid Prioritize process and cultural shifts over tool acquisition. When rolling out new tools, make sure you re adhering to DevOps principles. Don t avoid stakeholders. Engage early and keep open lines of communication. Seek feedback at every opportunity, and then act on it! Avoid analysis paralysis and big bang efforts. Have a bias for action, encourage experimentation and lateral thinking, and hold blameless post mortems.

Frequently asked questions Q: Will DevOps bring value to my organization if we re using waterfall? A: Yes! Q: Do I need to use the Cloud in order to transition to DevOps? A: No! Q: Is DevOps compatible with ITIL? A: Yes!

MN Department of Health: a case study Andrew Will-Holmberg | Cloud Operations Supervisor

Overview In 2017, the MN Department of Health completed their initial governance and planning phases for their AWS cloud migration project, and began the work of migrating their applications. The first application migration was completed in October of 2017. It was determined that transitioning to a DevOps approach would greatly accelerate their cloud migration initiative. I was brought on to the team in February of 2018 to oversee the operations team and to accelerate the DevOps transition.

Initial assessment Stakeholder s primary goals were: migration of all MDH applications to the Amazon Cloud, maintain service availability and development work with the same staff conducting the migration, security remediation on all migrating applications. Priority was security and reliability, then speed, then cost containment. Large number of small to medium sized applications, small number of larger applications. Large number of internal-facing applications. Public-facing applications were highly visible and provided mission-critical services. Mostly Java-based applications ran on Linux environments, but technology stack spanned nearly two decades.

Collaboration assessment Nearly all of the teams involved with the MDH application lifecycle were in-house, with shared vision and goals. Teams were in traditional silos, with much of the communication occurring via email and tickets. An ops resource had been embedded with one of the dev teams. Some shared documentation and logs.

Infrastructure as Code assessment Source control system was available, and use was high but not universal across all teams. Some configuration management via Ansible and Cloud Formation templates.

Automation assessment Overall a low amount of automation. Operations automation was mostly related to patch management. Most developer automation related to the build process very intermittent use of test automation.

Self-service assessment Most tools not capable of or configured for self-service. Cloud Formation templates were used to allow developers to provision their own test and dev environments. Developer teams were given high levels of access to a non-production environment in AWS to facilitate testing and experimentation.

Iteration assessment Continual Integration (CI) achieved for a small number of applications, no use of Continual Delivery (CD). Overall cloud migration project was laid out as waterfall, but PMO was very flexible when responding to shifting business and IT priorities. On an app by app basis, most developer teams were able to define their own dev and test processes. Significant bottlenecks with operations when existing Cloud Formation templates were insufficient. Long feedback loops with user acceptance and security testing.

Our approach Source control system was identified as a significant constraint. We implemented a new system within two months of my arrival this was the only new tool introduced for my first nine months. Focus was on ease of use and self-service. All code was consolidated within this system. We heavily encouraged continued use of our cloud formation templates, but didn t make them a hard requirement. Operations team remained flexible. We introduced working sessions that involved all of the teams working on a particular application. Only people involved in troubleshooting were allowed to attend these sessions. New services and procedures were directly tied to meeting organizational goals. Steady incremental improvements and successes were communicated across all stakeholders. Different developer teams were used when introducing new capabilities to distribute work and to encourage cross training between teams. Only after the migration project was completed in July of 2019, was time spent on updating our toolchain, containerization push, and CI/CD push.

Questions & Panel Discussion

Thank You! Andrew Will-Holmberg| Cloud Operations Supervisor Andrew.Will-Holmberg@state.mn.us Brenda Gabriel| Director IT Projects & Applications Brenda.Gabriel@state.mn.us Joseph Pugh| Development & Data Strategy Joseph.Pugh@state.mn.us Shawn Kammerud| CISO Shawn.Kammerud@state.mn.us