Patching and DevOps: A Match for Today's IT
At this year's Technology Exchange, Nick Lewis and I gave a presentation on Rugged DevOps, the concept of incorporating security into the DevOps pipeline. This post represents an extension of that presentation and extends it to a specific topic — patching.
For a level-set on the definition of DevOps in this post, we will use the definition from The Agile Admin:
"DevOps is the practice of operations and development engineers participating together in the entire service lifecycle, from design through the development process to production support."
Now that we have that out of the way, lets dig into the meat of the conversation. One of the key components of a good DevOps practice is the idea of "shifting left." In an IBM blog post, "shifting left" is described as a "Focus on quality, work on prevention instead of detection, and begin testing earlier than ever before." The idea being: the earlier and more often bugs are addressed (left being the start of the DevOps pipeline) the less bugs there will be in the final product.
So, how does shifting left help with application patching? Well, patching software can be thought of as addressing bugs in software too. Patching is essentially a collection of bug fixes. The software provider is just squashing the bugs for you to deploy the patch to your environment. By using the latest build of a software component, as early as possible in your run up to production, it automatically becomes part of your testing process. The patches can also be incorporated in a consistent, versioned way, as part of a regular build process. Patching as part of a build process also documents what changes were made and creates a delta from the previous version. This new version can also more easily roll back to a prior version if issues arise because the patch is incorporated as part of a build pipeline.
An especially important, and oft overlooked benefit is by building and deploying consistently in a dev/test environment, it leaves the prior version running production until they are ready to roll out. It also creates a process where ad hoc production changes and hotfixes are deeply discouraged and can be tested in dev/test. I personally suppress many a waking nightmare where some critical patch at 2am did not go as planned and have to race against the clock to get it fixed before the next morning. And while we are on the topic, wouldn't it be great if you got to the point that you could patch during the day! This is the first step to that achievable dream.
Taking this process, then, how might it apply to an ERP system? ERP system are commonly complex, three-tier web apps (web, app, and database). While in this scenario, database roll backs can be very difficult, the process described above can work quite well for packaged app and web instances. Each tier can be patched, built, run, and scaled independently. Cornell University, in fact, has had a good deal of success along these lines.
This process opens up some interesting techniques that could be used into the future. Canary builds can be used to test out new features with a small percentage of the user base before rolling out more fully. For a modern app with well-defined APIs, each service, rather than tier, could be managed independently, with even more greatly increased flexibility to deploy and run the application.
Modern apps, service driven architectures, and containers are means to an end. Ultimately universities will experience better service with this approach, and will achieve better results for their constituents. By slowly adding one element at a time, like containerizing an ERP's web app as a start, we can pursue that mission. We can also can also use the technology change to teach ourselves how to more flexibly serve these needs.
The community's use of DevOps techniques is still evolving, and our tooling to support the pipeline is still in development. As the market coalesces around a few of these tools, they will be adopted by campuses and be excellent candidates for NET+ services. If you would like to engage with the community around these services, or sponsor them for inclusion in the NET+ program, please reach out to me at firstname.lastname@example.org.
Get Regular Cloud Strategies and NET+ News!
Sign up now to receive regular updates and resources on leading Cloud services that are easing the challenges, costs, and risks of Cloud migration for research and education.