How Hotpatching on Windows Server is changing the game for Xbox

Explore how Xbox drives efficiency using Windows Server and SQL Server on Azure

Would you like to avoid spending your weekends patching servers? The new Hotpatch feature in Windows Server 2022 Datacenter: Azure Edition addresses this pain point—it can reduce many IT teams’ headaches including reboot failures and coordinating multitier workloads. It increases productivity and end-user uptime and can reduce the vulnerability window that would result if an update is delayed.

To demonstrate how Hotpatching works, we’ve brought in an example from our very own Xbox team. In this article you’ll learn how Microsoft has been using Hotpatch with Windows Server 2022 Azure Edition to substantially reduce downtime for SQL Server databases running on Windows Server Azure virtual machines on an important set of backend services for the Xbox network.

What is Hotpatch?

Hotpatch for Windows Server 2022 Datacenter: Azure Edition allows you to apply every month’s “patch Tuesday” security updates, but does not require the server operating system to restart two out of three months.

While Hotpatch has been available on the Server Core option of Windows Server 2022 Azure Edition for some time, it has just become available in summer 2023 for the more widely used Desktop Experience option. You can see a demo of it in this on-demand session from Ignite.

Here’s what’s great about it:

• Higher availability and fewer restarts.
• Faster deployment of updates because the packages are smaller, install faster, and have easier patch orchestration using Azure Update Management.
• Better protection because the Hotpatch update packages are scoped to Windows security updates that install faster without restarting.

 When you enable Hotpatch, a baseline Cumulative Update is applied to the server. This update does require a reboot. After this point, your team can update easily, with fewer restarts, which can greatly reduce any vulnerability window. Check out this release documentation for details on the Hotpatch calendar.

How the Xbox network team uses Hotpatch

The Xbox network relies on several critical backend services hosted in SQL Server databases running on Windows Server Azure virtual machines. There are 18 different services hosted in this manner, with some services handled by two SQL Servers and others up to 120 SQL Servers. Some of these workloads have been in production for 15 years.

Of course, when you’re running backend services for a group of passionate gamers like Xbox network customers, it’s imperative to patch and restore services with as little downtime as possible.

Approximately 1,000 servers hosting these services started their journey on physical hardware when the services were first deployed, and more than 15 years later, through a process of rolling upgrades and migration, are now running in Azure hosted as infrastructure as a service (IaaS) Virtual Machines (VMs). According to senior service engineer Tim Dreyling, the team has found it “magnitudes easier to manage Windows Server on Azure VMs, over relying on data center support to address ‘machine’ issues.”

After migrating the backend Xbox network services from the earlier version of Windows Server 2022 Azure Edition to the version that supported Hotpatch, the team that supported these specific backend services went from an update cycle every month that could take weeks of careful orchestration to being able to apply Hotpatch updates across a fleet of nearly 1,000 servers in less than 48 hours two months out of every three.

“As a database administrator (DBA) this is the biggest thing to increase our service reliability and uptime since SQL Server Availability Groups were introduced with SQL Server 2012,” says Tim.

Hotpatch with Windows Server 2022 Datacenter Azure Edition isn’t just used with SQL Server with Xbox network backend services, but is also used on IaaS VMs running Active Directory DS Domain Controllers and VMs hosting web services roles.

While your services might not have the complexity and scale of the Xbox network, we think you’ll quickly see the Hotpatch advantage of minimizing reboot downtimes while ensuring the services you host are reliable, protected, and available.

Hotpatch is currently available on Azure Edition (see below for details), but the team has more innovations in the works, and many ways to access cloud innovation in your hybrid cloud environment by connecting your servers to Azure Arc.

In case you weren’t able to join us at Ignite, you can watch two Windows Server-focused sessions on-demand. These talks cover Hotpatching and the Xbox example discussed above, along with a number of new and upcoming features for our Windows Server and SQL Server customers:

1. Do More with Windows Server and SQL Server on Azure—Bob Ward, Principal Architect in the Azure Data team, and Jeff Woolsey Principal PM Manager in Windows Server, do a quick-fire session with descriptions of the latest innovations across these technologies.
2. What’s New in Windows Server v.Next—Elden Christensen, Principal Group PM Manager, joins Jeff Woolsey to explain and demo the features that our engineering team is working on for the next Windows Server.

If you’re interested in being hands-on and trying what’s coming next for Windows Server, you can get early access to the latest features in the works by joining the Windows Insider program.

Learn more about Windows Server and Hotpatch

• Try the latest versions of SQL Server and Windows Server 2022 on Azure.
• You can find out more about Hotpatch for Windows Server 2022 Datacenter: Azure Edition in the documentation and in this blog post.
• Download the image here to get the new edition with Hotpatch (Azure account required).