Operations Management Suite (OMS): Change & Update Management

Introduction

Amongst the many features where OMS is growing in capability is the area of configuration management. This is an area where organizations look to develop the capability to quickly identify drift and shift in configuration and automate remediation of shifts from the desired state. Closely related to configuration management is update management, where in order to stay ahead of security threats, organizations strive to patch as quickly and with as little effort as possible. OMS now includes solutions in both configuration and update management, offering support for Windows and Linux operating systems.

In this chapter, we will explore the capabilities of OMS in tracking changes to key elements of Windows and Linux operating system configurations with the Change Tracking solution, as well as explaining how to use OMS for installing updates to supported versions of Windows and Linux systems with the Update Management solution.

Change Tracking Solution

The Change Tracking solution monitors several elements of Windows configuration, including changes to:

  • Installed software on Windows and Linux systems.
  • Windows services and Linux daemons.
  • User-specified files on Windows and Linux systems.
  • More than a dozen Windows registry keys by default, as well as user-defined keys.

The solution collects and sends data to the Log Analytics service in Azure for processing. Once received, solution logic is applied to the data and results are presented in the solution dashboard. Using the data in the Change Tracking dashboard, you can easily see recent changes made to your server infrastructure, and optionally, send alerts to system administrators.

Installation and Configuration

Before you can configure the Change Tracking solution, you need to:

  • Install an agent. A Microsoft Monitoring Agent for Windows or Linux needs to be installed on each computer where you want to monitor changes. For OMS agent deployment guidance, refer to Chapter 1.
  • Add the solution to your workspace. You will need to add the Change Tracking solution to your OMS workspace through the Azure marketplace or by adding it directly from the Solutions Gallery section of the OMS portal.

While no further configuration is required for the Change Tracking solution once it has been added, there is some optional configuration that enables additional monitoring beyond the default settings, specifically in the areas of file and registry key tracking.

File Tracking

The file tracking feature will track files on both Windows and Linux systems with the OMS agent installed. File tracking can be useful in a variety of configuration management and security scenarios, including monitoring for changes in content and permissions (ACLs) in custom line-of-business application files, or even Windows and Linux system binaries.

For Linux

Use the following steps to configure the files you wish to track on Linux computers.

  1. In the OMS portal, click Settings (the gear symbol).
  2. On the Settings page, click Data, and then click Linux File Tracking.
  3. Under Linux File Change Tracking, type the entire path, including the name of the file that you want to track and then click the Add symbol. A sample path is shown in Figure 1.

FIGURE 1. LINUX FILE CHANGE TRACKING PROPERTIES

Linux File Collection Properties

When you configure files for change tracking on Linux systems, there are a number of configurable properties, which deserve additional explanation:

  • Type - Defines the type of collection that will be monitored and you can choose from the following two options:
    • File. Report file metadata (size, modification date, hash, etc.) o     Directory. Report directory metadata (size, modification date, etc.)
  • Links - Defines how to handle Linux symbolic link (symlink) references to other files or directories. There are three options to choose from here:
    • Ignore - Ignore symlinks during recursions to not include the files/directories referenced.
    • Follow - Follow the symlinks during recursion to also include the files/directories referenced.
    • Manage - Follow the symlinks and alter the treatment of returned content.

Note: The "Manage" links option is not recommended, because file content retrieval is not currently supported.

  • Recurse - Recursively parse folder levels and track all files matching the path statement.
  • Sudo - Enable access to files or directories that require the sudo privilege.
For Windows

Use the following steps to configure files to track on Windows computers.

  1. In the OMS portal, click Settings (the gear symbol).
  2. On the Settings page, click Data, and then click Windows File Tracking.
  3. Under Windows File Change Tracking, type the entire path, including the name of the file that you want to track and then click the Add (+) button. There is no need to put quotes around the path and name, even if it includes spaces.
  4. Click Save.

Limitations

The Change Tracking solution does not currently support the following operations on the Windows platform:

  • Directories for Windows File Tracking
  • Recursion for Windows File Tracking
  • Wild cards for Windows File Tracking
  • Path variables
  • Network file systems
  • File Content

Note: Because these operations are unsupported, some of the aforementioned settings will have no effect in the current version.

  • The Max File Size column and values are unused in the current implementation.
  • If you collect more than 2500 files in the 30-minute collection cycle, solution performance might be degraded.

The following limitations should be considered for both the Windows and Linux platforms:

  • When network traffic is high, change records may take up to a maximum of six hours to display.
  • If you modify the configuration while a computer is shut down, the computer might post file changes that belonged to the previous configuration. This should correct itself on the next run, once the computer has received the updated solution configuration.

Registry Keys

OMS Log Analytics performs Windows registry monitoring and tracking with the Change Tracking solution. The purpose of monitoring changes to registry keys is to pinpoint extensibility points where third-party code and malware can activate. There are more than 15 keys tracked by default. A complete list of default registry keys is available at https://docs.microsoft.com/en-us/azure/log-analytics/log-analytics-change-tracking in the "Registry key change tracking" section.

Note: Presently, only the HKLM hive is supported by this feature.

You can also configure OMS to track Windows registry keys of your own choosing. Use the following steps to configure registry keys to track on Windows computers.

  1. In the OMS portal, click Settings (the gear symbol).
  2. On the Settings page, click Data, and then click Windows Registry Tracking.
  3. Under Windows Registry Change Tracking, type the entire key that you want to track and then click the Add (+) symbol as shown in Figure 2.

FIGURE 2. ADD WINDOWS REGISTRY KEY INTERFACE

  1. Click Save.

IMPORTANT: At the time this was written, the 'Recursive' option has been disabled when adding your own Windows registry keys. This is expected to become available again in a future update.

Software and Packages

Tracking installation or removal of software (Windows) or packages (Linux) requires no configuration beyond adding the Change Tracking solution to your workspace and deploying the agent.

Services and Daemons

Like software and packages, tracking changes to services and daemons does not require additional configuration.

Change tracking frequency

The following table shows the data collection frequency for each type of change, which gives you a better idea of the intended use cases for this solution.

Change type

Frequency

Does agent send differences when found?

Windows registry

50 minutes

No.

Windows file

30 minutes

Yes. If there is no change in 24 hours, a snapshot is sent.

Linux file

15 minutes

Yes. If there is no change in 24 hours, a snapshot is sent.

Windows services

30 minutes

Yes, every 30 minutes when changes are found. Every 24 hours a snapshot is sent, regardless of change. So, the snapshot is sent even where there are no changes.

Linux daemons

5 minutes

Yes. If there is no change in 24 hours, a snapshot is sent.

Windows software

30 minutes

Yes, every 30 minutes when changes are found. Every 24 hours a snapshot is sent, regardless of change. So, the snapshot is sent even where there are no changes.

Linux software

5 minutes

Yes. If there is no change in 24 hours, a snapshot is sent.

TABLE 3. FREQUENCY OF DIFFERENT CHANGE TRACKING OPERATIONS

Based on these times, you can see Change Tracking is designed more for retrospective troubleshooting than for real time monitoring. However, Microsoft is looking into improving the collection frequency for some elements, such as Windows Services. In the meantime, there is a workaround event logs.

Event logs reach OMS within 5-10 minutes of being created on the source computers. For the computers being monitored by OMS, in the OMS portal in the Settings > Data > Windows Event Logs section, you can enable the transfer of System logs to OMS. In OMS log search, you can query against these logs to find all stopped services (event IDs 7024 and 7034 for unexpected stoppage, 7036 for clean stoppage), and then you can take action or set up an alert to take action, like restarting the service using an automation runbook.

Custom fields, which will be useful in extracting event details of service stoppage events, are covered in "Chapter 6: Extending OMS Using Log Search". Alerting is covered in "Chapter 7: Alert Management".

Using the Change Tracking Solution

After the solution is installed, you can view the summary of changes for your monitored servers by using the Change Tracking tile on the Overview page in the OMS portal.

You can view changes to your infrastructure and then drill into details for the following categories:

  • Changes by configuration type for software and Windows services.
  • Windows service changes for individual servers.
  • Software changes to applications and updates for individual servers.
  • Total number of software changes for each application.
  • Linux packages.
  • Linux daemon changes.

Changes of any monitored type can be viewed in a number of different ways:

On the Overview page, click the Change Tracking tile, as shown in Figure 3.

FIGURE 3. CHANGE TRACKING TILE

On the Change Tracking dashboard, shown in Figure 4, you can review the summary information in one of the change type blades (the 'Changes over time by type' view). When you click on any of the areas reflecting recent changes, you are taken to an appropriate view of detailed information in the log search page.

FIGURE 4. CHANGE TRACKING DASHBOARD (PARTIAL)

On any of the log search pages, you can view results by time, detailed results, and your log search history. You can also filter by facets to narrow the results.

Files

How changes to files are tracked depends on the type of change you are talking about. For example, changes in file content are identified by a change in the MD5 hash of the file. The solution tracks not only changes in file content, last modified date, and size but also changes in the access control list (ACL), which defines file permissions. When a file is deleted, the Change Category column shows as 'Removed', as shown in Figure 5.

Note: While changes in file content will be surfaced due to change in the file's MD5 hash, the details of the content itself are not captured by this feature.

FIGURE 5. FILE MONITORING IN THE CHANGE TRACKING SOLUTION

Note: In the authors use of this feature in test and production scenarios, when a directory is specified for file monitoring, new files were not reported with any form of 'new file' event. Only changes and deletes were reported.

Windows Services or Linux Daemons

Changes in service or daemon state or configuration are also surfaced in the Change Tracking dashboard. If you click on the Services Change Tracking tile, you will see details of service changes, including status (starts and stops) or configuration, such as the service startup type, as shown in Figure 6.

FIGURE 6. SERVICE CHANGES SHOWN IN THE CHANGE TRACKING SOLUTION

Note: In the authors testing, new services do not appear as new services. When new services appear (as with those shown in Figure 6 above), they are generally shown as services with changes.

IMPORTANT: While you will notice the SvcState property present in the list of service properties, the Change Tracking solution is not the best way to drive timely notifications of service stoppage. Instead, use service status change events captured in Log Analytics, as detailed in "Change tracking frequency" earlier in this chapter.

Software and Packages

The Change Tracking solution also monitors for changes in installed software by taking snapshots of the list in Program and Features in the Windows Control Panel, or list of installed packages (Linux). Applications that have been added or removed since the previous snapshot will be reflected in the Log Analytics data behind the Change Tracking dashboard.

Note: Applications that are uninstalled then reinstalled between snapshots, or vice versa, will not be reflected as changes in the Change Tracking dashboard.

After a cycle in which changes to the list of install software are detected, you will see an update in 'Changes over time by type' in the dashboard. When you click on a blue box in the Software category, you will see adds and removes, as shown in Figure 7.

FIGURE 7. SOFTWARE CHANGES SURFACED BY THE CHANGE TRACKING SOLUTION

Update Management Solution

The Update Management solution in OMS allows you to manage operating system security updates for your Windows and Linux computers deployed in Azure, on-premises environments, or other cloud providers. The solution enables administrators to assess the status of available updates on all Windows Server and Linux computers, and then manage the process of installing required updates for servers.

Solution Architecture and Process Flow

Computers managed by OMS use the following for performing assessment and update deployments:

  • OMS agent for Windows or Linux.
  • PowerShell Desired State Configuration (DSC) for Linux.
  • Automation Hybrid Runbook Worker.
  • Microsoft Update or Windows Server Update Services for Windows computers.

Figure 8 shows a conceptual view of the solution architecture, rocess, and data flow with how the solution assesses and applies security updates to all connected Windows Server and Linux computers in an OMS workspace.

FIGURE 8. UPDATE MANAGEMENT SOLUTION ARCHITECTURE

The process flow, illustrated in Figure 8, is as follows:

Step 1: Compliance scan. After the computer performs a scan for update compliance, the OMS agent forwards the scan results to OMS. On a Windows computer, the compliance scan is performed every 12 hours by default. With a Linux computer, the compliance scan is performed every 3 hours by default,

Note: On both Windows and Linux systems, an update compliance scan is initiated within 15 minutes if the Microsoft Monitoring Agent (MMA) is restarted.

The compliance information is then processed and summarized in the dashboards included in the solution or searchable using user-defined or pre-defined queries. The solution reports how up-to-date the computer is based on what source you are configured to synchronize with. If the Windows computer is configured to report to WSUS, depending on when WSUS last synchronized with Microsoft Update, the results may differ from what Microsoft Updates shows. The same for Linux computers that are configured to report to a local repository versus a public repository.

Note: For Windows systems, the Update Management solution will only report on the updates approved in WSUS.

Step 2: Update selection and targeting. You can deploy and install software updates on computers that require the updates by creating a scheduled deployment. Updates classified as optional are not included in the deployment scope for Windows computers, only required updates. The scheduled deployment defines what target computers will receive the applicable updates, either by explicitly specifying computers or selecting a computer group that is based on log searches of a particular set of computers. You also specify a schedule to approve and designate a maintenance window during which updates are allowed to be installed. The solution allows maintenance windows from 30 minutes up to 6 hours.

Step 3: Update deployment initiated in Azure Automation. Updates are installed by runbooks in Azure Automation. You cannot view these runbooks, and they do not require any configuration. When an update deployment is created, it creates a schedule that starts a master update runbook at the specified time for the targeted computers. This master runbook starts a child runbook on each agent that performs the installation of required updates.

Step 4: Deployment on targeted computers. At the date and time specified in the update deployment, the target computers execute the deployment in parallel. The agent performs a scan first to verify the updates are still required and installs them.

Note: On Windows computers that are also WSUS clients, if the updates are not approved in WSUS, the update deployment will fail.

Step 5: Results processing and reporting. The results of the applied updates are forwarded by the OMS agents to OMS, where they are processed and summarized in the dashboards or by searching the events.

Install & Configuration

The prerequisites for the solution are fundamentally the same for Windows and Linux computers:

A supported OS

  • For Windows Server, Windows 2008 R2 SP1 and higher is supported. Client operating systems are not supported, nor are Server Core or Nano Server.

WARNING: Pay close attention to the lack of support for Server Core. This is a commonly chosen option for Windows Server 2016 deployments.

  • For Linux, CentOS 6 and 7 (x64 only), Red Hat Enterprise 6 and 7, SUSE Linux Enterprise Server 11 and 12 (x64 only), Ubuntu 12.04 LTS and newer. Unless otherwise mentioned, both x86 and x64 versions are supported.

Note: An OMS Agent for Linux configured to report to multiple OMS workspaces is not supported with this solution.

Update source

  • Windows agents must either be configured to communicate with a Windows Server Update Services (WSUS) server or have access to Microsoft Update.
  • Linux agents must have access to an update repository. Updates are installed with the platform's package manager, which may be Yum, Apt or Zypp, depending on the platform.

Note: The Update Management solution does not integrate with the update management feature in System Center Configuration Manager.

Adding the solution to your OMS workspace

You can deploy the solution in either the Azure Portal or the OMS portal or an Azure Resource Manager template. Refer to the "Adding solutions to your OMS workspace" section in "Chapter 1: Introduction and Onboarding" if you are unfamiliar with how to add a solution to OMS.

Hybrid Runbook Worker Role & Update Management

After you enable the Update Management solution, any Windows computer directly connected to your OMS workspace is automatically configured as a Hybrid Runbook Worker to support the runbooks included in this solution. For each Windows computer managed by the solution, it will be listed under the Hybrid Runbook Worker Groups blade of the Automation account following the naming convention Hostname FQDN_GUID.

Note: These groups are only intended to support the management solution. If you target these groups with runbooks, the runbook job will fail.

However, you can add the Windows computers to a Hybrid Runbook Worker group in your Automation account to support Automation runbooks, as long as you are using the same account for both the solution and Hybrid Runbook Worker group membership. This functionality has been added to version 7.2.12024.0 of the Hybrid Runbook Worker.

Using the Update Management Solution

When you add the Update Management solution to your OMS workspace, the Update Management tile (shown in Figure 9), will be added to your OMS dashboard. This tile displays a count and graphical representation of the number of computers in your environment and their current update compliance.

FIGURE 9. UPDATE MANAGEMENT TILE

Click on the Update Management tile to open the Update Management dashboard shown in Figure 10, which presents additional detail on computer update compliance, a link to the deployment configuration interface, as well as a list of linked sampled queries for use in Log Analytics.

FIGURE 10. THE UPDATE MANAGEMENT DASHBOARD

You can use this dashboard to get a detailed breakdown of update status categorized by type of operating system and update classification - critical, security, or other (such as a definition update). The Manage Update Deployments tile redirects you to the update deployments page where you can view schedules, deployments currently running, completed deployments, or schedule a new deployment.

You can run a log search that returns all records by clicking on the specific tile, or to run a query of a category and pre-defined criteria, select one from the list available under the Common Update Queries column.

Installing updates with the Update Management Solution

Once updates have been assessed for the Linux and Windows computers in your workspace, you can install required updates by creating an update deployment. An update deployment is a scheduled installation of required updates for one or more computers. You specify the date and time for the deployment in addition to a computer or group of computers that should be included in the scope of a deployment.

IMPORTANT: When you include computer groups in your update deployment, group membership is evaluated only once at the time of schedule creation. Subsequent changes to a group are not reflected. To work around this, delete the scheduled update deployment and recreate it.

Notes on default update management behavior:

  • Windows systems. Windows VMs deployed from the Azure Marketplace by default are set to receive automatic updates from Windows Update Service. This behavior does not change after adding this solution or Windows VMs to your workspace. If you do not actively manage updates with this solution, the default behavior (automatically apply updates) will apply.
  • Linux systems. For virtual machines created from the on-demand Red Hat Enterprise Linux (RHEL) images available in Azure Marketplace, they are registered to access the Red Hat Update Infrastructure (RHUI) deployed in Azure. Any other Linux distribution must be updated from the distributions online file repository following their supported methods.

Creating an update deployment

To create a new update deployment, complete the following steps:

  • Click the Manage Update Deployments tile in the Update Management dashboard, then click the Add (+) button at the top of the screen, shown in Figure 11.

FIGURE 11. UPDATE DEPLOYMENT SCHEDULE & STATUS

  • This opens the New update deployment page, shown in Figure 13. You must provide values for the following properties:
  • Name. Unique name to identify the update deployment.
  • Time Zone. Time zone to use for the start time.
  • Schedule Type. Type of schedule. Options available are One Time, Recurring Weekly, or Recurring Monthly.
  • Start Time. Date and time to start the update deployment. Updates should be scheduled at least 30 minutes in the future, though planning much further ahead is advisable.
  • Duration. Number of minutes the update deployment is allowed to run. If all updates are not installed within this duration, then the remaining updates must wait until the next update deployment.
  • Computers. Names of computers or computer groups to include and target in the update deployment. Select one or more entries from the drop-down list.

FIGURE 12. THE "NEW UPDATE DEPLOYMENT" SCREEN

Time Range

By default, the scope of the data analyzed in the Update Management solution is from all connected management groups and direct agents generated within the last 1 day.

To change the time range of the data, select Data based on at the top of the dashboard. You can select records created or updated within the last 7 days, 1 day, or 6 hours. Or you can select Custom and specify a custom date range.

Tracking update deployments

Click the Update Deployment tile (shown in Figure 13) to view the list of existing update deployments. They are grouped by status Scheduled, Running, and Completed.

FIGURE 13. EXISTING UPDATE DEPLOYMENTS

On the Scheduled tab, you will see the following properties for each update deployment you configure: Name, Schedule, Start Time, and Duration. On the Completed tab, you will see the details of completed update deployments. These columns will not be populated if the update deployment has not yet started.

Searching Update Management Log Analytics records

The Update Management solution creates two types of records in the OMS repository:

  • Update. A record for each update that is either installed or needed on each computer.
  • UpdateSummary. A record of this type is created for each Windows agent computer and is updated each time the computer is scanned.

Sample search queries: You can find sample OMS log analytics searches for the Update Management solution in the "Sample log searches" section at https://docs.microsoft.com/en-us/azure/operations-management-suite/oms-solution-update-management

Troubleshooting the Update Management solution

You can troubleshoot the Update Management solution using the Patch-MicrosoftOMSComputer runbook, which targets a specific managed computer, and produces a verbose data stream with detailed information for that deployment. The output will display which required updates are applicable, download status, installation status, and additional details.

Rolling back updates:

Currently, the Update Management solution does not include a rollback option to undo problematic updates. Deploy updates to a test group before deploying to production.

Recommendation: If deploying to production without testing, allow some time for other organizations to test updates and report issues through TechNet forums and other public support channels.

Summary

At the start of this chapter, we introduced you to the OMS Change Tracking solution and its features for tracking configuration of Windows and Linux systems, along with the configurable options for tracking Windows file and registry changes. Then, we moved into the details of the Update Management solution for delivering required updates to Windows Server operating systems, as well as patches for supported Linux distributions.

While there is certainly work that remains to be done on these solutions to bring them to the level of comprehensive enterprise solutions, they have certainly shown considerable improvement the last few months.