Amongst the many features where OMS is growing in capability is the area of configuration management. This is an area where organizations look to develop the capability to quickly identify drift and shift in configuration and automate remediation of shifts from the desired state. Closely related to configuration management is update management, where in order to stay ahead of security threats, organizations strive to patch as quickly and with as little effort as possible. OMS now includes solutions in both configuration and update management, offering support for Windows and Linux operating systems.
In this chapter, we will explore the capabilities of OMS in tracking changes to key elements of Windows and Linux operating system configurations with the Change Tracking solution, as well as explaining how to use OMS for installing updates to supported versions of Windows and Linux systems with the Update Management solution.
The Change Tracking solution monitors several elements of Windows configuration, including changes to:
The solution collects and sends data to the Log Analytics service in Azure for processing. Once received, solution logic is applied to the data and results are presented in the solution dashboard. Using the data in the Change Tracking dashboard, you can easily see recent changes made to your server infrastructure, and optionally, send alerts to system administrators.
Before you can configure the Change Tracking solution, you need to:
While no further configuration is required for the Change Tracking solution once it has been added, there is some optional configuration that enables additional monitoring beyond the default settings, specifically in the areas of file and registry key tracking.
The file tracking feature will track files on both Windows and Linux systems with the OMS agent installed. File tracking can be useful in a variety of configuration management and security scenarios, including monitoring for changes in content and permissions (ACLs) in custom line-of-business application files, or even Windows and Linux system binaries.
Use the following steps to configure the files you wish to track on Linux computers.
FIGURE 1. LINUX FILE CHANGE TRACKING PROPERTIES
Linux File Collection Properties
When you configure files for change tracking on Linux systems, there are a number of configurable properties, which deserve additional explanation:
Note: The "Manage" links option is not recommended, because file content retrieval is not currently supported.
Use the following steps to configure files to track on Windows computers.
Limitations
The Change Tracking solution does not currently support the following operations on the Windows platform:
Note: Because these operations are unsupported, some of the aforementioned settings will have no effect in the current version.
The following limitations should be considered for both the Windows and Linux platforms:
OMS Log Analytics performs Windows registry monitoring and tracking with the Change Tracking solution. The purpose of monitoring changes to registry keys is to pinpoint extensibility points where third-party code and malware can activate. There are more than 15 keys tracked by default. A complete list of default registry keys is available at https://docs.microsoft.com/en-us/azure/log-analytics/log-analytics-change-tracking in the "Registry key change tracking" section.
Note: Presently, only the HKLM hive is supported by this feature.
You can also configure OMS to track Windows registry keys of your own choosing. Use the following steps to configure registry keys to track on Windows computers.
FIGURE 2. ADD WINDOWS REGISTRY KEY INTERFACE
IMPORTANT: At the time this was written, the 'Recursive' option has been disabled when adding your own Windows registry keys. This is expected to become available again in a future update.
Tracking installation or removal of software (Windows) or packages (Linux) requires no configuration beyond adding the Change Tracking solution to your workspace and deploying the agent.
Like software and packages, tracking changes to services and daemons does not require additional configuration.
The following table shows the data collection frequency for each type of change, which gives you a better idea of the intended use cases for this solution.
Change type | Frequency | Does agent send differences when found? |
Windows registry | 50 minutes | No. |
Windows file | 30 minutes | Yes. If there is no change in 24 hours, a snapshot is sent. |
Linux file | 15 minutes | Yes. If there is no change in 24 hours, a snapshot is sent. |
Windows services | 30 minutes | Yes, every 30 minutes when changes are found. Every 24 hours a snapshot is sent, regardless of change. So, the snapshot is sent even where there are no changes. |
Linux daemons | 5 minutes | Yes. If there is no change in 24 hours, a snapshot is sent. |
Windows software | 30 minutes | Yes, every 30 minutes when changes are found. Every 24 hours a snapshot is sent, regardless of change. So, the snapshot is sent even where there are no changes. |
Linux software | 5 minutes | Yes. If there is no change in 24 hours, a snapshot is sent. |
TABLE 3. FREQUENCY OF DIFFERENT CHANGE TRACKING OPERATIONS
Based on these times, you can see Change Tracking is designed more for retrospective troubleshooting than for real time monitoring. However, Microsoft is looking into improving the collection frequency for some elements, such as Windows Services. In the meantime, there is a workaround – event logs.
Event logs reach OMS within 5-10 minutes of being created on the source computers. For the computers being monitored by OMS, in the OMS portal in the Settings > Data > Windows Event Logs section, you can enable the transfer of System logs to OMS. In OMS log search, you can query against these logs to find all stopped services (event IDs 7024 and 7034 for unexpected stoppage, 7036 for clean stoppage), and then you can take action or set up an alert to take action, like restarting the service using an automation runbook.
Custom fields, which will be useful in extracting event details of service stoppage events, are covered in "Chapter 6: Extending OMS Using Log Search". Alerting is covered in "Chapter 7: Alert Management".
After the solution is installed, you can view the summary of changes for your monitored servers by using the Change Tracking tile on the Overview page in the OMS portal.
You can view changes to your infrastructure and then drill into details for the following categories:
Changes of any monitored type can be viewed in a number of different ways:
On the Overview page, click the Change Tracking tile, as shown in Figure 3.
FIGURE 3. CHANGE TRACKING TILE
On the Change Tracking dashboard, shown in Figure 4, you can review the summary information in one of the change type blades (the 'Changes over time by type' view). When you click on any of the areas reflecting recent changes, you are taken to an appropriate view of detailed information in the log search page.
FIGURE 4. CHANGE TRACKING DASHBOARD (PARTIAL)
On any of the log search pages, you can view results by time, detailed results, and your log search history. You can also filter by facets to narrow the results.
How changes to files are tracked depends on the type of change you are talking about. For example, changes in file content are identified by a change in the MD5 hash of the file. The solution tracks not only changes in file content, last modified date, and size but also changes in the access control list (ACL), which defines file permissions. When a file is deleted, the Change Category column shows as 'Removed', as shown in Figure 5.
Note: While changes in file content will be surfaced due to change in the file's MD5 hash, the details of the content itself are not captured by this feature.
FIGURE 5. FILE MONITORING IN THE CHANGE TRACKING SOLUTION
Note: In the authors use of this feature in test and production scenarios, when a directory is specified for file monitoring, new files were not reported with any form of 'new file' event. Only changes and deletes were reported.
Changes in service or daemon state or configuration are also surfaced in the Change Tracking dashboard. If you click on the Services Change Tracking tile, you will see details of service changes, including status (starts and stops) or configuration, such as the service startup type, as shown in Figure 6.
FIGURE 6. SERVICE CHANGES SHOWN IN THE CHANGE TRACKING SOLUTION
Note: In the authors testing, new services do not appear as new services. When new services appear (as with those shown in Figure 6 above), they are generally shown as services with changes.
IMPORTANT: While you will notice the SvcState property present in the list of service properties, the Change Tracking solution is not the best way to drive timely notifications of service stoppage. Instead, use service status change events captured in Log Analytics, as detailed in "Change tracking frequency" earlier in this chapter.
The Change Tracking solution also monitors for changes in installed software by taking snapshots of the list in Program and Features in the Windows Control Panel, or list of installed packages (Linux). Applications that have been added or removed since the previous snapshot will be reflected in the Log Analytics data behind the Change Tracking dashboard.
Note: Applications that are uninstalled then reinstalled between snapshots, or vice versa, will not be reflected as changes in the Change Tracking dashboard.
After a cycle in which changes to the list of install software are detected, you will see an update in 'Changes over time by type' in the dashboard. When you click on a blue box in the Software category, you will see adds and removes, as shown in Figure 7.
FIGURE 7. SOFTWARE CHANGES SURFACED BY THE CHANGE TRACKING SOLUTION
The Update Management solution in OMS allows you to manage operating system security updates for your Windows and Linux computers deployed in Azure, on-premises environments, or other cloud providers. The solution enables administrators to assess the status of available updates on all Windows Server and Linux computers, and then manage the process of installing required updates for servers.
Computers managed by OMS use the following for performing assessment and update deployments:
Figure 8 shows a conceptual view of the solution architecture, rocess, and data flow with how the solution assesses and applies security updates to all connected Windows Server and Linux computers in an OMS workspace.
FIGURE 8. UPDATE MANAGEMENT SOLUTION ARCHITECTURE
The process flow, illustrated in Figure 8, is as follows:
Step 1: Compliance scan. After the computer performs a scan for update compliance, the OMS agent forwards the scan results to OMS. On a Windows computer, the compliance scan is performed every 12 hours by default. With a Linux computer, the compliance scan is performed every 3 hours by default,
Note: On both Windows and Linux systems, an update compliance scan is initiated within 15 minutes if the Microsoft Monitoring Agent (MMA) is restarted.
The compliance information is then processed and summarized in the dashboards included in the solution or searchable using user-defined or pre-defined queries. The solution reports how up-to-date the computer is based on what source you are configured to synchronize with. If the Windows computer is configured to report to WSUS, depending on when WSUS last synchronized with Microsoft Update, the results may differ from what Microsoft Updates shows. The same for Linux computers that are configured to report to a local repository versus a public repository.
Note: For Windows systems, the Update Management solution will only report on the updates approved in WSUS.
Step 2: Update selection and targeting. You can deploy and install software updates on computers that require the updates by creating a scheduled deployment. Updates classified as optional are not included in the deployment scope for Windows computers, only required updates. The scheduled deployment defines what target computers will receive the applicable updates, either by explicitly specifying computers or selecting a computer group that is based on log searches of a particular set of computers. You also specify a schedule to approve and designate a maintenance window during which updates are allowed to be installed. The solution allows maintenance windows from 30 minutes up to 6 hours.
Step 3: Update deployment initiated in Azure Automation. Updates are installed by runbooks in Azure Automation. You cannot view these runbooks, and they do not require any configuration. When an update deployment is created, it creates a schedule that starts a master update runbook at the specified time for the targeted computers. This master runbook starts a child runbook on each agent that performs the installation of required updates.
Step 4: Deployment on targeted computers. At the date and time specified in the update deployment, the target computers execute the deployment in parallel. The agent performs a scan first to verify the updates are still required and installs them.
Note: On Windows computers that are also WSUS clients, if the updates are not approved in WSUS, the update deployment will fail.
Step 5: Results processing and reporting. The results of the applied updates are forwarded by the OMS agents to OMS, where they are processed and summarized in the dashboards or by searching the events.
The prerequisites for the solution are fundamentally the same for Windows and Linux computers:
A supported OS
WARNING: Pay close attention to the lack of support for Server Core. This is a commonly chosen option for Windows Server 2016 deployments.
Note: An OMS Agent for Linux configured to report to multiple OMS workspaces is not supported with this solution.
Update source
Note: The Update Management solution does not integrate with the update management feature in System Center Configuration Manager.
You can deploy the solution in either the Azure Portal or the OMS portal or an Azure Resource Manager template. Refer to the "Adding solutions to your OMS workspace" section in "Chapter 1: Introduction and Onboarding" if you are unfamiliar with how to add a solution to OMS.
After you enable the Update Management solution, any Windows computer directly connected to your OMS workspace is automatically configured as a Hybrid Runbook Worker to support the runbooks included in this solution. For each Windows computer managed by the solution, it will be listed under the Hybrid Runbook Worker Groups blade of the Automation account following the naming convention Hostname FQDN_GUID.
Note: These groups are only intended to support the management solution. If you target these groups with runbooks, the runbook job will fail.
However, you can add the Windows computers to a Hybrid Runbook Worker group in your Automation account to support Automation runbooks, as long as you are using the same account for both the solution and Hybrid Runbook Worker group membership. This functionality has been added to version 7.2.12024.0 of the Hybrid Runbook Worker.
When you add the Update Management solution to your OMS workspace, the Update Management tile (shown in Figure 9), will be added to your OMS dashboard. This tile displays a count and graphical representation of the number of computers in your environment and their current update compliance.
FIGURE 9. UPDATE MANAGEMENT TILE
Click on the Update Management tile to open the Update Management dashboard shown in Figure 10, which presents additional detail on computer update compliance, a link to the deployment configuration interface, as well as a list of linked sampled queries for use in Log Analytics.
FIGURE 10. THE UPDATE MANAGEMENT DASHBOARD
You can use this dashboard to get a detailed breakdown of update status categorized by type of operating system and update classification - critical, security, or other (such as a definition update). The Manage Update Deployments tile redirects you to the update deployments page where you can view schedules, deployments currently running, completed deployments, or schedule a new deployment.
You can run a log search that returns all records by clicking on the specific tile, or to run a query of a category and pre-defined criteria, select one from the list available under the Common Update Queries column.
Once updates have been assessed for the Linux and Windows computers in your workspace, you can install required updates by creating an update deployment. An update deployment is a scheduled installation of required updates for one or more computers. You specify the date and time for the deployment in addition to a computer or group of computers that should be included in the scope of a deployment.
IMPORTANT: When you include computer groups in your update deployment, group membership is evaluated only once at the time of schedule creation. Subsequent changes to a group are not reflected. To work around this, delete the scheduled update deployment and recreate it.
Notes on default update management behavior:
To create a new update deployment, complete the following steps:
FIGURE 11. UPDATE DEPLOYMENT SCHEDULE & STATUS
FIGURE 12. THE "NEW UPDATE DEPLOYMENT" SCREEN
Time Range
By default, the scope of the data analyzed in the Update Management solution is from all connected management groups and direct agents generated within the last 1 day.
To change the time range of the data, select Data based on at the top of the dashboard. You can select records created or updated within the last 7 days, 1 day, or 6 hours. Or you can select Custom and specify a custom date range.
Click the Update Deployment tile (shown in Figure 13) to view the list of existing update deployments. They are grouped by status – Scheduled, Running, and Completed.
FIGURE 13. EXISTING UPDATE DEPLOYMENTS
On the Scheduled tab, you will see the following properties for each update deployment you configure: Name, Schedule, Start Time, and Duration. On the Completed tab, you will see the details of completed update deployments. These columns will not be populated if the update deployment has not yet started.
The Update Management solution creates two types of records in the OMS repository:
Sample search queries: You can find sample OMS log analytics searches for the Update Management solution in the "Sample log searches" section at https://docs.microsoft.com/en-us/azure/operations-management-suite/oms-solution-update-management
You can troubleshoot the Update Management solution using the Patch-MicrosoftOMSComputer runbook, which targets a specific managed computer, and produces a verbose data stream with detailed information for that deployment. The output will display which required updates are applicable, download status, installation status, and additional details.
Rolling back updates:
Currently, the Update Management solution does not include a rollback option to undo problematic updates. Deploy updates to a test group before deploying to production.
Recommendation: If deploying to production without testing, allow some time for other organizations to test updates and report issues through TechNet forums and other public support channels.
At the start of this chapter, we introduced you to the OMS Change Tracking solution and its features for tracking configuration of Windows and Linux systems, along with the configurable options for tracking Windows file and registry changes. Then, we moved into the details of the Update Management solution for delivering required updates to Windows Server operating systems, as well as patches for supported Linux distributions.
While there is certainly work that remains to be done on these solutions to bring them to the level of comprehensive enterprise solutions, they have certainly shown considerable improvement the last few months.