Improper default configuration of the Message Queuing Management Pack for Operations Manager

During recent work of fine-tuning Operations Manager infrastructure for the customer, high number of collected events in the SCOM database has come to my attention. So, I spent some time in analysing where did all collected events come from in order to better understand the root cause.

First thing that caught my eye was the fact that majority of collected events are the ones which have “Health Service Script” as Event Source property.

Next step was trying to understand which rule collects “Health Service Script” events. It turned out that 92% of all collected events were collected by the same rule, which was “Collect MSMQ Log Detail Script Events”.

Structure of collected events (per rule)

This rule is configured with an expression which collects all events from the Operations Manager event log where Event Source equals Health Service Script.

Collect MSMQ Log Detail Script Events rule configuration
Collect MSMQ Log Detail Script Events rule configuration

Needless to say that most of the “Health Service Script” events are not related to the MSMQ management pack, nor relevant for understanding the health state of managed devices from the MSMQ perspective. Collection of all “Health Service Script” events might cause event flooding on the Operations Manager database side.

“Collect MSMQ Log Detail Script Events” rule is included in the Message Queuing Management Pack for Operations Manager, enabled by default. To my knowledge, it is included in the following management packs:

  • Microsoft MSMQ 2003 management pack,
  • Microsoft MSMQ 2008 management pack,
  • Microsoft MSMQ 2008 R2 management pack and
  • Message Queuing 6.0 Management Pack.

In order to prevent event flooding, you should create an override and disable the “Collect MSMQ Log Detail Script Events” rule for all applicable target classes. In order to do that, change the value of the Enabled parameter from True (default value) to False (override value).

Override configuration
Override configuration
Improper default configuration of the Message Queuing Management Pack for Operations Manager

List of events included in Audit Collection Services reports

The following table contains list of events that are included in Audit Collection Services reports that are shipped with System Center Operations Manager 2007 R2:

Access Violation – Account Locked report: 539, 534, 4740, 6279
Access Violation – Unsuccessful Logon Attempts report: 529, 530, 531, 532, 533, 534, 535, 536, 537, 539, 4625
Account Management – Domain and Built-in Administrators Changes report: 632, 633, 636, 637
Account Management – Passwords Change Attempts by Non-owner report: 627, 628, 4723, 4724
Account Management – User Accounts Created report: 624, 4720
Account Management – User Accounts Deleted report: 630, 4726
Usage – Object Access report: 560, 567, 4656, 4663
Usage – Privileged logon report: 575, 4672
Usage – Sensitive Security Groups Changes report: 631, 632, 633, 634, 635, 636, 637, 638, 639, 641, 658, 659, 660, 661, 662, 4727, 4728, 4729, 4730, 4731, 4732, 4733, 4734, 4735, 4737, 4754, 4755, 4756, 4757, 4758
Usage – User Logon report: 528, 540, 4624
System Integrity – Audit Log Cleared report: 517, 1102
System Integrity – Audit Failure report: 516, 4612
Policy – Privilege Added Or Removed report: 609, 622, 4704, 4705
Policy – Object Permissions Changed report: 4670
Policy – Audit Policy Changed report: 612, 4719
Policy – Account Policy Changed report: 643, 4739
Planning – Logon Counts of Privileged Users report: 576, 4672

This information might be helpful to anyone interested in setting up filter on the ACS Collector side in order to limit the list of events that are stored in the ACS database. By default, all security events from ACS forwarders are stored in the ACS database. In my test environment, approximately 57% events stored in the ACS database are not included in any of above-mentioned ACS reports. This means that a lot of data in the ACS database is irrelevant for reporting purposes and that ACS database size can be reduced significantly by setting up ACS Collector filter.

To set up ACS Collector filter and to limit the list of events that are stored in the ACS database, AdtAdmin.exe /SetQuery command line utility must be used. It uses Windows Management Instrumentation (WMI) Query Language (WQL) to filter events before they hit the ACS database.

For more information on how to use AdtAdmin.exe command line utility, you may refer to the following Microsoft TechNet article:
AdtAdmin.exe /SetQueryhttp://technet.microsoft.com/en-us/library/bb381343.aspx

List of events included in Audit Collection Services reports

How to configure security events collection by using Audit Collection Services from computers in untrusted environment?

As you may know, in order to enable SCOM Agent-based monitoring of devices in untrusted environment (such as workgroup computers), SCOM Agent must be configured for certificate-based authentication.

If you want to go further and decide to collect security events from such computers by leveraging Audit Collection Services feature (assuming that the appropriate infrastructure for ACS is in place), you will soon notice that it does not work without additional configuration of ACS Collector, as well as ACS Forwarder components.

The following event in Operations Manager event log indicates that AdtAgent service running on ACS Forwarder is unable to authenticate against the ACS Collector:

Event Type:       Warning
Event Source:   AdtAgent
Event Category:               None
Event ID:             4369
User:                    NT AUTHORITY\NETWORK SERVICE
Description:      Forwarder unsuccessfully tried to connect to the following collector(s):
<<ACS Collector computer name>>:51909, status: 0x40 (TCP connect), source: registry addresses tried: <<ACS Collector IP address>>:51909

Similar to the System Center Management service (HealthService) which you have probably already configured for the certificate-based authentication by using the MOMCertImport.exe tool, ACS Collector and ACS Forwarder(s) must also be configured for certificate-based authentication.

To configure “Operations Manager Audit Collection Service” (AdtServer) running on the ACS Collector for certificate-based authentication, run AdtServer.exe –c command (by default, it is located in the %SYSTEMROOT%\System32\Security\AdtServer folder.

To configure “Operations Manager Audit Forwarding Service” (AdtAgent) running on the ACS Forwarder for certificate-based authentication, run AdtAgent.exe –c command (by default, it is located in the %SYSTEMROOT%\System32 folder.

Once this is done, AdtAgent will still be unable to authenticate. However, in this case you will see the following error in the Operations Manager event log (note that error number is different).

Event Type:       Warning
Event Source:   AdtAgent
Event Category:               None
Event ID:             4369
User:                    NT AUTHORITY\NETWORK SERVICE
Description:      Forwarder unsuccessfully tried to connect to the following collector(s): <<ACS Collector computer name>>:51909, status: 0x2746 (TCP connect), source: registry addresses tried: <<ACS Collector IP address>>:51909

Last step required to resolve this problem is to create name mapping to the certificate in the Active Directory environment where ACS Collector is located. In order to do that, you must first export computer certificate from the ACS Forwarder into the DER encoded binary X.509 (.CER) file. Then, create an “empty” computer account whose name must match NetBIOS name of ACS Forwarder. Right-click on the newly created computer account and then select Name Mappings option. Under the Security Identity Mapping option, import X.509 certificate. After that, make sure to restart AdtAgent service on the ACS Forwarder. It should now successfully authenticate against the ACS Forwarder, followed by this event in the Operations Manager event log:

Event Type:       Information
Event Source:   AdtAgent
Event Category:               None
Event ID:             4368
User:                    NT AUTHORITY\NETWORK SERVICE
Description:      Forwarder successfully connected to the following collector: <<ACS Collector computer name>>:51909, status: 0x0 (success), source: registry addresses tried: <<ACS Collector IP address>>:51909

For more information, refer to the following Microsoft TechNet articles:

This whole process is also explained in the Operations Manager 2007 Security Guide which is available as free download from Microsoft, as well as in the System Center Operations Manager 2007 Unleashed book.

How to configure security events collection by using Audit Collection Services from computers in untrusted environment?

Workflow Runtime: Failed to run a WMI query alert description

Today I have received the following alert from SCOM:

Alert: Workflow Runtime: Failed to run a WMI query
Alert description: Object enumeration failed
Query: ‘SELECT StartMode, State FROM Win32_Service WHERE Name = ‘MSSQLSERVER”
HRESULT: 0x800705af
Details: The paging file is too small for this operation to complete.
One or more workflows were affected by this.

This alert was raised by Windows Server 2008 R2-based computer and I believe that it is caused by an error explained in KB981314: The “Win32_Service” WMI class leaks memory in Windows Server 2008 R2 and in Windows 7http://support.microsoft.com/kb/981314.

This hotfix contains updated Cimwin32.dll which is located in %SYSTEMROOT%\System32\wbem folder. Updated file version is 6.1.7600.20683.

NOTE: Installation of KB981314 requires computer restart.

Workflow Runtime: Failed to run a WMI query alert description

How to export list of Monitors and Rules included in the Management Pack?

In many cases it is useful to have exported list of all monitors and rules included in specific SCOM management pack.

Two cmdlets are available in the Operations Manager Shell for that purpose. They are: Get-Monitor and Get-Rule.

In order to export list of all monitors from specific management pack into CSV file, run the following command in Operations Manager Shell:

Get-Monitor -ManagementPack ManagementPackFileName.mp | Select DisplayName, Description, Name, Enabled, Target | Export-Csv

In order to export list of all rules from specific management pack into CSV file, run the following command in Operations Manager Shell:

Get-Rule -ManagementPack ManagementPackFileName.mp | Select DisplayName, Description, Name, Enabled, Target | Export-Csv

 In both cases replace the value of the ManagementPack parameter accordingly.

How to export list of Monitors and Rules included in the Management Pack?

No graphs (charts) are included in scheduled SCOM reports

Today I have noticed that SCOM reports that are scheduled to be delivered as e-mail attachments do not contain chart data.

Scheduled SCOM report - chart missing
Scheduled SCOM report - chart missing

At the time when affected scheduled report is executed, the following event is logged in Application log of the server running SQL Reporting Services:
Log Name: Application
Source: Report Server Windows Service (MSSQLSERVER)
Event ID: 108
Level: Error
Description:
Report Server Windows Service (MSSQLSERVER) cannot load the EnterpriseManagementChartControl extension.

After some investigation, it turned out that this is a know issue which is documented in the following knowledge base article:

Scheduled reports that you create by using SQL Server 2008 Reporting Services (SSRS) do not display chart data in System Center Operations Manager 2007 R2http://support.microsoft.com/kb/972821



After editing ReportingServicesService.exe.config file, you should be able to see chart inlcuded in SCOM report.

Scheduled SCOM report - chart included
Scheduled SCOM report - chart included
No graphs (charts) are included in scheduled SCOM reports