Monday, November 19, 2012

Base OS MP Version 6.0.6989.0 Imported? And Now Some Reports Are Partially Empty?

Issue
I have seen this issue on multiple locations: The customer has imported the latest version of the Windows Server OS MP (aka Base OS MP) version 6.0.6989.0 and after some time these Reports (found under Reports > Windows Server Operating System Reports) Performance By System and Performance By Utilization fail to show the data related to the Processor performance, like this for instance:
image

Cause
First of all the good news. When your Reports did show data related to the Processor before you updated the Base OS MP to version 6.0.6989.0 changes are your SCOM R2/OM12 environment isn’t having issues at all. So before the Base OS MP was updated the Performance By System Report looked like this (based on the Base OS MP version 6.0.6958.0):
image

With the latest version of the Base OS MP, Microsoft has changed some Rules for Performance Collection, among them the Rule which collects the Total Percentage Processor Time. Kevin Holman already blogged about it: ‘…Several monitoring workflows were change from Processor, to “Processor Information” perf object.  This change was made because a new perf counter/object (Processor Information) was added to the OS to support more than 64 logical processors.  The old perf counter object (Processor) was limited to 64 CPU’s.  As physical hardware is starting to ship 6+ core systems, with HT, and multiple sockets, this was a problem for measuring utilization for VERY large boxes…’

And he continues to explain it further: ‘…NOTE:  This might BREAK your existing reports and dashboard views that are expecting “Processor” object, as we no longer collect that.…. so be prepared to make some changes there…’

And guess what? The two mentioned Reports aren’t adjusted to reflect this change. So the Reports want to show performance data which isn’t longer collected.

Old situation (Base OS MP before version 6.0.6989.0)
Let’s check the performance collection Rule Processor % Processor Time 2008:
image

As you can see, the Object is Processor and the Counter is % Processor Time, which matches 1:1 with the related Reports, for instance the Performance By System Report:
image

New situation (Base OS MP version 6.0.6989.0)
Let’s check the performance collection Rule Processor % Processor Time 2008:
image

As you can see, the Object is changed to Processor Information with the same counter (% Processor Time). On the other hand, the Report itself is still using the ‘old’ Performance Collection Rule which isn’t used anymore, thus resulting in partially empty Reports Bedroefde emoticon.

Solutions
There are two solutions: either changing the underlying code of the Reports involved and upload the modified Reports – under a different name – to the SSRS instance being used by SCOM R2/OM12.

The other solution is faster and requires less ‘magic’: rebuilding the old Perfomance Collection Rule and disable the new one. After a couple of days the earlier mentioned Reports will start showing data again. In this posting I’ll give you a quick explanation how to do this.

Creating a new Performance Collection Rule & Disabling the old one
When you have ‘beasts’ of Servers in place where many CPUs are present, it’s better to disable the Performance Collection Rule Processor % Processor Time 2008 using a Group which is dynamically populated with all Windows Servers which has those ‘beasts’ as excluded members. This way this Performance Collection Rule will run against those Windows Servers and will be disabled against all other Windows Servers.

Also it’s a good idea to put these modifications in a dedicated MP. Simply because I expect Microsoft to repair this glitch in a next version of the Base OS MP. When you put all these modifications in a special dedicated MP you only have to delete it and be done with it. Otherwise you have to go through it step by step which is a more time consuming process…

  1. Open the SCOMR2/OM12 Console with an account which has sufficient permissions to create/modify Rules;
  2. Go to Authoring > Authoring > Management Pack Objects;
  3. Hit ‘Change Scope’ and type Windows Server 2008 Processor;
  4. Select the Rule Processor % Processor Time 2008 and disable it through an override (put it in a dedicated MP like Temporary MP for Processor Performance);
  5. Right click Type: Windows Server 2008 Processor (3) and select Create a new Rule;
  6. As stated before put this Rule in a dedicated MP (example: Temporary MP for Processor Performance);
  7. Select the correct type of Rule to create (Collection Rules > Performance Based > Windows Performance);
    image
    > Next
  8. Give it a proper name, like Processor % Processor Time 2008 – TEMPORARY and a proper description:
    image
    Because of Step 4 and 5 the Rule Category and Rule Target are set correctly. Double check them though. > Next;
  9. In this screen, hit the Select button. Select a server which is running Windows Server 2008 R2 and select the correct items according the screen dump. Hit the OK button and the fields Object (Processor), Counter (% Processor Time) and Instance (_Total) will be filled with the correct information.
    image
    Set the interval to 5 minutes > Next
  10. Place a checkmark for Use Optimization, select the option Absolute number and set it to 5;
    image
    > Create. The temporary new Performance Collection Rule will be created now.

Within a few days the earlier mentioned Reports will show data again for the processor object Glimlach.

1 comment:

Stanislav Zhelyazkov said...

This is not the first time that counters of this MP were changed and some reports didn't show data. If I remember right there was similar situation when current queue length counter was disabled and average queue length counter was enabled. This caused one report to not show data but I cannot remember the name of the report.