The newest Core MP for SCOM R2: The New Road Ahead. Part II: Agent Management

--------------------------------------------------------------------------------- 
Postings in the same series:
Part I: The System Center Core Monitoring Reports 
---------------------------------------------------------------------------------

As promised in the previous posting of this series, this posting will be all about Agent Management. Besides the newly added worth while Reports, the management of the SCOM R2 Agents has been hugely improved and simplified as well.

Some examples:

01 – Agent Version and Patch Level
Until now you had to create such a View yourself. Now this View is present at default so with a few mouse clicks and a single glance one can see at what Version the SCOM Agents are and whether or not a certain hotfix/CU has been applied.

Go to Monitoring > Operations Manager > Agent > Agents By Version:

02 – Agent Performance
Also the performance of the SCOM R2 Agents can be easily tracked. So whenever an Agent is flooding the SCOM R2 Environment it is easily to be detected. Go to Monitoring > Operations Manager > Agent Peformance > Agent Performance:

Select one of the performance views and then an Agent in the underneath pane. Soon a graph will be displayed showing how that Agent performs.

03 – Script issues
As we all know, a script triggered from a MP on a certain Agent can go wrong from time to time. Now a new  View shows this in detail. The list within that View is a bit long but it helps to pinpoint exactly what goes wrong. With this a deeper insight into the engine room of the Agent is available which helps to find the exact cause.

Go to Monitoring > Operations Manager > Health Service Module Events:

04 - Agent Version and Architecture Check
As we all know, a 32-bit SCOM Agent installed on a 64-bit Operating System is not a happy marriage. It will produce unreliable results and many performance counters (x64 based) cannot be accessed by a 32-bit SCOM Agent. On top of that, when running an ‘old’ SCOM Agent can also be problematic.

So the new Core MP for SCOM R2 has introduced two new monitors: the ‘Agent and OS architecture are the same’ monitor and the ‘Agent Version Monitor’. Both are enabled by default and trigger a Warning with Priority level Medium. Which is good since it hasn’t to be a Critical Alert nor an Informational since it is a bit more serious than just that.

Both Monitors can be changed:

The ‘Agent and OS architecture are the same’ Monitor:
 

The ‘Agent Version Monitor’:

Nice to know is that also the Version upon which the Monitor checks can be changed as well.

And both Monitors have the Auto Resolve ability standard enabled. Which is nice as well.

So as you can see, Agent Management has become way much more easy and flexible with the new Core MP for SCOM R2. A job well done by Microsoft.

The newest Core MP for SCOM R2: The New Road Ahead. Part I: The System Center Core Monitoring Reports

--------------------------------------------------------------------------------- 
Postings in the same series:
Part II: Agent Management
---------------------------------------------------------------------------------

The newest core MP for SCOM R2 is not just an update. It is much more. It is the road Microsoft envisions for the SCOM (R2) MPs and the generations to come.

Wow! That is a lot to say. But wait and read on since I will explain why I say this so strongly. First of all, the MP has really grown up. The MP monitors SCOM R2 itself but does a way much better job than ever before. Simply because it does not only Alert better when things tend to go wrong but also since new Reports have been added which are a great help in order to start targeted trouble shooting.

In a small series of blog postings I will high light some of the new features of this Core MP for SCOM R2. The first posting will be about the new Reports found in this MP, the System Center Core Monitoring Reports since they are a special breed on their own.

Back in the days of SCOM RTM I had an issue going on at a customers site. The RMS stopped almost on a daily basis. With the help from PSS the issue was found. They sent me some queries to run against the SCOM Database and soon the culprit was found. But it took some time, mostly caused by the time difference between the US and Europe where I reside. When I had these Reports which are to be found in the new Core MP the cause had been found way much faster and the resolution as well.

These are the Reports I am talking about: System Center Core Monitoring Reports.

These Reports provide a deeper insight in the inner workings of SCOM R2. And not just that. We all know that getting a filled and understandable Report can be a challenge. This has been heard by Microsoft and look what they have done:

Finally! Some good and thorough explanation about what the Report does, how it works and how to configure it. Great!

Remember the blog posting of Kevin Holman about Config Churn? It is still a good posting but now much of it can be done by these Reports in order to see whether it is really happening. (The examples below are taken from one of mine test environments I carry with me on my laptop. So these Reports are a bit empty, but still good enough to show what I mean.)
 

and:

So with a few mouse clicks one can see really fast what is happening in the engine room of SCOM R2 and whether there is a ‘All hands on deck’ situation happening or not. These reports also have drill down functionality, so much information is to be found there as well. One can take a look on a per Management Pack basis or at Class level.

Try it yourself and be impressed like I am. The reports work fast, are easily used and show really good information without the need of being a rocket scientist in order to understand it.  So whenever SCOM R2 behaves different than expected, run these Reports in order to gain a deeper insight.

The next posting will be about Agent Management with the newest Core MP for SCOM R2.

SCOM R2 Console takes minutes to open

Got this from the Microsoft TechNet OpsMgr Forums. Even though none of my customers has experienced this issue, it is still good to know what to do when this happens.

Sometimes it can take many minutes for the SCOM R2 Console to open. When that happens a restart of the SDK Service on the RMS will do the trick. Microsoft knows about this issue and is making a hotfix for it which will placed into the next CU (Cumulative Update) for SCOM R2.

Error 25151. Configuring MOM database failed. Error Code: –2147023564 when installing SCOM R2 Reporting component

Got this error message today at a customers site where a total reinstall of the SCOM R2 Reporting component was at order. The wizard completed almost successfully but then this message popped up:

Hmm. Time for some investigation. This was shown in the log while searching for value 3:
MSI (s) (B4!0C) [13:33:18:838]: Product: System Center Operations Manager 2007 R2 Reporting Server -- Error 25151.Configuring MOM database failed. Error Code: -2147023564 (No mapping between account names and security IDs was done.).

The related accounts were OK, had enough permissions: within SCOM, SQL and on the related servers. To make it more interesting, it is a split installation: the Data Warehouse database resides on another server than the SQL Reporting Server Services (SSRS) instance. Which can be a challenge.

The AD servers were running smoothly as well, no issues there. The accounts were OK, and not locked. So what else could be the matter here?

Time to go through the installation log again. And now more detailed.

Before I could run the reinstall of the Reporting Component I had to revive SSRS first which took me some time. Also I needed to change some credentials for which I used the format <domain name>\<account>. In SSRS this is a valid format. But NOT in the wizard for the installation for the SCOM Reporting component.

And that is what I did.

I entered for the account the credentials in the format <domain name>\<account> instead of <account>. Since there is already a field in place displaying the domain where the account resides the format <domain name>\<account> isn’t needed . Where in other applications that field gets grayed out when the <domain name>\<account> format is being used, but not here in this wizard.

So the installation log told me that it was unable to validate the accounts with the format <domain name>\<domain name>\<account>. Which is normal! Look here:
AddPrivilegeToAccount: Giving privilege to account. DOMAIN\domain\account
AddPrivilegeToAccount: LookupAccountName failed. Error Code: 0x80070534.

So I ran the installation again and now I provided the account name without the domain name as prefix. And all ran just fine…

CEIP, ODR and the lot. What are they and why should I use them? Part I: ODR explained

--------------------------------------------------------------------------------- 
Postings in the same series:
Part  II: CEIP explained 
Part III: AEM explained, its origin
Part IV: AEM explained, how to configure it
---------------------------------------------------------------------------------

Many companies do have questions about CEIP (Customer Experience Improvement Program) and ODR (Operational Data Reports). With the most recent update of the core MPs for SCOM R2 the ODR reports have been extended as well. So it is a good time to write a couple of blog postings on these topics.

The first posting will be about ODR, what it is, how it operates and why to use it. Also there is one small caveat (a bit far fetched one as well) to reckon with. When that is properly taken care of, there is not a good reason why NOT to use ODR and helping Microsoft to make SCOM (R2) a better product.

First of all, ODR is an abbreviation of Operational Data Reports. These reports gather a summary about  the SCOM (R2) usage related to the Management Group. These reports are used by Microsoft to improve the quality of the Management Packs and SCOM in general. In order for this to work, Reporting must be installed and running. These reports are sent out in xml-format to Microsoft on a weekly basis when ODR is enabled.

Very important to know as well, the data is anonymous.

So no usernames are sent out nor specific information about your company. But wait. Here is the one and only caveat. Suppose your company is preparing a mercer with another company. And you, being the SCOM Admin and Guru are instructed to monitor some applications/services of the other company. So you install a SCOM Gateway Server and start monitoring as instructed. Now some additional hand made MPs are needed. When you name these MPs like ‘MP to monitor Company B with whom we are going to merge’, the name of this MP will appear in the ODR report.

I know, it is a long shot, but it could happen. So keep the names of the MPs you have made yourself in a tight naming format without exposing sensitive information. This is not so difficult at all.

OK, lets take a look at the ODR Reports themselves. Run them one by one in order to see what goes out. These reports are based on SCOM R2 and the latest SCOM R2 Core MPs. The names of the reports themselves are self explanatory:
 
The Report Alerts Per Day is new here. The name tells it all.

Also the Report Details Pane tells you more what a Report shows:

Actually, these Reports are very easy to run since all needed parameters are pre-defined. Even the start date (default is today – 7 days) and end date (default the day on which the report is run) are already set. So just double clicking a Report is enough to get it filled.

Just run the Reports your self in order to get a better insight and feeling about what is being sent out to Microsoft. By default this happens on a Saturday at 10:00 PM. For most companies this is a timeframe where not many people nor processes are being disturbed in any kind of way.

Still in doubt? Then read the Privacy Statement by Microsoft about these ODR Reports.

But how to enable these ODR Reports? Normally during the installation of SCOM (R2) Reporting this is already asked for. Many times I see customers not enabling it:

So how to enable it afterwards? Easily done. Start the SCOM R2 Console with sufficient permissions (on W2K08 servers it is better to start the SCOM R2 Console with elevated permissions since changing the setting of the ODR Reports involves a registry change).

Go to Administration > Settings > Privacy. Double click on it and go to the second tab, Operational Data Reports:
 

Select the option Yes, send operation data reports to Microsoft (recommended) and click Apply.

On Windows 2008 (R2) servers this error might pop up:

UAC is at play here. So close the SCOM R2 Console and run it with Administrator privileges

(of course, the admin account needs Admin access within SCOM as well…), and now all goes well:

Click OK. Now all ODR Reports will run once a week and will be sent out to Microsoft. Thank you so much for making SCOM (R2) a better product. :)

Newest Core MP for SCOM R2 has been released: version 6.1.7599.0

Just released: the newest core MP for SCOM R2, version 6.1.7599.0.

This MP takes a deep dive into the very core of SCOM R2 itself and keeps a watchful eye on it. New reports are added as well which can be very handy when troubleshooting of SCOM R2 is needed.

MP to be downloaded from here. Cory Delamarter from Microsoft has written a good posting about this MP and its details, found here.

SCOM R2 Technical Documentation

As we all know their are many good resources about the technical aspects of SCOM R2, written by Microsoft. Much of this can be viewed online or downloaded as a Word document. For some days now Microsoft has put all the offline content in one place so it is easily downloaded.

On top of it all Stefan Stranger has written a PS script (PS 2.0 is needed) which downloads all the separate files in one run. That’s nice! Thanks Stefan.

Resources:

1. SCOM R2 Product Documentation on- and offline:
   http://technet.microsoft.com/en-us/opsmgr/bb498235.aspx 
   
   
2. SCOM R2 Product Documentation, offline and per topic downloadable:
   http://www.microsoft.com/downloads/details.aspx?displaylang=en&FamilyID=19bd0eb5-7ca0-41be-8c0f-2d95fe7ec636
   
   
3. The PS 2.0 script written by Stefan Stranger:
   http://blogs.technet.com/stefan_stranger/archive/2010/02/19/download-opsmgr-guides-in-one-click.aspx

OpsMgr R2 Cross Platform Cumulative Update 2 has been released

For a few days now Microsoft has released CU #2 for the Xplat (Cross Platform) functionality of OpsMgr R2.

Locations of CU #2 for Xplat are:

1. KB article describing CU #2:
   KB979490
   
   
2. Download location of update file containing CU #2:
   To be found here.
   
   
3. Download location of updated Xplat MPs:
   To be found here.

Advice:
As always RTFM. The related KB article mentioned in #1 gives good information to run a successful upgrade. So use that information to your own advantage.

Bringing monitors back to a healthy state: the difference between ‘Reset Health’ and ‘Recalculate Health’.

As stated in some postings of mine, when an Alert is generated by a Monitor it is NOT sufficient to close that particular Alert and be done with it. After all, a Monitor is a ‘State Machine’ which can have three different states (Healthy, Warning or Critical).

When a Monitor changes State (it goes from Healthy to Critical for instance) it will raise an Alert. But closing that same Alert will not set the Monitor back to its Healthy State. So when a condition occurs which is Critical for that same Monitor, an Alert will NOT be raised since that same Monitor is already in a Critical State which raised the old (and already closed) Alert.

So the Health Explorer is needed in order to set the related Monitor back to its Healthy State in order to get a new Alert when a Critical or Warning situation (State Change) occurs. And now one finds two options to be available: Reset Health and Recalculate Health.

Even though both options can have the same end result (a Monitor which is Healthy again), there are certain things to reckon with:

Reset Health
This option always works. It resets the Health of the Monitor back to Healthy, no matter what the current state is. So even when the Monitor is already in a Healthy State, it will be forced into a reset thus a Healthy State.

Recalculate Health
This option will not always work. Depends on the Monitor involved. When the Monitor has on-demand detection, the option to recalculate its Health will work.

Why?
Good question. Glad you asked. The ‘on-demand-detection’ option means that the monitor is allowed to recalculate its Health at any moment and is NOT dependant of the interval set for these calculations to run.

How to differentiate between such Monitors you ask?
Well, you could take a look into the xml-code involved for that particular Monitor. But that is not very user friendly. Marius Sutara has written a blog posting about a rule of thumb. The video he refers to has been withdrawn unfortunately. Basically what he means is that when checking out the initial State Change of a Monitor in the Health Explorer and goes from Unmonitored to any State but Healthy OR goes from Unmonitored to Healthy AND the initial State Change Context isn’t like ‘The monitor has been initialized for the first time or it has exited maintenance mode’, then the changes are very likely that on-demand detection is present.

OK. Now what?
Well, that depends on you. A Reset will always work. So when an Alert has been raised by a Monitor and the cause has been solved and the Alert has been closed, the related Monitor can be reset. Or – when on-demand-detection – is being used, a Recalculate can be used as well.

Used sources
As stated before, Marius Sutara has blogged about it, to be found here. And Boris Yanushpolsky has written a good posting about on-demand-detection as well, to be found here.

MVP Summit 2010

I am attending the MVP Summit 2010. So this blog will be a bit silent for a week . Of course, much will be told during the MVP Summit. But everything is strictly under NDA (Non-Disclosure Agreement) so I won’t blog about it.

SCOM Gateway Approval Tool stalls

Yesterday at a customers site I bumped into an issue where I had to deploy a SCOM Gateway Management Server. For this some steps have to be taken. But when done properly, nothing is amiss.

However, this time I bumped into an issue with the tool which approves the SCOM Gateway Management Server. Somehow it hung. I used the correct syntax but after hitting <enter> the screen only showed the message ‘Copyright 2009 Microsoft Corporation. All rights reserved’. But the message that the SCOM Gateway Server had been approved successfully never appeared.

It took me some time but after a couple of hours (!) I got it working, thanks to the help of a local administrator.

Two issues seemed to be at play here: first, the file was ‘downloaded’ from the OS perspective. So the file was blocked since it happened on a Windows 2008 Server. This issue was found – and dealt with – within a couple of minutes.

However, the second cause took me much much longer.

The account I used had admin privileges on the SCOM servers and Admin permissions within SCOM as well. Finally, a systems engineer for this company ran the tool for me with the credentials of his admin account. The only difference here was that this account had also admin permissions on that SQL instance where as the account I used has only read access.

And now the tool ran just fine! The SCOM Gateway Server got approved within a matter of seconds!

I checked the documentation about this tool but nothing is to be found. But the deal is:

When running the SCOM Gateway Approval Tool do this with an account which has also admin permissions on the SQL instance hosting the SCOM Database.

A Day Later:
Yesterday I searched the internet to see whether any one else had experienced the same issue as I did. But all to no avail. And now when searching again, the FIRST hit I got is a blog posting of Tim McFadden describing the exact same issue as I experienced. So many credits (if not all) go to him. Apparently I ran a better search query now than yesterday…

New MP released: DFS Namespaces Management Pack

Yesterday Microsoft released a new MP for monitoring the health of the DFS Namespace service on Windows Server 2003 and Windows Server 2008.

Taken directly from the website:

Overview
The DFS Namespaces Management Pack monitors servers running Windows Server® 2008, Windows Server 2003 R2, and Windows Server 2003. It can also be configured to monitor the health of DFS Namespaces from client computers running Windows Vista® or Windows® XP. This management pack monitors events that are recorded in the System event log by DFS Namespaces. It also monitors the overall health of DFS Namespaces and alerts you to critical issues.

Feature Summary
The DFS Namespaces Management Pack is designed to provide valuable monitoring information about the health of DFS Namespaces by monitoring the following:

• Namespace health
• DFS service health
• Namespace server health
• DFS folder health
• Folder target health
• Namespace server Active Directory communication
• Namespace root directory health
• DFS client-based monitoring

Release History

• 2/2/2010 - Original release of the English version, version 6.0.6321.0

MP to be found here.

New KB article: The Windows Operating System column may have a Not Monitored state for a new agent.

Microsoft has released a new KB article which describes this issue: ‘The Windows Operating System column may have a Not Monitored state for a new agent.’

Cause and workaround are described in KB979387.

Windows 2003 Server and SCOM (R2): What hotfixes do I apply?

This posting is provided "AS IS" with no warranties, and confers no rights.

I see this question popping up many times so I have decided to write a blog posting about it.

Question is, whether there are some hotfixes needed/advised when SCOM (R2) is going to monitor a server which is Windows Server 2003 based.

And the answer is YES. These hotfixes are not SCOM based but are targeted at some Windows Server OS components, like WMI and Windows Scripting Host (WSH). These are components the SCOM (R2) Agent uses a lot. So it is better to have these components at a good level.

1. WMI Hotfix
   Solves (among other things): Many WMI errors reported in the SCOM Console.
   Related KB Article: KB933061
   
   
2. WSH version 5.7
   Solves (among other things): High CPU usage.
   Related KB Article: KB955360
   
   
3. SP1 voor MSXML6.0
   Solves (among other things): High CPU usage.
   Related KB Article: KB968967
   Additional Info: This KB can be applied to W2K08 servers as well.

It goes without saying, like applying any other hotfix, to test it first in a test environment. Until now I haven’t seen any negative side effects of these hotfixes BUT it is better to be SAFE then SORRY.

Twitter & Updates for MPs

Dan Rogers from Microsoft has a Twitter account which is all about MPs and their newest releases. Whenever a new/updated MP comes out, Dan tweets about it.

All you need to do is to subscribe to the MPreleases Twitter identity and you are the ‘first’ one to know when a new/updated MP gets out.

Management Pack Authoring Guide v2, next section is ready!

As stated in this blog posting, Brian Wren is rewriting the Management Pack Authoring Guide.

The second section, about the Composition has just been released.

Section Overview
Service Model
Composition

Where are my MPs?

As we all know the MP Catalog has been moved to PinPoint. Many people must get used to the way it works and Microsoft is working hard on it to make it work better as well.

In the mean time on the OpsMgr TechNet Forums there are some threads about this new Catalog. In these threads people working for Microsoft jump in as well and give some good advice.

One of them, Dan Rogers, has given some good advice which I would like to share with the rest of the OpsMgr Community. Taken directly from this thread, coming from Dan Rogers:

…Finding all of the Management Packs in a single, alphabetized (by name) sort, there is an easy way to get this in just two clicks. 1. First, click here to get to the System Center landing page on the Pinpoint products and services portal. 2. Next, click on middle tab that is labeled "Apps + Services".  This gets you a list of all Management Packs that Microsoft has produced. 3. Change the sort to "by name" and you have all Microsoft Management Packs on one web page (CTRL-F and print friendly), sorted alphabetically Finding Management Packs from other companies is just as easy.  If you know the name of the third party, first find their Pinpoint landing page.  Then click on Products and Services. Or Click on the category link "Management Pack" in any Management Pack description in the list above.  This changes the search to be all items in Pinpoint that are categorized as management packs…

Hopefully this makes it easier for people using PinPoint.