PowerShell: Identifying SCOM Operational Alerts & Modifying Resolution State

Case
In my line of work I see quite a few SCOM 2012x environments. And most of the times these environments do share many similarities, besides the obvious ones of course. The one that stands out the most are Alerts related to the operation of SCOM itself.

Some examples of the Alerts I am talking about:

• Operations Manager Failed to convert performance data
• Operations Manager failed to run a WMI query
• Operations Manager failed to start a process
• Power Shell Script failed to run

The ‘funny’ thing is that Alerts like these do have a huge change of slipping through the system. Not because the people want them to slip through, but because it’s just how the system ‘works’. Meaning, SQL Alerts are forwarded to the DBA’s, Exchange/Lync Alerts are forwarded to the communication experts and so on.

But many times the management of SCOM itself is done by people who have tons of other stuff on their plate. And since the SCOM Console still functions, and the other disciplines (like DBA’s and the communication experts) don’t complain about SCOM, those people think SCOM is just okay, so they just ignore those Alerts.

And that’s bad since many times these Alerts are tell tale signs SOMETHING is not 100% okay with your environment. And additional attention is required. When not, changes are more than likely sooner or later SCOM will start having serious issues, like a performance hit or a ‘dent’ in the availability or total coverage of the monitoring solution.

Therefore it’s important to filter out those Alerts, pipe them into their own View and have a HTML Report on them. Much of this can be done by using two PS scripts. One of them needs to be run by the Task Scheduler on a regular basis, another can be run manually when required.

What it does and DOESN’T do…
No. These PS scripts WON’T solve the operational SCOM Alerts for you. However, it will help you to identify them much quicker and get a count on them as well in order to see what type of operational SCOM Alerts are happening the most, enabling you to quickly identify the most urgent matters at hand.

How these scripts work
As stated before, two PS scripts are needed here. The first one is the PS script (Flip Alerts to SCOM Operational Alert ResState.ps1) which must be run by the Task Scheduler or software which does just that (System Center Orchestrator for instance).

This PS script identifies the SCOM operational Alerts and ‘flips’ their Resolution State to one specifically made for SCOM operational Alerts. This PS script reads the text file (SCOMOpertionalAlerts.txt) containing the names of all SCOM operational Alerts.

Reason for this separate text file is simple. Every SCOM environment is different, so you might have different SCOM operational Alerts at hand. So the PS script has to check for different Alert names. By putting those Alert names in a dedicated text file, it’s way much easier to modify it. Instead of modifying the PS script, all you’ve got to do is to modify the related text file.

The second PS script (Sort all open SCOM Operational Alerts - Highest on top - HTML Report.ps1) is run manually. It will enumerate the SCOM operation Alerts, count them, put the highest count on top and pipes the output into a CSS formatted HTML file (Overview SCOM Operational Alerts.html) which will opened by the same PS script as well.

Required preparations
Before you start using these PS scripts some preparations are required. I will only share the highlights since I safely presume you know perfectly how to do this. So no need to repeat the obvious…

1. Create a new Resolution State, especially for the SCOM operational Alerts, like ‘SCOM Operational Alert’:
   
2. Create a new View (not in the Default MP!) showing these new SCOM Operational Alerts:
   

All set, ready to rock!
Now all is set. Just download the required PS scripts, modify them as required (name of the SCOM 2012x MS server to connect to, text file location, Resolution State number and so on…), and run them.

The PS script Flip Alerts to SCOM Operational Alert ResState.ps1 is the one to be scheduled to run once per our , 24/7. It uses the text file SCOMOpertionalAlerts.txt, which must be saved in a folder. Don’t forget to modify the scheduled PS script so it’s capable to locate it in the correct folder!

The PS script Sort all open SCOM Operational Alerts - Highest on top - HTML Report.ps1 is run manually when required.

Some examples (screenshots)
The Alert View itself in the SCOM Console, fed by the scheduled PS script:

The HTML report Overview SCOM Operational Alerts.html, shown after the other PS script is manually run:

Feel free to download these PS script (don’t forget the related txt file!) and use them as required. Please let me know when you’ve modified it into something better. I’ll update this posting accordingly.