Monday, September 22, 2014

Lync Server 2010 MP Bug Alert: ‘Consolidator Module Failed Initialization’

Issue
A customer of mine runs a SCOM 2012 R2 environment and has many servers and workloads in place, all monitored by SCOM. So far so good. Also a SINGLE Lync 2012 Server is in place and being monitored by SCOM.

But soon the Alert came in: ‘Consolidator Module Failed Initialization’.  Even though I had seen this Alert in the past on a Lync 2010 server, the fix I found back then didn’t work. Simply because the fix for a WMI memory issue, didn’t apply here since it was already outdated by newer file versions. So something else was at play here.

Cause
Some time ago I was on an assignment for a customer running many many Lync Server environments, among them many Lync 2010 servers as well. And NO WHERE this issue popped up. So what was different here?

Soon I found it. As it turned out, Lync isn’t really made to be installed on a single server. For a POC perhaps – even then I would not recommend it since you can’t test all the functionality of Lync – or a lab. But not for full production.

So in environments with just ONE Lync server this issue pops up! And the Alert is caused by just ONE single Monitor which is configured in the wrong way. But more about that one later on.

This is what the OpsMgr event log on the single Lync 2012 server told me:
clip_image002

Log Name:      Operations Manager
Source:        Health Service Modules
Event ID:      11112
Task Category: None
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      XYZ.domain.com

Description:
The Microsoft Operations Manager Consolidator Module failed to initialize because the specified compare count is less than the minimum allowed.

Compare count value: 1
Minimum value allowed: 2

One or more workflows were affected by this. 

Workflow name: Microsoft.LS.2010.Monitoring.UnitMonitor.TimerResetEvent.PDP_PDP_IPADDRESS_NOT_CONFIGURED_IN_NCS
Instance name: LS Policy Decision Point Component [XYZ.domain.com]
Instance ID: {D97677B9-BA09-426C-2FFF-EE8CB1F6A774}
Management group: ABC

In this case the compare count value and the minimum value allowed triggered an Alert for me. When comparing items, you require at least TWO of them, You can’t compare just ONE item with itself!

So apparently this specific Unit Monitor was trying to compare at least TWO items but only found ONE (understandable since it’s only ONE Lync 2010 Server) and failed because of it!

Now my search on the internet became much simpler because I knew now what to look for. And soon on the TechNet Forums I found this topic where I found the cause of it all, as described by ZYEngineer: ‘…On simple deployments of Lync, such as mine, there is no explicitly defined Region, Site, Subnet, Region Link or Region Route.  There's no need since there's only one front-end server.  This monitor is looking for the PDP IP Address in Network Configuration Settings (PDP_PDP_IPADDRESS_NOT_CONFIGURED_IN_NCS). …’

Tada!

Solution
So the only solution here is to disable the Unit Monitor. After some searching in the XML code of this MP I located it, it’s Unit Monitor: IP addresses missed from Network Configuration Settings targeted against LSPolicy Decision Point Component.
clip_image002[10]

As it turned out, this Unit Monitor has a serious issue (allowing a compare count of 1, DUH!) causing it to crash the SCOM Console when trying to look at the properties of it:
clip_image002[6]

And the details show the cause of this Console crashConfused smile:
clip_image002[8]
So the SCOM Console detects the defect of this Unit Monitor without any issue, so why did it slip through QC before this MP was published?

GLADLY, an Override wasn’t an issue at all. So I disabled this Unit Monitor and some minutes later this error in the event log on the Lync 2010 Server and the related SCOM Alert were gone.

Tool: SMART Documentation & Conversion 2.0 Helper For Orchestrator Runbooks

Update, 09-22-2014
Based on this posting I got feedback from Bruno Saille, the Program Manager who’s responsible for this tooling (among other things). I’ve decided to incorporate his feedback in blue. Thanks Bruno for your feedback, awesome!

Already some months ago Microsoft released an updated version their tool to document your Orchestrator  Runbooks, Orchestrator Visio and Word Generator.

However, as it turns out, this is just more than an update of the tool since it incorporates an update of the tool SMART Runbook Conversion Helper as well. This results in the tool SMART Documentation and Conversion Helper 2.0.

The tool itself – with a long description about how to use it with some good examples – can be found here.

So far, so good. But since I’ve used this tool I want to share some of my personal experiences:

  1. Icons please?
    Bumped into it just once. Visio and Word output ended up without icons, even though the export PS script for Runbook icons (this is a 32 bits PS script!) had run successfully and all icon files (jpeg) were present in the same folder the script was run from. Only fix for me was to remove all the icon files, rerun the export PS script for Runbook icons again. Afterwards Visio and Word output had icons again. Don’t know whether this was a one time glitch.

  2. Output to Word is dead slow
    On a well dimensioned server it took a long time to convert some Runbooks to Word format, especially compared to the conversion to Visio. Something which I couldn’t solve and just had to ride it out.

    Feedback from Bruno Saille, Microsoft PM responsible for this software: ‘…Yes, Word output is unfortunately slow, and this is due to how PowerShell works with COM automation. The same code used to be much faster in .NET, and the Visio automation code is not slower with PowerShell. Good thing is that WMF 5.0 is supposed to have a few enhancements in COM automation speed, so this may help (I have not had a chance to give it a good try)…’

  3. Running multiple PS instances on the same computer doesn’t work for conversion to Word
    Tried to run two PS instances in order to make the conversion go faster. For Visio and PS this works great, but when converting Runbooks to Word documents, it doesn’t work. Somehow the document gets corrupted and only the last line is kept. The rest is overwritten by newer output of the conversion. No title, no table. Just a single line of text.

  4. Configure Visio before you start the conversion
    Visio needs some additional configuration before you start your first conversion to it. In my case Visio was installed on the D:\ drive, so I had to modify the path referring to a specific Visio startup file.

  5. Be patient
    The tool works, but has some quirks, one of them is lacking speed. Also when you click something, sometimes there will be a lag. Just be patient and wait. Until now the tool didn’t crash on me which is far more unwelcome.

  6. Placement of the tool window
    When running the tool and starting a conversion (especially to Word), don’t forget to place the window of the tool to a place which doesn’t ‘eat away’ most of your screen. Because when a conversion is started, this windows can’t be dragged to another position. Just something to be aware off.

    Feedback from Bruno Saille, Microsoft PM responsible for this software: ‘…That is one of the limitation of doing this via a PowerShell XAML/WPF "GUI" : The tool will not give focus back until the processing has been finished. This is why we tried to add more extensive feedback in the console window at the same time, so you can confirm something is happening. You can move the console window around even when the GUI is "waiting". Adding multi-threading, etc. might be possible, but then it may bring more complexity than anything, vs a more "simple" PS script that you can modify to your needs…’

Besides this I am happy with the tool since it allows me to gain a good insight in an Orchestrator environment and more important, the Runbooks present. But don’t forget that this tool is the START of your journey in how the Runbooks are made, and not the end of it.

Having a cup of coffee with the persons who built the Runbooks provides tons of information as well, which can’t be captured by any tooling. But than of course, those people must be still around AND available…

Friday, September 19, 2014

Core OS MP Version 6.0.7230.0: Undocumented Change

Already some people pointed it out to me, so the credit of this posting go to them. However, I did not find the time to double check it. But it’s true: the latest version of the Core OS MP (Server OS MP), version 6.0.7230.0 has some changes which AREN’T documented but can affect your monitoring of available free disk space.

What’s happening?
Some versions of the Core OS MP back, there were those really good free disk space Monitors which when they fired an Alert told you directly what’s going on. So the Alert told you exactly how much disk space was still left (in MBs and %).

However, those Monitors were replaced by new ones (for some good reasons). BUT now the Alerts didn’t tell you anymore how much disk space was left Sad smile. Kevin Holman made a special MP which fixed this annoying issue.

But now with the latest version of this Core OS MP (version 6.0.7230.0) these Monitors are back and turned on by default! The previous ones are disabled now by default.

Another thing to reckon with is the time frame in which they run. The reinstated Monitors runs once per hour (3600 seconds) by default. Of course by using an Override you can modify it as required.

More details
In order to make it more clear what I am talking about, I give you more details.

  1. In total there are 4 logical disk free space monitors per OS version now. In this screenshot I’ve 12 of them (WS 2003x, 2008x and 2012x):
    clip_image002
    Only ONE per OS version is enabled by default.

  2. The PREVIOUS ONE with the Alert lacking good information is turned off now by default,
    the Logical Disk Free Space Monitor:
    clip_image002[5]

    This one ran once per 15 minutes:
    clip_image002[7]

    But the Alert showed nothing about how much disk space was left:
    clip_image002[9]

  3. The ONLY Monitor which is enabled by default is the
    Windows 20xy Logical Disk Free Space Monitor. This is the 2012 server OS version:
    clip_image002[11]

    It runs once per hour (3600) seconds:
    clip_image002[13]

    AND it generates an Alert with GOOD information about how much disk space (MBs and %) is left:
    clip_image002[15]

So check your environment and overrides when this MP is in place. Hopefully the MP guide for this MP will properly updated.

SCCM 2012x: The package data in WMI is not consistent to PkgLib

Issue
I’ve bumped into this issue quite a few times now. The DPs seem to be fine BUT in the monitoring pane of the SCCM 2012x Console the DPs have a warning icon. When looking in the relevant log file (smsdpmon.log) on the DPs involved this entry points to the cause of it: The package data in WMI is not consistent to PkgLib.

Cause
As it turns out, it happens when some packages are removed but their entry still ‘lives’ in WMI of the DPs involved. Already the SCCM Team posted an article how to solve it, to be found here. Even though it works, it’s time consuming. So I searched for another solution and found it on the TechNet Forums.

Solution
In this thread member JT_DPS posted some powerful PS scripts, helping to solve this issue really fast. His PS scripts come in three parts.

Part I: This PS script shows you what packages are in WMI and not in the Content Library AND vice versa.

$WMIPkgList = Get-WmiObject -Namespace Root\SCCMDP -Class SMS_PackagesInContLib | Select -ExpandProperty PackageID | Sort-Object
$ContentLib = (Get-ItemProperty -path HKLM:SOFTWARE\Microsoft\SMS\DP -Name ContentLibraryPath)
$PkgLibPath = ($ContentLib.ContentLibraryPath) + "\PkgLib"
$PkgLibList = (Get-ChildItem $PkgLibPath | Select -ExpandProperty Name | Sort-Object)
$PkgLibList = ($PKgLibList | ForEach-Object {$_.replace(".INI","")})
$PksinWMIButNotContentLib = Compare-Object -ReferenceObject $WMIPkgList -DifferenceObject $PKgLibList -PassThru | Where-Object { $_.SideIndicator -eq "<=" }
$PksinContentLibButNotWMI = Compare-Object -ReferenceObject $WMIPkgList -DifferenceObject $PKgLibList -PassThru | Where-Object { $_.SideIndicator -eq "=>" }
Write-Host Items in WMI but not the Content Library
Write-Host ========================================
$PksinWMIButNotContentLib
Write-Host Items in Content Library but not WMI
Write-Host ====================================
$PksinContentLibButNotWMI

Part II: This PS script removes the package from WMI (using the list from Part I):

Foreach ($Pkg in $PksinWMIButNotContentLib){ Get-WmiObject -Namespace Root\SCCMDP -Class SMS_PackagesInContLib -Filter "PackageID = '$Pkg'" | Remove-WmiObject -Confirm }

Part III: This PS script removes the INI file (using the list from Part I):

Foreach ($Pkg in $PksinContentLibButNotWMI){ Remove-Item -Path "$PkgLibPath\$Pkg.INI" -Confirm }

When you revalidate the content on your DPs they should turn to green icons again.

Credits
All credits for the PS scripts in this posting go to JT_DPS.

Wednesday, September 17, 2014

Updated MP: Service Manager 2012x, version 7.5.3079.183

Yesterday Microsoft released an updated version of the Service Manager 2012x MP, version 7.5.3079.183.

This MP requires SCOM 2012x. MP can be downloaded from here.

Tuesday, September 16, 2014

SCOrch: What SQL Server & Database Is Being Used?

Issue
Bumped into a System Center 2012 – Orchestrator (SCOrch) environment where it was unknown what SQL server and database was being used. Nowhere this information was found in the registry. So where to look now?

Solution
As it turned out, the solution was a simple one (even though it still eludes me why this information isn’t stored in the registry, but apparently it’s stored in the SQL database as well….).

  1. Start the tool Deployment Manager (System Center 2012 R2 Orchestrator Deployment Manager);
  2. Right click on Orchestrator Management Server > select Properties;
    image
  3. Go to the second tab Orchestrator Management Server. On this tab the Database Server and Data Store Name (SQL database) are shown:
    image

Credits
I found the solution to this relative simple question in
this posting of the blog Kick That Computer.

Friday, September 12, 2014

SCOM 2012x & UX Monitoring. Guided walkthrough for troubleshooting UNIX and Linux agent discovery

KB2993901 is a comprehensive troubleshooting guide for anyone having issues with UNIX/Linux Agent discoveries and installations.

Glad to see Microsoft puts this kind of effort in it since monitoring UX systems with SCOM 2012 is still a challenge Smile.