Troubleshooting Event ID’s 33333, 11502 & 4502

Issue
A SCOM 2012 R2 environment was logging many Event ID’s 33333, 11502 & 4502 in the OpsMgr event log of the Management Server designated to send out the Alerts. And believe me, you DON’T want to see these events, ever:

• Event ID 33333
  Data Access Layer rejected retry on SqlError.
  
  
• Event ID 11502
  The Microsoft Operations Manager Connector Framework Alert Forwarding module failed to mark an alert for forwarding because the connector the module is configured for no longer exists.
  
  
• EventID 4502
  A module of type "Microsoft.EnterpriseManagement.Mom.Modules.McAlertWriteAction" reported an exception System.Data.SqlClient.SqlException (0x80131904).

And these events were logged almost every minute since Alerts are filtered for Notifications once per minute. So the OpsMgr event log of this particular server looked like a X-mas tree. Red & yellow all over the place…

Time for some deep troubleshooting.

Cause
First thing I noticed was that these events were ‘only’ logged on the MS server responsible for the Notifications. So the issue was directly related to it.

Secondly, the three earlier mentioned Event ID’s gave more information in their descriptions:

• Event ID 33333
  …the UPDATE statement conflicted with FOREIGN KEY constraint "FK_Alert_ConnectorId". The conflict occurred in database "OperationsManager", table "dbo.Connector", column "ConnectorId".
  
  
• Event ID 11502
  …the connector the module is configured for no longer exists. Connector Id: [GUID].
  
  
• EventID 4502
  …the UPDATE statement conflicted with the FOREIGN KEY constraint "FK_Alert_ConnectorId:. The conflict occured in database "OperationsManager", table "dbo.Connector", column 'ConnectorId'.

Apparently this SCOM 2012 R2 MG is trying to use Channels/Connectors/Subscriptions which aren’t present (anymore), causing the Notifications component to turn sour and  light up the OpsMgr event log of the particular MS server like a X-mas tree…

Close to finding the cause of it all…
In SCOM all the Notifications you create (Channels, subscribers, subscriptions AND the subscriptions of the Connectors) are saved in this MP: Notifications Internal Library (ID: Microsoft.SystemCenter.Notifications.Internal).

As it turned out, this MP was exported from an old SCOM 2007x environment and imported straight into the SCOM 2012 R2 environment. So ALL the configuration for the Notifications in SCOM 2007x was carried straight into the SCOM 2012 R2 environment, whether all Connectors were in place or not…

So finally the culprit was found. But now the question was asked how to solve it? The Quick & Dirty way or the more challenging way?

Resolution
Yes, you CAN import the default Notifications Internal Library MP from the SCOM 2012 R2 installation media. This way the ‘old’ MP will be overwritten and replaced by a completely empty Notifications model.

For this customer totally unacceptable since many other Channels, Subscribers, Subscriptions and Connector config was already being used. So a total reconfiguration – involving missing Notifications – was out of the question.

So no Quick & Dirty approach here. Time for the more challenging approach, editing the Notifications Internal Library in Notepad++ and removing all the misconfigurations one by one.

The VERY helpful Event ID 33333
Even though I DON”T like Event ID 33333 one bit, in this particular case this very same event provided me with very good information: the GUID of the ConnectorId, masked in yellow in this screendump out of respect for this customer:

This GUID is also present in the  Notifications Internal Library. So I exported this MP, saved a copy of it (in case I would wreck this MP), opened the copy and searched for the GUID of the ConnectorId.

For every entry I found I deleted the WHOLE Rule section, so from <Rule ID up to </Rule>. This can take up to 100+ rows…

But this wasn’t enough since every Rule also has a Rule ID or a ProductConnectorSubscription[GUID]. All those entries must be deleted as well. When you don’t you break this MP. Believe me, I’ve been there .

So the first deletion took me some time in order to get it 100% right. But when I imported that MP EventID 33333, 11502 & 4502 for that particular ConnectorId were GONE! By then I knew what to do…

1. Checked ALL Event ID’s 33333 and copied the related unique ConnectorId;
2. For every unique ConnectorId I adjusted the MP accordingly by removing the related Rule sections AND the related Rule ID or a ProductConnectorSubscription[GUID] entries;
3. Repeated Step 2 for every unique ConnectorId I found in Event ID 33333;
4. When done I incremented the version number of the MP, saved it and imported it;
5. And PRESTO, when the new configuration became active, NO MORE event id’s 33333, 11502 & 4502. Yeah!!!

Recap
When moving from SCOM 2007x to SCOM 2012 where SCOM 2012x is a brand new environment, think twice before importing core MP’s like Notifications Internal Library straight from SCOM 2007x into SCOM 2012x.

Changes are things won’t work and will even break stuff. So be careful here.

And when all is working as expected and you remove a Connector, first make sure there are NO subscriptions any more for it. When there are AND you remove the Connector WITHOUT removing the related Subscriptions before that action, you’ll find yourself in the same situation. At least you know now how to solve it…

New MP: Windows Server Storage Spaces 2012 R2

Yesterday Microsoft released a new MP in order to monitor Windows Server Storage Spaces 2012 R2, version 1.1.253.0.

This VERY basic MP allows you to monitor your Windows Server Storage Spaces 2012 R2 environment. Supported configurations:

• 16 Storage Nodes;
• 12 Storage Pools;
• 120 File Shares.

The related MP guide (just as BIG as the MP itself…) reveals more about this MP:

And:

As you can see, Microsoft has worked hard in order to get the job done and without any care about a budget (if there was any…).

But wait! As it turns out there is ONE View. Wow! And yes, MP Viewer confirms it:

The MP can be downloaded from here. However, be warned since a MP like this will have a huge hit on the available bandwidth, so download it outside the regular business hours

Updated MP: Monitoring WS2012 R2 Hyper-V, Version 7.1.10107.0

A few days ago Microsoft released the updated version of the Windows Server 2012 R2 Hyper-V MP, version 7.1.10107.0.

Updates and fixes:

1. Changed the format of management pack file from MP type to MPB type
2. Fixed bug that prevents the free disk space from being collected and therefore prevents the monitors tracking disk space from firing correctly. This could lead to disks filling up and outages as a result of this not being detected correctly.
3. Fixed bug that was generating 11903 events with the exception “The Microsoft Operations Manager Expression Filter Module could not convert the received value to the requested type.”. These events are raised by Microsoft.Windows.HyperV.2012.R2.VirtualDrive.FreeSpaceMonitor

MP can be downloaded from here.

Teaser: How To Solve Event ID‘s 11502, 33333 & 4502

Quiet recently I had to solve an issue where the SCOM 2012x Management Server logged these nagging and BAD Event ID’s:

1. 11502
   The Microsoft Operations Manager Connector Framework Alert Forwarding module failed to mark an alert for forwarding because the connector the module is configured for no longer exists. Connector Id: [GUID]
   
   
2. 33333
   Data Access Layer rejected retry on SqlError. The UPDATE statement conflicted with FOREIGN KEY constraint "FK_Alert_ConnectorId". The conflict occurred in database "OperationsManager", table "dbo.Connector", column "ConnectorId"
   
   
3. 4502
   A module of type "Microsoft.EnterpriseManagement.Mom.Modules.McAlertWriteAction" reported an exception System.Data.SqlClient.SqlException (0x80131904): The UPDATE statement conflicted with the FOREIGN KEY constraint "FK_Alert_ConnectorId:. The conflict occured in database "OperationsManager", table "dbo.Connector", column 'ConnectorId'.

It took me a while to get to the bottom of it all, but finally I FIXED it! The last days I didn’t have the time to write an article about it. But soon I’ll write a posting for it. What caused it, and how to solve it. So stay tuned!

Free eBook: Microsoft System Center Operations Manager Field Experience

A few days ago Microsoft released a new FREE ebook, titled: Microsoft System Center Operations Manager Field Experience.

The book is written by people who have a lot of experience with SCOM. They work on a daily basis in the ‘trenches’ so they know what they’re talking about.

Intended audience (taken directly from the website): ‘…If you’re responsible for designing, configuring, implementing, or managing a Microsoft System Center Operations Manager environment, then this book is for you. This book will help you understand what you can do to enhance your Operations Manager environment, and will give you the opportunity to better understand the inner workings of the product, even if you are a seasoned Operations Manager administrator…’

And: ‘…This book assumes that you have a deep working knowledge of the Operations Manager product and its concepts, that you understand the concept of management packs, and that you are basically familiar with Microsoft Azure as an infrastructure-as-a-service platform. This is a book about best practices, design concepts, how-tos, and in-depth technical troubleshooting. It covers the role of the Operations Manager product, the best practices for working with management packs, how to use the reporting feature to simplify managing the product, how to thoroughly troubleshoot, and how to use and install Operations Manager in a Microsoft Azure Public Cloud environment…’

Book can be downloaded from here. The book also has companion content, to be downloaded from here.

What’s The Diff Between SCOM 2007 R2 Default MP & SCOM 2012 R2 Default MP?

Issue
Bumped into a SCOM 2012 R2 environment which was built from the ground up in order to migrate away from an existing SCOM 2007 R2 environment. So many unsealed MPs from the SCOM 2007 R2 environment were exported from it and imported into the SCOM 2012 R2 environment.

Among all those unsealed MPs it turned out that the Default MP of SCOM 2007 R2 slipped through and was imported into the SCOM 2012 R2 environment as well. And yes, even though the version number of that MP is lower than the default MP which was present in the SCOM 2012 R2 environment.

However, versioning is ONLY enforced when the MP is sealed. For unsealed MPs version numbers aren’t enforced. Ever…

Is it really an issue?
However, since the SCOM 2007 R2 MP contained many custom Views and so on, it was hard for me to really know the difference between these two Default MPs. So it was time to run a comparison between both Default MPs as they’re to be found on the installation media.

For SCOM 2012 R2 it’s really easy. Simply mount the iso and browse to the folder ~:\ManagementPacks and copy the file Microsoft.SystemCenter.OperationsManager.DefaultUser.xml to a dedicated temp folder.

For SCOM 2007 R2 it’s a bit more work. Mount the iso and browse to the folder ~:\Server\AMD64 and open the file OMServer.cab. Copy the file F_Microsoft.SystemCenter.OperationsManager.DefaultUser.xml.540EA3C0_A5E9_41EA_A585_822C09EA2650 to a dedicated folder and rename it to Microsoft.SystemCenter.OperationsManager.DefaultUser.xml by removing all the ‘trimmings’.

So now we’ve got the two ORIGINAL Default MPs. One from SCOM 2007 R2 RTM and another from SCOM 2012 R2 RTM. Time to run a comparison.

Tooling required
For this you can use many different tools. Even though I am big fan of Notepad++, in this particular case XML Notepad 2007 does a better job. Simply because this really monolithic tool is aimed only at XML for MPs, so here it works great with the additional tool XmlDiff started within XML Notepad 2007. Kevin Holman has written a posting about how to use this tool just for this, so no need for me to repeat it here.

And with the tool XmlDiff it turned out there aren’t that many differences at all.

Findings
Besides the obvious (version numbers) there are two  major differences:

The ColumnInfo entry is set to 30 in SCOM 2012 R2, compared to SCOM 2007 R2:

And the Default MP for SCOM 2012 R2 contains a filled out LanguagePack section whereas the one for the Default MP for SCOM 2007 R2 RTM is completely empty.

So as you can see, it’s not that much of a difference.

What to do?
Easy. SCOM 2007 R2 is dead (mainstream support ended on 7/8/2014!) and SCOM 2012 R2 is alive and kicking (mainstream support until 7/11/2017). So no need to drag on all that old legacy stuff into SCOM 2012 R2. Time to leave it behind.

Also THE opportunity the put all the ‘lessons learned’ into the real world. And one of those Best Practices is NOT to use the Default MP for ANYTHING, also not for Views in the Console.

So first all the Views present in the SCOM 2007 default MP were put into a dedicated MP of it’s own in SCOM 2012 R2. Afterwards the SCOM 2007 R2 Default MP was deleted and replaced by the original SCOM 2012 R2 Default MP as found on the installation media.

End of story.

Cross Post: Ex2010 MP Correlation Engine Doesn’t Generate Alerts

As we all know the Exchange 2010 MP is a MP of a rare breed. It uses the ‘Correlation Engine’ (CE) in order to generate Alerts. Even though the theory behind it is good, the execution has ‘some’ issues and nags, like using most of the Custom fields of an Alert generated by the CE, causing many automated solutions, depending on those fields to turn sour.

So I was glad when Microsoft introduced Exchange Server 2013 MP and dropped the Correlation Engine for the MP in order to monitor it.

However, there are many organizations out there who still run Exchange Server 2010 and use SCOM 2007 to monitor it. When you want to migrate to SCOM 2012x changes are CE will stop functioning by not generating alerts anymore.

Gladly PFE Jimmy Harper bumped into this situation already. He solved it – with help of one of his colleagues – and blogged about it.

The posting is to be found here.

New MP: SCDPM 2012 R2 MP Version 4.2.1276.0

Yesterday Microsoft released the MP for monitoring System Center Data Protection Manager 2012 R2, version 4.2.1276.0.

This MP also contains a Reporting MP, enabling you to report about your SCDPM environment. Want to know more about the Reports in this MP? Read this posting on the System Center Data Protection Manager blog.

The MP itself can be downloaded from here.

KB3035262: Troubleshooting MMA Connectivity Issues

Microsoft has updated KB3035262 all about solving connectivity issues with your SCOM 2012x Agents (Microsoft Monitoring Agents).

Whenever you want to troubleshoot MMA connectivity issues this KB is an excellent starting point.

KB3054347: Error 800706D3 Occurs & Push Installation Fails a MMA To WS2012

Microsoft has published a new KB all about troubleshooting error 800706D3 occurs and a push installation fails for an Operations Manager 2012 agent to Windows Server 2012.

See KB3054347 about this issue, it’s cause and how to solve it.

Repost: Fix For Not Starting Data Access/Config Service After KB2677070 Is Installed

This week on Twitter there was a tweet all about not starting System Center services after KB26770 is installed. Many people retweeted it or added it to their favorites. So apparently this issue is current and harassing some organizations.

So I did some digging on my blog and found this posting, dated October 9th 2012. Normally I won’t repost my previous posts, but for this one I make an exception:

---

Yesterday I posted about KB2677070 causing issues when installing an (additional) OM12 Management Server.

As it turns out Microsoft has acknowledged it and posted a new KB article how to workaround this issue. KB2730040 tells one exactly what to do in order to keep KB2677070 AND to successfully install an (additional) OM12 Management Server.

Thanks to Vincent who mentioned this KB to me.

---

Cross Post: SCOM Build Numbers

Quite recently I needed to run a check on a SCOM 2007x version. Gladly I found a blog which is all about build numbers for a wide range of Microsoft Products.

On that blog there is also a page all about the build numbers for SCOM, starting at MOM 2000(!) up to SCOM 2012 R2 UR#5. This enabled me to pinpoint the exact version of SCOM 2007x and the CU level. Awesome!

So I’ve added this blog to my favorites: https://buildnumbers.wordpress.com/scom/.