Dell MP 5.1: Where Are The Dell OEM Servers?!

Issue
For a customer the Dell MP (only the Scalable MP, not the Detailed MP…) is in place. However, many Dell servers aren’t discovered, even though all prereqs are in place AND operational. But no matter what, these servers weren’t do be discovered as Dell servers. Time for some troubleshooting.

Cause
The Dell Server MP (Dell Windows Server (Scalable Edition) contains some Discoveries. The Discovery Dell Server Discovery is the main Discovery here, targeted against the Windows Computer class. This Discovery runs once per 24 hours.

This Discovery can be modified through the regular mechanism of setting Overrides. In this case I set the Discovery to run once per minute for a particular Dell server having these issues AND I enabled logging of this Discovery by setting the Parameter Name Log Level to 1:

Because of this the Discovery will create the log file ‘C:\Windows\Temp\Discovery_DellServer.log’. Soon when the Overrides were distributed to this server and processed by the Microsoft Monitoring Agent, the log file was created and neatly filled with information. And this is what came out of it:

So besides the hardware manufacturer Dell, there is also a manufacturer DELLOE… .

Time for a deeper investigation of the Discovery involved. I exported the MP to a plain XML file and used Notepad++ to open it. Soon I found the Discovery and it’s script, DellServerDiscovery.vbs. This is a huge script containing many functions, among them the function GetManufacturerName:

So this function runs this  WMI query: Set wbemObjectSet = GetObjectSet(wbemServices, "select * from Win32_ComputerSystem", "Manufacturer", "Win32_ComputerSystem"). Since the log file contains the entry INFO: HardwareManufacturerName Name retrieved successfully, I know this function run just fine. However, the value it contains isn’t accepted… Back to the Discovery script.

And yes, here I found the pain: Instr is case sensitive:

So this causes the Discovery not to work for servers with DELLOE as hardware manufacturer…

Where does DELLOE come from?
As is turns out, DELLOE is related to Dell servers running an OEM BIOS. Apparently this information is piped into WMI as DELLOE.

However, installing a Dell BIOS on servers like these isn’t the way to go since I don’t have any idea to what extend it will impact the operations of this server in a negative kind of way.

Solution?
As this moment I am thinking about multiple ‘solutions’ or better, workarounds. Ranging from dirty to acceptable but involving a lot of work. First I have to discuss them with other people and look into them. When I know more, I’ll update this posting accordingly. For now these Dell servers running an OEM BIOS won’t be covered by SCOM .

When Googling (sorry, here in Europe I don’t use Bing) with this query Hardware Manufacturer Name query : DELLOE, I found that I am not the only one having this issue:

Recap
Looking back at previous encounters I had with the Dell MP I must say I am not surprised. But ‘only’ disappointed…

The Dell MP has become far more better compared to the past but it’s still a MP to be careful with and not to load all available Dell MPs since before you know it, you’re in deep troubles.

CU#2 SCCM 2012 R2 Is Out!

Last Friday Microsoft released Cumulative Update #2 for SCCM 2012 R2. KB2970177 describes this CU#2 in more detail.

Check yourself before you wreck yourself…
Please take your time to read this KB article, all about CU#2 for SCCM 2012 R2. It states:

Please take notion of these comments, since they can save you and your organization a lot of pain and shame…

More information about CU#2 for SCCM 2012 R2
Only a few hours ago I found this posting on the Windows Intune blog, all about CU#2 for SCCM 2012 R2, in relation to Windows Intune.

This information isn’t to be found on KB2970177. For me this is an additional showcase that only a KB article tells it all. As you can see, CU#2 contains more then expected.

SCOM 2012x Community Must Haves – The Real Community Beasts & Efforts

For any SCOM 2012x environment there are some community efforts out there which are just awesome and add that little bit of extra which makes it far more easier to manage and/or enables deeper monitoring for a certain service/product.

Please feel free to comment with your own recommendations. I’ll update this posting accordingly. And PLEASE don’t be angry when you’re not named here. Just send in your comments and I’ll update this posting.

1. Tao Yang
   No, he’s a  person so you can’t import him into your environment, even though SCOM would really benefit from it . However, the results of his hard work and dedication to the SCOM community are really awesome and impressive. Not only because they fill up many gaps but also of the level of quality and ongoing improvements. Therefore Tao has earned my deepest respect.
   
   Some of his pearls:
   - The SCOM 2012 Self Maintenance MP
   - The Remote Maintenance Mode Scheduler MP
   - SCCM 2012 Clients MP
   
   There is far more to be found on his blog. All I can say: Go to his blog and CHECK IT OUT YOURSELF.
   
   
2. Stefan Roth
   Former MVP, SCOM guru and PowerShell Master. He shares his insights, ideas, code and tools with the community. His blog is always packed with really good information.
   
   - MAST Tool. I already blogged about it. A must have for anyone using SCOM to send out Notifications.
   
   
3. Raphael Burri
   Another Master of SCOM! Switzerland makes more besides watches and chocolate. Also HIGH quality MPs and this one comes for FREE! And yes, I am talking about the
   
   - PKI Verification MP.
   Recently updated (totally rewritten from the ground up). My customers love this MP since it has saved them from embarrassing situations and loosing tons of money (like: website for customers down because of an expired SSL certificate).
   
   
4. System Center Central > SCC Health Check Reports MP
   Written by Pete Zerger and Oskar Landman, two other SCOM Heroes. Even though the SCC Health Check Reports MP isn’t updated for a very long time, it works great with SCOM 2012x, up to R2! And it gives you a deep insight in SCOM itself and its health.

I know. There’s more. But my intention with this posting is to start a process where YOU (the reader of this blog) adds his/her comments, all about other great tooling, scripts and so on to be used in SCOM. And again, I’ll update this posting accordingly.

SCCM W7 OSD Task Sequence With ‘Install Software Updates’: Mind Patch Tuesday June 2014 – SCCM Client In Provisioning Mode :(

Issue
This one was hard to crack: it took me a whole day to identify the culprit, the underlying mechanisms, the why and the possible solutions/workarounds…

Because of Patch Tuesday some W7 based images had to be updated with the latest security updates. For some good reasons we don’t use Offline Services for OS Images. Instead we deploy the W7 image and install the software updates provided through ADR, all by a single Task Sequence.

So far so good. But when it was time to capture the newly build image, trouble started. For some reason the SCCM Client didn’t contact the SCCM MP. Even though the boundaries were okay, it just didn’t work. The SCCM servers involved looked upon the system as one without a SCCM Client, even though it was installed and running.

Time for additional checks. The SCCM Client on the newly build system was running but anything from okay.

The Client certificate wasn’t present:

Most components weren’t enabled so the list of Actions was very limited. And when Software Center was run, it showed the default values instead of the values set for this particular SCCM Site.

The log files PolicyAgent.log and LocationServices.log weren’t okay either. The first one contained serious errors last one missed crucial configuration information. And the log file ccmsetup-ccmeval.log was really alarming. Normally this file isn’t present at all! Also the last two entries in that log file told me something was seriously wrong:

So the SCCM Client was uninitialized!

Provisioning Mode
As it turns out the SCCM Client was in PROVISIONING MODE. Basically this leaves the SCCM Client fully installed and running but without an active configuration. So it won’t contact any SCCM MP, ever.

By the way this is the way the registry looks like when the SCCM Client is stuck in Provisioning Mode:

• HKLM\Software\Microsoft\CCM\CcmExec\ProvisionMode > True (should be FALSE)
• HKLM\Software\Microsoft\CCM\CcmExec\SystemTaskExcludes > Many values (should be empty)

But what caused it? Time for a test. I ran the same build Task Sequence again, but now with the Install Software Updates Task disabled. And now the SCCM Client was fully initialized and functional. The SCCM MP was found by the SCCM Client and all policies came in and were processed. A self signed was made and loaded. Software Center was configured as intended.

So somewhere the cause of the SCCM Client ending up in Provisioning Mode was caused by the software updates present in the Patch Tuesday release of June 2014.

But before I continue some additional information about the why a SCCM Client is normally put into Provisioning Mode (and later removed from that mode) and why it’s a GOOD thing when running a Task Sequence.

The mechanism of SCCM Client during a Task Sequence
When running a build Task Sequence the Task Setup Windows and Configuration Manager will install the SCCM Client. So this is a good thing.

However, when this SCCM Client becomes fully active while the Task Sequence is still running it might become a huge show blocker because it contacts the SCCM MP, downloads all kinds of policies and will process them. Changes are it will collide with the running Task Sequence resulting in a wrecked system, which is bad.

So therefore the SCCM Client is set into Provisioning Mode while the Task Sequence is running. At the end of it the SCCM Client will be removed from that mode, resulting in a fully functional SCCM Client.

However, in order to get there, the Task Sequence has to be AND stay in control of the whole build process. When something happens resulting in a (partially) broken or interrupted Task Sequence, changes are the SCCM Client won’t be removed from Provisioning Mode.

Time to move on to the cause…

The Big Internet and the BAD thing of it…
Yes, I know what you’re thinking. Why not modify the mentioned registry keys (can even be done by the SAME Task Sequence(!)) and be done with it.

HOWEVER, that won’t fly. Simply because it won’t fix it for EVER. Every time a Scheduled Task is run on every system running the SCCM Client. This Task ‘Configuration Manager Health Evaluation’ will reconfigure the SCCM Client BACK into the Provisioning Mode.

Why? Well for this particular Scheduled Task the file MobileClient.TCF is used. And guess what? It contains (among many other entries) this entry:

So all the blog postings out there telling you to modify the registry keys in order to fix this issue are simply wrong! It will only fix it for some time, leaving you later with the same issue all over again.

There is also another blog posting telling you not only to modify the registry keys but also to fix the mentioned file. And yes, this will fix the issue for good. BUT… it’s still not a solution only a workaround at best and the real cause is still there, unaddressed and still wrecking havoc.

I for myself don’t like solutions like these because you’re not in control of the whole process.

The ROOT cause
Like the title already mentions, the REAL cause of the SCCM Client being stuck in Provisioning Mode is a security update released in the Patch Tuesday cycle of June 2014.

The Security Update isn’t the real culprit but the way it works. Since this particular update requires MULTIPLE reboots. And a Task Sequence can handle ONE reboot for an update just fine. But a second reboot for that SAME update will cause the Task Sequence to FAIL and/or the SCCM Client being stuck in Provisioning Mode.

KB2894518 tells it all:

And:

Ouch!!!

Even though the same KB article states to be updated with updates requiring multiple restarts, it’s not up to date:

So how to find it? Luckily the internet has GOOD information as well and soon I found this blog posting from the company Ardamis, all about an update released with the Patch Tuesday release of June 2014, requiring MULTIPLE reboots!

However, this isn’t only at play for x64 systems and x64 based update. But ALSO for the x86 version of it!!! See MS14-030.

Solutions/ workarounds
So now we’ve located the culprit and the root cause of it all. But how to fix it? Let’s make a list of the available solutions:

1. No, I won’t use this ‘solution’ or workaround
   Modifying the registry keys.
   The pain: Is only a temporary fix but won’t address the root cause.
   
   
2. Nor will I use this ‘solution’
   Modifying the registry keys AND related file entries.
   The pain: Even though it fixes the SCCM Client being stuck in Provisioning Mode, it still doesn’t address the root cause NOR does it guarantee the related Task Sequence has run for a full 100% without ANY glitches…
   
   
3. Nope, this one isn’t a good one either
   Excluding the updates causing multiple restarts from the Task Sequences and run them through the normal channels, using the SCCM Client.
   The pain: I don’t want to TEST/EXAMINE ALL updates released monthly by Microsoft in order to see whether one of them or more, causes the system to reboot more than once.
   
   
4. Could be a possible workaround
   Excluding ALL updates from the Task Sequences and run them through the normal channels, using the SCCM Client.
   This resolves this issue WITHOUT the pain mentioned in the third option. But it’s not a solution.
   
   
5. OS Image creation/updating is something which shouldn’t be done solely by SCCM 2102x
   But MDT 2013 instead.
   This is a whole different discussion but something I’ve heard from multiple highly respected sources who have far more DEEP SCCM knowledge and experience compared to me.
   At WMUG Netherlands we had the honor to have Johan Arwidmark presenting a webinar about this topic in English. This webinar is recorded and to be found here.
   
   

Cross Post: Free ebooks: Microsoft System Center family of titles

Last months a whole bunch of FREE ebooks has been released by Microsoft Press, all about the System Center family.

Here is an overview of all of these titles. These books are a great opportunity to learn more about these technologies even though some of these books contain a lot marketing mumbo jumbo. None the less, they are still a great way to start learning new stuff, paving the way for the bigger and better stuff out there .

Want to know more about these FREE ebooks? Go here.

Awesome Tool: Meet MASTool!

Former MVP Stefan Roth made an awesome tool, filling up a serious gap in SCOM 2012x: a quick way to enable/disable Subscriptions and close the related Alerts with a few simple mouse clicks.

Since Stefan knows his way around in the dark and forsaken world of developing , he has made this tool himself and gave it the proper title MASTool (Modify Alert & Subscription Tool).

And like the true community beast he is (why isn’t he MVP anymore?) he shares this tool with the community for FREE!

Want to know more about this tool? Go here, download it, use it and be AMAZED! Just like me.

Remark:
All credits go to Stefan for making this awesome tool AND sharing it with the community.

Installing SCOM 2012x Gateway Server From CMD Prompt? Don’t Forget EULA!!!

Issue
When following this TechNet article all about installing a SCOM 2012x Gateway Server, also by using the command prompt (header To Install the gateway server by using the Command Prompt window), you’ll certainly bump into the situation where the Gateway Server won’t install. Ever!

Problems
The log file only becomes a few KBs big and it will every time show the message that the installation failed without stating a clear reason why.

And double checking the earlier mentioned KB article will show you nothing since you followed that TechNet article to the smallest detail.

Cause
Gladly the Application event log of the failing SCOM Gateway Server told me more:

Solution
Add the switch AcceptEndUserLicenseAgreement=1 to the syntax used for installing the SCOM Gateway Server and you’ll be fine:

%WinDir%\System32\msiexec.exe /i path\Directory\MOMGateway.msi /qn /l*v path\Logs\GatewayInstall.log ADDLOCAL=MOMGateway MANAGEMENT_GROUP="<ManagementGroupName>" IS_ROOT_HEALTH_SERVER=0 ROOT_MANAGEMENT_SERVER_AD=<ParentMSFQDN> ROOT_MANAGEMENT_SERVER_DNS=<ParentMSFQDN> ACTIONS_USE_COMPUTER_ACCOUNT=0 ACTIONSDOMAIN=<DomainName> ACTIONSUSER=<ActionAccountName> ACTIONSPASSWORD=<Password> ROOT_MANAGEMENT_SERVER_PORT=5723 [INSTALLDIR=<path\Directory>] AcceptEndUserLicenseAgreement=1

SCOM 2007x Retro Style Dashboards In SCOM 2012x

Widgets or dead wood?
With SCOM 2012x new styled dashboards were released as well, based on widgets. Even though these widgets may have a sexy look & feel to them, they aren’t flawless.

Even in SCOM 2012 R2 UR#2 there are still issues like widgets which time out (the underlying SQL queries aren’t ‘spot on’ so the widget times out…), making those same widgets pretty useless in bigger environments.

Where as other widgets don’t allow scoping for a a particular set of data, like the Alert widgets making them far to generic for proper usage by scoped end user groups.

Back to basics?
IMHO the widgets don’t live up to their promises yet, resulting in a quest for SCOM dashboards in the old SCOM 2007x style. These kind of dashboards are actually nothing but an aggregation of other already existing Views in the Console, enabling one to combine different Views to be presented on one single BIG screen.

The SCOM 2012x dashboard wizard won’t allow it…
However, the biggest party crusher here is the wizard which enables you to build dashboards. In SCOM 2012x this wizard won’t allow you to build these old style SCOM 2007x dashboards. Instead you’re forced to use widgets…

The workaround
In order to enable you to build SCOM 2007x style dashboards, I’ve built in my SCOM 2007 R2 CU#7 environment at home a simple MP containing most of the dashboards in SCOM 2007x style. None of these dashboards contains any view, only the placeholder enabling you to add the required SCOM Views.

Simply import this MP into your SCOM 2012x environment. Now in the Monitoring pane you’ll get the folder _SCOM 2007 Dashboards with the most used SCOM 2007x dashboard lay-outs:

The names of the Views are self explanatory. Simply select the dashboard you want to use, right click it, select Copy and paste it into the Unsealed Management Pack / View where you want to use it.

When the dashboard is pasted you can edit it by clicking Click to add a view in order to add an existing View to it.

MP can be downloaded from my OneDrive..