SCCM W7 OSD Task Sequence With ‘Install Software Updates’: Mind Patch Tuesday June 2014 – SCCM Client In Provisioning Mode :(

Issue
This one was hard to crack: it took me a whole day to identify the culprit, the underlying mechanisms, the why and the possible solutions/workarounds…

Because of Patch Tuesday some W7 based images had to be updated with the latest security updates. For some good reasons we don’t use Offline Services for OS Images. Instead we deploy the W7 image and install the software updates provided through ADR, all by a single Task Sequence.

So far so good. But when it was time to capture the newly build image, trouble started. For some reason the SCCM Client didn’t contact the SCCM MP. Even though the boundaries were okay, it just didn’t work. The SCCM servers involved looked upon the system as one without a SCCM Client, even though it was installed and running.

Time for additional checks. The SCCM Client on the newly build system was running but anything from okay.

The Client certificate wasn’t present:

Most components weren’t enabled so the list of Actions was very limited. And when Software Center was run, it showed the default values instead of the values set for this particular SCCM Site.

The log files PolicyAgent.log and LocationServices.log weren’t okay either. The first one contained serious errors last one missed crucial configuration information. And the log file ccmsetup-ccmeval.log was really alarming. Normally this file isn’t present at all! Also the last two entries in that log file told me something was seriously wrong:

So the SCCM Client was uninitialized!

Provisioning Mode
As it turns out the SCCM Client was in PROVISIONING MODE. Basically this leaves the SCCM Client fully installed and running but without an active configuration. So it won’t contact any SCCM MP, ever.

By the way this is the way the registry looks like when the SCCM Client is stuck in Provisioning Mode:

• HKLM\Software\Microsoft\CCM\CcmExec\ProvisionMode > True (should be FALSE)
• HKLM\Software\Microsoft\CCM\CcmExec\SystemTaskExcludes > Many values (should be empty)

But what caused it? Time for a test. I ran the same build Task Sequence again, but now with the Install Software Updates Task disabled. And now the SCCM Client was fully initialized and functional. The SCCM MP was found by the SCCM Client and all policies came in and were processed. A self signed was made and loaded. Software Center was configured as intended.

So somewhere the cause of the SCCM Client ending up in Provisioning Mode was caused by the software updates present in the Patch Tuesday release of June 2014.

But before I continue some additional information about the why a SCCM Client is normally put into Provisioning Mode (and later removed from that mode) and why it’s a GOOD thing when running a Task Sequence.

The mechanism of SCCM Client during a Task Sequence
When running a build Task Sequence the Task Setup Windows and Configuration Manager will install the SCCM Client. So this is a good thing.

However, when this SCCM Client becomes fully active while the Task Sequence is still running it might become a huge show blocker because it contacts the SCCM MP, downloads all kinds of policies and will process them. Changes are it will collide with the running Task Sequence resulting in a wrecked system, which is bad.

So therefore the SCCM Client is set into Provisioning Mode while the Task Sequence is running. At the end of it the SCCM Client will be removed from that mode, resulting in a fully functional SCCM Client.

However, in order to get there, the Task Sequence has to be AND stay in control of the whole build process. When something happens resulting in a (partially) broken or interrupted Task Sequence, changes are the SCCM Client won’t be removed from Provisioning Mode.

Time to move on to the cause…

The Big Internet and the BAD thing of it…
Yes, I know what you’re thinking. Why not modify the mentioned registry keys (can even be done by the SAME Task Sequence(!)) and be done with it.

HOWEVER, that won’t fly. Simply because it won’t fix it for EVER. Every time a Scheduled Task is run on every system running the SCCM Client. This Task ‘Configuration Manager Health Evaluation’ will reconfigure the SCCM Client BACK into the Provisioning Mode.

Why? Well for this particular Scheduled Task the file MobileClient.TCF is used. And guess what? It contains (among many other entries) this entry:

So all the blog postings out there telling you to modify the registry keys in order to fix this issue are simply wrong! It will only fix it for some time, leaving you later with the same issue all over again.

There is also another blog posting telling you not only to modify the registry keys but also to fix the mentioned file. And yes, this will fix the issue for good. BUT… it’s still not a solution only a workaround at best and the real cause is still there, unaddressed and still wrecking havoc.

I for myself don’t like solutions like these because you’re not in control of the whole process.

The ROOT cause
Like the title already mentions, the REAL cause of the SCCM Client being stuck in Provisioning Mode is a security update released in the Patch Tuesday cycle of June 2014.

The Security Update isn’t the real culprit but the way it works. Since this particular update requires MULTIPLE reboots. And a Task Sequence can handle ONE reboot for an update just fine. But a second reboot for that SAME update will cause the Task Sequence to FAIL and/or the SCCM Client being stuck in Provisioning Mode.

KB2894518 tells it all:

And:

Ouch!!!

Even though the same KB article states to be updated with updates requiring multiple restarts, it’s not up to date:

So how to find it? Luckily the internet has GOOD information as well and soon I found this blog posting from the company Ardamis, all about an update released with the Patch Tuesday release of June 2014, requiring MULTIPLE reboots!

However, this isn’t only at play for x64 systems and x64 based update. But ALSO for the x86 version of it!!! See MS14-030.

Solutions/ workarounds
So now we’ve located the culprit and the root cause of it all. But how to fix it? Let’s make a list of the available solutions:

1. No, I won’t use this ‘solution’ or workaround
   Modifying the registry keys.
   The pain: Is only a temporary fix but won’t address the root cause.
   
   
2. Nor will I use this ‘solution’
   Modifying the registry keys AND related file entries.
   The pain: Even though it fixes the SCCM Client being stuck in Provisioning Mode, it still doesn’t address the root cause NOR does it guarantee the related Task Sequence has run for a full 100% without ANY glitches…
   
   
3. Nope, this one isn’t a good one either
   Excluding the updates causing multiple restarts from the Task Sequences and run them through the normal channels, using the SCCM Client.
   The pain: I don’t want to TEST/EXAMINE ALL updates released monthly by Microsoft in order to see whether one of them or more, causes the system to reboot more than once.
   
   
4. Could be a possible workaround
   Excluding ALL updates from the Task Sequences and run them through the normal channels, using the SCCM Client.
   This resolves this issue WITHOUT the pain mentioned in the third option. But it’s not a solution.
   
   
5. OS Image creation/updating is something which shouldn’t be done solely by SCCM 2102x
   But MDT 2013 instead.
   This is a whole different discussion but something I’ve heard from multiple highly respected sources who have far more DEEP SCCM knowledge and experience compared to me.
   At WMUG Netherlands we had the honor to have Johan Arwidmark presenting a webinar about this topic in English. This webinar is recorded and to be found here.