SCOM R2 Agent won’t start. System event log shows EventID 7024 with Description ‘The System Center Management service terminated with service-specific error 123 (0x7B)’.

Cracked a tough one today.

The Case
A SCOM R2 Agent with CU#4 didn’t start at all. Agent tried to start but died within seconds, showing this message:

So I checked the OpsMgr event log, but it turned out empty. Checked the System event log and it showed Event ID 7024:

Easy One? Bummer!
Hmm, EventID 7024. I know that one. Even blogged about it. And Jimmy Harper, SCOM PFE, also blogged about it. But no matter what I did, nothing helped. My posting didn’t help nor Jimmy’s posting for that matter.

A bit frustrating it was.

The issue did resemble the postings though, because the SCOM R2 Agent wasn’t able to create a self-signed certificate as well. The local certificate store of the computer didn’t contain any SCOM related certificate. So the problem was known, but not the cause…

And somewhere the cause of it all was to be found in the description of the EventID: The System Center Management service terminated with service-specific error 123 (0x7B).

But I needed more information to go on. This was just too shallow.

Final Attempt, let’s take a DEEP dive
So I tried another approach. I started diagnostic tracing on the SCOM R2 Agent, as described in KB942864, hoping to find more information to go on.

After having run the command FormatTracing.cmd, many readable log files were created. Most of them were almost empty, but one of them, the file TracingGuidsNative.log, told me that the SCOM R2 Agent tried to access a TMP folder which was totally wrong.

Normally the TEMP and TMP System Variables of any Windows system do have this format: %SystemRoot%\TEMP. But on this problematic server the TMP System Variable looked like this: C:\Program Files\<Other Software>\<Folder>;%SystemRoot%\TEMP.

Windows will not differentiate between these two folders even when a semicolon is in place. It will look upon it as a whole path, like “C:\Program Files\<Other Software>\<Folder>;%SystemRoot%\TEMP”. And paths like these do not work. Ever.

After having removed the corrupt entry (basically bringing it back to the default value of %SystemRoot%\TEMP) and saving the changes, the SCOM R2 Agent started without any errors at all!

Yeah! Nice one!

Recap
So whenever a SCOM R2 Agent doesn’t start and the Event log of the problematic server doesn’t give too many details, try diagnostic tracing in order to obtain more information. But do only so when you know what you’re doing.