Wednesday, June 13, 2012

How to crack EventID 21405 for any given monitored server

A key customer of mine had this strange issue. Almost ALL (Discovery) scripts were exiting so almost nothing of the Management Packs landed on a particular server.

Issue
EventID 21405 containing message ‘The process started at x failed to create y, no errors detected in the output. The process exited with 1’ was shown many times in the OpsMgr eventlog of that server:
image

CSS to the rescue
And no matter what we did and tried, like this solution, helped. So finally Microsoft Customer Support Services (CSS) was contacted and a case opened. It took some time since the server itself had to be traced and logged TOTALLY. Which couldn’t be done during production hours.

Cause & Solution
Finally a giant log file was created and sent out to CSS for thorough investigation. Soon the answer came in: ‘…changing the size of the desktop heap could fix it…’. CSS referred to two KB articles on how to do that, based on various OS types:

After applying the fix as described in the KB article, all was well again.

So somehow somewhere the heap size wasn’t correct anymore which caused the scripts to fail. It turned out that it wasn’t a SCOM issue at all, but that the SCOM Agent made the issue with the server visible.

Using SCOM to detect heap size issues
The key customer has created a Monitor in SCOM which scans all servers for EventID 21405 and alerts upon it. So whenever a server is having heap size issues they’ll know it and know the fix for it as well.

2 comments:

devi said...

Hi Marnix,

I am not sure how to check the heap size and in the KB, there is no resolution steps.

Could you please provide the steps to check heap size and steps

Marnix Wolf said...

Hi Devi.

The related KB articles do contain steps how to solve this issue. However, don't think to easy about it, since editing the registry it can wreck havoc on your servers. So it's better to contact CSS and follow their instructions. The case I describe in this posting was a tough one and only with good dedicated support from Microsoft CSS and following their instructions, this matter was solved.

Cheers,
Marnix