Friday, October 18, 2013

SCOM 2012 Disaster Recovery: It Works BUT Forget The Line Breaks!!!

Have had my piece of SCOM today.

Wanted to upgrade a very important environment and it went bad. The single Management Server (I know…) was wrecked. The upgrade rolled back leaving me with nothing: the SCOM 2012 SP1 Management Server functionality was gone. Totally.

So it was time for Disaster Recovery as described here. First I restored the SCOM 2012 SP1 SQL databases from backup. And now I did something which I shouldn’t have done: I COPIED and PASTED this section into a cmd file:
image

Of course I added my information to it and ran the file. But all I got were errors in the log file (OpsMgrSetupWizard.log):
image

And this surprised me! And no matter what I tried I didn’t get passed it. Contacted some fellow MVP buddies and they also didn’t know what went wrong.

Finally I had a clear moment: remove all the line breaks so it becomes a single command line, like this:
image

And guess what? Now it worked! So within ten minutes I had myself a new SCOM 2012 SP1 Management Server enabling me to reinstall the Console, Web Console and Reporting Smile.

Back in business.

Lessons learned:

  1. Never EVER copy & paste but put it into a single command line;
  2. I do have some really good friends, in random order: Bob Cornelissen, Stanislav Zhelyazkov, Cameron Fuller, Kevin Holman, Flemming Riis and Oskar Landman. Awesome guys! Thanks for your effort and assistance in cracking this one.
  3. AND PLEASE READ THIS POSTING AS WELL SINCE YOU NEED IT TO GET IT ALL UP AND RUNNING AGAIN AND TO CRACK NAGGING EVENTID 29120

7 comments:

X Wls said...

I'm new to SCOM and your blog is great!

I've been commissioned to reinstall a SCOM 2012 SP1 test environment and CHANGE the Management server group . I thought I had everything covered, but when I performed the uninstall and reinstall of the Mgt server, I ran into a problem when it was time to add the databases. I want to use the original "OperationsManager" database from before, but it won't allow me to...I get the error "specific name of DB in use please provide a different DB name".
I'm wondering how do I recreate this environment, with a different Management group name, and use the SAME operations database since it has all the data?

I read that I may need to move the database to a new server, but this database has to stay on this particular SQL server so I can't use this option.

In your opinion, how do you think I should handle this situation?

Thanks again for your time and input.

Marnix Wolf said...

Hi X Wis.

Renaming a Management Group doesn't work at all. That will never work. The only way is brand new installation using NEW databases.

MGs have two things which can't be changed afterwards: domain membership and their MG name. Neither one can't be modified in any kind of way.

Cheers
Marnix

Aaron Jones said...

Wow, so I'm not the only one this happened to!!

Tried the upgrade to R2 and it bombed. Rolled back and wiped out everything!
Luckily this is a Dev environment and I had a second management server.

The problem I am having is that when I do the recovery I get an error saying that the DB is incompatible with the version of SCOM I was trying to recover with. (2012 SP1) Quick Google search says that the DB is now R2, while all the other SCOM elements seem to be gone completely.

Any thoughts on this? Did you see this at all when you did your recovery?

Marnix Wolf said...

Hi Aaron.

No, you're not the only one.

What you have makes it more difficult since the OpsMgr DB is already on OM12 R2 level.

You could try to do a disaster recovery based on those bits. However I don't know whether the OpsMgr DB is totally on 2012 R2 level or only partially.

Same goes for the DW database. So best is to restore those DBs (both of them) from backup so you're sure those DBs are on the same level and not on R2.

Then run a disaster recovery.

Hopefully that works out.

Cheers
Marnix

Aaron Jones said...

Thanks for the response.

I went ahead and followed the disaster recovery procedure and built a new server. Restored DBs, ran recovery etc...

It's back up and running. Now to get the management server I didn't have issues with out of the grey state!

Michael Brown said...

Marnix Wolf mentioned that domain membership was another restriction on restoring the MG. So does this mean i cannot restore my production SCOM to a test domain? The DR document recommends using the same MS name when rebuilding SCOM for a restore. This limit the ability to test the DR.

Any advise would be appreciated

Mike

Marnix Wolf said...

Hi Micheal,

This is Marnix here :).

Yes, SCOM is fully AD integrated. So you CAN'T restore your Management Group in any other domain.

When you want to test DR, it's far more better to build a second MG in your test AD and run a DR there.

Also this environment can (better: should) be used for testing new MPs and so on.

Cheers,
Marnix