Wednesday, January 28, 2015

OM12x: Changing Passwords OM12x Service Accounts

Why this posting?
Normally I advise my customers NOT to change the OM12x Service Accounts. Instead create the required OM12x Service Accounts with passwords which are impossible to remember, store them in the dedicated account/password application, like KeePass. Set the accounts so their passwords never expire and be done with it.

In many situations this works like a charm. However, there are organizations where this approach is simply forbidden. They have to adhere to all kinds of laws, legislations, rules and so on. In worlds like this service accounts with a ‘set & forget’ approach aren’t allowed at all and will make their security audits go wrong.

In a world like that it’s required that the passwords used by the OM12x service accounts are renewed, once per 100 days for instance. And yes, this can be done and Microsoft even provides procedures for it about how to do just that. But like many other procedures, they aren’t spot on. Therefore I’ve decided to write this posting, all about changing the passwords of the OM12x service accounts AND to have a fully functional SCOM 2012x environment again.

What to do?
There is a lot to do. As well as preparing everything and executing it. Therefore I’ve divided it in three parts: Preparation, Execution and Aftercare.

A: Preparation
During this phase you’re not going to change anything. You just make sure you’ve got everything in place and ready for the change itself. This isn’t about technology only, but more about processes, people & organization.

01: Change Management
Yes. I know. Change Management can be a pain in the backside. But still many companies use it (thankfully!) and it helps to prevent outages and other not so nice things. So simply follow the rules of your company. Or be toast…

Modification of the passwords of the SCOM 2012x Service Accounts IS a change. So treat it a such and plan to do it outside the regular production time. Or plan it during a time frame when a short outage of the SCOM environment is acceptable.

Since it’s something which comes back on a regular basis, make a template out of the Change Request. This way it’s far more easier to follow the change management procedure. Store that document in your SharePoint portal and notify your team about the existence of this document.

During this phase you allocate resources (who’s going to do what) and also plan a date and a time (who’s going to it when). It also helps you to see whether or not this change doesn’t coincide with other changes/actions which don’t allow the monitoring solution to be unavailable for some time.

02: Run an inventory
Know what you have and where. So make an inventory of your SCOM 2012x MG:

  1. What are the SCOM 2012x Management Servers (MS) ?
  2. What SCOM 2012x MS servers host the SCOM Web Console?
  3. Are there any SCOM 2012x Gateway Servers?
  4. Which SQL servers/instances host the Operational SQL database?
  5. Which SQL servers/instances host the Data Warehouse database?
  6. Which SQL server/instance hosts the Reporting Services for SCOM 2012x Reporting?
  7. What is the Active Directory SCOM Agent Action account?
  8. What is the Active Directory SCOM SDK/Data Access/ Configuration account?
  9. What is the Active Directory SCOM DW Read account?
  10. What is the Active Directory SCOM DW Write account?
  11. Are there any 3rd party service accounts (e.g. Veeam) which require password modifications as well?
  12. Are there any 3rd party servers used for extended monitoring (e.g. Veeam) which require attention as well?

Since this kind of information is required every time when a password change is due, simply DOCUMENT it, use good versioning, also store it on your SharePoint portal (DON’T write the passwords down in the same document!!!) and notify your team about this document. And when something changes (like newly added Gateway or MS Servers) update this document accordingly.

This way you’ve got a single place where you can find this information.

03: Write a procedure for the password change
Make a document, describing high level the required steps and their order. In the same document describe every step in details – with screenshots of the most ‘exciting’ parts – so this knowledge/experience is transferable to members of your team.

And when executing the change, use this document and modify it as required. Also store this document on your SharePoint portal and notify your team about the existence of this document.

04: Read, read and read
There are many resources available on the internet about this password change. So READ it. This helps you not to wander into the unknown.

05: Test yourself before you wreck yourself
When you’re lucky to have a SCOM 2012x test environment in place, you can test the procedures there before using them in your production environment. But please be aware that many times SCOM 2012x test environment many times do not represent your SCOM 2012x production environment. Things can be missing or less distributed, like a single SQL Server instance, hosting BOTH SCOM 2012x databases and running the SQL Server Reporting Services for SCOM Reporting as well, while in production this is hosted by many more SQL Server instances and perhaps even in a high available scenario.

06: Garbage in, garbage out…
Meaning: When your SCOM environment is already having issues. DON’T change the passwords!!! First FIX those issues so your SCOM environment is healthy BEFORE you change the passwords. Might sound stupid to mention it, but you would be surprised….. Sad smile

Recap phase A – Preparation
As you can see – except for Step 05 that is – you haven’t done anything technical. No geek stuff. Writing documents, filing a request for change and stuff like that. But this is VERY important. So do it for a full 100% since it will be the foundation for the actual password change itself.

B: Execution
In this phase you’re going to change the passwords of the SCOM 2012x service accounts. High level overview of the steps involved:

  1. Change the passwords of the SCOM 2012x service accounts in AD;
  2. Change the passwords of the SCOM 2012x service accounts for the related services on the SCOM 2012x MS servers;
  3. Change the passwords of the SCOM 2012x service accounts in the SCOM Console (Run-As-Accounts);
  4. Change the passwords for SCOM Reporting in Reporting Services Configuration Manager Console;
  5. Change the passwords in IIS for the related Application Pools (when affected);
  6. Change the passwords in 3rd party SCOM 2012x tooling/MPs/Add-ons (when affected).

Before you start, empty the Operations Manager event logs on the SCOM 2012x MS servers. So it’s easier to see what’s happening after the password change in AD. Also open multiple services.msc consoles, each one connected to one SCOM 2012x MS server. This makes it easier to change the passwords for the related SCOM 2012x services in a single RDP session, instead of switching to different RDP sessions.

01: Change the passwords in AD for the SCOM 2012x service accounts
This one is straight forward. I am NOT going to explain this one Smile
image

02: Change the passwords for the SCOM 2012x services on the SCOM 2012x MS servers
In the services.msc console, modify the passwords as required for the services System Center Data Access Service and System Center Management Configuration. Do this for ALL SCOM 2012x MS servers!

  1. Open the properties of the service, go to the tab Log on, and change the password
    image
  2. Apply it and restart the service.
  3. Repeat Steps 1 and 2 for the other service. Repeat this per SCOM 2012x MS server.

03: Change the passwords in the SCOM 2012x Console (Run-As-Accounts)
Now you need to open the SCOM 2012x Console, log on with SCOM admin permissions.

  1. Administration pane > Administration > Run As Configuration > Accounts
  2. Select Action Account > your SCOM Action Account (in this example SC\OM12Action) > open it > tab Credentials > modify the passwords > Apply
    image
  3. Repeat this for all SCOM 2012x accounts in your Console like Windows > Data Warehouse Action Account and Windows > Data Warehouse Report Deployment Account.
  4. Check ALL the Run As Accounts since this configuration can be different per MG!
  5. When done, RESTART ALL SCOM 2012x MS Services on your SCOM 2012x MS servers, Health Service (Microsoft Monitoring Agent) included. This will make SCOM process the changes faster. Also some errors will go away since SCOM will apply the password changes.

04: Change the passwords for SCOM Reporting in Reporting Services Configuration Manager Console
Now it’s time to open Reporting Services Configuration Manager Console for the SQL Server instance hosting this service for SCOM 2012x Reporting and to change the passwords there as well.

  1. Open Reporting Services Configuration Manager Console and connect to the SQL Server instance hosting SSRS for SCOM 2012x Reporting;
  2. Go to Service Account > modify the password > Apply
    image
  3. Check the Results pane in order to see all went well:
    image
  4. This is a beast when you forget it because Reporting won’t run anymore. You’ll get errors like these when trying to connect to Report Manager:
    image

    And when connecting to Reporting Services:image

    So go to Database > Change Credentials
    image

    Hit the button Test Connection and when you get an all is okay message > OK > Next
    image

    Enter the new password > Next
    image

    Summary page > Next > Check the status when finished, it MUST be Success > Finish.
    image
  5. Go to Execution Account > modify the password > Apply
    image
  6. Check the Results pane in order to see all went well:
    image
  7. Open the services.msc console and open the properties for the SQL Server Reporting Services (YOUR SQL SERVER INSTANCE), in this example SQL Server Reporting Services (MSSQLSERVER). Go to the tab Log on and modify the password > Apply. Don’t forget to restart that service!
    image
  8. All should be okay now. So simply test it in the Reporting Services Configuration Manager Console at the Web Service URL entry and Report Manager URL entry by clicking on their URLs:
    image
    Result (when all is okay Smile):
    image

    And:
    image
    Result (when all is okay Smile):
    image

    When something isn’t okay, check all previously mentioned steps again. This approach should work.
  9. Close the Reporting Services Configuration Manager Console.

05: Change the passwords in IIS for the related Application Pools
Only when affected. So when you’re running 3rd party add-ons like the Remote Maintenance Mode Scheduler for instance.

  1. Open IIS Manager and connect to the server which is hosting the Application Pool which requires the password modification > Application Pools. Now you see a complete list of Application Pools and under what account they run. Mostly it’s ApplicationPoolIdentity but sometimes a SCOM 2012x service account is being used:
    image
  2. Right click on that Application Pool using the SCOM 2012x service account credentials > Advanced Settings > Process Model > Identity > click on the button with the three dots > Set and enter the correct credentials
    image
  3. > OK > OK > OK. Restart this Application Pool.
  4. Test it in the IIS Admin console: Go to Sites > Default Web Site (or any other main web site under which the SCOM 2012x web applications/sites are placed) > select the Application Pool you just modified > right click > Manage Application > Browse. IE will open the URL now and when all is okay, the website will be properly loaded.
  5. Repeat Step 4 for ALL applications in order to see all is okay:
    image
  6. When you’re having issues with the SCOM 2012x Web Console: relax. That website is a pain but easy to fix:
    - Open an elevated cmd-prompt and type IISRESET. Wait until IIS is running again. Test the SCOM Web Console again. When not okay, go to this posting of mine, run the commands as discussed AND run an IISRESET afterwards. Then all will be okay again. When the Web Console is still broken, go to this blog posting of Bob Cornelissen and follow his advice. And again, relax. The SCOM 2012x Web Console is a pain and breaks very easily.

06: Change the passwords in 3rd party SCOM 2012x tooling/MPs/Add-ons
This one is straight forward. Simply follow the guidelines outlined by the 3rd party about how to do that Smile. When you can’t find those guidelines, call their support team and let them send you the required manual.

Recap phase B – Execution
Wow! That was some serious work. As you can see, changing the passwords of the SCOM 2012x service accounts isn’t like a walk in the park. It needs some serious attention. In the last phase you’re going to test whether SCOM is operating like it was before the password change.

C: Aftercare
In this phase you’re going to check AND double check in order to see EVERYTHING is all right before you sign it of and tell the team the change has gone well while some stuff is broken… Ouch! Bad for your reputation and perhaps even your career.

So: CHECK YOURSELF BEFORE YOU WRECK YOURSELF. (Pete Zerger and Anders Bengtsson taught me this slogan during a MMS conference, and I love it).

01: OperationsManager event logs on the SCOM 2012x MS servers
Those logs? I love them since they tell me so much about the current state and health of the SCOM environment. No need to open the SCOM Console (just yet!).

So check the event log and see whether there are no Warnings or Criticals requiring immediate attention like not being able to store data in the Data Warehouse. Run this check on ALL your SCOM MS servers.

02: SCOM Console
Open the SCOM Console. Does it work? No SDK errors? Phew! At least you got that part right Smile. But this is only the start of all your checks, there is much more:

  1. Monitoring > Active Alerts
    Check whether new Alerts do come in. How old are they? Can you open them? Can you modify the resolution state? Can you reset them (Monitors), can you close them (Rules)?
  2. Monitoring > Operations Manager > Management Group Health.
    Is everything green? And when not, open Health Explorer and investigate AND fix it.
  3. Reporting > Reporting
    Give the reporting tree a refresh (F5). Does it show the available reports? Can you run the reports? No errors?
  4. Administration > Administration > Settings > Web Address
    Can you open the SCOM Web Console? No errors? Can you browse in it? Can you open the Web Console based Health Explorer? Can you open the properties screen of an Alert?
  5. Administration > Administration > Settings > Reporting
    Open it, copy and paste the URL in IE. Can you open it in IE? Can you browse it?

03: IIS Admin console
Even though you have already tested during the execution phase, test it AGAIN. Always better to know for a full 100% all is okay than some else is telling you it doesn’t… Ouch!

  1. Go to Sites > Default Web Site (or any other main web site under which the SCOM 2012x web applications/sites are placed) > right click the first application on top of the list > Manage Application > Browse. IE will open the URL now and when all is okay, the website will be properly loaded.
  2. Repeat Step 1 for ALL applications in order to see all is okay:
    image

04: 3rd party add-ons, consoles, MPs, tooling and what ever you’ve got
Test those as well so you know those are okay as well.

Recap phase C – Aftercare
Congrats! Time to sign it off. The password change of the SCOM 2012x service accounts didn’t wreck your SCOM environment. Awesome! Now you can tell your team SCOM is up & running again AND fully functional.

Total recap
As you can see, modifying the passwords of the SCOM 2012x service accounts isn’t rocket science and yet it requires thorough preparations and once started, you need to follow a certain order and not stop when half way through since it will result in a (temporary) broken SCOM environment.

The first time is always the biggest challenge. But when you make good documentation, keep them up to date, store them in your companies SharePoint portal AND notify your team about the existence of those documents, you’ll see that the next time it will be easier to do.

4 comments:

screamer333 said...

I suggest to use MSA or gMSA on SSRS but unfortunattely we can't use with scom itself :(

Richard Kinser said...

This is a timely article. I just went through this process last weekend. Everything went smoothly except for the Reporting server. Mind you, I had done this once a few years ago already, so I was familiar with the process.

Sanaullah Mir said...

Could not get easier than this...
Thumbs up!!!

gopi krishnan said...

Good...