SCOM/OM12 Notification Errors: ‘Failed to send notification’ & ‘Failed to send notification using server/device’

The Case
Bumped into this issue at a customers location. SCOM R2 CU#5 is in place and SMTP as Channel fully configured and operational for sending out Notifications through e-mail by using an Exchange 2003 environment. However, when the new Exchange 2010 environment was used, and the SMTP Channel reconfigured in order to use the new Exchange 2010 environment the Alerts weren’t send out any more and these two errors were shown in the SCOM Console:

1. Failed to send notification
   Notification subsystem failed to send notification over 'Smtp' protocol to 'xxx@yyyy.nl'. Rule id: Subscription [GUID]
   
   
2. Failed to send notification using server/device
   Notification subsystem failed to send notification using device/server '[FQDN MAIL SERVER]' over 'Smtp' protocol to 'xxx@yyyy.nl'. Microsoft.EnterpriseManagement.HealthService.Modules.Notification.SmtpNotificationException: Mailbox unavailable. The server response was: 5.7.1 Client does not have permissions to send as this sender. Smtp status code 'MailboxUnavailable'. Rule id: Subscription [GUID]

What Caused it?
Even though one would blame Exchange 2010 at a first glance (Duh! It worked with Exchange 2003!) SCOM needs some investigation as well. Yes, Exchange 2010 is different compared to Exchange 2003. Also for it’s security which is far more sophisticated compared to Exchange 2003. And we do NEED that additional security since we don’t want too much spam nor any kind of mail relaying and other not so funny stuff.

The second Alert tells it all actually: ‘…The server response was: 5.7.1 Client does not have permissions to send as this sender…’. So tightened security is causing this issue.

The Solutions – First Phase
Many times in SCOM e-mail notifications work out of the box without additional configuration. But with Exchange 2010 some additional steps might be required in SCOM, like these ones:

1. The RMS server has to be trusted by the Exchange 2010 environment for sending out e-mail Alerts;
2. The RMS server has to use a special AD account for authenticating to the Exchange server.

Step 1 is easily configured by the Exchange Admins. They know what you mean and better, now how to go about it.

Step 2 has to be done within SCOM itself. A new Run-As-Account (Windows based!) has to be added and mapped to the Run-As-Profile Notification Account. This AD based account will be used by the RMS to authenticate itself to the Exchange server. When this Run-As-Profile isn’t configured, the RMS will use the RMS Action Account by default.

Normally these two steps are the solution to the earlier mentioned issue. However, in this case it wasn’t. The two errors kept coming back. The Exchange admins stated all was well and OK on their side of this story. So before blaming their environment additional investigation in SCOM was at order.

The Solutions – Final Phase
When a Channel for e-mail is configured, one needs to configure a Return Address as well. In 99% of those case, a bogus return address works just fine. Many times I use a bogus Return Address in this format noreply@scom.nl:

Since all the other components for sending out Alerts through e-mail were configured by the book, I decided to replace the Return Address by a real one. Saved the Channel and YES!!! It works!

Conclusion:
When ever your SCOM environment refuses to send out Alerts through e-mail and both Alerts are shown in the SCOM/OM12 Console try these THREE steps and you’re OK:

1. The RMS server has to be trusted by the Exchange 2010 environment for sending out e-mail Alerts;
2. The RMS server has to use a special AD account for authenticating to the Exchange server;
3. Replace the Return Address for a real one.