Tuesday, July 27, 2010

Empty graphs and reports for NICs based on HP Teamed NIC

28-07-2010 Update: This blog posting describes how to work around it.

Bumped into this issue at a customer. SCOM R2 is deployed, many MPs in place, among them the Server OS MP (of course). Also the Reporting component is installed.

Now came the time to create/publish some good reports, one of them being a Performance Report. This report shows on a per server basis the performance of the disks, cpu’s, memory and NICs. All went just fine until some Server reports did not show any data related to the NICs:

Strange. Time to take a deeper dive. I opened the Console the Monitoring Wunderbar > Microsoft Windows Server > Performance > Network Adapter Utilization. I selected a Chart at random and selected the related NICs of the server which showed no data in the earlier mentioned Report. And indeed, no data to be shown. Nothing. Selected another server and now data was shown.

What I noticed however is that the Chart gave me no option to check the relevant NIC of the server. Some NICs were shown but not all. So time to see whether the Discovery process was neatly in place.

In the Console > Monitoring Wunderbar > Discovered Inventory > right click in the middle pane and select Change Target Type. Set it to Windows Server 2008 Network Adapter and click OK. Now the View will show only the discovered Windows Server 2008 Network Adapters.

And here the related NIC was shown! So the NIC was neatly discovered by SCOM R2 but no Collection Rules are running against it. But now I started to see a pattern since the NIC having issues is a teamed NIC, created with 3rd party software. So officially speaking, it is NOT a NIC as seen from a Windows standpoint of View, but a virtual one created by 3rd party software and presented to the OS as a normal one…

So I went back to the reports and the same issue was happening when I targeted teamed NICs.  All other NICs were just fine and made nice graphs in the Reports.

Time for another check. I started PerfMon on the server with the teamed NICs and selected the Counter Network Interface. And now it all became clear: only the physical NICs were shown not the teamed NICs (two in total). Even ran the tool Extensible Counter List tool, (Exctrlst.exe) just to be sure, but it changed nothing.

What is basically comes down to is this:

The Server OS does not monitor teamed NICs since these are created by 3rd party software and therefore not ‘real’ Windows NICs as such.

The server experiencing this issue is a HP server. Even with the HP Proliant MP loaded in SCOM R2 it does not collect any performance data at all :(. So unfortunately, for this server no performance collection for its NICs will take place. On the other hand, many times NICs aren’t really bottle necks any more. CPU’s, disks and RAM are and performance data for those parts is neatly collected.

After story:
After some searching I found this thread on the TechNet OpsMgr Forums. Good to know I came to the right conclusions.


Anonymous said...

Hmmm... Is this a Windows 2008 thing perhaps, because we've got a good 36 HP ProLiant's showing their Teamed NIC stats in the Monitoring view but they're all running Win 2003. All our Win 2008/R2 servers are virtual. :)

Kevin Holman said...

One thing to add - you will see a LOT of errors in tghe OpsMgr event log when the Healthservice initializes about these - in the form of a 10102 event.... because we are trying to collect performance data about a discovered instance - but the instance does not exist in perfmon.

It would be good to create a group of these instances... and disable monitoring for them, or better yet find a way to not discover them.

Marnix Wolf said...

Hi Steve.

Hmm. That is interesting to know. So perhaps it is an issue with W2K08 servers only. Not the MP since the MP discovers the teamed NIC but Windows since the related instance of the perfom counter is not shown.


Marnix Wolf said...

Hi Kevin.

You are right. The OpsMgr eventlog shows those events indeed. But I am working on workaround without taking a deep dive into the programmatic side of SCOM. As soon I get any results I will let you know and when time permits, I will blog about it.


Lars Oeschey said...

This definately is a 2008R2 problem. I'm currently facing the same thing, however I try to use nagios with performance Counters, and the Team doesn't show up. Guess there's no solution for this besides the OpsMgr Solution (which doesn't help me ;))

Marnix Wolf said...

Even though I haven't seen on it Windows 2003 servers, since many environments I work with are mostly 2008 based, I think the same issue is at play there as well.