Main Contents

Advanced WMI Performance Monitor ZenPack for Zenoss Core

March 26, 2008

Zenoss Core (http://www.zenoss.com/) is a capable open source monitoring solution at no cost. They also have Zenoss Enterprise Edition available at a price with more features, including WMI performance monitoring. Before Zenoss, I’ve been using WMI to monitor Windows servers and desktops for quite a while, but the challenge for me has always been finding a good interface to create reports, graphs, monitor, alert on thresholds, etc. I’ve used my own scripts combined with tools such as Cacti, Nagios, and even my own web interfaces to manage WMI data. I finally got around to creating a ZenPack for Zenoss Core to monitor several key performance counters from Windows servers such as CPU utilization and queue, memory paging and usage, disk IOPS and queue length, terminal sessions and more. These monitors are completely agentless, unlike many other methods. The challenge with WMI is most stats are not provided as simple SNMP-type counters or gauges, but have to be calculated using two or more properties and often factor in the previous interval’s values. The result is a very accurate number, regardless of the time interval between queries. This is why most WMI monitoring scripts you’ll find only provide limited performance statistics. Here are some screenshots.

Zenoss WMI CPU Utilization Graph Zenoss Core WMI Memory Usage Graph Zenoss Core WMI Terminal Sessions Graph

Zenoss Core WMI Disk Queue Graph Zenoss Core WMI Memory Paging Graph Zenoss Core WMI Disk IOPS Graph

Zenoss Core WMI Disk Busy Graph

Attached below is the WMI Performance Monitor ZenPack. I’ve tested it with Zenoss Core 2.1.2 and 2.1.3. It also includes a few Multi-Graph Reports.

WMIPerformanceMonitor.zip

Instructions for use: Install it the same way you install any other ZenPack, either from a command line or in Settings - ZenPacks. Once installed, navigate to /Devices/Server/Windows and go to the Templates tab. Click the Bind Templates menu option and highlight WMI Performance Monitors (be sure to CTRL-Click if you have other templates bound already). Also, on the zProperties tab, be sure to set the zWinPassword and zWinUser fields with an account that has administrator access. You should see the new graphs under the Perf tab of your Windows devices. It can take up to 15 minutes to start getting valid data.

This ZenPack is just a sample of what you can do with WMI. For example, you can also do SQL or Exchange monitoring (look here), IIS, 3rd-party products or even your own developed applications. If you like this ZenPack and if you would like to do more with WMI and Zenoss Core, use this ZenPack as a base to make your own or let me know. My company, Sandbox Consulting, Inc., would be glad to offer our services to create custom ZenPacks.

UPDATE Nov 18, 2008: Just tested this with Zenoss 2.3 with great results. Zenoss fixed WMI to work with Windows 2008. To address the most common issues I hear about:

Depending on the Zenoss installation method (rpm vs stack vs source, etc), you may or may not have to modify the path to ‘wmic’ in the Products/WMIPerformanceMonitor/libexec/wmi_stats.pl script, as many have noted in the comments below.

You don’t need SNMP Informant. I normally disable the built-in “Device” template that uses Informant.

Be careful when binding templates to a group or device. Only bind templates you need (e.g. don’t bind them all!) or you will get errors. Verify bound templates with the zDeviceTemplates property in zProperties of each groups/server.

To test WMI, find the wmic command (usually in $ZENHOME/bin/wmic), and run this command:

wmic -U domain/administrator%password //host "Select * from Win32_ComputerSystem"

Run winmgmt /resyncperf on your windows servers

Filed under: Microsoft, Windows, Zenoss |

28 Comments

  1. Ken Granger April 9, 2008 @ 5:09 pm

    Cool stuff - I uploaded via the ZenPack- pointed to the zip file and I get an error everytime I access the ZenPack page:

    An error was encountered while publishing this resource. Please use the form below to submit details of this error to Zenoss, Inc. This information helps us identify and fix issues with the software, though we are unable to respond individually to all submissions.

    The Zenoss community forums are very active and a good resource for solving problems and answering questions. Zenoss also provides commercial services and support packages.

    Type: AttributeError
    Value: ‘BTrees._IIBTree.IITreeSet’ object has no attribute ‘__of__’

    Traceback (innermost last):

    * Module ZPublisher.Publish, line 114, in publish
    * Module ZPublisher.mapply, line 88, in mapply
    * Module ZPublisher.Publish, line 40, in call_object
    * Module Shared.DC.Scripts.Bindings, line 311, in __call__
    * Module Shared.DC.Scripts.Bindings, line 348, in _bindAndExec
    * Module Products.CMFCore.FSPageTemplate, line 195, in _exec
    * Module Products.CMFCore.FSPageTemplate, line 134, in pt_render
    * Module Products.PageTemplates.PageTemplate, line 104, in pt_render

    * Module TAL.TALInterpreter, line 206, in __call__
    * Module TAL.TALInterpreter, line 250, in interpret
    * Module TAL.TALInterpreter, line 711, in do_useMacro
    * Module TAL.TALInterpreter, line 250, in interpret
    * Module TAL.TALInterpreter, line 426, in do_optTag_tal
    * Module TAL.TALInterpreter, line 411, in do_optTag
    * Module TAL.TALInterpreter, line 406, in no_tag
    * Module TAL.TALInterpreter, line 250, in interpret
    * Module TAL.TALInterpreter, line 711, in do_useMacro
    * Module TAL.TALInterpreter, line 250, in interpret
    * Module TAL.TALInterpreter, line 426, in do_optTag_tal
    * Module TAL.TALInterpreter, line 411, in do_optTag
    * Module TAL.TALInterpreter, line 406, in no_tag
    * Module TAL.TALInterpreter, line 250, in interpret
    * Module TAL.TALInterpreter, line 734, in do_defineSlot
    * Module TAL.TALInterpreter, line 250, in interpret
    * Module TAL.TALInterpreter, line 426, in do_optTag_tal
    * Module TAL.TALInterpreter, line 411, in do_optTag
    * Module TAL.TALInterpreter, line 406, in no_tag
    * Module TAL.TALInterpreter, line 250, in interpret
    * Module TAL.TALInterpreter, line 734, in do_defineSlot
    * Module TAL.TALInterpreter, line 250, in interpret
    * Module TAL.TALInterpreter, line 426, in do_optTag_tal
    * Module TAL.TALInterpreter, line 411, in do_optTag
    * Module TAL.TALInterpreter, line 406, in no_tag
    * Module TAL.TALInterpreter, line 250, in interpret
    * Module TAL.TALInterpreter, line 426, in do_optTag_tal
    * Module TAL.TALInterpreter, line 411, in do_optTag
    * Module TAL.TALInterpreter, line 406, in no_tag
    * Module TAL.TALInterpreter, line 250, in interpret
    * Module TAL.TALInterpreter, line 711, in do_useMacro
    * Module TAL.TALInterpreter, line 250, in interpret
    * Module TAL.TALInterpreter, line 426, in do_optTag_tal
    * Module TAL.TALInterpreter, line 411, in do_optTag
    * Module TAL.TALInterpreter, line 406, in no_tag
    * Module TAL.TALInterpreter, line 250, in interpret
    * Module TAL.TALInterpreter, line 734, in do_defineSlot
    * Module TAL.TALInterpreter, line 250, in interpret
    * Module TAL.TALInterpreter, line 426, in do_optTag_tal
    * Module TAL.TALInterpreter, line 411, in do_optTag
    * Module TAL.TALInterpreter, line 406, in no_tag
    * Module TAL.TALInterpreter, line 250, in interpret
    * Module TAL.TALInterpreter, line 653, in do_loop_tal
    * Module TAL.TALInterpreter, line 250, in interpret
    * Module TAL.TALInterpreter, line 677, in do_condition
    * Module TAL.TALInterpreter, line 250, in interpret
    * Module TAL.TALInterpreter, line 426, in do_optTag_tal
    * Module TAL.TALInterpreter, line 411, in do_optTag
    * Module TAL.TALInterpreter, line 406, in no_tag
    * Module TAL.TALInterpreter, line 250, in interpret
    * Module TAL.TALInterpreter, line 501, in do_insertText_tal
    * Module Products.PageTemplates.TALES, line 227, in evaluateText
    * Module Products.PageTemplates.TALES, line 221, in evaluate
    URL: file:ZenModel/skins/zenmodel/viewZenPacks.pt
    Line 46, Column 6
    Expression:
    Names:

    {’container’: ,
    ‘context’: ,
    ‘default’: ,
    ‘here’: ,
    ‘loop’: ,
    ‘modules’: ,
    ‘nothing’: None,
    ‘options’: {’args’: ()},
    ‘repeat’: ,
    ‘request’: ,
    ‘root’: ,
    ‘template’: ,
    ‘traverse_subpath’: [],
    ‘user’: admin}

    * Module Products.PageTemplates.ZRPythonExpr, line 47, in __call__
    __traceback_info__: len(pack.packables())
    * Module Python expression “len(pack.packables())”, line 1, in
    * Module Products.ZenRelations.ToManyRelationship, line 67, in __call__

    AttributeError: ‘BTrees._IIBTree.IITreeSet’ object has no attribute ‘__of__’

  2. Russ April 16, 2008 @ 6:50 am

    Excellent!! This is exactly what I love about community products!! I have just installed this ZenPack and I’m loving it so far.

    Just had one issue… For whatever reason, there were two extra characters at the end of every color in the graph definitions. It was cause the RenderServer to die. I just went through each def and removed the last to hex characters, and all is well!!

    Thanks again for an awesome ZenPack!!

  3. James Dastrup April 16, 2008 @ 8:58 am

    Regarding the extra characters at the end of every color, that’s the alpha layer (transparency), which apparently causes problems for some. I originally was doing some transparent color layers, but it wasn’t necessary, so I just removed it. The ZenPack download link has been updated.

  4. stwainer April 18, 2008 @ 10:46 am

    Could you maybe tag these entries “Zenoss” or “Zenpack” ? They’d be easier to find.

    Thanks for the great work,

    Steve

  5. Josh Baird June 16, 2008 @ 12:29 am

    Quick fix for those running ZenOSS Core 2.2.0:

    - Edit $ZENHOME/Products/WMIPerformanceMonitor/libexec/wmi_stats.pl
    and replace Line 12 with:

    $cmd = “/usr/local/zenoss/common/bin/wmic -U ‘@ARGV[3]‘%’@ARGV[4]‘ //@ARGV[2] “;

  6. giochi casin online June 22, 2008 @ 2:50 am

    Great website, it was actually quite real helpful.

  7. Roberta Dikeman July 14, 2008 @ 1:24 pm

    Hmm - I just keep getting Missing RRD File
    I installed SNMP Informant
    I made the change above for the newer version
    And then made the subsequent changes for the Virtual machine.
    The other (CPU, Free Memory and Paging) performance monitors are working

  8. nuggetbro July 20, 2008 @ 8:54 pm

    Love the pack and works great in most parts. One problem I’m having is the memory usage graph. It appears that it grabs the memory and page file, but measures it again the physical memory only. So it thinks it is running out of memory, when it is barely touching the page file. Any help would be great, thanks.

  9. vijay July 30, 2008 @ 1:20 am

    Hi I am trying to create zenpack to monitor Equllogic,pls help me out to do this.

    Regards,
    vijay

  10. Yuk August 2, 2008 @ 8:17 am

    It works with 2.2.3 after change wmi_stats.pl following above instruction.

    Many thanks for this great work,

    Yuk.

  11. David August 7, 2008 @ 3:44 am

    I am new to zenoss, I installed zenpack, but i am getting Missing RRD file error on the performance graphs, All the basic configurations(SNMP informant,WMI) are set properly.

  12. caio August 7, 2008 @ 1:45 pm

    I would like to know about this Zenoss WMI plugin. I’ve studied how to collect data by WMI from WMS servers using perl scripts on a linux server, but I almost gave up because it seemed to be impossible. I didn’t get nothing about it until this one of Zenoss Core.

    How does it work? Any idea?

    Thanks in advance.

    Caio

  13. stinger August 13, 2008 @ 8:07 am

    Great Tool, Thanks!

    Is it possible to have the TerminalSession
    DataPoints in a Multi-Graph Report.
    While it worked with CPU Load i got no
    results for the total number of terminal sessions.

    regards

  14. David Gitman August 13, 2008 @ 12:51 pm

    I’m also receiving the “Missing RRD File” error. Did anyone manage to resolve this?

  15. stinger August 14, 2008 @ 3:05 am

    worked for me after i bound the template to /Devices/Server/Windows.

    … didn’t read the manual at the first try i guess ;-)

  16. Trevor August 15, 2008 @ 2:33 pm

    This is a great zenpack. One thing, it seems that if I view CPU stats over time(and others), the stats diminish. For example, the CPU is quite high on a server reaching 100% at times when viewing it for hourly/daily. But looking at the weekly stats the spikes are not at 100 anylonger but rather about 60%. Can there be something done to retain the original readings over a long period?

  17. Matt August 25, 2008 @ 12:06 pm

    This zenpack looks great, exactly what I was looking for. Unfortunately, I’m having some difficulties getting it to work right. I’m running 2.2.3 and I made the changes outlined above, but I’m still not getting any information from the graphs.

    The CPU, Memory Paging and Disk Busy Time graphs are all getting a missing RRD file error, and none of the other graphs show up at all. That is, I can see the category, but there’s no graph to go with it.

    I’m running Zenoss on a Debian server, trying to manage several Server 2000/2003 boxes. SNMP and SNMP Informant are installed.

    Any help would be much appreciated, I really would like to use this Zenpack.

  18. Darren Sellers September 8, 2008 @ 1:22 pm

    I’m seeing the following errors in the events of Zenoss.

    Cmd: $ZENHOME/Products/WMIPerformanceMonitor/libexec/wmi_stats.pl MemPage “LC1.CENTRAL.LOCAL” “10.10.0.46″ “central\usernameremoved” “Passwordremoved” - Code: 2 - Msg: Misuse of shell builtins

    severity 4
    eventState 0
    eventClassKey
    eventGroup
    stateChange 2008/09/08 19:33:49.000
    firstTime 2008/09/08 19:33:49.000
    lastTime 2008/09/08 19:33:49.000
    count 1
    prodState 1000
    suppid
    manager localhost
    agent zencommand
    DeviceClass /Server/Windows
    Location
    Systems |
    DeviceGroups |
    ipAddress 10.10.0.46
    facility unknown
    priority -1
    ntevid 0
    ownerid
    clearid
    DevicePriority 3
    eventClassMapping
    monitor

    I’ve run the script manually from the shell and it appears to work fine. I’m confused. Any help with this excellent script would be appreciated.

  19. gianluca September 18, 2008 @ 11:03 am

    Hello all,
    just downloaded and installed zenpack and bound
    (with all the other ones except for the two connected to snmp-informant)
    In my case with 2.2.3 stack installation on a 64 bit rh el 5 I get this after doing the steps above and going to the perf tab of the windows device (a win 2003 R2 SP2 server)
    At the end of the trace you find
    AttributeError: speed
    Any hints?
    thanks in advance,
    Gianluca

    Traceback (innermost last):

    * Module ZPublisher.Publish, line 114, in publish
    * Module ZPublisher.mapply, line 88, in mapply
    * Module ZPublisher.Publish, line 40, in call_object
    * Module Shared.DC.Scripts.Bindings, line 311, in __call__
    * Module Shared.DC.Scripts.Bindings, line 348, in _bindAndExec
    * Module Products.CMFCore.FSPageTemplate, line 195, in _exec
    * Module Products.CMFCore.FSPageTemplate, line 134, in pt_render
    * Module Products.PageTemplates.PageTemplate, line 104, in pt_render

    * Module TAL.TALInterpreter, line 206, in __call__
    * Module TAL.TALInterpreter, line 250, in interpret
    * Module TAL.TALInterpreter, line 711, in do_useMacro
    * Module TAL.TALInterpreter, line 250, in interpret
    * Module TAL.TALInterpreter, line 426, in do_optTag_tal
    * Module TAL.TALInterpreter, line 411, in do_optTag
    * Module TAL.TALInterpreter, line 406, in no_tag
    * Module TAL.TALInterpreter, line 250, in interpret
    * Module TAL.TALInterpreter, line 711, in do_useMacro
    * Module TAL.TALInterpreter, line 250, in interpret
    * Module TAL.TALInterpreter, line 426, in do_optTag_tal
    * Module TAL.TALInterpreter, line 411, in do_optTag
    * Module TAL.TALInterpreter, line 406, in no_tag
    * Module TAL.TALInterpreter, line 250, in interpret
    * Module TAL.TALInterpreter, line 734, in do_defineSlot
    * Module TAL.TALInterpreter, line 250, in interpret
    * Module TAL.TALInterpreter, line 426, in do_optTag_tal
    * Module TAL.TALInterpreter, line 411, in do_optTag
    * Module TAL.TALInterpreter, line 406, in no_tag
    * Module TAL.TALInterpreter, line 250, in interpret
    * Module TAL.TALInterpreter, line 734, in do_defineSlot
    * Module TAL.TALInterpreter, line 250, in interpret
    * Module TAL.TALInterpreter, line 426, in do_optTag_tal
    * Module TAL.TALInterpreter, line 411, in do_optTag
    * Module TAL.TALInterpreter, line 406, in no_tag
    * Module TAL.TALInterpreter, line 250, in interpret
    * Module TAL.TALInterpreter, line 711, in do_useMacro
    * Module TAL.TALInterpreter, line 250, in interpret
    * Module TAL.TALInterpreter, line 426, in do_optTag_tal
    * Module TAL.TALInterpreter, line 411, in do_optTag
    * Module TAL.TALInterpreter, line 406, in no_tag
    * Module TAL.TALInterpreter, line 250, in interpret
    * Module TAL.TALInterpreter, line 711, in do_useMacro
    * Module TAL.TALInterpreter, line 250, in interpret
    * Module TAL.TALInterpreter, line 426, in do_optTag_tal
    * Module TAL.TALInterpreter, line 411, in do_optTag
    * Module TAL.TALInterpreter, line 406, in no_tag
    * Module TAL.TALInterpreter, line 250, in interpret
    * Module TAL.TALInterpreter, line 734, in do_defineSlot
    * Module TAL.TALInterpreter, line 250, in interpret
    * Module TAL.TALInterpreter, line 426, in do_optTag_tal
    * Module TAL.TALInterpreter, line 411, in do_optTag
    * Module TAL.TALInterpreter, line 406, in no_tag
    * Module TAL.TALInterpreter, line 250, in interpret
    * Module TAL.TALInterpreter, line 651, in do_loop_tal
    * Module Products.PageTemplates.TALES, line 206, in setRepeat
    * Module Products.PageTemplates.TALES, line 221, in evaluate
    URL: file:ZenModel/skins/zenmodel/viewPerformanceDetail.pt
    Line 85, Column 4
    Expression:
    Names:

    {’container’: ,
    ‘context’: ,
    ‘default’: ,
    ‘here’: ,
    ‘loop’: ,
    ‘modules’: ,
    ‘nothing’: None,
    ‘options’: {’args’: ()},
    ‘repeat’: ,
    ‘request’: ,
    ‘root’: ,
    ‘template’: ,
    ‘traverse_subpath’: [],
    ‘user’: tekka}

    * Module Products.PageTemplates.ZRPythonExpr, line 47, in __call__
    __traceback_info__: here.getDefaultGraphDefs(drange=drange)
    * Module Python expression “here.getDefaultGraphDefs(drange=drange)”, line 1, in
    * Module Products.ZenModel.RRDView, line 202, in getDefaultGraphDefs
    * Module Products.ZenModel.RRDView, line 64, in getGraphDefUrl
    * Module Products.ZenModel.PerformanceConf, line 233, in performanceGraphUrl
    * Module Products.ZenModel.GraphDefinition, line 427, in getGraphCmds
    * Module Products.ZenModel.ThresholdGraphPoint, line 111, in getGraphCmds
    * Module Products.ZenModel.MinMaxThreshold, line 90, in createThresholdInstance
    * Module Products.ZenModel.MinMaxThreshold, line 110, in getMaxval
    * Module Products.ZenUtils.ZenTales, line 33, in talesEval
    * Module Products.PageTemplates.ZRPythonExpr, line 47, in __call__
    __traceback_info__: here.speed / 8 * .75
    * Module Python expression “here.speed / 8 * .75″, line 1, in

    AttributeError: speed

  20. gianluca September 18, 2008 @ 11:27 am

    Probably it was an error to bind
    ethernetCsmacd template

    but at this point the question is:
    by default how do I understand which templates are bounded????
    I tried resetting the values and binding only this one and it works.
    I get these warning events for
    Perf/Memory, Perf/Disk, ecc
    do I have to install snmp-informant to have this working?

    Cmd: $ZENHOME/Products/WMIPerformanceMonitor/libexec/wmi_stats.pl MemUse “winserver” “winserver ip addr” “.\zenuser” “zenpwd” - Code: 9 - Msg: Unknown error code: 9

  21. David October 15, 2008 @ 10:03 am

    I’m getting the MISSING RRD FILE in the graphs.

    I’m running 2.2.4 (stack installer) on RHEL5, and nothing else except for this performance pack loaded.

    I’ve made the changes to the wmi_stats.pl, and I can run it from the cli:

    ./wmi_stats.pl DiskQ “server” “172.25.5.62″ “domain\user” “pw”

    OK|AvgDiskReadQueueLength=0 AvgDiskWriteQueueLength=0 AvgDiskQueueLength=0

    ./wmi_stats.pl MemUse “server” “172.25.5.62″ “domain\user” “pw”

    OK|CommittedBytes=558985216 AvailableBytes=1333239808 TotalPhysicalMemory=2146713600 PercentPhysicalMemoryUsed=26.04

    I have the snmp-informant loaded on the two servers, and I can view all of the Data Sources and Graph Definitions of the template. I’ve bound the template again to /Devices/Servers/Windows

    I’m not entirely sure what is happening, but I don’t have any error in the logs about a problem(s).

    Thanks.

  22. Alex October 22, 2008 @ 1:18 pm

    I’m having a problem running the 2.2.4 VM on windows, where the wmi data for graphs quit’s collecting….i can sometimes reboot and get it to work, other times have to re-set wmi credentials….either way, will collect chart data for a while, then stop. any ideas on what i need to adjust to prevent this from happening? any ideas on where to look to determine the problem?

    -ac

  23. rahul November 16, 2008 @ 10:11 am

    Hello,

    I have used this tool and it is really good and appreciate the great work. I have a question on this. If I have three different windows domains in my company, how can i monitor the machines that come under all 3. I was able to key in zWinPassword and zWinUser only for one domain.

    Please help. thanks

  24. Fiss November 27, 2008 @ 9:03 am

    Rahul,

    You must create a class for each different Windows domain.

    Fiss

  25. Linda December 11, 2008 @ 12:03 pm

    The ZenPack is great! Thank you!
    I was wondering if there is a way to hide the password from being logged/emailed when there is an event related to this ZenPack (e.g. if the command isn’t working for one reason or another) since its a security risk to have passwords emailed around, especially when they’re for an account with admin perms?

    Thanks!

  26. Mario December 24, 2008 @ 11:39 am

    I am having an issue with the script when I have a password that contains either a “$” or an “@” symbol. The password isn’t interpreted properly and the account gets locked out.

  27. John December 30, 2008 @ 7:26 am

    Grean ZenPack. I am running it on 2.3.2 but like Alex it quits collecting after a while. If I run zenoss restart, collection resumes and stops after a while. Again, any ideas much appreciated.

    Keep up the good work!

  28. Chinthaka January 12, 2009 @ 1:17 pm

    Hello,

    I am new to Zenoss Core - I installed this Zen Pack to obtain a better suite of WMI Monitoring. I followed the instructions said above but no new graphs came up on ‘Perf’

    I am using Core 2.3.

    Please help.