nmon2graphite : Graph nmon output in real time

Download nmon2graphite by clicking here

Current nmon2graphite version is v0.1a. This is the official page for nmon2graphite. This page will be updated with future releases of nmon2graphite. New features, bugs, and anything related to nmon2graphite will be present in this page. If you find bugs (and I’m sure there is a lot of bugs) feel free to left a comment on this page or send me an email. Current source will be also available on github at this address https://github.com/chmod666org/nmon2graphite


Since nmon is integrated to AIX and available on Linux, it is one of the most used performance tool. The community has grown and some third part tools are developed to graph nmon output. I’m sure everybody knows nmon analyzer, nmon consolidator and Nigel’s nmon2rrd. A few months ago Ben Rockwood from Cuddletech introduce me to graphite with a post talking about using graphite to graph dtrace metrics. The idea of using graphite to graph nmon’s output comes from this post. If you haven’t yet heard of graphite let me explain you what it is. Graphite is a tool to produce graphs ; just a like rrdtool data are stored in a round robin database called whisper. One cool thing about graphite is that you can feed the database trough a TCP port, just by sending some formatted data into a socket, you can use what ever you want, for example, netcat can be used to send data to this socket. I personally use a perl script for nmon2graphite. This collection daemon is called carbon.

nmon2graphite global overview

The idea of nmon2graphite is to feed graphite with nmon’s outputs. nmon2graphite client takes a nmon file as argument, you can use your existing files and send them to graphite. Even better nmon2graphite can feed graphite in real time. To do that you have to deploy a perl script on each lpar you want to graph and modify root user crontab, here is how nmon2graphite works :

  • A fifo stack is created at midnight every day.
  • nmon is launched to write into this fifo stack at midnight every day.
  • nmon2graphite is launched at midnight every day.
  • nmon2graphite unstack the fifo stack over time.
  • nmon2graphite parse every line of the fifo stack and transform it into a graphite data.
  • These data are send trough a socket to carbon, and are stored into whisper.
  • On another server, a jquery and perl script are used to query whisper and produce graphs into a web page trough graphite render.

Client Side

If you want to graph nmon output with nmon2graphite you’ll have to choose if you want to graph output in real time or not. Follow the steps described below : copy the nmon2graphite binary on the lpar and modify the crontab to launch it every day. I’m sure some of you are already running nmon and save nmon output every day in a file. If you do not want to use nmon2graphite with real time feature you can use those files to feed graphite database

Enabling nmon2graphite with real time feature

  • Upload nmon2graphite perl script on your lpar :
  • # ls /tools/perf/bin/nmon2graphite
    /tools/perf/bin/nmon2graphite
    
  • Verifiy you can “talk” with your graphite box on port TCP 2003 :
  • # GRAPHITE_BOX="10.10.10.10"
    #  telnet $GRAPHITE_BOX 2003
    Trying...
    Connected to 10.10.10.10.
    Escape character is '^]'.
    ^]
    
  • Modify root user crontab to create a fifo stack and to start nmon (specify fifo stack file as nmon output file) :
  • # crontab -l
    00 00 * * * /usr/bin/mkfifo /tools/perf/nmon/$(date +\%Y-\%m-\%d-\%H-\%M).nmon.fifo ; /usr/bin/nmon -F /tools/perf/nmon/$(date +\%Y-\%m-\%d-\%H-\%M).nmon.fifo -s30 -c2880 -t >/dev/null 2>&1
    
  • Modifiy root user crontab to start nmon2graphite 10 seconds after the creation of the nmon’s fifo stack (nmon2graphite reads standard entry) :
  • # crontab -l
    00 00 * * *  sleep 10 ; /tools/perf/bin/nmon2graphite -i 10.10.10.10 -p 2003 -l /var/tmp/$$.log < /tools/perf/nmon/$(date +\%Y-\%m-\%d-\%H-\%M).nmon.fifo
    
  • Modify root user crontab to wipe old fifo stacks :
  • # crontab -l
    0 1 * * * find /tools/perf/nmon -type f -mtime +30 | xargs rm -f >/dev/null 2>&1
    

Using existing nmon file to feed graphite

  • Go to your directory containing nmon's output files;
  • # ls *.nmon
    2013-04-13.nmon  2013-04-14.nmon  2013-04-15.nmon  2013-04-16.nmon
    # ls -l *.nmon | awk '{print "/tools/list/perf/bin/nmon2graphite -i 10.10.10.10 -p 2003 -l /var/tmp/$$.log < "$NF}' | sh
    

Server side : nmon2graphite web interface

On a web server copy all nmon2graphite files in a directory, an let the web server serve this directory. Mine is running in a Workload partition :

nmon2graphite web interface configuration :

  • Modify nmon2graphite.js with the graphite ip or url, you can also choose nmon interval, mine is set to 30 seconds
  • # cd /app/nmon2graphite
    # ls
    anytime.css        data               graphlist.txt      index.cgi          jquery-migrate.js  jquery-ui.js       nmon2graphite      nmon2graphite.js
    anytime.js         do.pl              images             jquery-form.js     jquery-ui.css      jquery.js          nmon2graphite.css
    # grep -E "var graphite_url|var nmon_interval" nmon2graphite.js
      var graphite_url = "10.10.10.10";
      var nmon_interval = "30";
    
  • Modify nmon2graphite with the graphite ip or url, graphite port, and log file :
  • # more nmon2graphite
    [..]
    my $graphite_box_ip    = '10.10.10.10';
    my $graphite_box_port  = '2003';
    my $graphite_box_proto = 'tcp';
    my $log_file           = '/tmp/nmon2graphite.log';
    my $ignorehdisk        = 0;
    [..]
    

httpd configuration :

Setup http server, in my example I'm using apache. Verify you can reach nmon2graphite web interface :

# more httpd.conf
[..]
  Alias /nmon2graphite/ /app/nmon2graphite/
  
    Options None
    DirectoryIndex index.cgi
    Options +ExecCGI
    Order allow,deny
    Allow from all
  
[..]

Graphite side

I will not tell you how to install graphite, you can find a nice documentation on graphite website (http://graphite.wikidot.com/documentation). I'm just giving here a few tricks to setup graphite for nmon2graphite.

storage-schemas.conf

Modify storage-schemas.conf to fits to your needs, I'm using an nmon interval of 30 seconds and I want to keep data for seven days. An higher retention time will results in bigger whisper files :

# pwd
/opt/graphite/conf
# tail -3 storage-schemas.conf
[nmon]
pattern = ^nmon\.
retentions = 30s:7d

Colors : graphTemplates.conf

Graph style can be modified here. Here is my configuration :

# cat graphTemplates.conf
[default]
background = black
foreground = white
majorLine = white
minorLine = grey
lineColors = #0033cc,#f8de81,#944b90,#e1316f,#fffe0d,#ff9a04,#59e898,#0395d0,#dd4218,#df1932,#888371,#413f4c,#e7dcd6
fontName = Sans
fontSize = 10
fontBold = False
fontItalic = False

Whisper file creation : MAX_UPDATES_PER_SECOND

If you are feeding graphite with already existing nmon files a lot of whisper databases are created in the same time, by default graphite does not allow to create more than 500 whisper databases in one second, raise this limit by adjusting MAX_UPDATES_PER_SECOND in carbon.conf file. These whisper databases creation can be I/O intensive :

# pwd
/opt/graphite/conf
# grep ^MAX_UPDATES_PER_SECOND carbon.conf
MAX_UPDATES_PER_SECOND = 100000

Set TIME_ZONE :

Change graphite TIME_ZONE with your timezone, mine is Europe/Paris :

# pwd
/opt/graphite/webapp/graphite
# grep ^TIME_ZONE local_settings.py
TIME_ZONE = 'Europe/Paris'

Set LEGEND_MAX_ITEMS

Graphite automatically hides legend when number of items is to important, raise this limitation by setting LEGEND_MAX_ITEMS :

# pwd
/opt/graphite/webapp/graphite
# grep ^LEGEND_MAX_ITEMS settings.py
LEGEND_MAX_ITEMS = 1000

Creating you own graphs

The nmon2graphite web interface provides you a list of graph working on AIX lpars, but you can add your own ones by modifying the file graphlist.txt. Add a line to this file to create you own graph and a new checkbox will appear in the nmon2graphite web interface :

# grep ^disk_busy graphlist.txt
disk_busy:Disk busy:target=cactiStyle(legendValue(aliasByNode(nmon.PPPPP.LLLLL.diskbusy.*,4),"avg"))&height=300&width=1000&format=png&title=Pserie PPPPP Lpar LLLLL | Disk busy&vtitle=Percent&from=SSSSS&until=UUUUU
  • A graph line is separated by ":", here are values needed :
  • A name with no spaces.
  • A title (display next to the checkbox in the web interface)
  • The graphite api target to reach the graph, PPPPP is the frame name, LLLLL is the lpar name.