nmon2graphite : Graph nmon output in real time

Download nmon2graphite by clicking here

Current nmon2graphite version is v0.1a. This is the official page for nmon2graphite. This page will be updated with future releases of nmon2graphite. New features, bugs, and anything related to nmon2graphite will be present in this page. If you find bugs (and I’m sure there is a lot of bugs) feel free to left a comment on this page or send me an email. Current source will be also available on github at this address https://github.com/chmod666org/nmon2graphite


Since nmon is integrated to AIX and available on Linux, it is one of the most used performance tool. The community has grown and some third part tools are developed to graph nmon output. I’m sure everybody knows nmon analyzer, nmon consolidator and Nigel’s nmon2rrd. A few months ago Ben Rockwood from Cuddletech introduce me to graphite with a post talking about using graphite to graph dtrace metrics. The idea of using graphite to graph nmon’s output comes from this post. If you haven’t yet heard of graphite let me explain you what it is. Graphite is a tool to produce graphs ; just a like rrdtool data are stored in a round robin database called whisper. One cool thing about graphite is that you can feed the database trough a TCP port, just by sending some formatted data into a socket, you can use what ever you want, for example, netcat can be used to send data to this socket. I personally use a perl script for nmon2graphite. This collection daemon is called carbon.

nmon2graphite global overview

The idea of nmon2graphite is to feed graphite with nmon’s outputs. nmon2graphite client takes a nmon file as argument, you can use your existing files and send them to graphite. Even better nmon2graphite can feed graphite in real time. To do that you have to deploy a perl script on each lpar you want to graph and modify root user crontab, here is how nmon2graphite works :

  • A fifo stack is created at midnight every day.
  • nmon is launched to write into this fifo stack at midnight every day.
  • nmon2graphite is launched at midnight every day.
  • nmon2graphite unstack the fifo stack over time.
  • nmon2graphite parse every line of the fifo stack and transform it into a graphite data.
  • These data are send trough a socket to carbon, and are stored into whisper.
  • On another server, a jquery and perl script are used to query whisper and produce graphs into a web page trough graphite render.

Client Side

If you want to graph nmon output with nmon2graphite you’ll have to choose if you want to graph output in real time or not. Follow the steps described below : copy the nmon2graphite binary on the lpar and modify the crontab to launch it every day. I’m sure some of you are already running nmon and save nmon output every day in a file. If you do not want to use nmon2graphite with real time feature you can use those files to feed graphite database

Enabling nmon2graphite with real time feature

  • Upload nmon2graphite perl script on your lpar :
  • # ls /tools/perf/bin/nmon2graphite
    /tools/perf/bin/nmon2graphite
    
  • Verifiy you can “talk” with your graphite box on port TCP 2003 :
  • # GRAPHITE_BOX="10.10.10.10"
    #  telnet $GRAPHITE_BOX 2003
    Trying...
    Connected to 10.10.10.10.
    Escape character is '^]'.
    ^]
    
  • Modify root user crontab to create a fifo stack and to start nmon (specify fifo stack file as nmon output file) :
  • # crontab -l
    00 00 * * * /usr/bin/mkfifo /tools/perf/nmon/$(date +\%Y-\%m-\%d-\%H-\%M).nmon.fifo ; /usr/bin/nmon -F /tools/perf/nmon/$(date +\%Y-\%m-\%d-\%H-\%M).nmon.fifo -s30 -c2880 -t >/dev/null 2>&1
    
  • Modifiy root user crontab to start nmon2graphite 10 seconds after the creation of the nmon’s fifo stack (nmon2graphite reads standard entry) :
  • # crontab -l
    00 00 * * *  sleep 10 ; /tools/perf/bin/nmon2graphite -i 10.10.10.10 -p 2003 -l /var/tmp/$$.log < /tools/perf/nmon/$(date +\%Y-\%m-\%d-\%H-\%M).nmon.fifo
    
  • Modify root user crontab to wipe old fifo stacks :
  • # crontab -l
    0 1 * * * find /tools/perf/nmon -type f -mtime +30 | xargs rm -f >/dev/null 2>&1
    

Using existing nmon file to feed graphite

  • Go to your directory containing nmon's output files;
  • # ls *.nmon
    2013-04-13.nmon  2013-04-14.nmon  2013-04-15.nmon  2013-04-16.nmon
    # ls -l *.nmon | awk '{print "/tools/list/perf/bin/nmon2graphite -i 10.10.10.10 -p 2003 -l /var/tmp/$$.log < "$NF}' | sh
    

Server side : nmon2graphite web interface

On a web server copy all nmon2graphite files in a directory, an let the web server serve this directory. Mine is running in a Workload partition :

nmon2graphite web interface configuration :

  • Modify nmon2graphite.js with the graphite ip or url, you can also choose nmon interval, mine is set to 30 seconds
  • # cd /app/nmon2graphite
    # ls
    anytime.css        data               graphlist.txt      index.cgi          jquery-migrate.js  jquery-ui.js       nmon2graphite      nmon2graphite.js
    anytime.js         do.pl              images             jquery-form.js     jquery-ui.css      jquery.js          nmon2graphite.css
    # grep -E "var graphite_url|var nmon_interval" nmon2graphite.js
      var graphite_url = "10.10.10.10";
      var nmon_interval = "30";
    
  • Modify nmon2graphite with the graphite ip or url, graphite port, and log file :
  • # more nmon2graphite
    [..]
    my $graphite_box_ip    = '10.10.10.10';
    my $graphite_box_port  = '2003';
    my $graphite_box_proto = 'tcp';
    my $log_file           = '/tmp/nmon2graphite.log';
    my $ignorehdisk        = 0;
    [..]
    

httpd configuration :

Setup http server, in my example I'm using apache. Verify you can reach nmon2graphite web interface :

# more httpd.conf
[..]
  Alias /nmon2graphite/ /app/nmon2graphite/
  
    Options None
    DirectoryIndex index.cgi
    Options +ExecCGI
    Order allow,deny
    Allow from all
  
[..]

Graphite side

I will not tell you how to install graphite, you can find a nice documentation on graphite website (http://graphite.wikidot.com/documentation). I'm just giving here a few tricks to setup graphite for nmon2graphite.

storage-schemas.conf

Modify storage-schemas.conf to fits to your needs, I'm using an nmon interval of 30 seconds and I want to keep data for seven days. An higher retention time will results in bigger whisper files :

# pwd
/opt/graphite/conf
# tail -3 storage-schemas.conf
[nmon]
pattern = ^nmon\.
retentions = 30s:7d

Colors : graphTemplates.conf

Graph style can be modified here. Here is my configuration :

# cat graphTemplates.conf
[default]
background = black
foreground = white
majorLine = white
minorLine = grey
lineColors = #0033cc,#f8de81,#944b90,#e1316f,#fffe0d,#ff9a04,#59e898,#0395d0,#dd4218,#df1932,#888371,#413f4c,#e7dcd6
fontName = Sans
fontSize = 10
fontBold = False
fontItalic = False

Whisper file creation : MAX_UPDATES_PER_SECOND

If you are feeding graphite with already existing nmon files a lot of whisper databases are created in the same time, by default graphite does not allow to create more than 500 whisper databases in one second, raise this limit by adjusting MAX_UPDATES_PER_SECOND in carbon.conf file. These whisper databases creation can be I/O intensive :

# pwd
/opt/graphite/conf
# grep ^MAX_UPDATES_PER_SECOND carbon.conf
MAX_UPDATES_PER_SECOND = 100000

Set TIME_ZONE :

Change graphite TIME_ZONE with your timezone, mine is Europe/Paris :

# pwd
/opt/graphite/webapp/graphite
# grep ^TIME_ZONE local_settings.py
TIME_ZONE = 'Europe/Paris'

Set LEGEND_MAX_ITEMS

Graphite automatically hides legend when number of items is to important, raise this limitation by setting LEGEND_MAX_ITEMS :

# pwd
/opt/graphite/webapp/graphite
# grep ^LEGEND_MAX_ITEMS settings.py
LEGEND_MAX_ITEMS = 1000

Creating you own graphs

The nmon2graphite web interface provides you a list of graph working on AIX lpars, but you can add your own ones by modifying the file graphlist.txt. Add a line to this file to create you own graph and a new checkbox will appear in the nmon2graphite web interface :

# grep ^disk_busy graphlist.txt
disk_busy:Disk busy:target=cactiStyle(legendValue(aliasByNode(nmon.PPPPP.LLLLL.diskbusy.*,4),"avg"))&height=300&width=1000&format=png&title=Pserie PPPPP Lpar LLLLL | Disk busy&vtitle=Percent&from=SSSSS&until=UUUUU
  • A graph line is separated by ":", here are values needed :
  • A name with no spaces.
  • A title (display next to the checkbox in the web interface)
  • The graphite api target to reach the graph, PPPPP is the frame name, LLLLL is the lpar name.

37 thoughts on “nmon2graphite : Graph nmon output in real time

  1. I am running nmon on x86 Linux, but it seems nmon2graphite is expecting to see pseries as key. Any idea what I would need to change to show the x86 nodes. Currently my pull downs are empty.

    • Hi ,

      First of all thanks for leaving a comment. If you check at the code if nmon2graphite didn’t find any Serial Number on the nmon the host will be classified as “undefined”. (this is working for Solaris host with sarmon).

      1/ On graphite can you see the tree populated ?
      2/ Can you send me a screen with a description off your problem (bleechneeded@gmail.com)
      3/ Can you give me the output of the logfile (nmon2graphite.log).

      Best.

      Benoit

      • You made me recheck my environment, which I really appreciate. That was one of the keys. I was experimenting on one of my .conf files and the settings were conflicting with nmon2graphite. I am not getting as many graphs on x86 as I am getting on pSeries HW. I will try to see if I can figure out what is going on. nmon_analizer is capable of producing the same graphs. I am not comparing the tools, just saying that it is possible to generate them…

        Thanks for your patience and help.

        • Hi Noa, comments are appreciated.

          Have you succeed in running nmon2graphite ? Does it work on your environement ? I know my documentation is not the best and any feedback are appreciated.
          Don’t forget you can edit your own graph by modifying graphlist.txt files. The one I give is just an example.

          Best. Benoit.

          • Apologize for the delay on my response. Work got crazy and time flew really fast. To answer your question: Yes, I have nmon2graphite running. I am able to render the graphs, though I dont see to be able to see charts, it could be that I dont have enough data or the system is idle. Still playing with it.
            In addition to the entries you posted for httpd, I had to use the following option:
            AddHandler cgi-script .cgi .pl

            Without it, it showed the content of the scripts.

            Thanks,

  2. Hello everyone
    Can I use nmon2graphite on linux? I’m using RHEL & Ubuntu on my servers, your instructions include lpar, and just wondering if there has been any successful installs on Linux.
    Thanks

    • Hi,

      Sure, nmon2graphite collector can run on linux, the agent is coded in perl so no problem to run it on linux. (have some feedback by a few user running it on linux).
      If you plan to use the web interface, the graphlist.txt provided with nmon2graphite is for aix, you can modify it for your own need by editing it. If you do not plan to use the web interface just create your own dashboard on graphite with the metrics you want. The next version of nmon2graphite will be provided with a graphlist.txt file for linux and for sarmon. Let me know if you need help. If you create your graphlist file you can send it to me and it will be integrated to the current nmon2graphite release.

      Best.

      Benoît.

  3. Hi,

    I have some Linux nodes that have nmon attached as a radiography of activity and was thinking into aggregating this data to some web chart viewer, thus arriving here.

    Here are two quick questions:
    – 1 I see that you are pushing real hard for the real time collection, but we could as well just scp the nmon files from remote hosts and simply feed that to the perl script as in your example (for aggregation)
    – 2 how will the charts look for let’s say 50-100 nodes, is the data aggregated between the nodes, do we have some aggregation functions available or is it just a per based chart?

    Sorry but I just picked this up and working to understand how it could fit in (just made the EPEL based graphite play nice with a Centos for testing)

    Thank you (for the time and piece of code),

    • Hi,

      1/ Yes you are right, if there is no need for real time, just copy nmon file to any place you want (graphite server can be a good choice) and use the perl script to feed the graphite database (with localhost as graphite server).
      2/ Well, my answer can be it depends what exactly your are trying to graph. If you are graphing metrics server per server it will be obviously unreadable. Graphite is provided with aggregation function (sum, average, product, min, max, diff, stacking, ….). If you are using graphite dashboard to make you graphs it will be very easy. If you’are using nmon2graphite interface, this version is not ready for aggregated graphs (but the next one will be). Anyway, you can edit the graphlist.txt file and make you own graphs. Check on graphite render documentation to understand how it works.

      Hope it helps.

      Best.

      Benoit.

  4. Benoit,

    Thankyou for your effort & sharing it with us. Keep it up !

    One thing I found the the perl path needs to be adjusted in your scripts according to where it is installed. I had to change from “/usr/opt/perl5/bin/perl5.10.1″ to “/usr/bin/perl”

    Thanks

  5. Hi… I used nmon files to feed the database. The graphs showup without any data.
    # whisper-fetch.py /opt/graphite/storage/whisper/nmon/02810CR/zama/lpar/physicalcpu.wsp
    1379100360 None
    1379100420 None
    1379100480 None

    I queried all of the nmon data and all of them come as “None”. Would you know where to look for problems or issues.

    Thanks

    • Hi,

      1/ Can you please send me your nmon file, i’ll check this issue on my machines.
      2/ Be sure the timezone are matching between nmon files and and graphite.
      3/ If you are trying to graph an old nmon file be sure retentions times are correctly configured in /opt/graphite/conf/storage-schemas.conf

      Hope it helps.

      • Yes I have the retention & timezone set properly.

        I just sent you the nmon files to your email.

        Thanks

  6. Benoit -> Thank you very much for the nice tool. A few little remarks; hope they can help some others (for dummies ;)):
    – Alias in httpd.conf not imperatively
    – I had to add AddHandler cgi-script .cgi in the httpd-directory-directive (apache2-2.2.22-4.7.1.x86_64) (perhaps you can put the … in the code-example.
    – perl-path -> on opensuse it is /usr/bin/perl (as shiva mentioned yet), changed it in do.pl and index.cgi
    – graphite-web-interface is on our installation not on port 80, so I have to add the port-number on nmon2graphite.js, perhaps you can update the default to IP:80
    – the jquery.form.js was missed. I copied jquery-form.js to jquery.form.js (no more log-entries , hope thats ok )
    – some pngs were missed. I downloaded jquery-ui-master.zip and unpacked the pngs

    Actual I only imported one test-nmon file but that looks good. More testing have to be done….
    Thanks!!!

    • Thank you for your support ! Always really cool to have feedbacks. I have still a lot of work to do about this, I’ve found a lot a bug, talked with the users, and the next version is still in development …., I hope I will have more time in the next months to continue the work and the next version will come with huge improvements (ppc linux appliance, workload partition appliance, stacked graphs for whole machine monitoring and better support for linux). If you have any idea feel free to submit it here :).

      Once again thanks for you support and the bug you found : perl path, port, and many other options will be present in a configuration file in the next version.

      Benoit.

  7. G’day, finally got this tool up and running, wondering though, which link should I use to get the latest version? github or the link at the top?
    I’m not that familiar with Graphite generally, but seems that any updates to graphlist.txt have no impact on the presentation of the graphs in the browser, for instance, because I have 8 fc adapters per lpar, when I select the ‘disk adapter’ report, the key takes up 90% of the article space, I updated the graphlist.txt from 300 to 3000 expecting it update the graph, but it doesn’t.

    Can I get some tips on modifying the graph sizes?

    Disk busy (multiple sessions) and Network stacked graphs don’t display a graph at all – are these VIO graphs?

    Hope you can help me, and thanks for all your work on this so far, awesome tool.

  8. Any way to build graphlist.txt dynmically based on the nmon data input? We are trying to use this to graph nmon data from RHEL, AIX and Solaris on 600+ machines.

    • Hi Chris,

      Graphlist has to be build ‘by hand’. I’m working on a version working on Linux (the provide graphlist is for AIX). I have tested myself the current graphlist with sarmon on Solaris, and nmon Linux, some graph are ok and some are not (specific to aix).

      This two graphlist files (rhel and solaris) will be provided in the next version (still in dev); feel free to submit your own graphlist. They can be added to the current version.

      Sorry for this answer but you just have to wait a little bit :-).

      Thanks a lot for your support.

      Benoit.

  9. My graphs are just showing interval dots rather than a line, I’ve gone through the instructions several times, is there an easy fix?

    • OK, first sorry for the huge delay, but I am in vacation with no internet access :-)

      By default the interval time is 30 seconds (using nmon file with a larger interval can result in gaps on your graphs). Check on your nmon file if this interval is 30 seconds. If it’s not align your time and the storage-schema.conf, nmon file interval an storage-schema.conf have to match (retentions line, time per point).

      The interval has also to be set in the nmon2graphite.js if your are using the web interface (here is the piece of code below, chang it with your own value) :
      /* # Put nmon interval here ; in seconds */
      var nmon_interval = “30”;

      Let me know if the problem is solved with these advices.

      Waiting for your feedback.

      Benoit.

  10. All working great except diskbusy – diskread – diskwrite – etc no graph at all, actually nmon2graphite send out disk data to carbon/whisper, but the data was not saved to DB, so no graph. Hopefully it can be fixed soon. Thanks, Regards, sunaus

    • Hi,

      All hdisk date probably “invalid line received from client, ignoring”.

      But it will be ok if I made the put comment in nmon2graphite for the following lines, so use standard line for disk date other than special case.
      if ( $current_key_norm =~ m/^disk/ )

      Do not understand why, and do not know if should be fixed like this.

      Thanks,
      Regards,
      sunaus

      • Hi,

        Here is the problem. I do not want to graph all my hdisk in the same graph, if you have to many disk configured on the lpar the graph will be unreadable.
        The idea is to create on graph per disk. By clicking on a “disk” checkbox on the webinterface I need a list of all disks, this operation is performed by a JSON call.
        The list attribute purpose is to create on leaf (all hdisk* are branchs), the json call is performed on this leaf.

        —–
        $.getJSON(“http://”+graphite_url+”/render?”+json_string,null,function(cust_result){
        $.each(cust_result, function(i, item) {
        var my_val = cust_result[i].target.split(“.”);
        var buffer_string = graph_string;
        var buffer_string_link = graph_string;
        buffer_string = buffer_string.replace(/ZZZZZ/g,my_val[cust_index]);
        buffer_string_link = buffer_string.replace(/width=1000/g,”width=”+$(“#width_box”).val());
        graphs_to_draw[i] = buffer_string;
        graphs_to_draw_link[i] = buffer_string_link;
        f_add_graph(graphite_url,’#draw_graphs’,buffer_string,buffer_string_link,wanted_id+”_”+my_val[cust_index],’#date_value_from’, ‘#time_value_from’, ‘#date_value_until’, ‘#time_value_until’, nmon_interval,’#real_time_checkbox’, real_time_intervals);
        });
        });
        ——

        On the graphlist file this specials cases are prefixed by “cust” for these items.

        1/ can you send me a nmon file your are trying to graph
        2/ can you send the nmon2graphite log file

        I will check your problem on my own nmon2graphite version.

        Waiting for you files.

        Cheers,

        Benoit.

  11. Hi,

    One more question for what max period the nmon graph can be?
    Now it is 24 hours although I choose more than 24 hours on Interval From/To (starting/ending date/time). Maybe I do not know how to set up it.

    Thanks,
    Regards,
    sunaus

    • Hi,

      Found what was wrong, and it was user’s configuration problem.

      with storage-schemas.conf, nmon section has to be in the beginning, other than in the end, and then all fixed.

      Thanks,
      Regards,
      sunaus

  12. Any updates on the graphlist.txt for sarmon? I have to review 1400+ nmon files from sarmon and I would love to use nmon2graphite

  13. Hi , has someone installed and configured graphite on AIX boxes? I think it has a lot of deps RPMs, most of which I’m not able to find at all? Can someone help?
    Thanks.
    Narain.

  14. [FATAL] Unable to import the ‘cairo’ module, do you have pycairo installed for python 2.7.5?
    [FATAL] Unable to import the ‘django’ module, do you have Django installed for python 2.7.5?
    [FATAL] Unable to import the ‘tagging’ module, do you have django-tagging installed for python 2.7.5?
    [WARNING] Unable to import Interface from zope.interface.
    Without it, you will be unable to run carbon on this server.
    [WARNING]
    Unable to import the ‘memcache’ module, do you have python-memcached installed for python 2.7.5?
    This feature is not required but greatly improves performance.

    [WARNING]
    Unable to import the sqlite module, do you have python-sqlite2 installed for python 2.7.5?
    If you plan on using another database backend that Django supports (such as mysql or postgres)
    then don’t worry about this. However if you do not want to setup the database yourself, you will
    need to install sqlite2 and python-sqlite2.

    [WARNING]
    Unable to import the ‘ldap’ module, do you have python-ldap installed for python 2.7.5?
    Without python-ldap, you will not be able to use LDAP authentication in the graphite webapp.

    [WARNING]
    Unable to import the ‘twisted’ package, do you have Twisted installed for python 2.7.5?
    Without Twisted, you cannot run carbon on this server.
    [WARNING]
    Unable to import the ‘txamqp’ module, this is required if you want to use AMQP.
    Note that txamqp requires python 2.5 or greater.
    3 necessary dependencies not met. Graphite will not function until these dependencies are fulfilled.
    6 optional dependencies not met. Please consider the warning messages before proceeding.

  15. I have just upgraded my lpars to 7100-03-03-1415 and it has broken nmon2graphite real time graphing. Nmon will not open the file created by mkfifo for writing. for example if i run
    mkfifo -m 777 test2.nmon; nmon -F test2.nmon -s30 -c2880 -t

    i get
    opening output file: File exists
    ERROR nmon filename=test2.nmon
    everything was working fine beforre the upgrade.

      • This is now fixed. There is a flag introduced in nmon on that accepts an existing named pipe as input with the flag -yoverwrite=1

        Regards

        Joel

  16. Hello.
    I’m actually trying to use nmon2graphite in real-time mode.
    I can see data being saved into /opt/graphite/storage/whisper under “nmon” folder, but I can’t create any graph (empty “Frame” and “LPAR” boxes).
    I can’t understand what is getting wrong, could you please help me ?
    Thank you,
    Davide

    • Can you please send me and example and your log files. It seems to be related with the json request made to build the list of frames.
      In nmon directory can you see if files are correctly created when data are pushed ?
      Can you give me an “ls -ltR nmon” output please.
      Regards,
      B.

      • Hello Benoit.

        This is the output of my nmon directory :
        ga-nmon01:/opt/graphite/storage/whisper/nmon # ll
        total 20
        drwxr-xr-x 4 root root 4096 Mar 3 16:43 0629E8R
        drwxr-xr-x 3 root root 4096 Feb 24 15:16 0629E9R
        drwxr-xr-x 3 root root 4096 Feb 25 10:25 1046C4P
        drwxr-xr-x 3 root root 4096 Mar 3 17:03 carbon
        drwxr-xr-x 4 root root 4096 Mar 4 16:43 nmon

        JSON nmon2graphite.js :
        function f_pseries_select(graphite_ip, select_pseries) {
        $.getJSON(“http://”+graphite_ip+”/render?target=nmon.*.list&format=json&jsonp=?”,null,function(result) {
        $.each(result, function(i, item) {
        var values = result[i].target.split(“.”);
        $(select_pseries).append(new Option(values[1],values[1],true,true));
        });
        });
        }

        Thank you !
        Davide

Leave a Reply to sunaus Cancel reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>