nmon2graphite : Graph nmon output in real time

Posted on by

Download nmon2graphite by clicking here

Current nmon2graphite version is v0.1a. This is the official page for nmon2graphite. This page will be updated with future releases of nmon2graphite. New features, bugs, and anything related to nmon2graphite will be present in this page. If you find bugs (and I’m sure there is a lot of bugs) feel free to left a comment on this page or send me an email. Current source will be also available on github at this address https://github.com/chmod666org/nmon2graphite


Since nmon is integrated to AIX and available on Linux, it is one of the most used performance tool. The community has grown and some third part tools are developed to graph nmon output. I’m sure everybody knows nmon analyzer, nmon consolidator and Nigel’s nmon2rrd. A few months ago Ben Rockwood from Cuddletech introduce me to graphite with a post talking about using graphite to graph dtrace metrics. The idea of using graphite to graph nmon’s output comes from this post. If you haven’t yet heard of graphite let me explain you what it is. Graphite is a tool to produce graphs ; just a like rrdtool data are stored in a round robin database called whisper. One cool thing about graphite is that you can feed the database trough a TCP port, just by sending some formatted data into a socket, you can use what ever you want, for example, netcat can be used to send data to this socket. I personally use a perl script for nmon2graphite. This collection daemon is called carbon.

nmon2graphite global overview

The idea of nmon2graphite is to feed graphite with nmon’s outputs. nmon2graphite client takes a nmon file as argument, you can use your existing files and send them to graphite. Even better nmon2graphite can feed graphite in real time. To do that you have to deploy a perl script on each lpar you want to graph and modify root user crontab, here is how nmon2graphite works :

  • A fifo stack is created at midnight every day.
  • nmon is launched to write into this fifo stack at midnight every day.
  • nmon2graphite is launched at midnight every day.
  • nmon2graphite unstack the fifo stack over time.
  • nmon2graphite parse every line of the fifo stack and transform it into a graphite data.
  • These data are send trough a socket to carbon, and are stored into whisper.
  • On another server, a jquery and perl script are used to query whisper and produce graphs into a web page trough graphite render.

Client Side

If you want to graph nmon output with nmon2graphite you’ll have to choose if you want to graph output in real time or not. Follow the steps described below : copy the nmon2graphite binary on the lpar and modify the crontab to launch it every day. I’m sure some of you are already running nmon and save nmon output every day in a file. If you do not want to use nmon2graphite with real time feature you can use those files to feed graphite database

Enabling nmon2graphite with real time feature

  • Upload nmon2graphite perl script on your lpar :
    • # ls /tools/perf/bin/nmon2graphite
/tools/perf/bin/nmon2graphite
  • Verifiy you can “talk” with your graphite box on port TCP 2003 :
    • # GRAPHITE_BOX="10.10.10.10"
#  telnet $GRAPHITE_BOX 2003
Trying...
Connected to 10.10.10.10.
Escape character is '^]'.
^]
  • Modify root user crontab to create a fifo stack and to start nmon (specify fifo stack file as nmon output file) :
    • # crontab -l
00 00 * * * /usr/bin/mkfifo /tools/perf/nmon/$(date +\%Y-\%m-\%d-\%H-\%M).nmon.fifo ; /usr/bin/nmon -F /tools/perf/nmon/$(date +\%Y-\%m-\%d-\%H-\%M).nmon.fifo -s30 -c2880 -t >/dev/null 2>&1
  • Modifiy root user crontab to start nmon2graphite 10 seconds after the creation of the nmon’s fifo stack (nmon2graphite reads standard entry) :
    • # crontab -l
00 00 * * *  sleep 10 ; /tools/perf/bin/nmon2graphite -i 10.10.10.10 -p 2003 -l /var/tmp/$$.log < /tools/perf/nmon/$(date +\%Y-\%m-\%d-\%H-\%M).nmon.fifo
  • Modify root user crontab to wipe old fifo stacks :
    • # crontab -l
0 1 * * * find /tools/perf/nmon -type f -mtime +30 | xargs rm -f >/dev/null 2>&1

Using existing nmon file to feed graphite

  • Go to your directory containing nmon's output files; 
    • # ls *.nmon
2013-04-13.nmon  2013-04-14.nmon  2013-04-15.nmon  2013-04-16.nmon
# ls -l *.nmon | awk '{print "/tools/list/perf/bin/nmon2graphite -i 10.10.10.10 -p 2003 -l /var/tmp/$$.log < "$NF}' | sh

Server side : nmon2graphite web interface

On a web server copy all nmon2graphite files in a directory, an let the web server serve this directory. Mine is running in a Workload partition :

nmon2graphite web interface configuration :

  • Modify nmon2graphite.js with the graphite ip or url, you can also choose nmon interval, mine is set to 30 seconds
    • # cd /app/nmon2graphite
# ls
anytime.css        data               graphlist.txt      index.cgi          jquery-migrate.js  jquery-ui.js       nmon2graphite      nmon2graphite.js
anytime.js         do.pl              images             jquery-form.js     jquery-ui.css      jquery.js          nmon2graphite.css
# grep -E "var graphite_url|var nmon_interval" nmon2graphite.js
  var graphite_url = "10.10.10.10";
  var nmon_interval = "30";
  • Modify nmon2graphite with the graphite ip or url, graphite port, and log file :
    • # more nmon2graphite
[..]
my $graphite_box_ip    = '10.10.10.10';
my $graphite_box_port  = '2003';
my $graphite_box_proto = 'tcp';
my $log_file           = '/tmp/nmon2graphite.log';
my $ignorehdisk        = 0;
[..]

httpd configuration :

Setup http server, in my example I'm using apache. Verify you can reach nmon2graphite web interface :

# more httpd.conf
[..]
  Alias /nmon2graphite/ /app/nmon2graphite/
  
    Options None
    DirectoryIndex index.cgi
    Options +ExecCGI
    Order allow,deny
    Allow from all
  
[..]

Graphite side

I will not tell you how to install graphite, you can find a nice documentation on graphite website (http://graphite.wikidot.com/documentation). I'm just giving here a few tricks to setup graphite for nmon2graphite.

storage-schemas.conf

Modify storage-schemas.conf to fits to your needs, I'm using an nmon interval of 30 seconds and I want to keep data for seven days. An higher retention time will results in bigger whisper files :

# pwd
/opt/graphite/conf
# tail -3 storage-schemas.conf
[nmon]
pattern = ^nmon\.
retentions = 30s:7d

Colors : graphTemplates.conf

Graph style can be modified here. Here is my configuration :

# cat graphTemplates.conf
[default]
background = black
foreground = white
majorLine = white
minorLine = grey
lineColors = #0033cc,#f8de81,#944b90,#e1316f,#fffe0d,#ff9a04,#59e898,#0395d0,#dd4218,#df1932,#888371,#413f4c,#e7dcd6
fontName = Sans
fontSize = 10
fontBold = False
fontItalic = False

Whisper file creation : MAX_UPDATES_PER_SECOND

If you are feeding graphite with already existing nmon files a lot of whisper databases are created in the same time, by default graphite does not allow to create more than 500 whisper databases in one second, raise this limit by adjusting MAX_UPDATES_PER_SECOND in carbon.conf file. These whisper databases creation can be I/O intensive : 

# pwd
/opt/graphite/conf
# grep ^MAX_UPDATES_PER_SECOND carbon.conf
MAX_UPDATES_PER_SECOND = 100000

Set TIME_ZONE :

Change graphite TIME_ZONE with your timezone, mine is Europe/Paris :

# pwd
/opt/graphite/webapp/graphite
# grep ^TIME_ZONE local_settings.py
TIME_ZONE = 'Europe/Paris'

Set LEGEND_MAX_ITEMS

Graphite automatically hides legend when number of items is to important, raise this limitation by setting LEGEND_MAX_ITEMS :

# pwd
/opt/graphite/webapp/graphite
# grep ^LEGEND_MAX_ITEMS settings.py
LEGEND_MAX_ITEMS = 1000

Creating you own graphs

The nmon2graphite web interface provides you a list of graph working on AIX lpars, but you can add your own ones by modifying the file graphlist.txt. Add a line to this file to create you own graph and a new checkbox will appear in the nmon2graphite web interface :

# grep ^disk_busy graphlist.txt
disk_busy:Disk busy:target=cactiStyle(legendValue(aliasByNode(nmon.PPPPP.LLLLL.diskbusy.*,4),"avg"))&height=300&width=1000&format=png&title=Pserie PPPPP Lpar LLLLL | Disk busy&vtitle=Percent&from=SSSSS&until=UUUUU
  • A graph line is separated by ":", here are values needed :
  • A name with no spaces.
  • A title (display next to the checkbox in the web interface)
  • The graphite api target to reach the graph, PPPPP is the frame name, LLLLL is the lpar name.

NIM Less known features : HANIM, nimsh over ssl, DSM

Posted on by

The Network Installation Manager server is one of the most important host in an environment. New machines installations, machines backups, backups restorations,software (filesets), third party products installations, in some cases volume group backups are made from the NIM server. Some best practices have to be respected. I’ll give you in this post a few tricks for NIM. First off all a NIM server has to be in your disaster recovery plan because it the first server needed when you have to re-build a crashed machine : my solution HANIM. It has to be secured (nimsh, and nimsh authentication over ssl), and it has to be flexible and automated (DSM).

NIM High Availability : HANIM

Finding documentation and information about NIM High Availability is not so easy. I recommend you to check the NIM from a to Z Redbook, it’s one of the only viable source for HANIM. HANIM simple to setup and simple to use, but there are a few things to know and to understand about it :

HANIM Overview

  • The alternate NIM master is a backup NIM build from the NIM master.
  • Takeover operations from master to alternate are manuals. PowerHA can be used to run these takeover operations but my advice is not to use it. Takeover can be performed even if the NIM master is down. HANIM does not perform any heartbeat.
  • HANIM only provides a method for replicating NIM database and resources. Resources can be replicated from master to alternate : NIM database AND resources data can be replicated (replicate=yes option).
  • My advice is to run every NIM operation from the master (even if it is possible to run a NIM operation from the alternate).
  • Disks are not shared between the master and the alternate, when a sync operation is done, missing resources are copied over NFS form the master to the alternate, or from the alternate to the master. HANIM does not provides a filesystem takeover.
  • A takeover operation modify all the nimclient’s /etc/niminfo files. The NIM_MASTER_HOSTNAME_LIST is modified by the takeover operation and the alternate NIM master is moved in first position. The NIM_MASTER_HOSTNAME is modified with the alternated NIM master hostname.


Initial setup

On the NIM master and on the alternate NIM master some filesets have to be installed, check the presence of : bos.sysmgt.nim.master, bos.sysmgt.nim.spot, bos.sysmgt.nim.client. NIM master and alternate NIM master must be one the same AIX version :

# lslpp -l | grep -i nim
  bos.sysmgt.nim.client     7.1.2.15  COMMITTED  Network Install Manager -
  bos.sysmgt.nim.master     7.1.2.15  COMMITTED  Network Install Manager -
  bos.sysmgt.nim.spot       7.1.2.15  COMMITTED  Network Install Manager - SPOT
  bos.sysmgt.nim.client     7.1.2.15  COMMITTED  Network Install Manager -
# oslevel -s
7100-02-02-1316

Configure the NIM master

Initialize the NIM master with the nimconfig command, you’ll need to name the first network used by NIM. nimesis daemons will be started at this step.

# nimconfig -a pif_name=en0 -a netname=10-10-20-0-s24-net -a master_port=1058 -a verbose=3 -a cable_type=N/A
[..]
Checking input attributes.
attr_ass:
        'cpuid' => '00F359164D00'
        'pif_name' => 'en0'
        'netname' => '10-10-20-0-s24-net'
        'master_port' => '1058'
        'cable_type' => 'N/A'
        'net_addr' => '10.10.20.1'
        'snm' => '255.255.255.0'
        'adpt_addr' => '667C70F7A904'
        'adpt_name' => 'ent0'
Making sure the NIM Master package is OK.
      set_state: id=1361463886; name=; state_attr=85; new_state=5;
   checking the object definition of ;
   checking interface info for master;
Built NIM infomation file.
      10.10.20.1 is known as nim_master
Adding default route 10.10.20.254 to network object
         0 - /usr/lpp/bos.sysmgt/nim/methods/m_mknet
         1 - -anet_addr=10.10.20.1
         2 - -asnm=255.255.255.0
         3 - -tent
         4 - -arouting1=default 10.10.20.254
         5 - 10-10-20-0-s24-net
Connecting NIM master to master network.
         0 - /usr/lpp/bos.sysmgt/nim/methods/m_chmaster
         1 - -aif1=10-10-20-0-s24-net nim_master 667C70F7A904
         2 - -amaster_port=1058
         3 - -aregistration_port=1059
         4 - -acable_type1=N/A
         5 - master
Adding NIM deamons to SRC and starting....
0513-071 The nimesis Subsystem has been added.
0513-071 The nimd Subsystem has been added.
0513-059 The nimesis Subsystem has been started. Subsystem PID is 9568296.
[..]

NIM resources such as spot, lpp_source and so on can be created right now, please refer to the NIM cheatsheet by chmod666.org ;-) . For the purpose of this post some resources (spot, lpp_source, mksysb, network) are created, these ones will be replicated later.

Configure the alternate NIM master

NIM alternate master is configured with the niminit command. If you check on the NIM from a to Z, page 124, a note is warning you about the synchronization : “At the time of writing, only rsh/rshd communication is supported for NIM synchronization.”.THIS STATEMENT IS FALSE : I’m using nimsh for the synchronization, and I recommend to use it. We are in 2013, do not use rsh anymore.

# niminit -a is_alternate=yes -a master=nim_master -a pif_name=en0 -a cable_type1=N/A -a connect=nimsh -a name=nim_alternate
0513-071 The nimesis Subsystem has been added.
0513-071 The nimd Subsystem has been added.
0513-059 The nimesis Subsystem has been started. Subsystem PID is 10944522.
nimsh:2:wait:/usr/bin/startsrc -g nimclient >/dev/console 2>&1
0513-044 The nimsh Subsystem was requested to stop.
0513-059 The nimsh Subsystem has been started. Subsystem PID is 5963998.

Verification

You’re done with the configuration, you can now start to synchronize, replicate and takeover… pretty easy. Here are some points you can verify :

  • On the NIM master, the attribute is_alternate is set to yes :
    • # lsnim -l master
[..]
   is_alternate        = yes
[..]
  • On the NIM master, a new machine object typed alternate_master is created :
    • # lsnim -t alternate_master
nim_alternate     machines       alternate_master
  • After the first database synchronization, on the alternate NIM master, a new machine object typed alternate_master is created, this the NIM master :
    • # lsnim -t alternate_master
nim_master     machines       alternate_master
  • On the alternate NIM master, the attribute is_alternate does not exists :
    • # lsnim -l master | grep alternate

Synchronization and replication

NIM master and alternate NIM master can now communicate with each others, some resources are created on the master, and it’s now time to synchronize. Remember : HANIM only provides a method for replicating NIM database and resources. You can -if you want- synchronize the NIM database only or the NIM database and its resources (data included). Remember : never perform a NIM synchronization from the alternate NIM master.

Database synchronization only

The database synchronization is useful, when objects are modified, for example when you are modifying a subnet mask for a network object. It also can be useful when objects “without files” are created ; for instance a machine. On the other hand if your are trying to synchronize the database if an object “with a file” exists such as an lpp_source, a spot, or an fb_script, this one will not be created, you have to copy the file before synchronize, or use the replicate attribute :

  • On NIM master two objects are created, an fb_script and a machine:
    • # nim -o define -t fb_script -a server=master -a location=/export/nim/others/postinstall/fb_script.ksh fb_script01
# ls -l /export/nim/others/postinstall/fb_script.ksh
-rw-r--r--    1 root     system           35 Mar  8 18:01 /export/nim/others/postinstall/fb_script.ksh
# lsnim ruby
ruby     machines       standalone
  • A database synchronization is performed :
    • # nim -o sync -a force nim_alternate
[..]
The level of the NIM master fileset on this machine is: 7.1.2.15
The level of the NIM database backup is: 7.1.2.15
[..]
Checking NIM resources
  Removing fb_script01
    0518-307 odmdelete: 1 objects deleted. from nim_attr (serves attr)
    0518-307 odmdelete: 0 objects deleted. from nim_attr (group memberships)
    0518-307 odmdelete: 5 objects deleted. from nim_attr (resource attributes)
    0518-307 odmdelete: 1 objects deleted. from nim_object (resource object)
  Finished removing fb_script01
  • On the alternate NIM master, the machine object is here but the fb_script was not replicated because the file was not present on the alternate NIM master :
    • # lsnim ruby
ruby     machines       standalone
# lsnim fb_script01
0042-053 lsnim: there is no NIM object named "fb_script01"
  • If you copy the file before synchronize the resource will be created :
    • master# scp fb_script.ksh nim_alternate:/export/nim/others/postinstall
fb_script.ksh                      100%   35     0.0KB/s   00:00

    master# nim -o sync -a force nim_alternate
[..]
Restoring the NIM database from /tmp/_nim_dir_13041674/mnt0
x ./etc/NIM.level, 9 bytes, 1 tape blocks
[..]
  Keeping fb_script01

    alternate# # lsnim fb_script01
fb_script01     resources       fb_script

    Synchronization with replication

    I encourage you not to use the database synchronization, but to use it with replication, it does the same job but copy the files for you. Much much easier, just add replicate=yes attribute to the nim command, it works like a charm :

    # lsnim -q sync alternate_master

the following attributes are optional:
        -a verbose=
        -a replicate=
        -a reset_clients=
# nim -o sync -a force=yes -a replicate=yes alternate_master

    Takeover

    If the NIM master is down a takeover operation allows the alternate NIM master to become NIM master for the clients. On clients /etc/niminfo file is modified (NIM_MASTER_HOSTNAME and NIM_MASTER_HOSTNAME_LIST attributes are modified).

    • /etc/niminfo and lsnim output file before a takeover operation :
      • client# grep -E "NIM_MASTER_HOSTNAME_LIST|NIM_MASTER_HOSTNAME" /etc/niminfo
export NIM_MASTER_HOSTNAME=nim_master
export NIM_MASTER_HOSTNAME_LIST="nim_master nim_alternate"
master# lsnim -l client | grep current_master
   current_master = nim_master
    • Takeover operation is initiated from the alternate NIM master :
      • alternate# nim -o takeover -a show_progress=yes nim_master
+-----------------------------------------------------------------------------+
                      Performing "reset" Operation
+-----------------------------------------------------------------------------+
+-----------------------------------------------------------------------------+
                      "reset" Operation Summary
+-----------------------------------------------------------------------------+
 Target                  Result
 ------                  ------
 client                   RESET
 client1                  RESET
 [..]
+-----------------------------------------------------------------------------+
                      Initiating "takeover" Operation
+-----------------------------------------------------------------------------+
 Initiating the takeover operation on machine 1 of 240: client ...

 Initiating the takeover operation on machine 2 of 240: client1...
[..]
+-----------------------------------------------------------------------------+
                      "takeover" Operation Summary
+-----------------------------------------------------------------------------+
 Target                  Result
 ------                  ------
 client                  SUCCESS
 client1                 SUCCESS
[..]
alternate# lsnim -l client | grep current_master
   current_master = nim_alternate
client# grep -E "NIM_MASTER_HOSTNAME_LIST|NIM_MASTER_HOSTNAME" /etc/niminfo
export NIM_MASTER_HOSTNAME=nim_alternate
export NIM_MASTER_HOSTNAME_LIST="nim_alternate nim_master"
    • When the NIM master is up, initiate the takeover for the master :
      • # nim -o takeover -a show_progress=yes nim_alternate

    Synchronization automation and other files ?

    I recommend to run a NIM synchronization every day, I personally have a cronjob doing it every day at eleven PM. Most of the time a NIM synchronization is not enough and you’ll need to synchronize others file in my case, my root .profile my etc/hosts file, in your case whatever you want. For this need I’m using a little script based over rsync which synchronize my master to my alternate everyday :

    # crontab -l
[..]
0 23 * * * /export/nim/others/tools/do_sync.ksh >/dev/null 2>&1
[..]
# cat /export/nim/others/tools/do_sync.ksh
[..]
    nim -o sync -a force=yes -a replicate=yes -a reset_clients=yes ${alternate}
    /export/nim/others/tools/sync_to_alternate.ksh
[..]
# cat /export/nim/others/tools/sync_to_alternate.ksh
[..]
  /usr/bin/rsync -ave ssh ${a_filesystem} ${alternate_nim_master}:${a_filesystem}
[..]

    NIM Security, use nimsh and use it over SSL

    nimsh over ssl

    NIM Master configuration form nimsh over SSL

    From the NIM master enable the SSL support trough the nimconfig command, certificates will be generated in /ssl_nimsh/keys, OpenSSL fileset has to be installed :

    • Check OpenSSL filesets :
      • # lslpp -l | grep openssl
  openssl.base            0.9.8.2400  COMMITTED  Open Secure Socket Layer
  openssl.license         0.9.8.2400  COMMITTED  Open Secure Socket License
  openssl.man.en_US       0.9.8.2400  COMMITTED  Open Secure Socket Layer
  openssl.base            0.9.8.2400  COMMITTED  Open Secure Socket Layer
    • Use nimconfig to enable SSL support :
      • # nimconfig -c
0513-029 The tftpd Subsystem is already active.
Multiple instances are not supported.
NIM_MASTER_HOSTNAME=nim_master
x - /usr/lib/libssl.so.0.9.8
x - /usr/lib/libcrypto.so.0.9.8
Target "all" is up to date.
Generating a 1024 bit RSA private key
......++++++
.++++++
writing new private key to '/ssl_nimsh/keys/rootkey.pem'
-----
Signature ok
subject=/C=US/ST=Texas/L=Austin/O=ibm.com/CN=Root CA
Getting Private key
Generating a 1024 bit RSA private key
...............++++++
.......++++++
writing new private key to '/ssl_nimsh/keys/clientkey.pem'
-----
Signature ok
subject=/C=US/ST=Texas/L=Austin/O=ibm.com
Getting CA Private Key
Generating a 1024 bit RSA private key
......++++++
.............++++++
writing new private key to '/ssl_nimsh/keys/serverkey.pem'
-----
Signature ok
subject=/C=US/ST=Texas/L=Austin/O=ibm.com
Getting CA Private Key
    • Check the NIM master : attribute ssl_support is now set to yes :
      • # lsnim -l master | grep ssl_support
   ssl_support         = yes

    NIM alternate master for nimsh over SSL

    If you’re using an alternate NIM master repeat the same operation (OpenSSL and nimconfig -r). Alternate NIM master is also a client of the NIM master, its client has to be configured :

    # nimclient -c
x - /usr/lib/libssl.so.0.9.8
x - /usr/lib/libcrypto.so.0.9.8
Received 2763 Bytes in 0.0 Seconds
0513-044 The nimsh Subsystem was requested to stop.
0513-077 Subsystem has been changed.
0513-059 The nimsh Subsystem has been started. Subsystem PID is 9502954.

    Client configuration

    Configure all nimclients to use ssl crypted authentication, if you are using alternate NIM master do not forget to download alternate certificates on clients :

    # rmitab nimsh 2>/dev/null
# rm -rf /etc/niminfo
# niminit -aname=$(hostname) -a master=nim_master -a master_port=1058 -a registration_port=1059 -a connect=nimsh
# nimclient -c
# nimclient -o get_cert -a master_name=nim_alternate
# stopsrc -s nimsh
# startsrc -s nimsh

    On the NIM server itself client’s connect attribute is now set to “nimsh (secure)” :

    # lsnim -l ruby | grep connect
   connect        = nimsh (secure)

    Are the data encrypted ?

    Check this statement in NIM from a to Z Redbook at page 434 :

    “Any communication initiated from the NIM client (pull operation) reaches the NIM master on the request for services and registration ports (1058 and 1059, respectively). This communication is not encrypted. For any communication initiated from the NIM master (push operations), the NIM master communicates with the NIM client using the NIMSH daemon. This allows an encrypted handshake dialog during authentication. However, data packets are not encrypted.”

    To sum up :

    • Only push operations can use secure nimsh.
    • Data packets are not encrypted.
    • Secure nimsh just add an encrypted handshake between NIM master and its clients.

    Have a look on this two screenshots, the first one is the tcp stream of a non-secure operation, the second one is secured :

    • Non secure tcp stream of a push operation :

    • Secure tcp stream of a push operation :

    Distributed Systems Management

    Distributed Systems Management (we’ll call it DSM until now), is a set of tools and programs used to enhance NIM capabilities. I personally use DSM for two main purposes, opening and monitoring consoles through the dconsole utility, and to automate my installations. DSM add new objects the NIM environment, and new attributes to the NIM objects. You can also gain more on control on your lpars and directly restart, maint_boot an lpar through NIM by using DSM. Hardware Management Console (HMC objects) and Pserie’s frames (CEC objects) can be added in NIM, profile management are added to standalone objects in order to take advantage of DSM with NIM.

    There are two main source of information for DSM

    • The dsm.core fileset comes with a pdf file named dsm_tech_note.pdf, page 161, chapter 5.
      • # lslpp -f dsm.core | grep dsm_tech_note.pdf
                        /opt/ibm/sysmgt/dsm/doc/dsm_tech_note.pdf
    • There are full detailed examples in the IBM AIX Version 7.1 Differences Guide .

    Filesets prerequisites

    Starting with AIX 6.1 TL3 base installation media are shipped with DSM packages (dsm.core). expect, tcl, tk, and xterm are needed by this DSM pacakges :

    # lslpp -l | grep -E "dsm|tcl|tk|expect|xterm"
  X11.apps.aixterm           7.1.2.0  COMMITTED  AIXwindows aixterm Application
  X11.apps.xterm            7.1.2.15  COMMITTED  AIXwindows xterm Application
  X11.msg.en_US.apps.aixterm
                             7.1.2.0  COMMITTED  AIXwindows aixterm Messages -
  dsm.core                  7.1.2.15  COMMITTED  Distributed Systems Management
  dsm.dsh                   7.1.2.15  COMMITTED  Distributed Systems Management
  expect.base               5.42.1.0  COMMITTED  Binary executable files of
  expect.man.en_US          5.42.1.0  COMMITTED  Expect man page documentation
  tcl.base                   8.4.7.0  COMMITTED  Binary executable files of Tcl
  tcl.man.en_US              8.4.7.0  COMMITTED  Tcl man page documentation
  tk.base                    8.4.7.0  COMMITTED  Binary executable files of Tk
  tk.man.en_US               8.4.7.0  COMMITTED  Tk man page documentation

    Defining HMC objects

    DSM is using HMC to start (poweron) lpars, stop (poweroff) lpars and open console on lpars. HMC can be defined on NIM. An HMC object is a management object. To avoid prompting password each time a NIM operations is performed, or each time dconsole is called, DSM provides a mechanism to manage SSH key sharing between the NIM and the HMC. Before adding an HMC object use dpasswd and dkeyexch command to enable SSH key authentication :

    • Create the authentication file with dpasswd command. File is by default stored in /etc/ibm/sysmgm/dsm/config :
      • # dpasswd -f hmc1_passwd -U hscroot
Password:
Re-enter password:
Password file created
# ls -l  /etc/ibm/sysmgt/dsm/config/
total 24
-r--r--r--    1 root     system           16 Mar 11 13:25 .key
-r--r--r--    1 root     system           24 Mar 11 13:25 hmc1_passwd
    • Share the key between NIM master and HMC using dkeyexch command :
      • # dkeyexch -f /etc/ibm/sysmgt/dsm/config/hmc1_passwd -I hmc -H hmc1
OpenSSH_6.0p1, OpenSSL 0.9.8x 10 May 2012
    • At this step you should be able to connect to the HMC without password prompting :
      • # ssh hscroot@hmc1
Last login: Mon Mar 11 13:51:35 2013 from 10.10.20.21
    • Define the new HMC object with nim command, the network on which the HMC is running must be defined as an NIM network :
      • # nim -o define -t ent -a net_addr=10.10.30.0 -a snm=255.255.254.0 -a routing1="default 10.10.31.254" 10-10-30-0-s23-net
# nim -o define -t hmc -a if1="find_net hmc1 0" -a passwd_file=/etc/ibm/sysmgt/dsm/config/hmc1_passwd hmc1
# lsnim -t hmc
hmc1     management       hmc
# lsnim -lF hmc1
hmc1:
   id          = 1363005068
   class       = management
   type        = hmc
   if1         = 10-10-30-0-s23-net hmc1 0
   Cstate      = ready for a NIM operation
   prev_state  =
   Mstate      = not running
   passwd_file = /etc/ibm/sysmgt/dsm/config/hmc1_passwd

    Defining CEC objects

    Defining HMC object allows to define CEC object, NIM CEC‘s object are requiring four mandatory attributes, hardware type (hw_type), hardware model (hw_model), hardware serial (hw_serial), and the HMC used to control this CEC object (mgmt_source). Query the HMC to get the attributes with lssyscfg command, and define the new CEC object with the nim command :

    • Querying HMC to get hw_model, hw_serial, and hw_type :
      • # ssh hscroot@hmc1 "lssyscfg -r sys -F name,type_model,serial_num"
# CEC1,8203-E4A,060CE99
    • lssyscfg output tells you that : hw_type=8203, hw_model=EA4 and hw_serial=060CE99
    • Create the CEC object :
      • # nim -o define -t cec -a hw_type=8203 -a hw_model=E4A -a hw_serial=060CE99 -a mgmt_source=hmc1 cec1
# lsnim -l cec1
cec1:
   class      = management
   type       = cec
   Cstate     = ready for a NIM operation
   prev_state =
   hmc        = hmc1
   serial     = 8203-E4A*060CE99

    Adding profile management to standalone object

    To define a standalone object with a management profile or to add a management profile to an existing standalone, MAC address and lpar id are needed, the lpar id can easily be learned by the HMC, for the MAC address use the dgetmacs command to get it:

    • Get the lpar id trough the HMC :
      • ssh hscroot@infmc102 "lssyscfg -r lpar -m CEC1 -F name,lpar_id"
lpar1,5
lpar2,4
vios1,3
vios2,2
lpar3,1
    • Define the machine and replace the MAC address by 0 :
      • # nim -o define -t standalone -a if1="10-10-20-0-s24-net lpar2 0" -a net_settings1="auto auto" -a mgmt_profile1="hmc1 4 CEC1" lpar2
    • Retrieve the machine MAC address by using the dgetmacs command, the host will booted on openfirmware. If the host is already installed get the MAC address with entstat command directly on the machine :
      • #  dgetmacs -n lpar2 -C NIM
Using an adapter type of "ent".
Could not dsh to node lpar2.
Attempting to use openfirmware method to collect MAC addresses.
Acquiring adapter information from Open Firmware for node lpar2.

# Node::adapter_type::interface_name::MAC_address::location::media_speed::adapter_duplex::UNUSED::install_gateway::ping_status::machine_type::netaddr::subnet_mask

lpar1::ent_v::::2643EEBC6C04::U8203.E4A.060CE99-V4-C4-T1::auto::auto::::::n/a::secondary::::
    • Modify the NIM object to add the MAC address :
      • # nim -o change -a if1="10-10-20-0-s24-net lpar2 2643EEBC6C04" lpar2

    Using dconsole to open and monitor machines consoles

    If the machine is already installed, or after the installation with a bos_inst operation, you can manage its console with the dconsole command. A few cool things comes with dconsole such as opening a console in read only mode, opening a console in text mode or through an xterm, and logging all consoles outputs into /var/ibm/sysmgt/dsm/log/console; here are a few examples :

    • Opening a text console in read-write mode and log the output in /var/ibm/sysmgt/dsm/log/console :
      • # dconsole -C NIM -n lpar2 -t -l
Starting console daemon
[read-write session]

 Open in progress

 Open Completed.
AIX Version 7
Copyright IBM Corporation, 1982, 2013.
Console login: root
# echo test
test
# tail -10 /var/ibm/sysmgt/dsm/log/console/lpar2.0
# echo test
test
# exit
    • Opening an xterm console in read-write mode and log the output in /var/ibm/sysmgt/dsm/log/console on greenclient1 :
      • # export DISPLAY=10.10.20.35:0
# dconsole -C NIM -n greenclient1  -l
Starting console daemon

    • Opening a text console in read-only mode :
      • # dconsole -C NIM -n lpar2  -l -t -r
Starting console daemon
[read only session, user input discarded]

 Open in progress

 Open Completed.
AIX Version 7
Copyright IBM Corporation, 1982, 2013.
Console login: [read only session, user input discarded]
[read only session, user input discarded]

    bos_inst operation through NIM with DSM

    Machine installation and bos_inst operation can be automated with DSM. If a machine has a management profile and a bos_inst operation is performed this one will be rebooted and automatically installed, I do install machine with this method and it works like a charm :

    • Install the machine lpar2 in aix 7100-02-02, a bosinst_data with no prompt stanza was created for this installation :
      • # nim -o bos_inst -a bosinst_data=hdisk0_noprompt-bosinst_data -a source=rte -a installp_flags=agX -a accept_licenses=yes -a spot=7100-02-02-1316-spot -a lpp_source=7100-02-02-1316-lpp_source lpar2
dnetboot Status: Invoking /opt/ibm/sysmgt/dsm/dsmbin/lpar_netboot lpar2
dnetboot Status: Was successful network booting node lpar2.
    • DSM is using HMC lpar_netboot command to install machines, the output of this command can be found in /tmp filesystem :
      • # cat /tmp/lpar_netboot.12124286.exec.log
lpar_netboot Status: process id is 12124286
lpar_netboot Status: lpar_netboot -i -t ent -D -S 10.10.20.140 -G 10.10.20.254 -C 10.10.20.202 -m 2643EEBC6C04 -s auto -d auto -F /etc/ibm/sysmgt/dsm/config/hmc1_passwd -j hmc -J 10.10.30.1 4 060C
E74 8203-E4A
[..]
IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM
IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM

          1 = SMS Menu                          5 = Default Boot List
          8 = Open Firmware Prompt              6 = Stored Boot List
[..]
10.10.20.202:    24  bytes from 10.10.20.140:  icmp_seq=7  ttl=? time=21  ms

10.10.20.202:    24  bytes from 10.10.20.140:  icmp_seq=8  ttl=? time=21  ms
PING SUCCESS.
[..]
38300 ^MPACKET COUNT = 38400 ^MPACKET COUNT = 38500 ^MPACKET COUNT = 38600 ^MPACKET COUNT = 38700 ^MPACKET COUNT = 38800 ^MPACKET COUNT = 38900 ^MFINAL PACKET COUNT = 38913
FINAL FILE SIZE = 19922944  BYTES
    • The installation progression can be monitored form the NIM itself :
      • # lsnim -l lpar2 |grep info
   info           = BOS install 39% complete : Installing additional software.

    Is it free ?

    Unlike CSM DSM is free, you do not need any licenses to use it. As you can see these tools can be very powerful to automate installations for standalone clients. VMControl is using DSM and NIM to automate installations. DSM is the right tool to industrialize your NIM installations.

    Cheatsheet

    I love cheat sheet ! NIM commands are complex and hard to remember, I’ve search over the internet if a NIM cheat sheet already exists but I haven’t found anything correct or anything that fits my needs. I’m sure that a lot of my readers already knows William Favorite’s Quicksheets. I’m a huge fan of this Quicksheets and I was inspired by Willam when creating my own one for NIM. Feel free to contact me if you want to add or correct something in my cheat sheet, you’ll be -of course- credited if you add some useful informations. Click here to download my NIM cheat sheet :chmod666 NIM Cheat Sheet

    No future ?

    I do love NIM, but in my opinion it’s a little bit outdated, everyone is calling for an update of the Redbook (click here to call for an update ;-) ) and of the product, me included. This part of the post was inspired by one of my AIX Guru, thanks to him, I’m sure he’ll recognize himself. If IBMers are reading this part of the post, please tell IBM to update NIM. Readers please react in the comments if you agree with me on this point. Here are a few points I want to see in a future NIM release :

    • Network package repository of software : publish lpp_source over http or https. IBM can publish an official repository, and customer can create its own one on the NIM server (this one can be synchronized with IBM official repository).
    • Create a client (updated nimclient) with search and download option. (Yes like yum).
    • Getting rid of bootp and tftp, download kernel (created in /tftpboot when a new SPOT is created) and ramdisk image trough http or https.
    • Replace nfs exports by http or https (or force nfsv4) for NIM resources sharing (SPOT, lpp_source, install_script, bosinst_data…)(easier for security, and firewall ruling).
    • Allow IPL menu to be setup in dhcp.
    • Automatic dependencies checking and resolution while installing a software.
    • Simplify postinstall (script) and firstboot (fb_script). My actual solution is to create a firstboot script, this one download a script and add an entry in /etc/inittab, the downloaded script do the job and remove the entry in /etc/inittab at the end of its execution.
    • Automatic multibos creation while updating a system trough NIM — or in option.
    • Keep mksysb the way it is, this is the best bare metal backup I ever known.
    • Getting rid of rsh, force user to use nimsh (for nimadm too).
    • Better design for high availability (HANIM auto sync for example).
    • NIM Database flexibility : Let user renaming an resource object (please do this !!!) — Who has never experienced this problem while creating a SPOT or an lpp_source with an erroneous name ?
    • Allow allocating multiple lpp_source for different installp_bundle for installation.
    • Allow nimadm migration to be performed without the exact same level for bos.alt_disk_install.rte fileset.
    • Allow nimsh to be configured over http or https (no more multiple ports for nimsh ; easier for security, and firewall ruling).
    • Automatically enabled cryptographic authentication for NIM service handler. (nimsh can uses SSL-encrypted certificates).
    • Easier NIM backup and restore, getting rid of m_backup_db and m_restore_db.


    Please comment and react I do need support ;-) . Hope this can help.

Improved control for Live Partition Mobility : Choose your destination fibre channel port

Posted on by

I’m working in an virtualized environnement fully based on NPIV. All NPIV lpars can move from one machine to another with Live Partition Mobility. A few months ago I had to move all lpars from one machine to another, and I realized one major drawback of Live Partition Mobility. All Virtual I/O Servers were configured with two or four fibre channel adapters. When a mobility operation is performed the moving lpar is always using the first fibre channel adapter. If you have 60 lpars on a machine it can be a big problem : fibre channel switch port can be saturated. With the new release of PowerVM, you can now choose the destination fibre channel adapter and all lpars can be distributed among all fibre channel adapters.

Prerequisites

Be sure Hardware Management Console and Virtual I/O Server are up to date with the latest version :

  • Hardware Management Console version has to be 7.6.0 :
    • # lshmc -V
"version= Version: 7
 Release: 7.6.0
 Service Pack: 1
HMC Build level 20121109.1
","base_version=V7R7.5.0
"
  • Source and destination Virtual I/O Servers have to be 2.2.2.1 :
    • # ioslevel
2.2.2.1

Mobility

Here is an example : all virtual fibre channel adapters are mapped on the same fibre channel adapter : fcs0 is mapped to seven virtual fibre channel adapters, and fcs1 to none, this is a result of multiple mobility operations :

# lsnports
name             physloc                        fabric tports aports swwpns  awwpns
fcs0             U5803.001.9ZZ03PZ-P1-C2-T1          1     64     57   2048    2021
fcs1             U5803.001.9ZZ03PZ-P1-C2-T2          1     64     64   2048    2048

# lsmap -all -npiv | grep "FC name"
FC name:fcs0                    FC loc code:U5803.001.9ZZ03PZ-P1-C2-T1
FC name:fcs0                    FC loc code:U5803.001.9ZZ03PZ-P1-C2-T1
FC name:fcs0                    FC loc code:U5803.001.9ZZ03PZ-P1-C2-T1
FC name:fcs0                    FC loc code:U5803.001.9ZZ03PZ-P1-C2-T1
FC name:fcs0                    FC loc code:U5803.001.9ZZ03PZ-P1-C2-T1
FC name:fcs0                    FC loc code:U5803.001.9ZZ03PZ-P1-C2-T1
FC name:fcs0                    FC loc code:U5803.001.9ZZ03PZ-P1-C2-T1

Choosing the destination fibre channel adapter can only be done using command line (I hope a dialog box will be available with the next Hardware Management Console release). You have to choose the destination adapter by your own :

  • Before the mobility operation 64 aports are available on fcs5 on this Virtual I/O Server :
    • # lsnports
name             physloc                        fabric tports aports swwpns  awwpns
fcs2             U5803.001.9ZZ03PZ-P2-C2-T1          1     64     64   2048    2048
fcs3             U5803.001.9ZZ03PZ-P2-C2-T2          1     64     64   2048    2048
fcs4             U5803.001.9ZZ03PZ-P2-C3-T1          1     64     64   2048    2048
fcs5             U5803.001.9ZZ03PZ-P2-C3-T2          1     64     64   2048    2048
  • The mobility operation is launched by the command line only, as you can see fibre channel adapter fcs5 is used on this mobility operation :
    • # migrlpar -o m -m P795-SRC -t P795-DST -p lpar-test -w 1 -i 'virtual_fc_mappings="10/vios1/15//fcs5,11/vios2/16//fcs5",source_msp_name=vios3,dest_msp_name=vios1,shared_proc_pool_name=shp_test'
  • After the mobility operation 63 aports are available on fcs5 on the destination Virtual I/O Server :
    • # lsnports
name             physloc                        fabric tports aports swwpns  awwpns
fcs2             U5803.001.9ZZ03PZ-P2-C2-T1          1     64     64   2048    2048
fcs3             U5803.001.9ZZ03PZ-P2-C2-T2          1     64     64   2048    2048
fcs4             U5803.001.9ZZ03PZ-P2-C3-T1          1     64     64   2048    2048
fcs5             U5803.001.9ZZ03PZ-P2-C3-T2          1     64     63   2048    2045

Use case

After moving all my lpars from one machine to another here is the result : all lpars fibre channel adapters are distributed among all the real fibre channel adapters.

# lsnports
name             physloc                        fabric tports aports swwpns  awwpns
fcs0             U5803.001.9ZZ03PZ-P2-C6-T1          1     64     43   2048    1967
fcs1             U5803.001.9ZZ03PZ-P2-C6-T2          1     64     58   2048    2030
fcs4             U5803.001.9ZZ03PZ-P2-C8-T1          1     64     55   2048    2017
fcs5             U5803.001.9ZZ03PZ-P2-C8-T2          1     64     59   2048    2033

# /usr/ios/cli/ioscli lsmap -all -npiv | grep "FC name" | sort | uniq -c
  12 FC name:fcs0                    FC loc code:U5803.001.9ZZ03PZ-P2-C6-T1
   8 FC name:fcs1                    FC loc code:U5803.001.9ZZ03PZ-P2-C6-T2
   8 FC name:fcs4                    FC loc code:U5803.001.9ZZ03PZ-P2-C8-T1
   9 FC name:fcs5                    FC loc code:U5803.001.9ZZ03PZ-P2-C8-T2

Hope this can help.

Adventures in IBM Systems Director in System P environment. Part 5: VMcontrol and Shared Storage Pool

Posted on by

I am working on IBM Systems Director since almost one year. I remember how I was frustrated when someone from IBM told me that I can’t use VMcontrol because of our SAN environment (CISCO Switches + EMC 40K Storage array). My first question after this statement was “OK no problem, i’ll use it over a Shared Storage Pool”; I was even more frustrated when the answer was “Uh, not yet supported”. Fortunately there was a “yet” in this sentence. This happens six months ago. With the IBM Systems Director update to 6.3.2, and the new 2.2.2.1 Virtual I/O Server version this fonctionality is now supported. I have successfully implemented VMcontrol over a Shared Storage Pool and I doesn’t have enough words to tell you how incredible it is. AWESOME. Here are my tips and tricks to setup VMcontrol over a Shared Storage Pool (v3). Enjoy :

Prerequisites

Before trying to deploy a Virtual appliance, or even to capture one, check all points one by one :

  • Update IBM System Director to 6.3.2 :
    • # smcli lsver
6.3.2
  • Ensure that all your Virtual I/O Servers participing in the Shared Storage Pool are 2.2.2.1 Virtual I/O Server (I highly encourage you to install ifixes IV31624m0a and IV32091s0a) :
    • # ioslevel
2.2.2.1
# oem_setup_env
# emgr -l
[..]
1    S    IV31624m0a 12/17/12 14:30:52            VIOS 2.2.2.1 + ifixes
2    S    IV32091s0a 12/17/12 14:31:10            cleanup fails on source vios
[..]
  • On all your Virtual I/O Servers ensure that the common agent is correctly installed with the viocluster subagent in 6.3.2 version :
    • # /opt/ibm/director/agent/bin/lwiupdatemgr.sh -listFeatures | grep -i vios
com.ibm.director.hw.power.vioscluster.agent.feature_6.2.1.10 Disabled
com.ibm.director.hw.power.vioscluster.agent.feature_6.3.2.0 Enabled
com.ibm.director.hw.power.vioscluster.agent.installer.feature_6.3.2 Enabled
  • On your NIM (Network Installation Manager) server, ensure that the common agent is correctly installed with the common repository subagent, and the nim subagent :
    • # /opt/ibm/director/agent/bin/lwiupdatemgr.sh -listFeatures | grep -E "nim|cr"
com.ibm.director.im.cr.agent.installer_2.4.2.0-201211131517 Enabled
com.ibm.director.im.rf.nim.subagent_2.4.1 Disabled
com.ibm.director.im.rf.nim.subagent_2.4.2.0-201211131517 Enabled
  • dsm, openssl, and openssh filesets have to be installed on the NIM server :
    • # lslpp -Lc | grep -E "dsm.core|openssh|openssl"
dsm:dsm.core:7.1.2.0: : :C:F:Distributed Systems Management Core: : : : : : :0:0:/:1241
openssh.base:openssh.base.client:6.0.0.6100: : :C: :Open Secure Shell Commands: : : : : : :0:0:/:
openssh.base:openssh.base.server:6.0.0.6100: : :C: :Open Secure Shell Server: : : : : : :0:0:/:
openssh.license:openssh.license:5.8.0.6102: : :C: :Open Secure Shell License: : : : : : :0:0:/:
openssh.man.en_US:openssh.man.en_US:6.0.0.6100: : :C: :Open Secure Shell Documentation - U.S. English: : : : : : :0:0:/:
openssl.base:openssl.base:0.9.8.2400: : :C: :Open Secure Socket Layer: : : : : : :0:0:/:
openssl.license:openssl.license:0.9.8.2400: : :C: :Open Secure Socket License: : : : : : :0:0:/:
openssl.man.en_US:openssl.man.en_US:0.9.8.2400: : :C: :Open Secure Socket Layer: : : : : : :0:0:/:

As always “discover, access, inventory”

On all IBM Systems Director objects involved in VMcontrol (Virtual I/O Servers, NIM Server, Pseries on which virtual appliance will be deployed, Pseries on which virtual appliance will be captured, HMC controlling theses Pseries) ensure that access is granted, and a full inventory has been collected :

  • Virtual I/O Server (if someone knows how to get “Last Collected Inventory” from command line it’ll be useful) :
    • # smcli lssys -oT -A AccessState vios1,vios2,vios3,vios4
vios1, Server, 0x5067e: Unlocked
vios1, OperatingSystem, 0x50646: Unlocked
vios2, OperatingSystem, 0x5064c: Unlocked
vios2, Server, 0x50707: Unlocked
vios3, Server, 0x506a2: Unlocked
vios3, OperatingSystem, 0x50654: Unlocked
vios4, Server, 0x506e3: Unlocked
vios4, OperatingSystem, 0x5065c: Unlocked

  • NIM Server :
    • # smcli lssys -oT -A AccessState nim
nim, OperatingSystem, 0x28a28: Unlocked
nim, Server, 0x47d73: Unlocked
  • Pseries :
    • # smcli lssys -oT -A AccessState P520-TST-1,P520-TST-2
P520-TST-1, Server, 0x18312: Unlocked
P520-TST-2, Server, 0x1830c: Unlocked

  • HMC :
    • # smcli lssys -oT -A AccessState hmc1
hmc1, HardwareManagementConsole, 0x33bfe: Unlocked

If everything is going well, IBM Systems Director has now discovered a new Virtual I/O Cluster and a new Shared Storage Pool associated to it :

  • Virtual I/O Server :
    • # smcli lssys -le vio000tst-cluster
vio000tst-cluster:
    DisplayName (Name) : vio000tst-cluster (vio000tst-cluster)
    Description (Description) : Storage Manageable Endpoint (Storage System)
    SerialNumber (Serial Number) : a549bcf4c77311e18f5400215e487480 (a549bcf4c77311e18f5400215e487480)
    MachineType (Machine Type) : VIOS Cluster (VIOS Cluster)
    PrimaryHostName (Primary Host Name) : 10.10.122.109 (10.10.122.109)
    Manufacturer (Manufacturer) : IBM (IBM)
    AccessState (Access State) : Unlocked (Full Access)
    CommunicationState (Communication State) : 2 (Communication OK)
    Model (Model) : VIOS Cluster (VIOS Cluster)
    CreatedDate (Created Date) : 2012-12-17T19:35:08+01:00 (2012-12-17T19:35:08+01:00)
    ChangedDate (Changed Date) : 2012-12-31T11:14:00+01:00 (2012-12-31T11:14:00+01:00)
    CurrentTimeZone (Agent Time Zone Offset) : -1 ()
    IPv4Address (IP Addresses) : { '10.10.122.109', '10.10.122.107', '10.10.122.108', '10.10.122.110' } (10.10.122.109, 10.10.122.107, 10.10.122.108, 10.10.122.110)
    HostName (IP Hosts) : { '10.10.122.109', '10.10.122.107', '10.10.122.108', '10.10.122.110' } (10.10.122.109, 10.10.122.107, 10.10.122.108, 10.10.122.110)
    OperatingState (State) : 0 (Unknown)
    DisplayPingTime (Query Vital Properties Interval) : 2 (Every hour)
    DisplayOperationalStatusTime (Verify Connection Interval) : 3 (Every 15 minutes)

  • Shared Storage Pool :
    • # smcli lssspstoragepool -C 0x4fb69 -l
vio000tst-ssp:
        OID : 0x4fb70
        Capacity : 65,280
        RemainingManagedSpace : 30,574
        Threshold : 95
        CreatedDate : 12/17/12 7:35 PM
        ChangedDate : 12/31/12 11:12 AM
# smcli lssharedstpool -l
vio000tst-cluster:
        OID : 0x4fb69
        PrimaryHostName : 10.10.122.109
        CreatedDate : 12/17/12 7:35 PM
        ChangedDate : 12/31/12 11:14 AM
# smcli lssspviosvs -C 0x4fb69
vios1, vios2, vios3, vios4
# smcli lssspphysvol -C 0x4fb69 -l
Repository Volumes
hdisk5:
        OID : 0x522bb
        UDID : 1D0667520609SYMMETRIX03EMCfcp
        TotalSize (MB) : 32768
        DeviceID on end-point : hdisk5
        CreatedDate : 12/18/12 11:12 AM
        ChangedDate : 12/31/12 11:15 AM

Storage Pool Volumes
hdisk6:
        OID : 0x522c3
        UDID : 1D0667520709SYMMETRIX03EMCfcp
        TotalSize (MB) : 32768
        DeviceID on end-point : hdisk6
        CreatedDate : 12/18/12 11:12 AM
        ChangedDate : 12/31/12 11:15 AM

hdisk7:
        OID : 0x522c1
        UDID : 1D0667520809SYMMETRIX03EMCfcp
        TotalSize (MB) : 32768
        DeviceID on end-point : hdisk7
        CreatedDate : 12/18/12 11:12 AM
        ChangedDate : 12/31/12 11:15 AM

  • Check the Shared Storage Pool is present in the storage managment tab :


Step by step VMcontrol workflow

Here is a step by step workflow for the deployment of a new Workload :

  • 1/ Prerequistes sum up :
    • Ensure that IBM Systems Director version is 6.3.2.
    • Install the IBM System Director Common Agent with the Virtual I/O Server Cluster Subagent on each Shared Storage Pool’s Virtual I/O Server.
    • Install the IBM System Director Common Agent with the NIM Subagent and Common Repository Subagent on NIM Server.
    • Run a full inventory on each IBM System Director objects used by VMcontrol.
    • Check that the Virtual I/O Cluster and the Shared Storage Pool are discovered and accessible.
  • 2/ Create a Common Repository on NIM Server using ‘mkrepos’ command on IBM Systems Director. (Repository can also be created directly on the Virtual I/O Cluster.)
  • 3/ Capture or import a Virtual Appliance using ‘captureva’ or ‘importva’ command on IBM Systems Director. A Virtual Appliance can be capture from :
    • a mksysb.
    • a lpp_source.
    • a Virtual Server (a lpar) (lpar must be powered off to be captured).
    • an existing Virtual appliance.
  • 4/ Using ‘deployva’ command from IBM Systems Director deploy the previously captured Virtual Appliance. Use ‘lscustomization’ command to check available and compatible parameters. A Virtual Appliance can be deployed on :
    • A Server (if deployed on a Server Virtual Appliance resilience will not be active).
    • A System Pool (if deployed on a System Pool Virtual Appliance resilience will be active, and Virtual Appliance can automatically be relocated based on user defined criteria)
  • 5/ Using the Hardware Management Console IBM Systems Director create the new logical partition :
    • If you’re using dual Virtual I/O Servers two client scsi adapters will be created on the new logical parition.
    • On each Virtual I/O Servers a server scsi adapter will be created.
    • A new backing device will be created in the Shared Storage Pool.
  • 6/ IBM System Director will ‘prepare’ the NIM server trough the NIM subagent :
    • Management Object will be created (a hmc object, a cec object).
    • If this is the first deploy of the Virtual Appliance an associated spot will be created.
    • A machine object will be created (be careful, NIM has to resolve the hostname).
    • scripts, resolv_conf, bosinst_data object will be created
  • 7/ Previously created resources are correctly exported to enable the installation of the new Workload.
  • 8/ Logical partition is booted and installed using Hardware Management Console (lpar_netboot).
  • 9/ Resources are unexported.
  • 10/ Inventory is collected on the new created Workload.

Click on the image to enlarge it, this is how VMcontrol is working :

Common image repository creation

VMcontrol needs an Image Repository to store captured Virtual Appliances, a Common Image Repository can be created on a NIM Server or on a Virtual I/O Server. I’ve created 2 repositories, one on the NIM server and the other one on a Virtual I/O Server. Here is an example : how to create a Common Image Repository on a Virtual I/O Server. My “main” Common Image Repository was created on my NIM server, this is the one used for the rest of this post.

  • Use the ‘mkrepos’ command to identify the storage OID, in this case 326512 :
    • # smcli mkrepos -C | grep -ip vios1
vios1 (329286)
repositorystorage
        Min:    1
        Max:    1
        Description:    null
        Options:
        Key,    Storage,        Storage location,       Type,   Available GB,   Total GB,       Description,    OID
        [vio000tst-ssp]         vio000tst-ssp vio000tst-cluster     SAN     23      68              326512
  • Use the ‘lssys’ command to identify the Operating System on which the Common Image Repository will be created :
    • # smcli lssys  -oT  vios1
vios1, Server, 0x5067e
vios1, OperatingSystem, 0x50646
  • Using the Operating System’s OID and the Storage’s OID build the ‘mkrepos’ command and create the common repository :
    • # smcli mkrepos -S 326512 -O 0x50646 -n vio-common-repository
  • All repositories can be listed with ‘lsrepos’ command :
    • # smcli lsrepos -l
nim
        Notifiable:true
        ClassName:com.ibm.usmi.datamodel.software.ImageRepository
        UniqueId:15b69cf1-433d-4bc8-98af-b7ec033797c1
        ImageRepositoryType:1
        ChangedDate:2012-12-19T11:08:29+01:00
        SourceTokens:{ 'NO_IR_DELETE' }
        DisplayName:nim
        CreatedDate:2012-12-19T11:08:29+01:00
        ImagingTool:DISCOVERY_NIM_REPOSITORY
        OID:340561
        Guid:07DFEC27B3763D56976642CD4CE0A493
        ObjectType:ImageRepository
        DisplayNameSpecified:true

vio-common-repository
        Notifiable:true
        ClassName:com.ibm.usmi.datamodel.software.ImageRepository
        UniqueId:6b51515c-e754-458a-b5ed-310fd4b1d380
        ImageRepositoryType:0
        ChangedDate:2013-01-07T16:53:40+01:00
        DisplayName:udivo-common-repository
        CreatedDate:2013-01-07T16:53:40+01:00
        ImagingTool:DISCOVERY_CR_REPOSITORY
        OID:369178
        Guid:7CDBB36682143AD1A4B12E91B2707215
        AgentType:1
        ObjectType:ImageRepository
        DisplayNameSpecified:true

Virtual appliance capture

A captured Virtual Appliance is stored on a repository. Before trying to capture a new Virtual Appliance the first thing to do is to identify the common repository that will be used. As always, I’m working with ID or OID not with names :

  • Identify the common repository with the ‘lsrepos’ command : 
    • # smcli lsrepos -o
nim, 340561 (0x53251)
vio-repository, 338309 (0x52985)

After the repository has been identify use the ‘captureva’ command to capture the virtual appliance, here are two examples, one using an mksysb the second one using an lpp_source on the NIM server :

  • Capturing a virtual appliance for an mksysb :
    • # smcli captureva -v -r 340561 -F repos://export/nim/images/moonstone -n 7100-02-00-1241-virtual_appliance -D "imported from mksysb 7100-02-00-1241" -A "cpushare=0.1,memsize=512"
Wed Dec 19 16:26:49 CET 2012  captureva Operation started.
Attempt to get capture object data from file repos://export/nim/image/moonstone
Update collection with user entered attributes.
Call captureFile function
Call capture command executed. Return code= 340,914
Wed Dec 19 16:27:40 CET 2012  captureva Operation took 51 seconds.
  • Capturing a virtual appliance from an lpp_source :
    • # smcli captureva -vvvv -r 340561 -F repos:6100-08-01-1245-lpp_source -n 6100-08-01-1245-virtual_appliance -D "imported from 6100-08-01-1245-lpp_source" -A "cpushare=0.1,memsize=512"
Thu Dec 27 18:01:05 CET 2012  captureva Operation started.
Attempt to get capture object data from file repos:6100-08-01-1245-lpp_source
Update collection with user entered attributes.
Call captureFile function
Call capture command executed. Return code= 350,520
Thu Dec 27 18:01:38 CET 2012  captureva Operation took 32 seconds.

If a mksysb is used for the capture, a new NIM Object is created :

# lsnim -l appliance-1_image-1
appliance-1_image-1:
   class          = resources
   type           = mksysb
   Rstate         = ready for use
   prev_state     = unavailable for use
   location       = /export/nim/appliances/84dd48b5-2eaa-416c-b70b-fe4fe3c5c6c1/moonstone
   version        = 7
   release        = 1
   mod            = 2
   oslevel_r      = 7100-02
   alloc_count    = 0
   server         = master
   extracted_spot = nimrf-0000000000000005-spot
   creation_date  = Wed Dec 19 16:28:48 2012

All captured Virtual Appliances are stored in /exports/nim/appliances:

# ls /export/nim/appliances
84dd48b5-2eaa-416c-b70b-fe4fe3c5c6c1  bf6c42a3-45c8-4764-835e-0b4dc10a90a4  d5227bc5-85ef-4e82-ba86-b7835652a5f7  lost+found
b907734c-84a5-41eb-a112-db7df014984d  d03f6420-d0e7-4756-8e02-7c2e350cfabb  da0b9051-4046-4341-9b41-b8c6dbefb9e6  version

Each Virtual Appliance is described in an ovf (open visualization format) file. This file can be edited by hand :

# more da0b9051-4046-4341-9b41-b8c6dbefb9e6.ovf

Deploy a new Virtual Appliance

Before trying to deploy a new Virtual Appliance you have to collect some information :

  • What is the OID of the Virtual Appliance to be deployed (use ‘lsva’ command to list virtual appliances):
    • # smcli lsva -o
5300-12-05-1140-virtual_appliance, 346436 (0x54944)
6100-08-01-1245-virtual_appliance, 350520 (0x55938)
7100-02-00-1241-virtual_appliance, 359718 (0x57d26)
  • Is this a new Virtual Appliance (deploy_new : the lpar will be created) or an existing virtual appliance (deploy_existing : an existing lpar will be used to deploy the Virtual Appliance).
  • On which host, or on which system pool will the virtual appliance be deployed (use ‘lsdeploytargets’ to check eligible hosts):
  • On a server :
    • # smcli lsdeploytargets -v -a deploy_new -V 340914 | grep TST
P520-TST-1, (0x18312) (P520-TST-1)
P520-TST-2, (0x1830c) (P520-TST-2)
  • On a systems pool :
    • # smcli lsdeploytargets -v -o -a deploy_new -V 350520 | grep TST
FRMTST-systempool, 359968 (0x57e20) (FRMTST-systempool)
  • Some parameters can be tuned (storage pool used, hostname, ip, etc..). Use ‘lscustomization’ command to check tunable parameters :
    • # smcli lscustomization -a deploy_new -V 350520 -s 0x1830c
[..]
virtualnetworks
        Description:    Network Mapping
        Changeable Columns:
                Column Name*    CLI Attribute
                Virtual Networks on Host        hostVnet

        Options:
        Key,    Network Name,   Description,    Virtual Networks on Host*
        [Network 1]     Network 1       Default network Discovered/1122/0

        Options:        Discovered/1122/0 (Discovered/1122/0 (VLAN 1122, Bridged)),
                        ETHERNET0/1122 (Discovered/1122/0 (VLAN 1122, Bridged)),
                        Discovered/4094/0 (Discovered/4094/0 (VLAN 4094, Not Bridged)),
                        ETHERNET0/4094 (Discovered/4094/0 (VLAN 4094, Not Bridged)),
                        Discovered/999/0 (Discovered/999/0 (VLAN 999, Bridged)),
                        ETHERNET0/999 (Discovered/999/0 (VLAN 999, Bridged))
[..]
poolstorages
        Min:    1
        Max:    1
        Description:    The storage pools available for virtual disk allocation. Used together with the storagemapping parameter.
        Options:
        Key,    Name,   Location,       VIOS Count,     Maximum Allocation (MB),        Description
        [326512]        vio000tst-ssp VIOS Cluster: vio000tst-cluster       2       18209   Shared Storage Pool accessed through one or more VIOS.
[..]
  • Here are two examples of a Virtual Appliance deployment, one on a Server, the second one on a System Pool :
  • On a Server :
    • # smcli deployva -v -s 0x1830c -V 340914 -a deploy_new -A "poolstorages=326512,product.vs0.com.ibm.ovf.vmcontrol.system.networking.hostname=carbon,product.vs0.com.ibm.ovf.vmcontrol.adapter.networking.ipv4addresses.5=10.10.122.239,product.vs0.com.ibm.ovf.vmcontrol.adapter.networking.ipv4netmasks.5=255.255.255.0,product.vs0.com.ibm.ovf.vmcontrol.system.networking.ipv4defaultgateway=10.10.122.254,product.vs0.com.ibm.ovf.vmcontrol.system.networking.dnsIPaddresses=10.20.74.196 10.20.2.251,product.vs0.com.ibm.ovf.vmcontrol.system.networking.domainname=domain.test"
Fri Dec 21 11:48:03 CET 2012  deployva Operation started.
Attempt to get the default customization data for deploy_new.
Attempt to get the deploy_new customization data.
Update collection with user entered attributes.
Attempt to validate the deploy request for 340,914.
Attempt to deploy new.
Workload 7100-02-00-1241-virtual_appliance_56750 was created.
DNZLOP412I Deploying virtual appliance 7100-02-00-1241-virtual_appliance to server P520-TST-2.
DNZLOP412I Deploying virtual appliance 7100-02-00-1241-virtual_appliance to server carbon.
DNZLOP401I Booting virtual server carbon to the Open Firmware state.
DNZLOP402I Gathering network adapter information for virtual server carbon.
DNZLOP405I Initiating deploy processing on the NIM master.
Virtual server carbon added to workload 7100-02-00-1241-virtual_appliance_56750.
Workload 7100-02-00-1241-virtual_appliance_56750 is stopped.
DNZIMC094I Deployed Virtual Appliance 7100-02-00-1241-virtual_appliance to new Server carbon hosted by system P520-TST-2.
Fri Dec 21 12:13:14 CET 2012  deployva Operation took 1510 seconds.
  • On a System Pool :
    • smcli deployva -v -V 350520 -g 0x57e20 -m -1093167946908409598_01 -a deploy_new -A "poolstorages=326512,product.vs0.com.ibm.ovf.vmcontrol.system.networking.hostname=carbon,product.vs0.com.ibm.ovf.vmcontrol.adapter.networking.ipv4addresses.5=10.10.122.231,product.vs0.com.ibm.ovf.vmcontrol.adapter.networking.ipv4netmasks.5=255.255.255.0,product.vs0.com.ibm.ovf.vmcontrol.system.networking.ipv4defaultgateway=10.10.122.254,product.vs0.com.ibm.ovf.vmcontrol.system.networking.dnsIPaddresses=10.20.74.196 10.20.2.251,product.vs0.com.ibm.ovf.vmcontrol.system.networking.domainname=domain.test"

As you can see, the new Virtual Appliance is created in almost 20 minutes, no so bad ….. Here is a screenshot, with some deployed Virtual Appliances :

What’s next

In my opinion, VMcontrol is very powerfull, deploying a new AIX lpar in 20 minutes is incredible. Combined with a Shared Storage Pool, VMcontrol can easily be used and installed by everyone. In the part 6 of “Adventure in IBM Systems Director” I’ll post about how to create a resilient workload. A resiliant workload has to be created on a System Pool and can be automatically relocated between the System Pool’s hosts. These workloads are monitored with resiliency policy, if some problems are detected, action and relocation are taken to maintain workload resilience. I do not want to talk too much about that in this post, you’ll have to wait the next one.

Hope this can help.