Enhance your AIX packages management with yum and nim over http

As AIX is getting older and older our old favorite OS is still trying to struggle versus the mighty Linux and the fantastic Solaris (no sarcasm in that sentence I truly believe what I say). You may have notice that -with time- IBM is slowly but surely moving from proprietary code to something more open (ie. PowerVC/Openstack projects, integration with Chef, Linux on Power and tons of other examples). I’m a little bit deviating from the main topic of this blog post but speaking about open source I have many things to say. If someone from my company is reading this post please note that it is my point of view … but I’m still sure that we are going the WRONG way not being more open, and not publishing on github. Starting from now every AIX IT shop in world must consider using OpenSource software (git, chef, ansible, zsh and so on) instead of maintaining homemade tools, or worse paying for tools that are 100 % of the time worse than OpenSource tools. Even better, every IT admin and every team must consider sharing their sources with the rest of the world for one single good reason: “Alone we can do so little, together we can do so much”. Every company not considering this today is doomed. Take example on Bloomberg, Facebook (sharing to the world all their Chef’s cookbooks), twitter, they’re all using github to share their opensource projects. Even military, police and banks are doing the same. They’re still secure but they are open to world ready work to make and create things better and better. All of this to introduce you to new things coming on AIX. Instead of reinventing the wheel IBM had the great idea to use already well implanted tools. It was the case for Openstack/PowerVC and it is also for the tools I’ll talk about in this post. It is the case for yum (yellowdog updater modified). Instead of installing rpm packages by hand you now have the possibility to use yum and to definitely end the rpm dependency nightmare that we all had since AIX 5L was released. Next instead of using the proprietary nimsh protocol to install filesets (bff package) you can now tell the nim server and nimclient to this over http/https (secure is only for the authentication as far as I know) (an open protocol :-) ). By doing this you will enhance the way you are managing packages on AIX. Do this now on every AIX system you install, yum everywhere and stop using NFS … we’re now in an http world :-)

yum: the yellow dog updater modified

I’m not going to explain you what yum is. If you don’t know you’re not in the right place. Just note that my advice starting from now is to use yum to install every software of the AIX toolbox (ftp://ftp.software.ibm.com/aix/freeSoftware/aixtoolbox/RPMS/ppc/). IBM is providing an official repository than can be mirrored on your own site to avoid having to use a proxy or having an access to the internet from you servers (you must admit that this is almost impossible and every big company will try to avoid this). Let’s start by trying to install yum:

Installing yum

IBM is providing an archive with all the needed rpm mandatory to use and install yum on an AIX server, you can find this archive here: ftp://public.dhe.ibm.com/aix/freeSoftware/aixtoolbox/ezinstall/ppc/yum_bundle_v1.tar. Just download it and install every rpm in it and yum will be available on you system, simple as that:

A specific version of rpm binary command is mandatory to use yum. Before doing anything update the rpm.rte fileset. As AIX is rpm “aware” it already have an rpm database, but this one will not be manageable by yum. The installation of rpm in a version greater than 4.9.1.3 is needed. This installation will migrate the existing rpm database to a new one usable by yum. The fileset in the right version can be found here ftp://ftp.software.ibm.com/aix/freeSoftware/aixtoolbox/INSTALLP/ppc/

  • By default the rpm command is installed by an AIX fileset:
  • # which rpm
    /usr/bin/rpm
    # lslpp -w /usr/bin/rpm
      File                                        Fileset               Type
      ----------------------------------------------------------------------------
      /usr/bin/rpm                                rpm.rte               File
    # rpm --version
    RPM version 3.0.5
    
  • The rpm database is located in /usr/opt/freeware/packages :
  • # pwd
    /usr/opt/freeware/packages
    # ls -ltr
    total 5096
    -rw-r--r--    1 root     system         4096 Jul 01 2011  triggerindex.rpm
    -rw-r--r--    1 root     system         4096 Jul 01 2011  conflictsindex.rpm
    -rw-r--r--    1 root     system        20480 Jul 21 00:54 nameindex.rpm
    -rw-r--r--    1 root     system        20480 Jul 21 00:54 groupindex.rpm
    -rw-r--r--    1 root     system      2009224 Jul 21 00:54 packages.rpm
    -rw-r--r--    1 root     system       647168 Jul 21 00:54 fileindex.rpm
    -rw-r--r--    1 root     system        20480 Jul 21 00:54 requiredby.rpm
    -rw-r--r--    1 root     system        81920 Jul 21 00:54 providesindex.rpm
    
  • Install the rpm.rte fileset in the right version (4.9.1.3):
  • # file rpm.rte.4.9.1.3
    rpm.rte.4.9.1.3: backup/restore format file
    # installp -aXYgd . rpm.rte
    +-----------------------------------------------------------------------------+
                        Pre-installation Verification...
    +-----------------------------------------------------------------------------+
    Verifying selections...done
    Verifying requisites...done
    Results...
    
    SUCCESSES
    ---------
      Filesets listed in this section passed pre-installation verification
      and will be installed.
    
      Selected Filesets
      -----------------
      rpm.rte 4.9.1.3                             # RPM Package Manager
    [..]
    #####################################################
            Rebuilding RPM Data Base ...
            Please wait for rpm_install background job termination
            It will take a few minutes
    [..]
    Installation Summary
    --------------------
    Name                        Level           Part        Event       Result
    -------------------------------------------------------------------------------
    rpm.rte                     4.9.1.3         USR         APPLY       SUCCESS
    rpm.rte                     4.9.1.3         ROOT        APPLY       SUCCESS
    
  • After the installation check you have the correct version of rpm, you can also notice some changes in the rpm database files:
  • # rpm --version
    RPM version 4.9.1.3
    # ls -ltr /usr/opt/freeware/packages
    total 25976
    -rw-r--r--    1 root     system         4096 Jul 01 2011  triggerindex.rpm
    -rw-r--r--    1 root     system         4096 Jul 01 2011  conflictsindex.rpm
    -rw-r--r--    1 root     system        20480 Jul 21 00:54 nameindex.rpm
    -rw-r--r--    1 root     system        20480 Jul 21 00:54 groupindex.rpm
    -rw-r--r--    1 root     system      2009224 Jul 21 00:54 packages.rpm
    -rw-r--r--    1 root     system       647168 Jul 21 00:54 fileindex.rpm
    -rw-r--r--    1 root     system        20480 Jul 21 00:54 requiredby.rpm
    -rw-r--r--    1 root     system        81920 Jul 21 00:54 providesindex.rpm
    -rw-r--r--    1 root     system            0 Jul 21 01:08 .rpm.lock
    -rw-r--r--    1 root     system         8192 Jul 21 01:08 Triggername
    -rw-r--r--    1 root     system         8192 Jul 21 01:08 Conflictname
    -rw-r--r--    1 root     system        28672 Jul 21 01:09 Dirnames
    -rw-r--r--    1 root     system       221184 Jul 21 01:09 Basenames
    -rw-r--r--    1 root     system         8192 Jul 21 01:09 Sha1header
    -rw-r--r--    1 root     system         8192 Jul 21 01:09 Requirename
    -rw-r--r--    1 root     system         8192 Jul 21 01:09 Obsoletename
    -rw-r--r--    1 root     system         8192 Jul 21 01:09 Name
    -rw-r--r--    1 root     system         8192 Jul 21 01:09 Group
    -rw-r--r--    1 root     system       815104 Jul 21 01:09 Packages
    -rw-r--r--    1 root     system         8192 Jul 21 01:09 Sigmd5
    -rw-r--r--    1 root     system         8192 Jul 21 01:09 Installtid
    -rw-r--r--    1 root     system        86016 Jul 21 01:09 Providename
    -rw-r--r--    1 root     system       557056 Jul 21 01:09 __db.004
    -rw-r--r--    1 root     system     83894272 Jul 21 01:09 __db.003
    -rw-r--r--    1 root     system      7372800 Jul 21 01:09 __db.002
    -rw-r--r--    1 root     system        24576 Jul 21 01:09 __db.001
    

Then install yum. Please note that I already have some rpm installed on my current system that’s why I’m not installing db, or gdbm. If your system is free of any rpm install all the rpm found in the archive:

# tar xvf yum_bundle_v1.tar
x curl-7.44.0-1.aix6.1.ppc.rpm, 584323 bytes, 1142 media blocks.
x db-4.8.24-3.aix6.1.ppc.rpm, 2897799 bytes, 5660 media blocks.
x gdbm-1.8.3-5.aix5.2.ppc.rpm, 56991 bytes, 112 media blocks.
x gettext-0.10.40-8.aix5.2.ppc.rpm, 1074719 bytes, 2100 media blocks.
x glib2-2.14.6-2.aix5.2.ppc.rpm, 1686134 bytes, 3294 media blocks.
x pysqlite-1.1.7-1.aix6.1.ppc.rpm, 51602 bytes, 101 media blocks.
x python-2.7.10-1.aix6.1.ppc.rpm, 23333701 bytes, 45574 media blocks.
x python-devel-2.7.10-1.aix6.1.ppc.rpm, 15366474 bytes, 30013 media blocks.
x python-iniparse-0.4-1.aix6.1.noarch.rpm, 37912 bytes, 75 media blocks.
x python-pycurl-7.19.3-1.aix6.1.ppc.rpm, 162093 bytes, 317 media blocks.
x python-tools-2.7.10-1.aix6.1.ppc.rpm, 830446 bytes, 1622 media blocks.
x python-urlgrabber-3.10.1-1.aix6.1.noarch.rpm, 158584 bytes, 310 media blocks.
x readline-6.1-2.aix6.1.ppc.rpm, 489547 bytes, 957 media blocks.
x sqlite-3.7.15.2-2.aix6.1.ppc.rpm, 1334918 bytes, 2608 media blocks.
x yum-3.4.3-1.aix6.1.noarch.rpm, 1378777 bytes, 2693 media blocks.
x yum-metadata-parser-1.1.4-1.aix6.1.ppc.rpm, 62211 bytes, 122 media blocks.
# rpm -Uvh curl-7.44.0-1.aix6.1.ppc.rpm glib2-2.14.6-2.aix5.2.ppc.rpm pysqlite-1.1.7-1.aix6.1.ppc.rpm python-2.7.10-1.aix6.1.ppc.rpm python-devel-2.7.10-1.aix6.1.ppc.rpm python-iniparse-0.4-1.ai
x6.1.noarch.rpm python-pycurl-7.19.3-1.aix6.1.ppc.rpm python-tools-2.7.10-1.aix6.1.ppc.rpm python-urlgrabber-3.10.1-1.aix6.1.noarch.rpm yum-3.4.3-1.aix6.1.noarch.rpm yum-metadata-parser-1.1.4-
1.aix6.1.ppc.rpm
# Preparing...                ########################################### [100%]
   1:python                 ########################################### [  9%]
   2:pysqlite               ########################################### [ 18%]
   3:python-iniparse        ########################################### [ 27%]
   4:glib2                  ########################################### [ 36%]
   5:yum-metadata-parser    ########################################### [ 45%]
   6:curl                   ########################################### [ 55%]
   7:python-pycurl          ########################################### [ 64%]
   8:python-urlgrabber      ########################################### [ 73%]
   9:yum                    ########################################### [ 82%]
  10:python-devel           ########################################### [ 91%]
  11:python-tools           ########################################### [100%]

Yum is now ready to be configured and used :-)

# which yum
/usr/bin/yum
# yum --version
3.4.3
  Installed: yum-3.4.3-1.noarch at 2016-07-20 23:24
  Built    : None at 2016-06-22 14:13
  Committed: Sangamesh Mallayya  at 2014-05-29

Setting up yum and you private yum repository for AIX

A private repository

As nobody wants to use the official IBM repository available directly on internet the goal here is to create your own repository. Download all the content of the official repository and “serve” this directory (the one where you download all the rpms) on an private http server (yum is using http/https obviously :-) ).

  • Using wget download the content of the whole official repository. You can notice here that IBM is providing the metadata needed (repodata directory) (if you don’t have this repodata directory yum can’t work properly. This one can be created using the createrepo command available on akk good Linux distros :-) ):
  • # wget -r ftp://public.dhe.ibm.com/aix/freeSoftware/aixtoolbox/RPMS/ppc/
    # ls -ltr
    [..]
    drwxr-xr-x    2 root     system         4096 Jul 11 22:08 readline
    drwxr-xr-x    2 root     system          256 Jul 11 22:08 rep-gtk
    drwxr-xr-x    2 root     system         4096 Jul 11 22:08 repodata
    drwxr-xr-x    2 root     system         4096 Jul 11 22:08 rpm
    drwxr-xr-x    2 root     system         4096 Jul 11 22:08 rsync
    drwxr-xr-x    2 root     system          256 Jul 11 22:08 ruby
    drwxr-xr-x    2 root     system          256 Jul 11 22:09 rxvt
    drwxr-xr-x    2 root     system         4096 Jul 11 22:09 samba
    drwxr-xr-x    2 root     system          256 Jul 11 22:09 sawfish
    drwxr-xr-x    2 root     system          256 Jul 11 22:09 screen
    drwxr-xr-x    2 root     system          256 Jul 11 22:09 scrollkeeper
    
  • Configure you web server (here it’s just an alias because I’m using my http server for other things):
  • # more httpd.conf
    [..]
    Alias /aixtoolbox/  "/apps/aixtoolbox/"
    
        Options Indexes FollowSymLinks MultiViews
        AllowOverride None
        Require all granted
    
    
  • Restart your webserver and check you repository is accessible:
  • repo

  • That’s it the private repository is ready.

Configuring yum

On the client just modify the /opt/freeware/etc/yum/yum.conf or add a file in /opt/freeware/etc/yum/yum.repos.d to point to your private repository:

# cat /opt/freeware/etc/yum/yum.conf
[main]
cachedir=/var/cache/yum
keepcache=1
debuglevel=2
logfile=/var/log/yum.log
exactarch=1
obsoletes=1

[AIX_Toolbox]
name=AIX ToolBox Repository
baseurl=http://nimserver:8080/aixtoolbox/
enabled=1
gpgcheck=0

# PUT YOUR REPOS HERE OR IN separate files named file.repo
# in /etc/yum/repos.d

That’s it the client is ready.

Chef recipe to install and configre yum

My readers all knows that I’m using Chef as a configuration management tools. As you are going to do this on every single system you have I think giving you the Chef recipe installing and configuring yum can be useful (if you don’t care about it just skip it and go to the next session). If you are not using a configuration management tool maybe this simple example will help you to move on and stop doing this by hand or writing ksh scripts. I have to do that on tons of system so for me it’s just mandatory. Here is my recipe to do all the job, configuring and installing yum, and installing some RPM:

directory '/var/tmp/yum' do
  action :create
end

remote_file '/var/tmp/yum/rpm.rte.4.9.1.3'  do
  source "http://#{node['nimserver']}/powervc/rpm.rte.4.9.1.3"
  action :create
end

execute "Do the toc" do
  command 'inutoc /var/tmp/yum'
  not_if { File.exist?('/var/tmp/yum/.toc') }
end

bff_package 'rpm.rte' do
  source '/var/tmp/yum/rpm.rte.4.9.1.3'
  action :install
end

tar_extract "http://#{node['nimserver']/powervc/yum_bundle_v1.tar" do
  target_dir '/var/tmp/yum'
  compress_char ''
  user 'root'
  group 'system'
end

# installing some rpm needed for yum
for rpm in [ 'curl-7.44.0-1.aix6.1.ppc.rpm', 'python-pycurl-7.19.3-1.aix6.1.ppc.rpm', 'python-urlgrabber-3.10.1-1.aix6.1.noarch.rpm', 'glib2-2.14.6-2.aix5.2.ppc.rpm', 'yum-metadata-parser-1.1.4-1.aix6.1.ppc.rpm', 'python-iniparse-0.4-1.aix6.1.noarch.rpm', 'pysqlite-1.1.7-1.aix6.1.ppc.rpm'  ]
  execute "installing yum" do
    command "rpm -Uvh /var/tmp/yum/#{rpm}"
    not_if "rpm -qa | grep $(echo #{rpm} | sed 's/.aix6.1//' | sed 's/.aix5.2//' | sed 's/.rpm//')"
  end
end

# updating python
execute "updating python" do
  command "rpm -Uvh /var/tmp/yum/python-devel-2.7.10-1.aix6.1.ppc.rpm /var/tmp/yum/python-2.7.10-1.aix6.1.ppc.rpm"
  not_if "rpm -qa | grep python-2.7.10-1"
end

# installing yum
execute "installing yum" do
  command "rpm -Uvh /var/tmp/yum/yum-3.4.3-1.aix6.1.noarch.rpm"
  not_if "rpm -qa | grep yum-3.4.3.1.noarch"
end

# changing yum configuration
template '/opt/freeware/etc/yum/yum.conf' do
  source 'yum.conf.erb'
end

# installing some software with aix yum
for soft in [ 'bash', 'bzip2', 'curl', 'emacs', 'gzip', 'screen', 'vim-enhanced', 'wget', 'zlib', 'zsh', 'patch', 'file', 'lua', 'nspr', 'git' ] do
  execute "install #{soft}" do
    command "yum -y install #{soft}"
  end
end

# removing temporary file
execute 'removing /var/tmp/yum' do
  command 'rm -rf /var/tmp/yum'
  only_if { File.exists?('/var/tmp/yum')}
end

chef_yum1
chef_yum2
chef_yum3

After running the chef recipe yum is fully usable \o/ :

chef_yum4

Using yum on AIX: what you need to know

yum is usable just like it is on a Linux system. You may hit some issues when using yum on AIX. For instance you can have this kind of errors:

# yum check
AIX-rpm-7.2.0.1-2.ppc has missing requires of rpm
AIX-rpm-7.2.0.1-2.ppc has missing requires of popt
AIX-rpm-7.2.0.1-2.ppc has missing requires of file-libs
AIX-rpm-7.2.0.1-2.ppc has missing requires of nss

If you are not aware of what is the purpose of AIX-rpm please read this. This rpm is what I call a meta package. It does not install anything. This rpm is used because the rpm database does not know anything about things (binaries, libraries) installed by standard AIX filesets. By default rpm are not “aware” of what is installed by a fileset (bff) but most of rpms depends on things installed by filesets. When you install a fileset … let’s say it install a library like libc.a AIX run the updtvpkg program to rebuild this AIX-rpm and says “this rpm will resolve any rpm dependencies issue for libc.a. So first, never try to uninstall this rpm, second it’s not a real problem is this rpm has missing dependencies …. as it is providing nothing. If you really want to see what dependencies resolve AIX-rpm run the following command:

# rpm -q --provides AIX-rpm-7.2.0.1-2.ppc | grep libc.a
libc.a(aio.o)
# lslpp -w /usr/lib/libc.a
  File                                        Fileset               Type
  ----------------------------------------------------------------------------
  /usr/lib/libc.a                             bos.rte.libc          Symlink

If you want to get rid of these messages just install the missing rpm … using yum:

# yum -y install popt file-libs

A few examples

Here are a few example a software installation using yum:

  • Installing git:
  • # yum install git
    Setting up Install Process
    Resolving Dependencies
    --> Running transaction check
    ---> Package git.ppc 0:4.3.20-4 will be installed
    --> Finished Dependency Resolution
    
    Dependencies Resolved
    
    ================================================================================================================================================================================================
     Package                                    Arch                                       Version                                         Repository                                          Size
    ================================================================================================================================================================================================
    Installing:
     git                                        ppc                                        4.3.20-4                                        AIX_Toolbox                                        215 k
    
    Transaction Summary
    ================================================================================================================================================================================================
    Install       1 Package
    
    Total size: 215 k
    Installed size: 889 k
    Is this ok [y/N]: y
    Downloading Packages:
    Running Transaction Check
    Running Transaction Test
    Transaction Test Succeeded
    Running Transaction
      Installing : git-4.3.20-4.ppc                                                                                                                                                             1/1
    
    Installed:
      git.ppc 0:4.3.20-4
    
    Complete!
    
  • Removing git :
  • # yum remove git
    Setting up Remove Process
    Resolving Dependencies
    --> Running transaction check
    ---> Package git.ppc 0:4.3.20-4 will be erased
    --> Finished Dependency Resolution
    
    Dependencies Resolved
    
    ================================================================================================================================================================================================
     Package                                   Arch                                      Version                                           Repository                                          Size
    ================================================================================================================================================================================================
    Removing:
     git                                       ppc                                       4.3.20-4                                          @AIX_Toolbox                                       889 k
    
    Transaction Summary
    ================================================================================================================================================================================================
    Remove        1 Package
    
    Installed size: 889 k
    Is this ok [y/N]: y
    Downloading Packages:
    Running Transaction Check
    Running Transaction Test
    Transaction Test Succeeded
    Running Transaction
      Erasing    : git-4.3.20-4.ppc                                                                                                                                                             1/1
    
    Removed:
      git.ppc 0:4.3.20-4
    
    Complete!
    
  • List available repo
  • yum repolist
    repo id                                                                                repo name                                                                                          status
    AIX_Toolbox                                                                            AIX ToolBox Repository                                                                             233
    repolist: 233
    

Getting rid of nimsh: USE HTTPS !

A new feature that is now available on latest version of AIX (7.2) allows you to use nim over http. It is a long awaited feature for different reasons (it’s just my opinion). I personally don’t like proprietary protocols such as nimsh and nimsh secure … security teams neither. Who has never experienced installation problems because of nimsh port not opened, because of ids, because of security teams ? Using http or https is the solution? No company is not allowing http or https ! This protocol is so used and secured, widely spread in a lot of products that everybody trust it. I personally prefer opening on single port than struggling opening all nimsh ports. You’ll understand that using http is far better than using nimsh. Before explaining this in details here are a few things you need to now. nimhttp is only available on latest version of AIX (7.2 SP0/1/2), same for the nimclient. If there is a problem using http the nimclient will automatically fallback in an NFS mode. Only certain nim operation are available over http:

Configuring the nim server

To use nim over http (nimhttp) you nim server must be at least deployed on an AIX 7.2 server (mine is updated to the latest service pack (SP2)). Start the service nimhttp on the nim server to allow nim to use http for its operations:

# oslevel -s
7200-00-02-1614
# startsrc -s nimhttp
0513-059 The nimhttp Subsystem has been started. Subsystem PID is 11665728.
# lssrc -a | grep nimhttp
 nimhttp                           11665728     active

The nimhttp service will listen on port 4901, this port is defined in the /etc/services :

# grep nimhttp /etc/services
nimhttp         4901/tcp
nimhttp         4901/udp
# netstat -an | grep 4901
tcp4       0      0  *.4901                 *.*                    LISTEN
# rmsock f1000e0004a483b8 tcpcb
The socket 0xf1000e0004a48008 is being held by proccess 14811568 (nimhttpd).
# ps -ef | grep 14811568
    root 14811568  4456760   0 04:03:22      -  0:02 /usr/sbin/nimhttpd -v

If you want to enable crypto/ssl to encrypt http authentication, just add the -a “-c” to your command line. This “-c” argument will tell nimhttp to start in secure mode and encrypt the authentication:

# startsrc -s nimhttp -a "-c"
0513-059 The nimhttp Subsystem has been started. Subsystem PID is 14811570.
# ps -ef | grep nimhttp
    root 14811570  4456760   0 22:57:51      -  0:00 /usr/sbin/nimhttpd -v -c

Starting the service for the first time will create an httpd.conf file in the root home directory :

# grep ^document_root ~/httpd.conf
document_root=/export/nim/
# grep ^service.log ~/httpd.conf
service.log=/var/adm/ras/nimhttp.log

If you choose to enable the secure authentication nimhttp will use the pem certificates file used by nim. If you are already using secure nimsh you don’t have to run the “nimconfig -c” command. If it is the first time this command will create the two pem files (root and server in /ssl_nim/certs) (check my blog post about secure nimsh for more information about that):

# nimconfig -c
# grep ^ssl. ~/httpd.conf
ssl.cert_authority=/ssl_nimsh/certs/root.pem
ssl.pemfile=/ssl_nimsh/certs/server.pem

The document_root of the http server will define the resource the nim http will “serve”. The default one is /export/nim (default nim place for all nim resources (spot, mksysb, lpp_source) and cannot be changed today (I think it is now ok on SP2, I’ll change the blog post as soon as the test will be done). Unfortunately for me one of my production nim was created by someone not very aware of AIX and … resources are not in /export/nim (I had to recreate my own nim because of that :-( )

On the client side ?

On the client side you just have nothing to do. If you’re using AIX 7.2 and nimhttp is enabled the client will automatically use http for communication (if it is enabled on the nim server). Just note that if you’re using nimhttp in secure mode, you must enable your nimclient in secure mode too:

# nimclient -c
Received 2788 Bytes in 0.0 Seconds
0513-044 The nimsh Subsystem was requested to stop.
0513-077 Subsystem has been changed.
0513-059 The nimsh Subsystem has been started. Subsystem PID is 13500758.
# stopsrc -s nimsh
# startsrc -s nimsh

Changing nimhttp port

You can easily change the port on which nimhttp is listening by modify the /etc/services file. Here is an example with the port 443 (I know this is not a good idea to use this one but it’s just for the example)

#nimhttp                4901/tcp
#nimhttp                4901/udp
nimhttp         443/tcp
nimhttp         443/udp
# stopsrc -s nimhttp
# startsrc -s nimhttp -a "-c"
# netstat -Aan | grep 443
f1000e00047fb3b8 tcp4       0      0  *.443                 *.*                   LISTEN
# rmsock f1000e00047fb3b8 tcpcb
The socket 0xf1000e00047fb008 is being held by proccess 14811574 (nimhttpd).

Same on the client side, just change the /etc/services file and use your nimclient as usual

# grep nimhttp /etc/services
#nimhttp                4901/tcp
#nimhttp                4901/udp
nimhttp         443/tcp
nimhttp         443/udp
# nimclient -l

To be sure I’m not using nfs anymore I’m removing any entries in my /etc/export file. I know that it will just work for some case (some type of resources) as nimesis is filling the file even if this one is empty:

# > /etc/exports
# exportfs -uav
exportfs: 1831-184 unexported /export/nim/bosinst_data/golden-vios-2233-08192014-bosinst_data
exportfs: 1831-184 unexported /export/nim/spot/golden-vios-22422-05072016-spot/usr
exportfs: 1831-184 unexported /export/nim/spot/golden-vios-22410-22012015-spot/usr
exportfs: 1831-184 unexported /export/nim/mksysb
exportfs: 1831-184 unexported /export/nim/hmc
exportfs: 1831-184 unexported /export/nim/lpp_source
[..]

Let’s do this

Let’s now try this with a simple example. I’m here installing powervp on a machine using a cust operation from the nimclient, on the client I’m doing like I have always do running the exact same command as before. Super simple:

# nimclient -o cust -a lpp_source=powervp1100-lpp_source -a filesets=powervp.rte

+-----------------------------------------------------------------------------+
                    Pre-installation Verification...
+-----------------------------------------------------------------------------+
Verifying selections...done
Verifying requisites...done
Results...

SUCCESSES
---------
  Filesets listed in this section passed pre-installation verification
  and will be installed.

  Selected Filesets
  -----------------
  powervp.rte 1.1.0.0                         # PowerVP for AIX

  << End of Success Section >>

+-----------------------------------------------------------------------------+
                   BUILDDATE Verification ...
+-----------------------------------------------------------------------------+
Verifying build dates...done
FILESET STATISTICS
------------------
    1  Selected to be installed, of which:
        1  Passed pre-installation verification
  ----
    1  Total to be installed

+-----------------------------------------------------------------------------+
                         Installing Software...
+-----------------------------------------------------------------------------+

installp: APPLYING software for:
        powervp.rte 1.1.0.0

0513-071 The syslet Subsystem has been added.
Finished processing all filesets.  (Total time:  4 secs).

+-----------------------------------------------------------------------------+
                                Summaries:
+-----------------------------------------------------------------------------+

Installation Summary
--------------------
Name                        Level           Part        Event       Result
-------------------------------------------------------------------------------
powervp.rte                 1.1.0.0         USR         APPLY       SUCCESS
powervp.rte                 1.1.0.0         ROOT        APPLY       SUCCESS

On the server side I’m checking the /var/adm/ras/nimhttp.log (log file for nimhttp) and I can check that files are transferred from the server to the client using the http protocol. So it works great.

# Thu Jul 21 23:44:19 2016        Request Type is GET
Thu Jul 21 23:44:19 2016        Mime not supported
Thu Jul 21 23:44:19 2016        Sending Response Header "200 OK"
Thu Jul 21 23:44:19 2016        Sending file over socket 6. Expected length is 600
Thu Jul 21 23:44:19 2016        Total length sent is 600
Thu Jul 21 23:44:19 2016        handle_httpGET: Entering cleanup statement
Thu Jul 21 23:44:20 2016        nim_http: queue socket create product (memory *)200739e8
Thu Jul 21 23:44:20 2016        nim_http: 200739e8 6 200947e8 20098138
Thu Jul 21 23:44:20 2016        nim_http: file descriptor is 6
Thu Jul 21 23:44:20 2016        nim_buffer: (resize) buffer size is 0
Thu Jul 21 23:44:20 2016        file descriptor is : 6
Thu Jul 21 23:44:20 2016        family is : 2 (AF_INET)
Thu Jul 21 23:44:20 2016        source address is : 10.14.33.253
Thu Jul 21 23:44:20 2016        socks: Removing socksObject 2ff1ec80
Thu Jul 21 23:44:20 2016        socks: 200739e8 132 <- 87 bytes (SSL)
Thu Jul 21 23:44:20 2016        nim_buffer: (append) len is 87, buffer length is 87
Thu Jul 21 23:44:20 2016        nim_http: data string passed to get_http_request: "GET /export/nim/lpp_source/powervp/powervp.1.1.0.0.bff HTTP/1.1

Let's do the same thing with a fileset coming from a bigger lpp_source (in fact an simage one for the latest release of AIX 7.2):

# nimclient -o cust -a lpp_source=7200-00-02-1614-lpp_source -a filesets=bos.loc.utf.en_KE
[..]

Looking on the nim server I notice that files are transfered from the server to the client, but NOT my fileset and it's dependencies .... but the whole lpp_source (seriously ? uh ? why ?)

# tail -f /var/adm/ras/nimhttp.log
Thu Jul 21 23:28:39 2016        Request Type is GET
Thu Jul 21 23:28:39 2016        Mime not supported
Thu Jul 21 23:28:39 2016        Sending Response Header "200 OK"
Thu Jul 21 23:28:39 2016        Sending file over socket 6. Expected length is 4482048
Thu Jul 21 23:28:39 2016        Total length sent is 4482048
Thu Jul 21 23:28:39 2016        handle_httpGET: Entering cleanup statement
Thu Jul 21 23:28:39 2016        nim_http: queue socket create product (memory *)200739e8
Thu Jul 21 23:28:39 2016        nim_http: 200739e8 6 200947e8 20098138
Thu Jul 21 23:28:39 2016        nim_http: file descriptor is 6
Thu Jul 21 23:28:39 2016        nim_buffer: (resize) buffer size is 0
Thu Jul 21 23:28:39 2016        file descriptor is : 6
Thu Jul 21 23:28:39 2016        family is : 2 (AF_INET)
Thu Jul 21 23:28:39 2016        source address is : 10.14.33.253
Thu Jul 21 23:28:39 2016        socks: Removing socksObject 2ff1ec80
Thu Jul 21 23:28:39 2016        socks: 200739e8 132 <- 106 bytes (SSL)
Thu Jul 21 23:28:39 2016        nim_buffer: (append) len is 106, buffer length is 106
Thu Jul 21 23:28:39 2016        nim_http: data string passed to get_http_request: "GET /export/nim/lpp_source/7200-00-02-1614/installp/ppc/X11.fnt.7.2.0.0.I HTTP/1.1

If you have a deeper look of what is nimclient doing when using nimhttp .... he is just transfering the whole lpp_source from the server to the client and then installing the needed fileset from a local filesystem. Filesets are storred into /tmp so be sure you have a /tmp bigger enough to store your biggest lpp_source. Maybe this will be changed in the future but it is like it is for the moment :-) . The nimclient is creating temporary directory named (prefix) "_nim_dir_" to store the lpp_source:

root@nim_server:/export/nim/lpp_source/7200-00-02-1614/installp/ppc# du -sm .
7179.57 .
root@nim_client:/tmp/_nim_dir_5964094/export/nim/lpp_source/7200-00-02-1614/installp/ppc# du -sm .
7179.74 .

More details ?

You can notice while running a cust operation from the nim client that nimhttp is also running in background (on the client itself). The truth is that the nimhttp binary running on client act as an http client. In the output below the http client is getting the file Java8_64.samples.jnlp.8.0.0.120.U and

# ps -ef |grep nim
    root  3342790 16253432   6 23:29:10  pts/0  0:00 /bin/ksh /usr/lpp/bos.sysmgt/nim/methods/c_installp -afilesets=bos.loc.utf.en_KE -alpp_source=s00va9932137:/export/nim/lpp_source/7200-00-02-1614
    root  6291880 13893926   0 23:29:10  pts/0  0:00 /bin/ksh /usr/lpp/bos.sysmgt/nim/methods/c_script -alocation=s00va9932137:/export/nim/scripts/s00va9954403.script
    root 12190194  3342790  11 23:30:06  pts/0  0:00 /usr/sbin/nimhttp -f /export/nim/lpp_source/7200-00-02-1614/installp/ppc/Java8_64.samples.jnlp.8.0.0.120.U -odest -s
    root 13500758  4325730   0 23:23:29      -  0:00 /usr/sbin/nimsh -s -c
    root 13893926 15991202   0 23:29:10  pts/0  0:00 /bin/ksh -c /var/adm/nim/15991202/nc.1469222947
    root 15991202 16974092   0 23:29:07  pts/0  0:00 nimclient -o cust -a lpp_source=7200-00-02-1614-lpp_source -a filesets=bos.loc.utf.en_KE
    root 16253432  6291880   0 23:29:10  pts/0  0:00 /bin/ksh /tmp/_nim_dir_6291880/script

You can use the nimhttp as a client to download file directly from the nim server. Here I'm just listing the content of /export/nim/lpp_source from the client

# nimhttp -f /export/nim/lpp_source -o dest=/tmp -v
nimhttp: (source)       /export/nim/lpp_source
nimhttp: (dest_dir)     /tmp
nimhttp: (verbose)      debug
nimhttp: (master_ip)    nimserver
nimhttp: (master_port)  4901

sending to master...
size= 59
pull_request= "GET /export/nim/lpp_source HTTP/1.1
Connection: close

"
Writing 1697 bytes of data to /tmp/export/nim/lpp_source/.content
Total size of datalen is 1697. Content_length size is 1697.
# cat /tmp/export/nim/lpp_source/.content
DIR: 71-04-02-1614 0:0 00240755 256
DIR: 7100-03-00-0000 0:0 00240755 256
DIR: 7100-03-01-1341 0:0 00240755 256
DIR: 7100-03-02-1412 0:0 00240755 256
DIR: 7100-03-03-1415 0:0 00240755 256
DIR: 7100-03-04-1441 0:0 00240755 256
DIR: 7100-03-05-1524 0:0 00240755 256
DIR: 7100-04-00-1543 0:0 00240755 256
DIR: 7100-04-01-1543 0:0 00240755 256
DIR: 7200-00-00-0000 0:0 00240755 256
DIR: 7200-00-01-1543 0:0 00240755 256
DIR: 7200-00-02-1614 0:0 00240755 256
FILE: MH01609.iso 0:0 00100644 1520027648
FILE: aixtools.python.2.7.11.4.I 0:0 00100644 50140160

Here I'm just downloading a python fileset !

# nimhttp -f /export/nim/lpp_source/aixtools.python.2.7.11.4.I -o dest=/tmp -v
[..]
Writing 65536 bytes of data to /tmp/export/nim/lpp_source/aixtools.python.2.7.11.4.I
Writing 69344 bytes of data to /tmp/export/nim/lpp_source/aixtools.python.2.7.11.4.I
Writing 7776 bytes of data to /tmp/export/nim/lpp_source/aixtools.python.2.7.11.4.I
Total size of datalen is 50140160. Content_length size is 50140160.
# ls -l /tmp/export/nim/lpp_source/aixtools.python.2.7.11.4.I
-rw-r--r--    1 root     system     50140160 Jul 23 01:21 /tmp/export/nim/lpp_source/aixtools.python.2.7.11.4.I

Allowed operation

All cust operations on nim objects type lpp_source, installp_bundle, fix_bundle, scripts, and file_res in push or pull are working great with nimhttp. Here are a few examples (from the official doc, thanks to Paul F for that ;-) ) :

  • Push:
  • # nim –o cust –a file_res=obj_name client_obj_name
    # nim –o cust –a script=obj_name client_obj_name
    # nim –o cust –a lpp_source=obj_name -a filesets=fileset names to install client_obj_name
    # nim –o cust –a lpp_source=obj_name -a installp_bundle=obj_name client_obj_name
    # nim –o cust –a lpp_source=obj_name ‐a fixes=update_all client_obj_name
    
  • Pull:
  • # nimclient -o cust -a lpp_source=obj_name -a filesets=fileset names to install
    # nimclient –o cust –a file_res=obj_name
    # nimclient –o cust –a script=obj_name nimclient –o cust –a lpp_source=obj_name -‐a filesets=fileset names to install
    # nimclient –o cust –a lpp_source=obj_name -a installp_bundle=obj_name
    # nimclient –o cust –a lpp_source=obj_name -a fixes=update
    

Proxying: use your own http server

You can use you own webserver to host nimhttp and the nimhttp binary will just act as a proxy between your client and you http server. I have tried to do it but didn't succeed with that I'll let you know if I'm finding the solution:

# grep ^proxt ~/httpd.conf
service.proxy_port=80
enable_proxy=yes

Conclusion: "about administration and post-installation"

Just a few words about best practices of post-installation and administration on AIX. On on the major purpose of this blog post is to prove to you than you need to get rid of an old way of working. The first thing to do is always to try using http or https instead of NFS. To give you an example of that I'm always using http to transfer my files whatever it is (configuration, product installation and so on ...). With an automation tool such as Chef it is so simple to integrate the download of a file from an http server that you must now avoid using NFS ;-) . Second good practice is to never install things "by hand" and using yum is one of the reflex you need to have instead of using the rpm command (Linux users will laugh reading that ... I'm laughing writing that, using yum is just something I'm doing for more than 10 years ... but for AIX admins it's still not the case and not so simple to understand :-) ). As always I hope it helps.

About blogging

I just wanted to say one word about blogging because I got a lot of questions about this (from friends, readers, managers, haters, lovers). I'm doing this for two reasons. The first one is that writing and explaining things force me to better understand what I'm doing and force me to always discover new features, new bugs, new everything. Second I'm doing this for you, for my readers because I remember how blogs were useful to me when I began AIX (Chris and Nigel are the best example of that). I don't care about being the best or the worst. I'm just me. I'm doing this because I love that that's all. Even if manager, recruiters or anybody else don't care about it I'll continue to do this whatever appends. I agree with them "It does not prove anything at all". I'm just like you a standard admin trying to do his job at his best. Sorry for the two months "break" about blogging but it was really crazy at work and in my life. Take care all. Haters gonna hate.

Updating AIX TL and SP using Chef

Creating something to automate the update of a service pack or a technology level has always been a dream that never come true. You can trust me almost every customers that I know tried to make that dream come true. Different customers, same story everywhere. They tried to do something and then tripped up in a miserable way. A fact that is always true in those stories is that the decision of doing that is always taken by someone that do not understand that AIX cannot be managed like a workstation or any other OS (who said windows). A good example of that is an IBM (and you know that I’m an IBM fan) tool call BigFix/TEM (for Tivoli Endpoint Manager): I’m not an expert about TEM (so maybe I am wrong) but you can use this one to update your Windows OS, your Linux, your AIX and even your Iphones or Android devices. LET ME LAUGH! How can it be possible that someone think about this: updating an Iphone the same way as you update an AIX. A good joke! (To be clear I am always and will always support IBM but my role is also to say what I think). Another good example is the utilization of IBM Systems Director (unfortunately … or fortunately this one has been withdrawn since a couple of days). I tried this one myself a few years ago (you can check this post). System Director was (in my humble opinion) the least worst solution to update an AIX or a Virtual I/O Server in a automated way. So how are we going to do this in a world that is always asking to do more with less people ?. I had to find a solution a few months ago to update more than 700 hosts from AIX 6.1 to AIX 7.1, the job was to create something that anybody can launch without knowing anything about AIX (one more time who can even think this is possible ?). I tried to do things like writing scripts to automate nimadm and I’m pretty happy with this solution (almost 80% were ok without any errors, but there were tons of prerequisites before launching the scripts and we faced some problems that were inevitable (nimsh error, sendmail configuration, broken filesets) forcing the AIX L3 team to fix tons of migrations). As everybody knows now I’m working on Chef since a few months and this can be the solution to what our world is asking today : replacing hundred of peoples by a single man launching a magical thing that can do everything without knowing anything about anything and save money! This is obviously ironical but unfortunately this is the reality of what happends today in France. “Money” and “resource” rules everything without having any plans about the future (to be clear I’m here talking about a generality, nothing here can reflect what’s going on in my place). It is like it is and as a good soldier I’m going to give you solutions to face the reality of this harsh world. But now it’s action time ! I don’t want to be too pessimistic but this is unfortunately the reality of what is happening today and my anger about that only reflects the fact that I’m living in fear, the fear of becoming bad or the fear of doing a job I really don’t like. I think I have to find a solution about this problem. The picture below is clear enough to give you a good a example of what I’m trying to do with Chef.

CF8j9_dWgAAOuyC

How do we update machines

I’m not here to teach you how to update a service pack or a technology level (I’m sure everybody know that) but in an automated way we need to talk about the method and identify each needed steps to perform an update. As there is always one more way to do it I have identified three ways to update a machine (the multibos way, the nimclient way and finally the alt_disk_copy way). To be able to update using Chef we obviously need to have an available provider for each method (you can do this with the execute resource, but we’re here to have fun and to learn some new things). So we need one provider capable of managing multibos, one capable of managing nimclient, and one capable of managing alt_disk_copy. All of these three providers are available now and can be used to write different recipes doing what is necessary to update a machine. Obviously there are pre-update and post-update steps needed (removing efixes, checking filesets). Let’s identify the step required first:

  • Verify with lppchk the consistency of all installed packages.
  • Remove any installed efixes (using emgr provider)
  • The multibos way:
    • You don’t need to create a backup of the rootvg using the multibos way.
    • Mount the SP or TL directory from the NIM server (using Chef mount resource).
    • Create the multibos instance and update using the remote mounted directory (using multibos resource).
  • The nimclient way:
    • Create a backup of your rootvg (using the altdisk resource).
    • Use nimclient to run a cust operation (using niminit,nimclient resource).
  • The alt_disk_copy way:
    • You don’t new to create a backup of the rootvg using the alt_disk_copy way.
    • Mount the SP or TL directory from the NIM server (using Chef mount).
    • Create the altinst_rootvg volume group and update it using the remote mounted directory (using altdisk provider).
  • Reboot the machine.
  • Remove any unwanted bos, old_rootvg.

Reminder where to download the AIX Chef cookbook:

Before trying to do all these steps in a single way let’s try to use the resources one by one to understand what each one is doing.

Fixes installation

This one is simple and allows you to install or remove fixes from your AIX machine, in the example below we are going to show how to do that with two Chef recipes: one for installing and the other one for removing! Super easy.

Installing fixes

In the recipe provides all the fixes name in an array and specify the name of the directory in which the filesets are (this can be an NFS mount point if you want to). Please note here that I’m using the cookbook_file resource to download the fixes, this resource allows you to download a file directly from the cookbook (so from the Chef server). Imagine using this single recipe to install a fix on all your machines. Quite easy ;-)

directory "/var/tmp/fixes" do
  action :create
end

cookbook_file "/var/tmp/fixes/IV75031s5a.150716.71TL03SP05.epkg.Z" do
  source 'IV75031s5a.150716.71TL03SP05.epkg.Z'
  action :create
end

cookbook_file "/var/tmp/fixes/IV77596s5a.150930.71TL03SP05.epkg.Z" do
  source 'IV77596s5a.150930.71TL03SP05.epkg.Z'
  action :create
end

aix_fixes "installing fixes" do
  fixes ["IV75031s5a.150716.71TL03SP05.epkg.Z", "IV77596s5a.150930.71TL03SP05.epkg.Z"]
  directory "/var/tmp/fixes"
  action :install
end

directory "/var/tmp/fixes" do
  recursive true
  action :delete
end

emgr1

Removing fixes

The recipe is almost the same but with the remove action instead of the install action. Please note that you can specify which fixes to remove or use the keyword all to remove all the installed fixes (in the case of our recipe to update our servers we will use “all” as we want to remove all fixes before trying launch the update).

aix_fixes "remove fixes IV75031s5a and IV77596s5a" do
  fixes ["IV75031s5a", "IV77596s5a]
  action :remove
end
aix_fixes "remove all fixes" do
  fixes ["all"]
end

emgr2

Alternate disks

In most AIX places I have seen the solution to backup your system before doing anything is to create an alternate disk using the alt_disk_copy command. Sometimes in some places where sysadmins love their job this disk is updated on the go to do a TL or SP upgrade. The altdisk resource I’ve coded for Chef take care of this. I’ll not detail with examples every actions available and will focus on create and cust:

  • create: This action create an alternate disk we will detail the attributes in the next section.
  • cleanup: Cleanup the alternate disk (remove it).
  • rename: Rename the alternate disk.
  • sleep: Put the alternate disk in sleep (umount every /alt_inst/* filesystem and varyoff the volume group)
  • wakeup: Wake up the alternate disk (varyon the volume group and mount every filesystems)
  • customize: Run a cust operation (the current resource is coded to use a directory to update the alternate disk with all the filesets present in a directory).

Creation

The alternate disk create action create an alternate disk an helps you to find an available disk for this creation. In any cases only free disks will be choosen (disks with no PVID and no volume group defined). Different types are available to choose the disk on which the alternate disk will be created:

  • Size: If type is size a disk by the exact same size of the value attribute will be used.
  • Name: If type is name a disk by the name of the value attribute will be used.
  • Auto: In auto mode available values for value are bigger and equals. If bigger is choose the first disk found with a size bigger than the current rootvg size will be used. If equals is choose the first disk found with a size equals to the current rootvg size is used.
aix_altdisk "cloning rootvg by name" do
  type :name
  value "hdisk3"
  action :create
end
aix_altdisk "cloning rootvg by size 66560" do
  type :size
  value "66560"
end
aix_altdisk "removing old alternates" do
  action :cleanup
end

aix_altdisk "cloning rootvg" do
  type :auto
  value "bigger"
  action :create
end

altdisk1

Customization

The customization action will update the previously created alternate disk with the filesets presents in an NFS mounted directory (from the NIM server). Please note in the recipe below that we are mounting the directory from NFS. The node[:nim_server] is an attribute of the node telling which nim server will be mounted. For instance you can define a nim server used for production environment and a nim server used for development environment.

# mounting /mnt
mount "/mnt" do
  device '#{node[:nim_server]}:/export/nim/lpp_source'
  fstype 'nfs'
  action :mount
end

# updating the current disk
aix_altdisk "altdisk_update" do
  image_location "/mnt/7100-03-05-1524"
  action :customize
end

mount "/mnt" do
  action :umount
end

altdisk_cust

niminit/nimclient

The niminit and nimclient resources are used to register the nimclient to the nim master and then run a nimclient operation from the client. In my humble opinion this is the best way to do the update at the time of writing this blog post. One cool thing is that you can specify on which adapter the nimclient will be configured by using some ohai attributes. It’s an elegant way to do that, one more time this is showing you the power of Chef ;-) . Let’s start with some examples:

niminit

aix_niminit node[:hostname] do
  action :remove
end

aix_niminit node[:hostname] do 
  master "nimcloud"
  connect "nimsh"
  pif_name node[:network][:default_interface]
  action :setup
end

nimclient1

nimclient

nimclient can first be used to install some filesets you may need. The provider is intelligent and can choose the good lpp_source for you. Please note that you will need lpp_source with a specific naming convention if you want to use this feature. To find the next/latest available sp/tl the provider is checking the current oslevel of the current machine and compare it with the available lpp_source present on you nim server. The naming convention needed is $(oslevel s)-lpp_source (ie. 7100-03-05-1524-lpp_source) (same principle is applicable to the spot when you need to use spot)

$ lsnim -t lpp_source | grep 7100
7100-03-00-0000-lpp_source             resources       lpp_source
7100-03-01-1341-lpp_source             resources       lpp_source
7100-03-02-1412-lpp_source             resources       lpp_source
7100-03-03-1415-lpp_source             resources       lpp_source
7100-03-04-1441-lpp_source             resources       lpp_source
7100-03-05-1524-lpp_source             resources       lpp_source

If your nim resources name are ok the lpp_source attribute can be:

  • latest_sp: the latest available service pack.
  • next_sp: the next available service.
  • latest_tl: the latest available technology level.
  • next_tl: the next available techonlogy level.
  • If you do not want to do this you can still specify the name of the lpp_source by hand.

Here are a few example to install packages

aix_nimclient "installing filesets" do
  installp_flags "aXYg"
  lpp_source "7100-03-04-1441-lpp_source"
  filesets ["openssh.base.client","openssh.base.server","openssh.license"]
  action :cust
end

aix_nimclient "installing filesets" do
  installp_flags "aXYg"
  lpp_source "7100-03-04-1441-lpp_source"
  filesets ["bos.compat.cmds", "bos.compat.libs"]
  action :cust
end

aix_nimclient "installing filesets" do
  installp_flags "aXYg"
  lpp_source "7100-03-04-1441-lpp_source"
  filesets ["Java6_64.samples"]
  action :cust
end

nimclient2

Please note that some filesets were already installed and the resource did not converge because of that ;-) . Let’s now try to update to the latest service pack:

aix_nimclient "updating to latest sp" do
  installp_flags "aXYg"
  lpp_source "latest_sp"
  fixes "update_all"
  action :cust
end

nimclient3

Tadam the machine was updated from 7100-03-04-1441 to 7100-03-05-1524 using a single a recipe and without knowing which service pack was available to update!

multibos

I really like the multibos way and I don’t know why today so few peoples are using it, anyway, I know some customers who are only working this way so I thought it was worth it working on a multibos resource. Here is a nice recipe creating a bos and updating it.

# creating dir for mount
directory "/var/tmp/mnt" do
  action :create
end

# mounting /mnt
mount "/var/tmp/mnt" do
  device "#{node[:nim_server]}:/export/nim/lpp_source"
  fstype 'nfs'
  action :mount
end

# removing standby multibos
aix_multibos "removing standby bos" do
  action :remove
end

# create multibos and updateit
aix_multibos "creating bos " do
  action :create
end

aix_multibos "update bos" do
  update_device "/var/tmp/mnt/7100-03-05-1524"
  action :update
end

# unmount /mnt
mount "/var/tmp/mnt" do
  action :umount
end

# deleting temp directory
directory "/var/tmp/mnt" do
  action :delete
end

multibos1
multibos2

Full recipes for updates

Let’s now write a big recipe doing all the things we need for an update. Remember that if one resource is failing the recipe stop by itself. For instance you’ll see in the recipe below that I’m doing an “lppchk -vm3″. If it returns something other than 0, the resources fail and the recipe fail. It’s obviously a normal behavior, it’s seems ok not to continue if there is a problem. So to sum up here are all the steps this recipe is doing: check fileset consistency, removing all fixes, committing filesets, creating an alternate disk, configuring the nimclient, running the update, deallocating resources

# if lppchk -vm return code is different
# than zero recipe will fail
# no guard needed here
execute "lppchk" do
  command 'lppchk -vm3'
end

# removing any efixes
aix_fixes "remvoving_efixes" do
  fixes ["all"]
  action :remove
end

# committing filesets
# no guard needed here
execute 'commit' do
  command 'installp -c all'
end

# cleaning exsiting altdisk
aix_altdisk "cleanup alternate rootvg" do
  action :cleanup
end

# creating an alternate disk using the
# first disk bigger than the actual rootvg
# bootlist to false as this disk is just a backup copy
aix_altdisk "altdisk_by_auto" do
  type :auto
  value "bigger"
  change_bootlist true
  action :create
end

# nimclient configuration
aix_niminit node[:hostname] do
  master "nimcloud"
  connect "nimsh"
  pif_name "en1"
  action :setup
end

# update to latest available tl/sp
aix_nimclient "updating to latest sp" do
  installp_flags "aXYg"
  lpp_source "latest_sp"
  fixes "update_all"
  action :cust
end

# dealloacate resource
aix_nimclient "deallocating resources" do
  action :deallocate
end

How about a single point of management “knife ssh”, “pushjobs”

Chef is and was designed on a pull model, it means that the client is asking to server to get the recipes and cookbooks and then execute them. This is the role of the chef-client. In a Linux environment, people are often running the client in demonized mode, it means that the client is waking up on a time interval basis and is executed (then every change to the cookbooks are run by the client). I’m almost sure that every AIX shop will be against this method because this one is dangerous. If you are doing that run the change first in test environment, then in dev, and finally in production. To be honest this is not the model I want to build where I am working. We want for some actions (like updates) a push model. By default Chef is delivered with a feature called push jobs. Push jobs is a way to run jobs like “execute the chef-client” from your knife workstation, unfortunately push jobs needs plugin to the chef-client and this one is only available on Linux OS …. not yet one AIX. Anyway we have an alternative, this one is the ssh knife plugin. This plugin that is in knife by default allows you to run commands on the nodes with ssh. Even better if you already have an ssh gateway with key sharing enabled knife ssh can use this gateway to communicate with the clients. Using knife ssh you’ll have the possibility to say “run chef-client on all my AIX 6.1 nodes” or “run this recipes installing this fix on all my AIX 7.1 nodes”, possibilities are infinite. Last note about knife ssh. This one is creating tunnels through your ssh gateway to communicate with the node, so if you use a shared key you have to copy the private key on the knife workstation (it tooks me time to understand that). Here are somes exmples:

knifessh

  • On two nodes check the current os level:
  • ssh1

  • Run the update with Chef:
  • update3

  • Alternates disk have been created:
  • update4

  • Both systems are up to date:
  • update5

Conclusion

I think this blog post helped you to better understand Chef and what is Chef capable of. We are still on the very beginning of the Chef cookbook and I’m sure plenty of new things (recipes, providers) will come in the next few months. Try it by yourself and I’m sure you’ll like the way it work. I must admit that it is difficult to learn and to start but if you are doing this right you’ll get the benefit of an automation tool working on AIX … and honestly AIX needs an automation tool. I’m almost sure it will be Chef (in fact we have no other choice). Help us to write postinstall recipes, updates recipes and any other recipes you can think about. We need your help and it is happening now! You have the opportunity to be a part of this, a part of something new that will help AIX in the future. We don’t want a dying os, Chef will give AIX the opportunity to be an OS with a fully working automation tool. Go give it a try now!

Use your NIM server to install Red Hat Enterprise Linux 6 for IBM Power

I’m actually working on a project including the deployement of PowerVC. Not one PowerVC but many. Like everybody know PowerVC is not running on AIX but is only -for the moment and forever- available on Red Hat Enterprise Linux 6.4. If you do not have the infrastructure or servers (check IBM Installation Toolkit for PowerLinux) to install Red Hat Enterprise Linux, you can choose to use your NIM server to do so. We will not use the NIM commands for that purpose because NIM is not designed to install Linux servers, we’ll just re-use tools and services used by NIM such as bootp and tftp to deploy our new Red Hat Enterprise Linux for Power. Here is how I did it with the errors and mistakes made while trying to do it.

Copying the Red Hat Enterprise Linux 6 DVD on the NIM Server

You’ll need to copy the whole Red Hat Enterprise Linux 6 DVD on the NIM server because you’ll somes files (initrd.img, vmlinuz) for booting the lpar with tftp and bootp, and all the RPM files for the installation itself. I personnally choose to copy the content of the DVD in /export/nim/lpp_source/nim/rhel/6.4 :

# mkdir -p /export/nim/lpp_source/rhel/6.4
# loopmount -i /export/iso/rhel-server-6.4-ppc64-dvd.iso -o "-V cdrfs -o ro" -m /mnt
# cp -ra /mnt /export/nim/lpp_source/rhel/6.4
# umount /mnt

tftpd configuration

By using the NIM server to install a Linux you’ll need to download files with tftp service. By default NIM is configured to allow dowloading files from tftp in the /tftpboot directory only. You’ll need to add (an)other location(s) for your own use, in my case I want to download yaboot.conf file initrd.img and vmlinuz files (I first wanted to add the ppc64.img file, but I had to remove it and replace it by the yaboot file -check the explication below). To allow downloading with tftp from different locations modify the /etc/tftpaccess.ctl configuration file. In my case I had to add /etc and /tftpboot/etc filesystems (I’ll will explain this later).

cat /etc/tftpaccess.ctl
# NIM access for network boot
allow:/tftpboot
allow:/tftpboot/etc
allow:/etc
allow:/export/lpp_source/rhel/6.4

Identify the file you’ll need to make your installation :

  • yaboot image and yaboot configuration file are needed to network load yaboot (you do not have to move yaboot image (it will not be modified)), just copy the yaboot.conf from the DVD to /etc (yaboot will search in /etc) :
  • # cp /export/nim/lpp_source/rhel/6.4/ppc/ppc64/yaboot.conf /etc/yaboot.conf
    
  • vmlinuz and initrd.img will be downloaded with tftp after yaboot will be loaded so move these files in /tftpboot/etc (I’ll explain this later on the yaboot configuration part) :
  • # mkdir -p /tftpboot/etc
    # cp /export/nim/lpp_source/rhel/6.4/ppc/ppc64/initrd.img /export/nim/lpp_source/rhel/6.4/ppc/ppc64/vmlinuz /tftpboot/etc 
    

bootptab modification

You cannot modifiy the /etc/bootptab file for you own use by using NIM commands so you had to do it by hand. The bootptab file will tell your client which files to download to begin the installation. Before Red Hat Enterprise Linux 6 you can use the ppc64.img to begin the installation directly by booting from this file. On Red Hat Enterprise Linux 6 you have to do it by using yaboot, so edit the /etc/bootptab file and add this parameters :

  • The name of the server, in my case : powervc.lab.chmod666.org
  • The boot file used to boot the server, in our case the yaboot file : /export/nim/lpp_source/rhel6/6.4/ppc/chrp/yaboot
  • The IP address of the server : 10.10.20.21
  • The type of network : ethernet
  • The IP address of the NIM server : 10.10.20.20
  • The gateway address :10.10.20.254
  • The netmask : 255.255.255.0

Fields are separated by ‘:’, do not forget to add the last ‘:’.

# tail -1 /etc/bootptab
powervc.lab.chmod666.org:bf=/export/nim/lpp_source/rhel6/6.4/ppc/chrp/yaboot:ip=10.10.20.21:ht=ethernet:sa=10.10.20.20:gw=10.10.20.254:sm=255.255.255.0:

It’s a matter of size : do not try to boot ppc64.img

If you already use this method to install a Red Hat Enterprise Linux 5 you probably just use the ppc64.img and do not have to be annoyed by all the yaboot configuration. You can’t do that with Red Hat Enterprise Linux 6 because the size of the ppc64.img file is greater than 32MB. tftp downloads are limited to 32MB, it is like this by design and you can verify it in the RFC. I’m not the only one who had this problem and a official bug was opened on the Red Hat Bugzilla : https://bugzilla.redhat.com/show_bug.cgi?id=613929 : Cannot install RHEL6 via TFTP netboot using ppc64.img. Both Red Hat Enterprise Linux 6 and tftp are defected by designed. If you try do so you’ll get an error while downloading the ppc64.img file “No operating system install”

Configuring yaboot

All of this comes from my own experience while trying to configure yaboot, yaboot boot process is not very well documented and I had to look in the source code to check how it works :

  1. yaboot image is loaded with tftp.
  2. yaboot will search it’s configuration file located in the /etc/, for example : /etc/00aabbccddee
  3. If yaboot does not find this file it will try fo find it’s configuration file by searching the IP address of your server in hexadecimal. If it does not find it, it’ll remove the last byte from the address, in our example /etc/0a0a1415 then /etc/0a0a14, and so on, until the first byte.
  4. If yaboot does not fine any mac address or IP address based file it will finally search for an /etc/yaboot.conf file.

I personnally chose to configure yaboot to let me install Red Hat Enterprise Linux with or without a kickstart file, so I have two blocks one called linux and the other one called linux-ks. The Linux one will let me install Linux by using the Red Hat installer and I have to answer the question for the installation. The seconde one : linux-ks let me install Linux with a kickstart based installation. In both case I have to load with tftp initrd.img and vmlinuz files, and in the kickstart case I have to download the ks.cfg file (In my case I have configured a web server to provide it but you can use AIX nfs if you want :-) ). Here is my /etc/yaboot.conf file

  • Configuration file header for /etc/yaboot.conf :
  • init-message = "\nWelcome to the 64-bit Red Hat Enterprise Linux 6.4 installer!\nHit  for boot options.\n\n"
    timeout=6000
    default=linux
    
  • linux block :
  • image=etc/vmlinuz
            label=linux
            initrd=etc/initrd.img
            read-only
    
  • linux-ks block :
  • image=etc/vmlinuz
            label=linux-ks
            initrd=etc/initrd.img
            append="ks=http://10.10.20.20:8080/rhel6/ks.cfg ksdevice=eth0 ip=10.253.112.146 netmask=255.255.255.254 gateway=10.253.112.129 noipv6"
            read-only
    

As you can see on the output above, yaboot needs initrd.img file and vmlinuz file, and it’ll search it in /tftpboot/etc do not add /tftpboot/ as the tftpboot client root is /tftpboot and you can use like me relative paths. I had to configure a web server to allow net based installation without NFS, but you can use NFS provided by the NIM server.

# tail -7 /etc/opt/freeware/apache/httpd.conf 
Alias /rhel6/ "/export/nim/lpp_source/rhel/6.4/"

    AllowOverride None
    Options ExecCGI Includes FollowSymLinks Indexes
    Order allow,deny
    Allow from all

My kickstart file is also provided by this web server :

# ls -l /export/nim/lpp_source/rhel/6.4/ks.cfg
-rw-r--r--    1 root     system         1394 Jan  2 11:06 /export/lpp_source/core/rhel/6.4/ks.cfg

Installation sequence

Here is the boot sequence after everything is configured and working as desired. Configure your IPL device as you always do, and boot it through the SMS menu :

  • First, yaboot image is downloaded with tftp :
  • TFTP BOOT ---------------------------------------------------
    Server IP.....................10.10.20.20
    Client IP.....................10.10.20.21
    Gateway IP....................10.10.20.254
    Subnet Mask...................255.255.255.0
    ( 1  ) Filename................./export/nim/lpp_source/rhel/6.4/ppc/chrp/yaboot
    TFTP Retries..................5
    Block Size....................512
    FINAL PACKET COUNT = 516
    
  • Then yaboot is searching for its configuration file, first trying by searching a configuration file named with the mac address of the machine :
  • System has 256 Mbytes in RMA
    Try to netboot
    
    claim of 0x2000000 at 0xe000000 returned 0xe000000
    
    TFTP BOOT ---------------------------------------------------
    Server IP.....................10.10.20.20
    Client IP.....................10.10.20.21
    Gateway IP....................10.10.20.254
    Subnet Mask...................255.255.255.0
    ( 1  ) Filename.................\etc\01-8a-70-76-c4-13-02
    TFTP Retries..................5
    Block Size....................512
    1
    2
    3
            !BA017021 !
    
    Error, can't read config file
    
  • If yaboot does not find this file it’ll try with the IP address of the server in hexadecimal :
  • Error, can't read config file
    
    
    claim of 0x2000000 at 0xe000000 returned 0xe000000
    
    TFTP BOOT ---------------------------------------------------
    Server IP.....................10.10.20.20
    Client IP.....................10.10.20.21
    Gateway IP....................10.10.20.254
    Subnet Mask...................255.255.255.0
    ( 1  ) Filename.................\etc\0a0a2021
    TFTP Retries..................5
    Block Size....................512
    1
    2
    3
           !BA017021 !
    
    Error, can't read config file
    
    
    claim of 0x2000000 at 0xe000000 returned 0xe000000
    
    TFTP BOOT ---------------------------------------------------
    Server IP.....................10.10.20.20
    Client IP.....................10.10.20.21
    Gateway IP....................10.10.20.254
    Subnet Mask...................255.255.255.0
    ( 1  ) Filename.................\etc\0a0a20
    TFTP Retries..................5
    Block Size....................512
    1
    2
    3
    
  • Finally yaboot will search for a file named yaboot.conf :
  • FTP BOOT ---------------------------------------------------
    Server IP.....................10.10.20.201
    Client IP.....................10.10.20.21
    Gateway IP....................10.10.20.254
    Subnet Mask...................255.255.255.0
    ( 1  ) Filename.................\etc\yaboot.conf
    TFTP Retries..................5
    Block Size....................512
    FINAL PACKET COUNT = 1
    FINAL FILE SIZE = 442  BYTES
    Config file read, 442 bytes
    
    Welcome to the 64-bit Red Hat Enterprise Linux 6.4 installer!
    Hit  for boot options.
    
  • Choose one option to boot on (in our case linux-ks) :
  • Welcome to yaboot version 1.3.14 (Red Hat 1.3.14-41.el6)
    Enter "help" to get some basic usage information
    boot:
    * linux                      linux-ks
    
    boot: linux-ks
    Please wait, loading kernel...
    
  • Kernel vmlinuz is loaded with tftp:
  • claim of 0x2000000 at 0xe000000 returned 0xe000000
    
    TFTP BOOT ---------------------------------------------------
    Server IP.....................10.10.20.20
    Client IP.....................10.10.20.21
    Gateway IP....................10.10.20.254
    Subnet Mask...................255.255.255.0
    ( 1  ) Filename.................etc\vmlinuz
    TFTP Retries..................5
    Block Size....................512
    FINAL PACKET COUNT = 32730
    FINAL FILE SIZE = 16757576  BYTES
       Elf64 kernel loaded...
    Loading ramdisk...
    
  • Ramdisk initrd.img is loaded with tftp :
  • claim of 0x2000000 at 0xe000000 returned 0xe000000
    
    TFTP BOOT ---------------------------------------------------
    Server IP.....................10.10.20.21
    Client IP.....................10.10.10.20
    Gateway IP....................10.10.20.254
    Subnet Mask...................255.255.255.0
    ( 1  ) Filename.................etc\initrd.img
    TFTP Retries..................5
    Block Size....................512
    FINAL PACKET COUNT = 54371
    
  • Finally the installation begin because with choose the linux-ks :
  • Welcome to Red Hat Enterprise Linux for ppc64
        +----------------------------¦ Retrieving +----------------------------+
        ¦                                                                      ¦
        ¦ Retrieving /install.img...                                           ¦
        ¦                                                                      ¦
        ¦                                  17%                                 ¦
        ¦                                                                      ¦
        +----------------------------------------------------------------------+
      / between elements  |  selects |  next screen
    

IBM Installation toolkit for PowerLinux

Finally if your are planning to install a lot of Linux Servers on Power Hardware IBM provide a solution to easily install and deploy Linux OS on Power Hardware. This tool is called IBM Installation Toolkit for PowerLinux and can be find here : http://www-304.ibm.com/webapp/set2/sas/f/lopdiags/installtools/home.html. You’ll need to dedicated (Linux ?) server for this product if you want to do so. I’ve never tested it but if I had to deploy more and more Linux servers I’ll give it a try, after all NIM server are used to install AIX and not Linux :-)

Updating and backuping Virtual I/O Servers with NIM : Story of APARs IV46060, IV????? and IV?????

I recently had to find the best solution to update a bunch of Virtual I/O Server at a time. Since a couple of months I’m intensively using NIM new features such as DSM and my first thought was to use NIM to update all my Virtual I/O Servers. You’ve probably notice that a new operation exists in latest NIM version called “updateios“. With this new operation comes two new types, vios (a Virtual I/O Server machine) and ios_mksysb (a mksysb created by the backupios command on the Virtual I/O Server). I’m probably the only guy using this because at the time of writing this post the updateios command does not work. For IBMers who are reading this post I had the chance to work with french L3 Virtual I/O Server support on two PMRs (a big thanks to them for their skills and efficiency), you can have a look on it :

  • PMR 84369,664,706 : NIM updateios operation hanging on NIM master resulting in two APARs (IV?????; and IV?????) (these two APARs are still in validation at the time of writing).
  • PMR 84152,664,706 : NIM updateios problem with /usr/lpp/bos.sysmgt/nim/methods/c_updateios resulting in one APAR (IV46060) (http://www-01.ibm.com/support/docview.wss?crawler=1&uid=isg1IV46060).

After a few weeks of work with the support we finally found two workarounds for these problems. This post will explain the solutions we found with the support. If you had one lesson to remember by reading this post keep this one : “Always subscribe to SWMA support because they are damn brillant”.

Defining Virtual I/O Server object

If you are reading this post I hope you’ve already read my post about NIM Less known features. If you have no time to read this one here is a reminder. Before running any operation on a Virtual I/O Server, you have to create management objects associated to it :

  • Create the HMC object :
  • # dpasswd -f foo  -U hscroot
    Password file is /etc/ibm/sysmgt/dsm/config/foo
    Password:
    Re-enter password:
    Password file created.
    # dkeyexch -f /etc/ibm/sysmgt/dsm/config/myhmc_passwd -I hmc -H myhmc
    OpenSSH_6.0p1, OpenSSL 0.9.8x 10 May 2012
    # nim -o define -t hmc -a if1="find_net myhmc 0" -a passwd_file=/etc/ibm/sysmgt/dsm/config/myhmc_passwd myhmc
    
  • Create the CEC object, I’m using in this example the nimquery command to find serial number and machine type :
  • # nimquery -a hmc=myhmc-p | grep ^CEC
    [..]
    CEC SERVER1 - 8202-E4B_6565655 :
    CEC SERVER2 - 8205-E6B_0606065 :
    [..]
    # nim -o define -t cec -a hw_type=8202 -a hw_model=E4B -a hw_serial=6565655 -a mgmt_source=myhmc SERVER1 
    
  • Created the vios object, I’m using in this example the nimquery command to find the identity field :
  • # nimquery -a cec=SERVER1 -p
    [..]
    LPAR my_vios - lpar_id 2 :
            allow_perf_collection = 1
            auto_start = 0
            curr_lpar_proc_compat_mode = POWER7
            curr_profile = my_vios
            default_profile = my_vios
            desired_lpar_proc_compat_mode = default
            logical_serial_num = 6565655
            lpar_avail_priority = 191
            lpar_env = vioserver
            lpar_id = 2
            lpar_keylock = norm
            msp = 1
            name = my_vios
            os_version = VIOS 2.2.2.1
            power_ctrl_lpar_ids = none
            redundant_err_path_reporting = 0
            resource_config = 1
            rmc_ipaddr = 10.10.20.107
            rmc_state = active
            shared_proc_pool_util_auth = 1
            state = Running
            time_ref = 0
            work_group_id = none
    [..]
    # nim -o define -t vios -a if1="1020-10-10-20-0-s24-net my_vios 0" -a mgmt_source="SERVER1" -a identity=2  my_vios
    
  • Check everything is ok by using lsnim command :
  • # lsnim -t hmc
    my_hmc      management       hmc
    # lsnim -t cec
    SERVER2     management       cec
    # lsnim -t vios
    my_vios           management       vios
    

Setup Virtual I/O Server as a nim client

Only a few people knows that a Virtual I/O Server can be a setup as a NIM Client. Remember that you never had to use oem_setup_env to perform administration tasks on Virtual I/O Server. To setup a Virtual I/O Server as a NIM client use a special command called remote_management as padmin. It’s the niminit command for a Virtual I/O Server. Keep in mind that the remote_management setup NIM client to use nimsh protocol (it’s important for the rest of this post.) :

  • You probably had to add NIM servers entries in your /etc/hosts file :
  • # hostmap -addr 10.10.20.140 -host my_nim1 my_nim1.lab.chmod666.org
    # hostmap -addr 10.10.20.141 -host my_nim2 my_nim2.lab.chmod666.org
    
  • Enable remote_management :
  • # remote_management -interface en0 my_nim1
    nimsh:2:wait:/usr/bin/startsrc -e "LIBPATH=/usr/lib" -g nimclient >/dev/console 2>&1
    0513-059 The nimsh Subsystem has been started. Subsystem PID is 7340278.
    
  • If you have to disable remote_management use the disable option :
  • # remote_management -disable
    0513-044 The nimsh Subsystem was requested to stop.
    
  • Check nimsh is running :
  • # ps -ef | grep nimsh
        root 5767198 5963976   0   Aug 23      -  0:00 /usr/sbin/nimsh -s
    

Backuping Virtual I/O Server by creating an ios_mksysb resource.

Before updating the Virtual I/O Server create a ios_mksysb. Most PowerVM administrator are running a script from the Virtual I/O Server but you can now invoke the backupios command from the NIM server. You can now do this for all your Virtual I/O Server and store the ios_mksysb on the NIM server, much easier than running a command on the Virtual I/O Server and mounting an NFS share on it …. :

# nim -o define -t ios_mksysb -a source=my_vios -a location=/export/nim/mksysb/my_vios/my_vios-ios_mksysb  -a server=master -a mk_image=yes my_vios-ios_mksysb
+---------------------------------------------------------------------+
                System Backup Image Space Information
              (Sizes are displayed in 1024-byte blocks.)
+---------------------------------------------------------------------+
Required = 7316181 (7145 MB)    Available = 386230180 (377178 MB)


/tmp/7274624.mnt0/myvios-ios_mksysb  doesn't exist.

Creating /tmp/7274624.mnt0/myvios-ios_mksysb
Backup in progress.  This command can take a considerable amount of time
to complete, please be patient...


Creating information file (/image.data) for rootvg.

Creating list of files to back up.
....
Backing up 169631 files............
51526 of 169631 files (30%)..............................
155443 of 169631 files (91%)..

169631 of 169631 files (100%)
0512-038 savevg: Backup Completed Successfully.

While running this command you can have a look on the Virtual I/O Server. By “proctreeing” the nimsh process you can check that the backupios with mksysb flag command is running :

# proctree -a  9240678
1    /etc/init
   3342492    /usr/sbin/srcmstr
      5046448    /usr/sbin/nimsh -s
         10813570    /usr/sbin/nimsh -s
            6160534    /bin/ksh /usr/lpp/bos.sysmgt/nim/methods/c_nimpush /usr/lpp/bos.sysmgt/nim/meth
               7274624    /bin/ksh /usr/lpp/bos.sysmgt/nim/methods/c_backupios -aserver=my_nim1 -al
                  9240678    /usr/ios/cli/ioscli backupios -file /tmp/7274624.mnt0/my_vios-ios_mksysb -mk
                     10158278    /bin/ksh /usr/bin/savevg -X -i -f /tmp/7274624.mnt0/my_vios-ios_mksysb rootv
                        8585348    /bin/ksh /usr/bin/savevg -X -i -f /tmp/7274624.mnt0/my_vios-ios_mksysb rootv
                           10223832    /usr/bin/sleep 10
                        9764964    /usr/bin/cat /tmp/mksysb.10158278/.archive.list.10158278
                        11337872    backbyname -i -q -v -Z -p -U -f /tmp/7274624.mnt0/my_vios-ios_mksysb

After the ios_mksysb creation you can check the source and the ioslevel of your backup :

# lsnim -l my_vios-ios_mksysb
my_vios-ios_mksysb:
   class         = resources
   type          = ios_mksysb
   arch          = power
   Rstate        = ready for use
   prev_state    = unavailable for use
   location      = /export/nim/mksysb/my_vios/my_vios-ios_mksysb
   version       = 6
   release       = 1
   mod           = 8
   oslevel_r     = 6100-07
   alloc_count   = 0
   server        = master
   creation_date = Mon Sep 30 11:52:35 2013
   source_image  = my_vios
   ioslevel      = 2.2.2.1

Committing existing updates on the Virtual I/O Server with updateios operation.

Commit all uncommitted updates on the Virtual I/O Server. The NIM command will invoke “ioscli updateios -commit” command on the Virtual I/O Server. Remember to remove all ifix/efix before commiting (use emgr)

# /usr/sbin/emgr -r -L IV16920s02
# nim -o updateios -a lpp_source=vios2223-fp26-sp02-lpp_source  -a accept_licenses=yes -a preview=no -a updateios_flags="-commit" -a force=yes my_vios

Updating Virtual I/O Server with updateios operation.

First of all if the Virtual I/O Server is member of a Shared Storage Pool cluster it can’t be updated. Leave the cluster before running the update :

#  clstartstop -stop -n my_cluster -m my_vios

You will face two problems when updating a Virtual I/O Server from NIM with the updateios operation. Running an updateios operation from the NIM server call the script /usr/lpp/bos.sysmgt/nim/methods/c_updateios on the Virtual I/O Server. If you perform the updateios operation this one will fail with this output :

# nim -o updateios -a lpp_source=vios2223-fp26-sp02-lpp_source  -a accept_licenses=yes -a preview=no -a updateios_flags="-install" -a force=yes my_vios
[..]
******************************************************************************
End of installp PREVIEW.  No apply operation has actually occurred.
******************************************************************************

Continue bos.rte.install installation [y|n]?
[..]
******************************************************************************
End of installp PREVIEW.  No apply operation has actually occurred.
******************************************************************************

Continue the installation [y|n]?
Command did not complete.

As you can see on the output the updateios command is interactive and ask TWO yes/no questions. On the Virtual I/O Server while running the updateios operation you can check that /usr/lpp/bos.sysmgt/nim/methods/c_updateios is called by nimsh process :

# proctree 15466556
4260044    /usr/sbin/srcmstr
   7340280    /usr/sbin/nimsh -s -c
      12451968    /usr/sbin/nimsh -s -c
         15466556    /bin/ksh /usr/lpp/bos.sysmgt/nim/methods/c_nimpush /usr/lpp/bos.sysmgt/nim/meth
            14352628    /bin/ksh /usr/lpp/bos.sysmgt/nim/methods/c_updateios -aaccept_licenses=yes -afo
               10944754    /usr/ios/cli/ioscli updateios -install -dev /tmp/_nim_dir_14352628/mnt0 -f -acc
                  5374158    installp -e install.log -a -d /tmp/_nim_dir_14352628/mnt0 bos.rte.install
                     9961620    installp -e install.log -a -d /tmp/_nim_dir_14352628/mnt0 bos.rte.install

If you edit the /usr/lpp/bos.sysmgt/nim/methods/c_updateios you can see at the line 130 that ‘y’ it just send one time :

# vi /usr/lpp/bos.sysmgt/nim/methods/c_updateios
[..]
                -install)
                        argument="-install -dev $lpp_access ${force:+-f} ${accept_licenses:+-accept}"
                        if [[ $preview = "no" ]]; then
                                command="eval echo 'y' | /usr/ios/cli/ioscli updateios $argument"
                        else
                                command="eval echo 'n' | /usr/ios/cli/ioscli updateios $argument"
                        fi
                        ;;
[..]

Modify the ‘y’ by ‘y\ny’ and the script will send two ‘y’, easy :-) :

# grep -n eval /usr/lpp/bos.sysmgt/nim/methods/c_updateios | head -1
130:                            command="eval echo 'y\ny' | /usr/ios/cli/ioscli updateios $argument"

Rerun the NIM operation and the update will start.

At the end of the installation you will probably face another problem. This one occurs only if the Virtual I/O Server NIM client is using nimsh protocol. The NIM operation will hang forever on the NIM server : on the Virtual I/O Server a socket remain opened between the NIM client and the NIM server:

# netstat -Aan |grep 3901
f1000e0001cb2bb8 tcp4       0      0  10.10.20.107.3901   10.10.20.140.1021   ESTABLISHED
f1000e00098bdbb8 tcp        0      0  *.3901                *.*                   LISTEN
# rmsock f1000e00098bdbb8 tcpcb
The socket 0xf1000e00098bd808 is being held by proccess 8126526 (accessprocess).
#  rmsock f1000e0001cb2bb8 tcpcb
The socket 0xf1000e0001cb2808 is being held by proccess 12386388 (cimserver).
#  proctree 12386388
12386388
   8323090    /usr/ios/lpm/sbin/eventhelper --events ref_code,lpar_state,not_ivm,migration_st
# proctree 8126526
15269920    /usr/bin/ksh /usr/ios/lpm/sbin/lparmgr all start
   8126526    /usr/ios/lpm/sbin/accessprocess
# ps -ef |grep 12386388
    root  8323090 12386388   0 15:44:31      -  0:00 /usr/ios/lpm/sbin/eventhelper --events ref_code,lpar_state,not_ivm,migration_state,vsp_state
    root 12386388        1   0 15:42:56      -  0:16 [cimserve]

The issue was found with the support, a command called by cimserve called climgr is not closing correctly its file descriptors a the end of the update, modify this script to close all opened file descriptor :

# grep -n exec /usr/ios/sbin/climgr
366:exec 1<&-
367:exec 2<&-
368:exec 5<&-

Rerun the operation and evrything will just work fine :-)

Conclusion

I assume these two problems will be fixed in the next Virtual I/O Server release, probably not the 2.2.3.0 version but the next one (I have to wait in average 6 months before the fix is applied to the current version). Once again I want to thanks the IBM Support for helping me on these cases and for their efficiency. I hope it helps.