NIM Less known features : HANIM, nimsh over ssl, DSM

The Network Installation Manager server is one of the most important host in an environment. New machines installations, machines backups, backups restorations,software (filesets), third party products installations, in some cases volume group backups are made from the NIM server. Some best practices have to be respected. I’ll give you in this post a few tricks for NIM. First off all a NIM server has to be in your disaster recovery plan because it the first server needed when you have to re-build a crashed machine : my solution HANIM. It has to be secured (nimsh, and nimsh authentication over ssl), and it has to be flexible and automated (DSM).

NIM High Availability : HANIM

Finding documentation and information about NIM High Availability is not so easy. I recommend you to check the NIM from a to Z Redbook, it’s one of the only viable source for HANIM. HANIM simple to setup and simple to use, but there are a few things to know and to understand about it :

HANIM Overview

  • The alternate NIM master is a backup NIM build from the NIM master.
  • Takeover operations from master to alternate are manuals. PowerHA can be used to run these takeover operations but my advice is not to use it. Takeover can be performed even if the NIM master is down. HANIM does not perform any heartbeat.
  • HANIM only provides a method for replicating NIM database and resources. Resources can be replicated from master to alternate : NIM database AND resources data can be replicated (replicate=yes option).
  • My advice is to run every NIM operation from the master (even if it is possible to run a NIM operation from the alternate).
  • Disks are not shared between the master and the alternate, when a sync operation is done, missing resources are copied over NFS form the master to the alternate, or from the alternate to the master. HANIM does not provides a filesystem takeover.
  • A takeover operation modify all the nimclient’s /etc/niminfo files. The NIM_MASTER_HOSTNAME_LIST is modified by the takeover operation and the alternate NIM master is moved in first position. The NIM_MASTER_HOSTNAME is modified with the alternated NIM master hostname.


Initial setup

On the NIM master and on the alternate NIM master some filesets have to be installed, check the presence of : bos.sysmgt.nim.master, bos.sysmgt.nim.spot, bos.sysmgt.nim.client. NIM master and alternate NIM master must be one the same AIX version :

# lslpp -l | grep -i nim
  bos.sysmgt.nim.client     7.1.2.15  COMMITTED  Network Install Manager -
  bos.sysmgt.nim.master     7.1.2.15  COMMITTED  Network Install Manager -
  bos.sysmgt.nim.spot       7.1.2.15  COMMITTED  Network Install Manager - SPOT
  bos.sysmgt.nim.client     7.1.2.15  COMMITTED  Network Install Manager -
# oslevel -s
7100-02-02-1316

Configure the NIM master

Initialize the NIM master with the nimconfig command, you’ll need to name the first network used by NIM. nimesis daemons will be started at this step.

# nimconfig -a pif_name=en0 -a netname=10-10-20-0-s24-net -a master_port=1058 -a verbose=3 -a cable_type=N/A
[..]
Checking input attributes.
attr_ass:
        'cpuid' => '00F359164D00'
        'pif_name' => 'en0'
        'netname' => '10-10-20-0-s24-net'
        'master_port' => '1058'
        'cable_type' => 'N/A'
        'net_addr' => '10.10.20.1'
        'snm' => '255.255.255.0'
        'adpt_addr' => '667C70F7A904'
        'adpt_name' => 'ent0'
Making sure the NIM Master package is OK.
      set_state: id=1361463886; name=; state_attr=85; new_state=5;
   checking the object definition of ;
   checking interface info for master;
Built NIM infomation file.
      10.10.20.1 is known as nim_master
Adding default route 10.10.20.254 to network object
         0 - /usr/lpp/bos.sysmgt/nim/methods/m_mknet
         1 - -anet_addr=10.10.20.1
         2 - -asnm=255.255.255.0
         3 - -tent
         4 - -arouting1=default 10.10.20.254
         5 - 10-10-20-0-s24-net
Connecting NIM master to master network.
         0 - /usr/lpp/bos.sysmgt/nim/methods/m_chmaster
         1 - -aif1=10-10-20-0-s24-net nim_master 667C70F7A904
         2 - -amaster_port=1058
         3 - -aregistration_port=1059
         4 - -acable_type1=N/A
         5 - master
Adding NIM deamons to SRC and starting....
0513-071 The nimesis Subsystem has been added.
0513-071 The nimd Subsystem has been added.
0513-059 The nimesis Subsystem has been started. Subsystem PID is 9568296.
[..]

NIM resources such as spot, lpp_source and so on can be created right now, please refer to the NIM cheatsheet by chmod666.org ;-). For the purpose of this post some resources (spot, lpp_source, mksysb, network) are created, these ones will be replicated later.

Configure the alternate NIM master

NIM alternate master is configured with the niminit command. If you check on the NIM from a to Z, page 124, a note is warning you about the synchronization : “At the time of writing, only rsh/rshd communication is supported for NIM synchronization.”.THIS STATEMENT IS FALSE : I’m using nimsh for the synchronization, and I recommend to use it. We are in 2013, do not use rsh anymore.

# niminit -a is_alternate=yes -a master=nim_master -a pif_name=en0 -a cable_type1=N/A -a connect=nimsh -a name=nim_alternate
0513-071 The nimesis Subsystem has been added.
0513-071 The nimd Subsystem has been added.
0513-059 The nimesis Subsystem has been started. Subsystem PID is 10944522.
nimsh:2:wait:/usr/bin/startsrc -g nimclient >/dev/console 2>&1
0513-044 The nimsh Subsystem was requested to stop.
0513-059 The nimsh Subsystem has been started. Subsystem PID is 5963998.

Verification

You’re done with the configuration, you can now start to synchronize, replicate and takeover… pretty easy. Here are some points you can verify :

  • On the NIM master, the attribute is_alternate is set to yes :
  • # lsnim -l master
    [..]
       is_alternate        = yes
    [..]
    
  • On the NIM master, a new machine object typed alternate_master is created :
  • # lsnim -t alternate_master
    nim_alternate     machines       alternate_master
    
  • After the first database synchronization, on the alternate NIM master, a new machine object typed alternate_master is created, this the NIM master :
  • # lsnim -t alternate_master
    nim_master     machines       alternate_master
    
  • On the alternate NIM master, the attribute is_alternate does not exists :
  • # lsnim -l master | grep alternate
    

Synchronization and replication

NIM master and alternate NIM master can now communicate with each others, some resources are created on the master, and it’s now time to synchronize. Remember : HANIM only provides a method for replicating NIM database and resources. You can -if you want- synchronize the NIM database only or the NIM database and its resources (data included). Remember : never perform a NIM synchronization from the alternate NIM master.

Database synchronization only

The database synchronization is useful, when objects are modified, for example when you are modifying a subnet mask for a network object. It also can be useful when objects “without files” are created ; for instance a machine. On the other hand if your are trying to synchronize the database if an object “with a file” exists such as an lpp_source, a spot, or an fb_script, this one will not be created, you have to copy the file before synchronize, or use the replicate attribute :

  • On NIM master two objects are created, an fb_script and a machine:
  • # nim -o define -t fb_script -a server=master -a location=/export/nim/others/postinstall/fb_script.ksh fb_script01
    # ls -l /export/nim/others/postinstall/fb_script.ksh
    -rw-r--r--    1 root     system           35 Mar  8 18:01 /export/nim/others/postinstall/fb_script.ksh
    # lsnim ruby
    ruby     machines       standalone
    
  • A database synchronization is performed :
  • # nim -o sync -a force nim_alternate
    [..]
    The level of the NIM master fileset on this machine is: 7.1.2.15
    The level of the NIM database backup is: 7.1.2.15
    [..]
    Checking NIM resources
      Removing fb_script01
        0518-307 odmdelete: 1 objects deleted. from nim_attr (serves attr)
        0518-307 odmdelete: 0 objects deleted. from nim_attr (group memberships)
        0518-307 odmdelete: 5 objects deleted. from nim_attr (resource attributes)
        0518-307 odmdelete: 1 objects deleted. from nim_object (resource object)
      Finished removing fb_script01
    
  • On the alternate NIM master, the machine object is here but the fb_script was not replicated because the file was not present on the alternate NIM master :
  • # lsnim ruby
    ruby     machines       standalone
    # lsnim fb_script01
    0042-053 lsnim: there is no NIM object named "fb_script01"
    
  • If you copy the file before synchronize the resource will be created :
  • master# scp fb_script.ksh nim_alternate:/export/nim/others/postinstall
    fb_script.ksh                      100%   35     0.0KB/s   00:00
    
    master# nim -o sync -a force nim_alternate
    [..]
    Restoring the NIM database from /tmp/_nim_dir_13041674/mnt0
    x ./etc/NIM.level, 9 bytes, 1 tape blocks
    [..]
      Keeping fb_script01
    
    alternate# # lsnim fb_script01
    fb_script01     resources       fb_script
    

    Synchronization with replication

    I encourage you not to use the database synchronization, but to use it with replication, it does the same job but copy the files for you. Much much easier, just add replicate=yes attribute to the nim command, it works like a charm :

    # lsnim -q sync alternate_master
    
    the following attributes are optional:
            -a verbose=
            -a replicate=
            -a reset_clients=
    # nim -o sync -a force=yes -a replicate=yes alternate_master
    

    Takeover

    If the NIM master is down a takeover operation allows the alternate NIM master to become NIM master for the clients. On clients /etc/niminfo file is modified (NIM_MASTER_HOSTNAME and NIM_MASTER_HOSTNAME_LIST attributes are modified).

    • /etc/niminfo and lsnim output file before a takeover operation :
    • client# grep -E "NIM_MASTER_HOSTNAME_LIST|NIM_MASTER_HOSTNAME" /etc/niminfo
      export NIM_MASTER_HOSTNAME=nim_master
      export NIM_MASTER_HOSTNAME_LIST="nim_master nim_alternate"
      master# lsnim -l client | grep current_master
         current_master = nim_master
      
    • Takeover operation is initiated from the alternate NIM master :
    • alternate# nim -o takeover -a show_progress=yes nim_master
      +-----------------------------------------------------------------------------+
                            Performing "reset" Operation
      +-----------------------------------------------------------------------------+
      +-----------------------------------------------------------------------------+
                            "reset" Operation Summary
      +-----------------------------------------------------------------------------+
       Target                  Result
       ------                  ------
       client                   RESET
       client1                  RESET
       [..]
      +-----------------------------------------------------------------------------+
                            Initiating "takeover" Operation
      +-----------------------------------------------------------------------------+
       Initiating the takeover operation on machine 1 of 240: client ...
      
       Initiating the takeover operation on machine 2 of 240: client1...
      [..]
      +-----------------------------------------------------------------------------+
                            "takeover" Operation Summary
      +-----------------------------------------------------------------------------+
       Target                  Result
       ------                  ------
       client                  SUCCESS
       client1                 SUCCESS
      [..]
      alternate# lsnim -l client | grep current_master
         current_master = nim_alternate
      client# grep -E "NIM_MASTER_HOSTNAME_LIST|NIM_MASTER_HOSTNAME" /etc/niminfo
      export NIM_MASTER_HOSTNAME=nim_alternate
      export NIM_MASTER_HOSTNAME_LIST="nim_alternate nim_master"
      
    • When the NIM master is up, initiate the takeover for the master :
    • # nim -o takeover -a show_progress=yes nim_alternate
      

    Synchronization automation and other files ?

    I recommend to run a NIM synchronization every day, I personally have a cronjob doing it every day at eleven PM. Most of the time a NIM synchronization is not enough and you’ll need to synchronize others file in my case, my root .profile my etc/hosts file, in your case whatever you want. For this need I’m using a little script based over rsync which synchronize my master to my alternate everyday :

    # crontab -l
    [..]
    0 23 * * * /export/nim/others/tools/do_sync.ksh >/dev/null 2>&1
    [..]
    # cat /export/nim/others/tools/do_sync.ksh
    [..]
        nim -o sync -a force=yes -a replicate=yes -a reset_clients=yes ${alternate}
        /export/nim/others/tools/sync_to_alternate.ksh
    [..]
    # cat /export/nim/others/tools/sync_to_alternate.ksh
    [..]
      /usr/bin/rsync -ave ssh ${a_filesystem} ${alternate_nim_master}:${a_filesystem}
    [..]
    

    NIM Security, use nimsh and use it over SSL

    nimsh over ssl

    NIM Master configuration form nimsh over SSL

    From the NIM master enable the SSL support trough the nimconfig command, certificates will be generated in /ssl_nimsh/keys, OpenSSL fileset has to be installed :

    • Check OpenSSL filesets :
    • # lslpp -l | grep openssl
        openssl.base            0.9.8.2400  COMMITTED  Open Secure Socket Layer
        openssl.license         0.9.8.2400  COMMITTED  Open Secure Socket License
        openssl.man.en_US       0.9.8.2400  COMMITTED  Open Secure Socket Layer
        openssl.base            0.9.8.2400  COMMITTED  Open Secure Socket Layer
      
    • Use nimconfig to enable SSL support :
    • # nimconfig -c
      0513-029 The tftpd Subsystem is already active.
      Multiple instances are not supported.
      NIM_MASTER_HOSTNAME=nim_master
      x - /usr/lib/libssl.so.0.9.8
      x - /usr/lib/libcrypto.so.0.9.8
      Target "all" is up to date.
      Generating a 1024 bit RSA private key
      ......++++++
      .++++++
      writing new private key to '/ssl_nimsh/keys/rootkey.pem'
      -----
      Signature ok
      subject=/C=US/ST=Texas/L=Austin/O=ibm.com/CN=Root CA
      Getting Private key
      Generating a 1024 bit RSA private key
      ...............++++++
      .......++++++
      writing new private key to '/ssl_nimsh/keys/clientkey.pem'
      -----
      Signature ok
      subject=/C=US/ST=Texas/L=Austin/O=ibm.com
      Getting CA Private Key
      Generating a 1024 bit RSA private key
      ......++++++
      .............++++++
      writing new private key to '/ssl_nimsh/keys/serverkey.pem'
      -----
      Signature ok
      subject=/C=US/ST=Texas/L=Austin/O=ibm.com
      Getting CA Private Key
      
    • Check the NIM master : attribute ssl_support is now set to yes :
    • # lsnim -l master | grep ssl_support
         ssl_support         = yes
      

    NIM alternate master for nimsh over SSL

    If you’re using an alternate NIM master repeat the same operation (OpenSSL and nimconfig -r). Alternate NIM master is also a client of the NIM master, its client has to be configured :

    # nimclient -c
    x - /usr/lib/libssl.so.0.9.8
    x - /usr/lib/libcrypto.so.0.9.8
    Received 2763 Bytes in 0.0 Seconds
    0513-044 The nimsh Subsystem was requested to stop.
    0513-077 Subsystem has been changed.
    0513-059 The nimsh Subsystem has been started. Subsystem PID is 9502954.
    

    Client configuration

    Configure all nimclients to use ssl crypted authentication, if you are using alternate NIM master do not forget to download alternate certificates on clients :

    # rmitab nimsh 2>/dev/null 
    # rm -rf /etc/niminfo
    # niminit -aname=$(hostname) -a master=nim_master -a master_port=1058 -a registration_port=1059 -a connect=nimsh
    # nimclient -c
    # nimclient -o get_cert -a master_name=nim_alternate
    # stopsrc -s nimsh
    # startsrc -s nimsh
    

    On the NIM server itself client’s connect attribute is now set to “nimsh (secure)” :

    # lsnim -l ruby | grep connect
       connect        = nimsh (secure)
    

    Are the data encrypted ?

    Check this statement in NIM from a to Z Redbook at page 434 :

    “Any communication initiated from the NIM client (pull operation) reaches the NIM master on the request for services and registration ports (1058 and 1059, respectively). This communication is not encrypted. For any communication initiated from the NIM master (push operations), the NIM master communicates with the NIM client using the NIMSH daemon. This allows an encrypted handshake dialog during authentication. However, data packets are not encrypted.”

    To sum up :

    • Only push operations can use secure nimsh.
    • Data packets are not encrypted.
    • Secure nimsh just add an encrypted handshake between NIM master and its clients.

    Have a look on this two screenshots, the first one is the tcp stream of a non-secure operation, the second one is secured :

    • Non secure tcp stream of a push operation :
    • Secure tcp stream of a push operation :

    Distributed Systems Management

    Distributed Systems Management (we’ll call it DSM until now), is a set of tools and programs used to enhance NIM capabilities. I personally use DSM for two main purposes, opening and monitoring consoles through the dconsole utility, and to automate my installations. DSM add new objects the NIM environment, and new attributes to the NIM objects. You can also gain more on control on your lpars and directly restart, maint_boot an lpar through NIM by using DSM. Hardware Management Console (HMC objects) and Pserie’s frames (CEC objects) can be added in NIM, profile management are added to standalone objects in order to take advantage of DSM with NIM.

    There are two main source of information for DSM

    • The dsm.core fileset comes with a pdf file named dsm_tech_note.pdf, page 161, chapter 5.
    • # lslpp -f dsm.core | grep dsm_tech_note.pdf
                              /opt/ibm/sysmgt/dsm/doc/dsm_tech_note.pdf
      
    • There are full detailed examples in the IBM AIX Version 7.1 Differences Guide .

    Filesets prerequisites

    Starting with AIX 6.1 TL3 base installation media are shipped with DSM packages (dsm.core). expect, tcl, tk, and xterm are needed by this DSM pacakges :

    # lslpp -l | grep -E "dsm|tcl|tk|expect|xterm"
      X11.apps.aixterm           7.1.2.0  COMMITTED  AIXwindows aixterm Application
      X11.apps.xterm            7.1.2.15  COMMITTED  AIXwindows xterm Application
      X11.msg.en_US.apps.aixterm
                                 7.1.2.0  COMMITTED  AIXwindows aixterm Messages -
      dsm.core                  7.1.2.15  COMMITTED  Distributed Systems Management
      dsm.dsh                   7.1.2.15  COMMITTED  Distributed Systems Management
      expect.base               5.42.1.0  COMMITTED  Binary executable files of
      expect.man.en_US          5.42.1.0  COMMITTED  Expect man page documentation
      tcl.base                   8.4.7.0  COMMITTED  Binary executable files of Tcl
      tcl.man.en_US              8.4.7.0  COMMITTED  Tcl man page documentation
      tk.base                    8.4.7.0  COMMITTED  Binary executable files of Tk
      tk.man.en_US               8.4.7.0  COMMITTED  Tk man page documentation
    

    Defining HMC objects

    DSM is using HMC to start (poweron) lpars, stop (poweroff) lpars and open console on lpars. HMC can be defined on NIM. An HMC object is a management object. To avoid prompting password each time a NIM operations is performed, or each time dconsole is called, DSM provides a mechanism to manage SSH key sharing between the NIM and the HMC. Before adding an HMC object use dpasswd and dkeyexch command to enable SSH key authentication :

    • Create the authentication file with dpasswd command. File is by default stored in /etc/ibm/sysmgm/dsm/config :
    • # dpasswd -f hmc1_passwd -U hscroot
      Password:
      Re-enter password:
      Password file created
      # ls -l  /etc/ibm/sysmgt/dsm/config/
      total 24
      -r--r--r--    1 root     system           16 Mar 11 13:25 .key
      -r--r--r--    1 root     system           24 Mar 11 13:25 hmc1_passwd
      
    • Share the key between NIM master and HMC using dkeyexch command :
    • # dkeyexch -f /etc/ibm/sysmgt/dsm/config/hmc1_passwd -I hmc -H hmc1
      OpenSSH_6.0p1, OpenSSL 0.9.8x 10 May 2012
      
    • At this step you should be able to connect to the HMC without password prompting :
    • # ssh hscroot@hmc1
      Last login: Mon Mar 11 13:51:35 2013 from 10.10.20.21
      
    • Define the new HMC object with nim command, the network on which the HMC is running must be defined as an NIM network :
    • # nim -o define -t ent -a net_addr=10.10.30.0 -a snm=255.255.254.0 -a routing1="default 10.10.31.254" 10-10-30-0-s23-net
      # nim -o define -t hmc -a if1="find_net hmc1 0" -a passwd_file=/etc/ibm/sysmgt/dsm/config/hmc1_passwd hmc1
      # lsnim -t hmc
      hmc1     management       hmc
      # lsnim -lF hmc1
      hmc1:
         id          = 1363005068
         class       = management
         type        = hmc
         if1         = 10-10-30-0-s23-net hmc1 0
         Cstate      = ready for a NIM operation
         prev_state  =
         Mstate      = not running
         passwd_file = /etc/ibm/sysmgt/dsm/config/hmc1_passwd
      

    Defining CEC objects

    Defining HMC object allows to define CEC object, NIM CEC‘s object are requiring four mandatory attributes, hardware type (hw_type), hardware model (hw_model), hardware serial (hw_serial), and the HMC used to control this CEC object (mgmt_source). Query the HMC to get the attributes with lssyscfg command, and define the new CEC object with the nim command :

    • Querying HMC to get hw_model, hw_serial, and hw_type :
    • # ssh hscroot@hmc1 "lssyscfg -r sys -F name,type_model,serial_num"
      # CEC1,8203-E4A,060CE99
      
    • lssyscfg output tells you that : hw_type=8203, hw_model=EA4 and hw_serial=060CE99
    • Create the CEC object :
    • # nim -o define -t cec -a hw_type=8203 -a hw_model=E4A -a hw_serial=060CE99 -a mgmt_source=hmc1 cec1
      # lsnim -l cec1
      cec1:
         class      = management
         type       = cec
         Cstate     = ready for a NIM operation
         prev_state =
         hmc        = hmc1
         serial     = 8203-E4A*060CE99
      

    Adding profile management to standalone object

    To define a standalone object with a management profile or to add a management profile to an existing standalone, MAC address and lpar id are needed, the lpar id can easily be learned by the HMC, for the MAC address use the dgetmacs command to get it:

    • Get the lpar id trough the HMC :
    • ssh hscroot@infmc102 "lssyscfg -r lpar -m CEC1 -F name,lpar_id"
      lpar1,5
      lpar2,4
      vios1,3
      vios2,2
      lpar3,1
      
    • Define the machine and replace the MAC address by 0 :
    • # nim -o define -t standalone -a if1="10-10-20-0-s24-net lpar2 0" -a net_settings1="auto auto" -a mgmt_profile1="hmc1 4 CEC1" lpar2
      
    • Retrieve the machine MAC address by using the dgetmacs command, the host will booted on openfirmware. If the host is already installed get the MAC address with entstat command directly on the machine :
    • #  dgetmacs -n lpar2 -C NIM
      Using an adapter type of "ent".
      Could not dsh to node lpar2.
      Attempting to use openfirmware method to collect MAC addresses.
      Acquiring adapter information from Open Firmware for node lpar2.
      
      # Node::adapter_type::interface_name::MAC_address::location::media_speed::adapter_duplex::UNUSED::install_gateway::ping_status::machine_type::netaddr::subnet_mask
      
      lpar1::ent_v::::2643EEBC6C04::U8203.E4A.060CE99-V4-C4-T1::auto::auto::::::n/a::secondary::::
      
    • Modify the NIM object to add the MAC address :
    • # nim -o change -a if1="10-10-20-0-s24-net lpar2 2643EEBC6C04" lpar2
      

    Using dconsole to open and monitor machines consoles

    If the machine is already installed, or after the installation with a bos_inst operation, you can manage its console with the dconsole command. A few cool things comes with dconsole such as opening a console in read only mode, opening a console in text mode or through an xterm, and logging all consoles outputs into /var/ibm/sysmgt/dsm/log/console; here are a few examples :

    • Opening a text console in read-write mode and log the output in /var/ibm/sysmgt/dsm/log/console :
    • # dconsole -C NIM -n lpar2 -t -l
      Starting console daemon
      [read-write session]
      
       Open in progress
      
       Open Completed.
      AIX Version 7
      Copyright IBM Corporation, 1982, 2013.
      Console login: root
      # echo test
      test
      # tail -10 /var/ibm/sysmgt/dsm/log/console/lpar2.0
      # echo test
      test
      # exit
      
    • Opening an xterm console in read-write mode and log the output in /var/ibm/sysmgt/dsm/log/console on greenclient1 :
    • # export DISPLAY=10.10.20.35:0
      # dconsole -C NIM -n greenclient1  -l
      Starting console daemon
      

    • Opening a text console in read-only mode :
    • # dconsole -C NIM -n lpar2  -l -t -r
      Starting console daemon
      [read only session, user input discarded]
      
       Open in progress
      
       Open Completed.
      AIX Version 7
      Copyright IBM Corporation, 1982, 2013.
      Console login: [read only session, user input discarded]
      [read only session, user input discarded]
      

    bos_inst operation through NIM with DSM

    Machine installation and bos_inst operation can be automated with DSM. If a machine has a management profile and a bos_inst operation is performed this one will be rebooted and automatically installed, I do install machine with this method and it works like a charm :

    • Install the machine lpar2 in aix 7100-02-02, a bosinst_data with no prompt stanza was created for this installation :
    • # nim -o bos_inst -a bosinst_data=hdisk0_noprompt-bosinst_data -a source=rte -a installp_flags=agX -a accept_licenses=yes -a spot=7100-02-02-1316-spot -a lpp_source=7100-02-02-1316-lpp_source lpar2
      dnetboot Status: Invoking /opt/ibm/sysmgt/dsm/dsmbin/lpar_netboot lpar2
      dnetboot Status: Was successful network booting node lpar2.
      
    • DSM is using HMC lpar_netboot command to install machines, the output of this command can be found in /tmp filesystem :
    • # cat /tmp/lpar_netboot.12124286.exec.log
      lpar_netboot Status: process id is 12124286
      lpar_netboot Status: lpar_netboot -i -t ent -D -S 10.10.20.140 -G 10.10.20.254 -C 10.10.20.202 -m 2643EEBC6C04 -s auto -d auto -F /etc/ibm/sysmgt/dsm/config/hmc1_passwd -j hmc -J 10.10.30.1 4 060C
      E74 8203-E4A
      [..]
      IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM
      IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM
      
                1 = SMS Menu                          5 = Default Boot List
                8 = Open Firmware Prompt              6 = Stored Boot List
      [..]
      10.10.20.202:    24  bytes from 10.10.20.140:  icmp_seq=7  ttl=? time=21  ms
      
      10.10.20.202:    24  bytes from 10.10.20.140:  icmp_seq=8  ttl=? time=21  ms
      PING SUCCESS.
      [..]
      38300 ^MPACKET COUNT = 38400 ^MPACKET COUNT = 38500 ^MPACKET COUNT = 38600 ^MPACKET COUNT = 38700 ^MPACKET COUNT = 38800 ^MPACKET COUNT = 38900 ^MFINAL PACKET COUNT = 38913
      FINAL FILE SIZE = 19922944  BYTES
      
    • The installation progression can be monitored form the NIM itself :
    • # lsnim -l lpar2 |grep info
         info           = BOS install 39% complete : Installing additional software.
      

    Is it free ?

    Unlike CSM DSM is free, you do not need any licenses to use it. As you can see these tools can be very powerful to automate installations for standalone clients. VMControl is using DSM and NIM to automate installations. DSM is the right tool to industrialize your NIM installations.

    Cheatsheet

    I love cheat sheet ! NIM commands are complex and hard to remember, I’ve search over the internet if a NIM cheat sheet already exists but I haven’t found anything correct or anything that fits my needs. I’m sure that a lot of my readers already knows William Favorite’s Quicksheets. I’m a huge fan of this Quicksheets and I was inspired by Willam when creating my own one for NIM. Feel free to contact me if you want to add or correct something in my cheat sheet, you’ll be -of course- credited if you add some useful informations. Click here to download my NIM cheat sheet :chmod666 NIM Cheat Sheet

    No future ?

    I do love NIM, but in my opinion it’s a little bit outdated, everyone is calling for an update of the Redbook (click here to call for an update ;-)) and of the product, me included. This part of the post was inspired by one of my AIX Guru, thanks to him, I’m sure he’ll recognize himself. If IBMers are reading this part of the post, please tell IBM to update NIM. Readers please react in the comments if you agree with me on this point. Here are a few points I want to see in a future NIM release :

    • Network package repository of software : publish lpp_source over http or https. IBM can publish an official repository, and customer can create its own one on the NIM server (this one can be synchronized with IBM official repository).
    • Create a client (updated nimclient) with search and download option. (Yes like yum).
    • Getting rid of bootp and tftp, download kernel (created in /tftpboot when a new SPOT is created) and ramdisk image trough http or https.
    • Replace nfs exports by http or https (or force nfsv4) for NIM resources sharing (SPOT, lpp_source, install_script, bosinst_data…)(easier for security, and firewall ruling).
    • Allow IPL menu to be setup in dhcp.
    • Automatic dependencies checking and resolution while installing a software.
    • Simplify postinstall (script) and firstboot (fb_script). My actual solution is to create a firstboot script, this one download a script and add an entry in /etc/inittab, the downloaded script do the job and remove the entry in /etc/inittab at the end of its execution.
    • Automatic multibos creation while updating a system trough NIM — or in option.
    • Keep mksysb the way it is, this is the best bare metal backup I ever known.
    • Getting rid of rsh, force user to use nimsh (for nimadm too).
    • Better design for high availability (HANIM auto sync for example).
    • NIM Database flexibility : Let user renaming an resource object (please do this !!!) — Who has never experienced this problem while creating a SPOT or an lpp_source with an erroneous name ?
    • Allow allocating multiple lpp_source for different installp_bundle for installation.
    • Allow nimadm migration to be performed without the exact same level for bos.alt_disk_install.rte fileset.
    • Allow nimsh to be configured over http or https (no more multiple ports for nimsh ; easier for security, and firewall ruling).
    • Automatically enabled cryptographic authentication for NIM service handler. (nimsh can uses SSL-encrypted certificates).
    • Easier NIM backup and restore, getting rid of m_backup_db and m_restore_db.


    Please comment and react I do need support ;-). Hope this can help.

Easy AIX installation using alt_disk_copy and PowerVM capabilities

Some days ago, I have to install 9 new AIX lpars, in a new VLAN and in a short time. Unfortunately I didn’t have time -and my network team too- to open NIM ports for theses new installations. So I ask myself a question : “How can I do to install a new lpar without NIM ports open ?”. First of all my environment is full NPIV. The first things I have to do is to create lpars and to generate new WWNPs for my new hosts. After doing this I ask my SAN team new Luns for my brand new lpars. Using PowerVM capabilities my idea is to create an additional Virtual client fibre channel adapter (forcing WWPNs of the new lpar) to allow an already installed lpar to see new Luns masked by my SAN team. When this already installed lpar can see theses new Luns the only thing I have to do is using alt_disk_copy command to clone my lpar on new disks, remove Virtual client fibre channel adapter, and re-add it to the new lpar. Here is the way I proceed.

In our case here are the id used for lpar and virtuals adapters :

  • VIO server id : 15.
  • already installed lpar id: 10.
  • new lpar to install id: 12.
  • temporarily Virtual client fibre channel adapter used on the already installed lpar : 99.
  • temporarily Virtual server fibre channel adapter used on the VIO Server : 99.
  • Virtual server fibre channel adapter used on the VIO Server for the new lpar: 10.
  • new lpar Virtual client fibre channel adapter removed at the begining and added on the end : 10.

Creating new Virtual client fibre channel adapter on an already installed lpar forcing WWNPs of the new lpar

On the new created lpar remove the Virtual client fibre channel adapter where new Luns are masked (you can’t create two adapters with same WWPNs). After this step go on the HMC and create the exactly same adapter on an already installed lpar forcing the same WWPNs :


# chhwres -m PSERIE-795-1 -r virtualio --rsubtype fc -o a --id 10 -s 99 'adapter_type=client,remote_lpar_name=vios1,remote_slot_num=99,wwpns="c0507604f2080088,c0507604f2080089"'

WWNPs are forced to theses values : c0507604f2080088,c0507604f2080089.

Creating new Virtual server fibre channel adapter on VIO Server to map your new client adapter

Client adapter is created, but we now have to map it on an Virtual server fibre channel adapter with id 99 (see command below) :

# chhwres -m PSERIE-795-1 -r virtualio --rsubtype fc -o a --id 15 -s 99 'adapter_type=server,remote_lpar_name=already_installed_lpar,remote_slot_num=99'

Mapping new adapters on the VIO Server

You now have to map the new Virtual server fibre channel adapter to a Physical fibre channel adapter using vfcmap command. Identify your adapter with its id (in our case id 99 is vfchost26) (discover it with cfgdev command).

# cfgdev
# vfcmap -vadapter vfchost26 -fcp fcs0

Clone the already installed lpar with alt_disk_copy

On the already installed lpar you can now see new luns, and use it to clone the system. Run cfgmgr command to discover new luns and alt_disk_copy to clone system :

# cfgmgr
# lsdev -C | grep 99 | grep hdisk
hdisk2
# alt_disk_copy -g -d hdisk2

Do not forget to update the bootlist. alt_disk_copy command set the bootlist automatically on the new hdisk. When alt_disk_copy has ended, remove the -now useless- Virtual client fibre channel adapter with rmdev command :

# lspv | grep rootvg
hdisk0
# bootlist -o -m normal hdisk0
# rmdev -Rdl fcs2

Removing now useless Virtual client fibre channel adapter on the already installed lpar

Through the HMC you can now remove the now useless Virtual client fibre channel adapter.

# chhwres -m PSERIE-795-1 -r virtualio --rsubtype fc -o d --id 10 -s 99

Run an cfgmgr on the already installed lpar, and check there is no additional adapter with id 99.

Removing now useless Virtual server fibre channel adapter on the VIO Server

Through the HMC remove the now useless Virtual server fibre channel adapter. First you have to remove it from the VIO Server, be very careful not to remove a wrong adapter :

# vfcmap -vadapter vfchost26 -fcp
# rmdev -dev vfchost26
# chhwres -m PSERIE-795-1 -r virtualio --rsubtype fc -o d --id 15 -s 99

Re-adding Virtual client fibre channel adapter on the new lpar forcing WWPNs

Last step, re-add the removed adapter on the new lpar forcing WWPNs :

chsyscfg -r proc -m PSERIE-795-1 -i 'name=newlpar,lpar_id=12,"virtual_fc_adapter+=""10/Client/15/vios1/10/c0507604f2080088,c0507604f2080089/0"""'

You can now boot your lpar using the new system created on new disks with alt_disk_copy. I hope this method can be supported by IBM, so IBM guys feel free to comment this awesome post and tell us if there is nothing wrong to clone systems using alt_disk_copy.

Hope this can help

Restoring mksysb image without nim using virtual optical disk and mkcd command

Sometimes, it can be very usefull to restore an lpar through CD/DVD. For example, your LPAR cannot access any known VLAN, or you don’t have any nim server. PowerVM virtualisation provides the ability to store images into repositories and to load them on Virtual Optic Devices. LPAR can be restored very easy with this method.

Creating image repository

First of all you need to create a Repository to store virtual media cd, use mkrep command create this Repository.
This Repository has to be created on an Storage Pool, by default, rootvg Storage Pool exists.

# lssp
Pool   Size(mb) Free(mb) Alloc Size(mb) BDs Type
rootvg 279552   201216         256      0 LVPOOL
# mkrep -sp rootvg -size 20G
Virtual Media Repository Created
Repository created within "VMLibrary" logical volume
# lsrep
Size(mb) Free(mb) Parent Pool Parent Size Parent Free
  20396    20396  rootvg           279552      201216

Adding virtual scsi adapter on LPAR and on VIO Server

On client LPAR

On lpar, create a new Client SCSI Adapter device as described below :
As you can see on image below (you can choose whatever you want) :

  • Client Vscsi adapter ID is set to 100.
  • Server Vscsi adapter ID is set to 100.

On VIO Server

On VIO Server, create a new Server SCSI Adapter device as described below :

As you can see on image below, Client Vscsi adapter ID (100) and Server Vscsi adapter ID are matching :

Post cheks on VIO Server

On VIO Server check that Virtual SCSI adapter is here, and is matching with our ID (100) :

# lsdev -type adapter | grep vhost0
vhost0           Available   Virtual SCSI Server Adapter
# lsdev -slots | grep vhost0
U9119.FHB.84F55B6-V3-C100    Virtual I/O Slot  vhost0

Creating a DVD mksysb form an AIX Lpar

On the host you want to restore create an mksysb image with mksysb command :

# mksysb -i /mksysb_images/my_node.mksysb
[..]

You now have to convert this mksysb into CD/DVDs, unfortunatly it’s not possible to create a big iso file and you’ll have to choose CD or DVD format. Use the mkcd command to convert mksysb file into bootable CD/DVDs. In our example we’ll create DVD sized iso files.

# mkcd -L -S -I /mksysb_images/mkcd -m /mksysb_images/my_node.mksysb
Initializing mkcd log: /var/adm/ras/mkcd.log...
Verifying command parameters...
Creating temporary file system: /mkcd/cd_fs...
Populating the CD or DVD file system...
Building chrp boot image...
Copying backup to the CD or DVD file system...
......
Creating Rock Ridge format image: /mksysb_images/mkcd/cd_image_11010246.vol1
Running mkisofs ...
.......
mkrr_fs was successful.

Making the CD or DVD image bootable...

Copying the remainder of the backup to the CD or DVD file system...
Creating Rock Ridge format image: /mksysb_images/mkcd/cd_image_11010246.vol2
Running mkisofs ...
.......
mkrr_fs was successful.

Copying the remainder of the backup to the CD or DVD file system...
Creating Rock Ridge format image: /mksysb_images/mkcd/cd_image_11010246.vol3
Running mkisofs ...
...
mkrr_fs was successful.
  • -L : this option is used to create DVD sized iso images.
  • -S : this option is used to keep image file, and avoid writing it on a real DVD.
  • -I : specify the directory where images will be stored.
  • -m : mksysb image file to convert into DVDs.

mkcd command as created three DVD files :

# ls -l
total 53815784
-rw-r--r--    1 root     system   4274950144 Apr 30 13:09 cd_image_22872126.vol1
-rw-r--r--    1 root     system   4293890048 Apr 30 13:12 cd_image_22872126.vol2
-rw-r--r--    1 root     system   4293890048 Apr 30 13:14 cd_image_22872126.vol3

Adding images to VIO Server repository

After creating DVDs files from an mksysb file you’ll have to put them on VIO Server repository, transfer it via scp or NFS, and add it into repository :

# mkvopt -name cd_image_22872126.vol1 -file ./cd_image_22872126.vol1
# mkvopt -name cd_image_22872126.vol2 -file ./cd_image_22872126.vol2
# mkvopt -name cd_image_22872126.vol3 -file ./cd_image_22872126.vol3
# lsrep
Size(mb) Free(mb) Parent Pool         Parent Size      Parent Free
   20397     7208 rootvg                   279552           201216

Name                                    File Size Optical         Access
cd_image_22872126.vol1                       4077 None            rw
cd_image_22872126.vol2                       4095 None            rw
cd_image_22872126.vol3                       4095 None            rw

Create Virtual Optic Device on VIO Server

On VIO Server create a Virtual Optic Device on Server Virtual SCSI Adapter with mkvdev command

# mkvdev -fbo -vadapter vhost0 -dev lpar_cdrom0
lpar_cdrom0 Available

Loading and Unloading Optic Device

You just have finished, load first DVD on Virtual Optic Device, boot the client on LPAR on Virtual SCSI Adapter and start you restore, then load DVDs one by one :

# loadopt -disk cd_image_22872126.vol1 -vtd lpar_cdrom0 -release
# lsrep
lsrep
Size(mb) Free(mb) Parent Pool         Parent Size      Parent Free
   20397     7208 rootvg                   279552           201216

Name                                    File Size Optical         Access
cd_image_22872126.vol1                       4077 lpar_cdrom0     rw
cd_image_22872126.vol2                       4095 None            rw
cd_image_22872126.vol3                       4095 None            rw
# unloadopt -release -vtd lpar_cdrom0
# loadopt -disk cd_image_22872126.vol2 -vtd lpar_cdrom0 -release
lsrep
Size(mb) Free(mb) Parent Pool         Parent Size      Parent Free
   20397     7208 rootvg                   279552           201216

Name                                    File Size Optical         Access
cd_image_22872126.vol1                       4077 None            rw
cd_image_22872126.vol2                       4095 lpar_cdrom0     rw
cd_image_22872126.vol3                       4095 None            rw

You’re done.

Upgrading from AIX 5.3 to AIX 6.1 using nimadm

It can be very usefull to have a minimum downtime when upgrading from AIX 5.3 to AIX 6.1. Using nimadm can be a really good method to do it. It seems that multibos can answer to this problem, but this is not the subject for this post. So let’s have a deeper look on nimadm.

In this post we’ll call :

  • nim_master/nim master : nim server used to perform migration.
  • nim_client/ nim client : nim client to upgrade from AIX5.3TL12 to AIX6.1TL06.

Prerequisites

Rsh configuration between nim server and and nim client

nimadm use rsh to transfer data from nim client to nim server before and after upgrade. I really don’t know why but theses backups are performed through rsh. I have to check if it can be done with ou beloved ssh

  • So, enable rsh connexion between nim master and nim client :
nim_client# chsubserver -a -v shell -p tcp6 -r inetd
nim_client# echo "nim_master root" >> /root/.rhosts
nim_client# chmod 600 .rhosts
  • Be sure rsh is fully working and test it :
nim_master# rsh nim_client hostname
nim_client

Nim server nimadmvg

Our nim server needs an nimadm volume group, we’ll call it nimadmvg. This volume group has be greater or equals than nim_client rootvg.

nim_master# mkvg -S -s 128M -y nimadmvg hdisk17 hdisk37 hdisk38 hdisk39
nim_master# lsvg -l nimadmvg
nimadmvg:
LV NAME TYPE LPs PPs PVs LV STATE MOUNT POINT
nim_master# lsvg -p nimadmvg
nimadmvg:
PV_NAME PV STATE TOTAL PPs FREE PPs FREE DISTRIBUTION
hdisk38 active 255 255 51..51..51..51..51
hdisk37 active 255 255 51..51..51..51..51
hdisk36 active 255 255 51..51..51..51..51
hdisk17 active 269 268 54..53..53..54..54

bos.alt_disk_install.rte fileset

To perform a nimadm migration, nim server, spot, and lpp_source needs bos.alt_disk_install.rte fileset. This fileset has to be in the exact same version on nim master, and in spot used for migration.

In our case we use a spot called spot_aix610TL06 and an lpp_source called lpp_aix610TL06, to migrate for AIX 5.3TL12 to AIX6.1TL06. We are going to check our bos.alt_disk_install.rte fileset on our nim and on our spot.

  • Let’s check on nim server :
nim_master# lslpp -l | grep -i bos.alt_disk_install.rte
bos.alt_disk_install.rte 6.1.6.1 COMMITTED Alternate Disk Installation
bos.alt_disk_install.rte 6.1.6.1 COMMITTED Alternate Disk Installation
  • Let’s check on spot_AIX610TL06 spot :
nim_master# nim -o showres 'spot_aix610TL06' | grep bos.alt_disk_install.rte
bos.alt_disk_install.rte 6.1.6.1 A F Alternate Disk Installation

Nim client free physical volumes for alt_inst_rootvg

Check that nim_client have enough free phyiscal volumes to cater it’s own rootvg for alt_inst_rootvg.

lpar#  lspv | grep -w rootvg | awk '{print $1}'
hdisk3
hdisk4
hdisk5

Running nimadm, it’s time to upgrade

Ok, we’re ready, all prerequisites are done, we are in strating blocks to begin our migration from AIX5.3TL12 to AIX6.1TL06.
Run nimadm command to start migration, use these arguments :

  1. -j : volume group used on nim_master.
  2. -c : nim_client hostname.
  3. -s : spot name.
  4. -l : lpp_source name.
  5. -d : disks used by nim_client for alt_inst_rootvg.
  6. -y : accept licences.

Here we go :

nimadm -j nimadmvg -c lpar-s spot_aix610TL06 -l lpp_aix610TL06 -d "hdisk3 hdisk4 hdisk5" -Y

nimadm is done in 12 steps, let’s have a look on each one :
All steps description are taken from http://www.ibm.com/developerworks/aix/library/au-migrate_nimadm/index.html

phase 1

“nim_master run alt_disk_install on nim client, alternate root volume group (altinst_rootvg) is created during this phase.”

+-----------------------------------------------------------------------------+
Executing nimadm phase 1.
+-----------------------------------------------------------------------------+
Cloning altinst_rootvg on client, Phase 1.
Client alt_disk_install command: alt_disk_copy -j -M 6.1 -P1 -d "hdisk3 hdisk4 hdisk5"
Calling mkszfile to create new /image.data file.
Checking disk sizes.
Creating cloned rootvg volume group and associated logical volumes.
Creating logical volume alt_hd5.
Creating logical volume alt_paging00.
Creating logical volume alt_tivolilv.
Creating logical volume alt_hd8.
Creating logical volume alt_hd4.
Creating logical volume alt_hd2.
Creating logical volume alt_hd9var.
Creating logical volume alt_hd3.
Creating logical volume alt_hd1.
Creating logical volume alt_hd10opt.
Creating logical volume alt_devdumphd3.
Creating logical volume alt_cdc.
Creating logical volume alt_ccm63lv.
Creating logical volume alt_adminlv.
Creating logical volume alt_sinagioslv.
Creating logical volume alt_sysloadlv.
Creating logical volume alt_samlv.
Creating logical volume alt_tempolv.
Creating logical volume alt_omnilv.
Creating logical volume alt_db2v9lv.
Creating logical volume alt_orascriptlv.
Creating logical volume alt_ora1020lv.
Creating logical volume alt_orabdumplv.
Creating logical volume alt_oratarlv.
Creating logical volume alt_oranetlv.
Creating logical volume alt_oraadumplv.
Creating logical volume alt_oracdumplv.
Creating logical volume alt_oraudumplv.
Creating logical volume alt_tinalv.
Creating logical volume alt_hd11admin.
Creating /alt_inst/ file system.
Creating /alt_inst/admin file system.
Creating /alt_inst/home file system.
Creating /alt_inst/opt file system.
Creating /alt_inst/tempo file system.
Creating /alt_inst/tmp file system.
Creating /alt_inst/tools/list/admin file system.
Creating /alt_inst/tools/list/ccm63 file system.
Creating /alt_inst/tools/list/cdc file system.
Creating /alt_inst/tools/list/db2/V9.5FP8 file system.
Creating /alt_inst/tools/list/omnivision/collectnode file system.
Creating /alt_inst/tools/list/oracle/adump file system.
Creating /alt_inst/tools/list/oracle/bdump file system.
Creating /alt_inst/tools/list/oracle/cdump file system.
Creating /alt_inst/tools/list/oracle/network file system.
Creating /alt_inst/tools/list/oracle/product/10.2.0.4 file system.
Creating /alt_inst/tools/list/oracle/scripts file system.
Creating /alt_inst/tools/list/oracle/tar file system.
Creating /alt_inst/tools/list/oracle/udump file system.
Creating /alt_inst/tools/list/sam file system.
Creating /alt_inst/tools/list/sinagios file system.
Creating /alt_inst/tools/list/sysload file system.
Creating /alt_inst/tools/list/tina file system.
Creating /alt_inst/tools/list/tivoli file system.
Creating /alt_inst/usr file system.
Creating /alt_inst/var file system.
Generating a list of files
for backup and restore into the alternate file system...
Phase 1 complete.

phase 2

“nim_master creates the cache file systems in the nimadmvg volume group. Some initial checks for the required migration disk space are performed.”

+-----------------------------------------------------------------------------+
Executing nimadm phase 2.
+-----------------------------------------------------------------------------+
Creating nimadm cache file systems on volume group nimadmvg.
Checking for initial required migration space.
Creating cache file system /lpar-s_alt/alt_inst
Creating cache file system /lpar-s_alt/alt_inst/admin
Creating cache file system /lpar-s_alt/alt_inst/home
Creating cache file system /lpar-s_alt/alt_inst/opt
Creating cache file system /lpar-s_alt/alt_inst/tempo
Creating cache file system /lpar-s_alt/alt_inst/tmp
Creating cache file system /lpar-s_alt/alt_inst/tools/list/admin
Creating cache file system /lpar-s_alt/alt_inst/tools/list/ccm63
Creating cache file system /lpar-s_alt/alt_inst/tools/list/cdc
Creating cache file system /lpar-s_alt/alt_inst/tools/list/db2/V9.5FP8
Creating cache file system /lpar-s_alt/alt_inst/tools/list/omnivision/collectnode
Creating cache file system /lpar-s_alt/alt_inst/tools/list/oracle/adump
Creating cache file system /lpar-s_alt/alt_inst/tools/list/oracle/bdump
Creating cache file system /lpar-s_alt/alt_inst/tools/list/oracle/cdump
Creating cache file system /lpar-s_alt/alt_inst/tools/list/oracle/network
Creating cache file system /lpar-s_alt/alt_inst/tools/list/oracle/product/10.2.0.4
Creating cache file system /lpar-s_alt/alt_inst/tools/list/oracle/scripts
Creating cache file system /lpar-s_alt/alt_inst/tools/list/oracle/tar
Creating cache file system /lpar-s_alt/alt_inst/tools/list/oracle/udump
Creating cache file system /lpar-s_alt/alt_inst/tools/list/sam
Creating cache file system /lpar-s_alt/alt_inst/tools/list/sinagios
Creating cache file system /lpar-s_alt/alt_inst/tools/list/sysload
Creating cache file system /lpar-s_alt/alt_inst/tools/list/tina
Creating cache file system /lpar-s_alt/alt_inst/tools/list/tivoli
Creating cache file system /lpar-s_alt/alt_inst/usr
Creating cache file system /lpar-s_alt/alt_inst/var

phase 3

“nim_master creates the cache file systems in the nimadmvg volume group.”

+-----------------------------------------------------------------------------+
Executing nimadm phase 3.
+-----------------------------------------------------------------------------+
Syncing client data to cache ...

phase 4

“If a pre-migration script resource has been specified, it is executed at this time.”

+-----------------------------------------------------------------------------+
Executing nimadm phase 4.
+-----------------------------------------------------------------------------+
nimadm: There is no user customization script specified for this phase.

phase 5

“System configuration files are saved. Initial migration space is calculated and appropriate file system expansions are made. The bos image is restored and the device database is merged (similar to a conventional migration). All of the migration merge methods are executed, and some miscellaneous processing takes place.”

+-----------------------------------------------------------------------------+
Executing nimadm phase 5.
+-----------------------------------------------------------------------------+
Saving system configuration files.
Checking for initial required migration space.
Setting up for base operating system restore.
/lpar-s_alt/alt_inst
Restoring base operating system.
Merging system configuration files.
Running migration merge method: ODM_merge Config_Rules.
Running migration merge method: ODM_merge SRCextmeth.
Running migration merge method: ODM_merge SRCsubsys.
Running migration merge method: ODM_merge SWservAt.
Running migration merge method: ODM_merge pse.conf.
Running migration merge method: ODM_merge vfs.
Running migration merge method: ODM_merge xtiso.conf.
Running migration merge method: ODM_merge PdAtXtd.
Running migration merge method: ODM_merge PdDv.
Running migration merge method: convert_errnotify.
Running migration merge method: passwd_mig.
Running migration merge method: login_mig.
Running migration merge method: user_mrg.
Running migration merge method: secur_mig.
Running migration merge method: RoleMerge.
Running migration merge method: methods_mig.
Running migration merge method: mkusr_mig.
Running migration merge method: group_mig.
Running migration merge method: ldapcfg_mig.
Running migration merge method: ldapmap_mig.
Running migration merge method: convert_errlog.
Running migration merge method: ODM_merge GAI.
Running migration merge method: ODM_merge PdAt.
Running migration merge method: merge_smit_db.
Running migration merge method: ODM_merge fix.
Running migration merge method: merge_swvpds.
Running migration merge method: SysckMerge.

phase 6

“all system filesets are migrated using installp. Any required RPM images are also installed during this phase.”

+-----------------------------------------------------------------------------+
Executing nimadm phase 6.
+-----------------------------------------------------------------------------+
Installing and migrating software.
Updating install utilities.
[..]
+-----------------------------------------------------------------------------+
Installing Software...
+-----------------------------------------------------------------------------+

installp: APPLYING software for:
bos.rte.mlslib 6.1.6.0

. . . . . << Copyright notice for bos >> . . . . . . .
Licensed Materials - Property of IBM

5765G6200
Copyright International Business Machines Corp. 1985, 2010.
Copyright AT&T 1984, 1985, 1986, 1987, 1988, 1989.
Copyright Regents of the University of California 1980, 1982, 1983, 1985, 1986, 1987, 1988, 1989.
Copyright BULL 1993, 2010.
Copyright Digi International Inc. 1988-1993.
Copyright Interactive Systems Corporation 1985, 1991.
Copyright ISQUARE, Inc. 1990.
Copyright Innovative Security Systems, Inc. 2001-2006.
Copyright Mentat Inc. 1990, 1991.
Copyright Open Software Foundation, Inc. 1989, 1994.
Copyright Sun Microsystems, Inc. 1984, 1985, 1986, 1987, 1988, 1991.

All rights reserved.
US Government Users Restricted Rights - Use, duplication or disclosure
restricted by GSA ADP Schedule Contract with IBM Corp.
. . . . . << End of copyright notice for bos >>. . . .

Filesets processed: 1 of 2 (Total time: 3 secs).

installp: APPLYING software for:
X11.compat.lib.X11R6_motif 6.1.6.0

. . . . . <>< ;; . . . . . . .
Licensed Materials - Property of IBM

5765G6200
Copyright International Business Machines Corp. 2007, 2010.
Copyright Massachusetts Institute of Technology, 1985, 1994.

All rights reserved.
US Government Users Restricted Rights - Use, duplication or disclosure
restricted by GSA ADP Schedule Contract with IBM Corp.
. . . . . << End of copyright notice for X11.compat >>. . . .

Finished processing all filesets. (Total time: 8 secs).
[..]

phase 7

"If a post-migration script resource has been specified, it is executed at this time."

+-----------------------------------------------------------------------------+
Executing nimadm phase 7.
+-----------------------------------------------------------------------------+
nimadm: There is no user customization script specified for this phase.

phase 8

"The bosboot command is run to create a client boot image, which is written to the client's alternate boot logical volume (alt_hd5)."

+-----------------------------------------------------------------------------+
Executing nimadm phase 8.
+-----------------------------------------------------------------------------+
Creating client boot image.
bosboot: Boot image is 45273 512 byte blocks.
Writing boot image to client's alternate boot disk hdisk3.

phase 9

"All the migrated data is now copied from the NIM master's local cache file and synced to the client's alternate rootvg via rsh."

+-----------------------------------------------------------------------------+
Executing nimadm phase 9.
+-----------------------------------------------------------------------------+
Adjusting client file system sizes ...
Adjusting size for /
Adjusting size for /admin
Adjusting size for /home
Adjusting size for /opt
Adjusting size for /tempo
Adjusting size for /tmp
Adjusting size for /tools/list/admin
Adjusting size for /tools/list/ccm63
Adjusting size for /tools/list/cdc
Adjusting size for /tools/list/db2/V9.5FP8
Adjusting size for /tools/list/omnivision/collectnode
Adjusting size for /tools/list/oracle/adump
Adjusting size for /tools/list/oracle/bdump
Adjusting size for /tools/list/oracle/cdump
Adjusting size for /tools/list/oracle/network
Adjusting size for /tools/list/oracle/product/10.2.0.4
Adjusting size for /tools/list/oracle/scripts
Adjusting size for /tools/list/oracle/tar
Adjusting size for /tools/list/oracle/udump
Adjusting size for /tools/list/sam
Adjusting size for /tools/list/sinagios
Adjusting size for /tools/list/sysload
Adjusting size for /tools/list/tina
Adjusting size for /tools/list/tivoli
Adjusting size for /usr
Adjusting size for /var
Syncing cache data to client ...

phase 10

"The NIM master cleans up and removes the local cache file systems."

+-----------------------------------------------------------------------------+
Executing nimadm phase 10.
+-----------------------------------------------------------------------------+
Unmounting client mounts on the NIM master.
forced unmount of /lpar-s_alt/alt_inst/var
forced unmount of /lpar-s_alt/alt_inst/usr
forced unmount of /lpar-s_alt/alt_inst/tools/list/tivoli
forced unmount of /lpar-s_alt/alt_inst/tools/list/tina
forced unmount of /lpar-s_alt/alt_inst/tools/list/sysload
forced unmount of /lpar-s_alt/alt_inst/tools/list/sinagios
forced unmount of /lpar-s_alt/alt_inst/tools/list/sam
forced unmount of /lpar-s_alt/alt_inst/tools/list/oracle/udump
forced unmount of /lpar-s_alt/alt_inst/tools/list/oracle/tar
forced unmount of /lpar-s_alt/alt_inst/tools/list/oracle/scripts
forced unmount of /lpar-s_alt/alt_inst/tools/list/oracle/product/10.2.0.4
forced unmount of /lpar-s_alt/alt_inst/tools/list/oracle/network
forced unmount of /lpar-s_alt/alt_inst/tools/list/oracle/cdump
forced unmount of /lpar-s_alt/alt_inst/tools/list/oracle/bdump
forced unmount of /lpar-s_alt/alt_inst/tools/list/oracle/adump
forced unmount of /lpar-s_alt/alt_inst/tools/list/omnivision/collectnode
forced unmount of /lpar-s_alt/alt_inst/tools/list/db2/V9.5FP8
forced unmount of /lpar-s_alt/alt_inst/tools/list/cdc
forced unmount of /lpar-s_alt/alt_inst/tools/list/ccm63
forced unmount of /lpar-s_alt/alt_inst/tools/list/admin
forced unmount of /lpar-s_alt/alt_inst/tmp
forced unmount of /lpar-s_alt/alt_inst/tempo
forced unmount of /lpar-s_alt/alt_inst/opt
forced unmount of /lpar-s_alt/alt_inst/home
forced unmount of /lpar-s_alt/alt_inst/admin
forced unmount of /lpar-s_alt/alt_inst
Removing nimadm cache file systems.
Removing cache file system /lpar-s_alt/alt_inst
Removing cache file system /lpar-s_alt/alt_inst/admin
Removing cache file system /lpar-s_alt/alt_inst/home
Removing cache file system /lpar-s_alt/alt_inst/opt
Removing cache file system /lpar-s_alt/alt_inst/tempo
Removing cache file system /lpar-s_alt/alt_inst/tmp
Removing cache file system /lpar-s_alt/alt_inst/tools/list/admin
Removing cache file system /lpar-s_alt/alt_inst/tools/list/ccm63
Removing cache file system /lpar-s_alt/alt_inst/tools/list/cdc
Removing cache file system /lpar-s_alt/alt_inst/tools/list/db2/V9.5FP8
Removing cache file system /lpar-s_alt/alt_inst/tools/list/omnivision/collectnode
Removing cache file system /lpar-s_alt/alt_inst/tools/list/oracle/adump
Removing cache file system /lpar-s_alt/alt_inst/tools/list/oracle/bdump
Removing cache file system /lpar-s_alt/alt_inst/tools/list/oracle/cdump
Removing cache file system /lpar-s_alt/alt_inst/tools/list/oracle/network
Removing cache file system /lpar-s_alt/alt_inst/tools/list/oracle/product/10.2.0.4
Removing cache file system /lpar-s_alt/alt_inst/tools/list/oracle/scripts
Removing cache file system /lpar-s_alt/alt_inst/tools/list/oracle/tar
Removing cache file system /lpar-s_alt/alt_inst/tools/list/oracle/udump
Removing cache file system /lpar-s_alt/alt_inst/tools/list/sam
Removing cache file system /lpar-s_alt/alt_inst/tools/list/sinagios
Removing cache file system /lpar-s_alt/alt_inst/tools/list/sysload
Removing cache file system /lpar-s_alt/alt_inst/tools/list/tina
Removing cache file system /lpar-s_alt/alt_inst/tools/list/tivoli
Removing cache file system /lpar-s_alt/alt_inst/usr
Removing cache file system /lpar-s_alt/alt_inst/var

phase 11

"The alt_disk_install command is called again to make the final adjustments and put altinst_rootvg to sleep. The bootlist is set to the target disk."

+-----------------------------------------------------------------------------+
Executing nimadm phase 11.
+-----------------------------------------------------------------------------+
Cloning altinst_rootvg on client, Phase 3.
Client alt_disk_install command: alt_disk_copy -j -M 6.1 -P3 -d "hdisk3 hdisk4 hdisk5"
## Phase 3 ###################
Verifying altinst_rootvg...
Modifying ODM on cloned disk.
forced unmount of /alt_inst/var
forced unmount of /alt_inst/usr
forced unmount of /alt_inst/tools/list/tivoli
forced unmount of /alt_inst/tools/list/tina
forced unmount of /alt_inst/tools/list/sysload
forced unmount of /alt_inst/tools/list/sinagios
forced unmount of /alt_inst/tools/list/sam
forced unmount of /alt_inst/tools/list/oracle/udump
forced unmount of /alt_inst/tools/list/oracle/tar
forced unmount of /alt_inst/tools/list/oracle/scripts
forced unmount of /alt_inst/tools/list/oracle/product/10.2.0.4
forced unmount of /alt_inst/tools/list/oracle/network
forced unmount of /alt_inst/tools/list/oracle/cdump
forced unmount of /alt_inst/tools/list/oracle/bdump
forced unmount of /alt_inst/tools/list/oracle/adump
forced unmount of /alt_inst/tools/list/omnivision/collectnode
forced unmount of /alt_inst/tools/list/db2/V9.5FP8
forced unmount of /alt_inst/tools/list/cdc
forced unmount of /alt_inst/tools/list/ccm63
forced unmount of /alt_inst/tools/list/admin
forced unmount of /alt_inst/tmp
forced unmount of /alt_inst/tempo
forced unmount of /alt_inst/opt
forced unmount of /alt_inst/home
forced unmount of /alt_inst/admin
forced unmount of /alt_inst
Changing logical volume names in volume group descriptor area.
Fixing LV control blocks...
Fixing file system superblocks...
Bootlist is set to the boot disk: hdisk3 blv=hd5

phase 12

"Cleanup is executed to end the migration."

+-----------------------------------------------------------------------------+
Executing nimadm phase 12.
+-----------------------------------------------------------------------------+
Cleaning up alt_disk_migration on the NIM master.
Cleaning up alt_disk_migration on client lpar-s

Final steps

Check bootlist on nim_client

Check nim client boot list is set on alt_inst_rootvg before rebooting

nim_client# lspv
hdisk0 00cf55c68810289c rootvg
hdisk1 00cf55c6a65524a1 datavg active
hdisk2 00cf55c6a655251d datavg active
hdisk3 00cf55c6a6552594 altinst_rootvg active
hdisk4 00cf55c6a655264d altinst_rootvg active
hdisk5 00cf55c6a65526f0 altinst_rootvg active
nim_client# bootlist -o -m normal
hdisk3 blv=hd5 pathid=0
hdisk3 blv=hd5 pathid=1
hdisk3 blv=hd5 pathid=2
hdisk3 blv=hd5 pathid=3

Reboot oslevel

Reboot nim client and check AIX version with oslevel command :

nim_client# shutdown -Fr
nim_client# oslevel -s
6100-06-06-1140

Congratulation you have successfully migrated an AIX lpar from 5.3TL12 to 6.1TL06.