Adventures in IBM Systems Director in System P environment. Part 2 : Updating hosts with update manager

With the new release 2.2.1.4 of VIO Server, I have to update more than 80 VIO Servers. I gave myself a challenge, using ISD Update manager to download and install all these updates. Using Update manager seems to be easy, but configure access to the Internet through a proxy, understanding relationships between AIX hosts, VIO Server, NIM Server, and ISD Server can be painful, and it really give me an headache. It takes me more than a week to configure Update manager. The first advice I can give to you is take a look at these two awesome videos by Nigel Griffiths :

To really understand how Update manager proceeds we first have to talk about theory, then we can practice and update two AIX boxes. Yes AIX, as I am writing this article I did not succeed in updating a VIO Server from 2.2.1.3 to 2.2.1.4. Any help will be appreciated, but i’ll talk about this a bit later in conclusion.

Theory

Here are the needed steps to upgrade a system using update manager. Remember to check all steps in this order while updating a system :

  • Step 1: On the destination system who has to be updated, a common agent has to be installed.
  • Step 2: Always discover, access, and inventory the system who has to be updated.
  • Step 3: Configure this system as a nim client of the nim server using smitty nim_config_services.
  • Step 4: Request fix central to acquire udpate definitions on IBM Systems Director.
  • Step 5: Check system compliance with freshly downloaded update definitions.
  • Step 6: If an update is needed, download it on IBM Systems Director.
  • Step 7: Push downloaded update to nim server.
  • Step 8: Create nim object on nim server with the new downloaded update.
  • Step 9: Update system, reboot it if needed, and re-inventory it.

To sum up all theses steps have a look on the image below. Most of theses steps are automatised by IBM Systems Director itself, but I strongly recommand to check it manually while updating process.

Practice, using GUI interface, updating to 6.1TL05SP08

Let’s update an AIX client named blueclient2 in AIX 6.1TL05SP08, as I told you before we can check these 9 steps :

  • Step 1 & 2 : Is blueclient2, discovered, accessible, and inventoried ? Is Common Agent installed ?
  • # smcli lssys blueclient2
    blueclient2
    # smcli lssys -A Protocols -n blueclient2
    blueclient2: { 'CAS', 'CIM', 'SSH' }
    
    # smcli lssys -A AccessState -n blueclient2
    blueclient2: Unlocked
    
    # smcli lsinv -n blueclient2
    blueclient2:
        CacheMemory.ElementType = com.ibm.usmi.services.manageablecomponent.IManageableComponent
        CacheMemory.ObjectType = CacheMemory
        CacheMemory.DisplayName = L2 Cache Memory
    [..]
    
  • Step 3 : Is blue client configured as a nim client ?
  • # nimclient -l -L blueclient2
    lpp_base530                                lpp_source
    lpp_base520                                lpp_source
    lpp_base610                                lpp_source
    [..]
    
  • Step 3 : If step 3 is unsuccessfull, run nim_config_services :
  • # smitty nim_config_services
    * Communication Protocol used by client              [shell]
    
      NIM Service Handler Options
    *   Enable Cryptographic Authentication              [disable]
          for client communication?
    
        Install Secure Socket Layer Software (SSLv3)?    [no]
          Absolute path location for RPM package         [/dev/cd0]
                      -OR-
          lpp_source which contains RPM package          []
    
        Alternate Port Range for Secondary Connections
           (reserved values will be used if left blank)
          Secondary Port Number                          []
          Port Increment Range                           []
    
  • Step 4,5,6,7,8,9 are automatically performed by GUI interface.
  • On blue client, select Show and Install Updates :
  • Available updates are listed by ISD :
  • If an update is not Downloaded, you can choose to download it on ISD server, if you do not choose to download it, ISD will automatically download it for you :
  • If a restart is needed ISD will warn you, you can choose to restart system a bit later :
  • You can check process is running well by going in to task manager, and by checking steps :
  • While process is running you can have a look on NIM Server, NIM Object are created by ISD, in this example a new lpp_source object is beeing created named 6100-05-08-1207_lppsrc :
  • # lsnim -l 6100-05-08-1207_lppsrc
    6100-05-08-1207_lppsrc:
       class       = resources
       type        = lpp_source
       arch        = power
       Rstate      = ready for use
       prev_state  = unavailable for use
       location    = /export/um_lpp_source/6100.05.08.1207
       alloc_count = 1
       server      = master
    # lsnim | tail -1
    6100-05-08-1207_lppsrc                     resources       lpp_source
    
  • On blueclient2, NIM export is mounted, and a custom NIM operation is currently running :
  • # mount
      node       mounted        mounted over    vfs       date        options
    -------- ---------------  ---------------  ------ ------------ ---------------
             /dev/hd4         /                jfs2   Jul 11 16:01 rw,log=/dev/hd8
             /dev/hd2         /usr             jfs2   Jul 11 16:01 rw,log=/dev/hd8
             /dev/hd9var      /var             jfs2   Jul 11 16:01 rw,log=/dev/hd8
             /dev/hd3         /tmp             jfs2   Jul 11 16:01 rw,log=/dev/hd8
             /dev/hd1         /home            jfs2   Jul 11 16:01 rw,log=/dev/hd8
             /dev/hd11admin   /admin           jfs2   Jul 11 16:01 rw,log=/dev/hd8
             /proc            /proc            procfs Jul 11 16:01 rw
             /dev/hd10opt     /opt             jfs2   Jul 11 16:01 rw,log=/dev/hd8
             /dev/toolsadminlv /tools/admin jfs2   Jul 11 16:01 rw,log=INLINE
    nim /export/um_lpp_source/6100.05.08.1207 /tmp/_nim_dir_10682408/mnt0 nfs3   Jul 11 16:54 hard,intr
    
    # ps -ef | grep cust
        root  6357106  8192198   0 17:00:49  pts/0  0:00 grep cust
        root 11075790 11337794   0 16:54:40      -  0:00 nimclient -o cust -a lpp_source=6100-05-08-1207_lppsrc -a installp_flags=agXYv -a fixes=update_all
    
  • After updating process, blueclient2 is automatically rebooted in TL05SP08 :
  • #
    Broadcast message from root@blueclient2 (tty) at 17:04:30 ...
    
    PLEASE LOG OFF NOW ! ! !
    System maintenance in progress.
    All processes will be killed now.
    
    Broadcast message from root@blueclient2 (tty) at 17:04:30 ...
    
    ! ! ! SYSTEM BEING BROUGHT DOWN NOW ! ! !
    
    # oslevel -s
    6100-05-08-1207
    

    Practice, using command line, updating to 6.1TL07SP03

    As you know, I’m a big fan of command line, so let’s try using smcli command line interface to update our blueclient2 from AIX 6.1TL01 to 6.1TL07.

  • First step is to check needed updates for this server, using installneeded command :
  • # smcli installneeded -n blueclient2 -u AIX -v
    ATKUPD489I Collecting inventory for one or more systems.
    ATKUPD470I The check for updates task has started.
    ATKUPD421I A new update acquisition request is active.
    ATKUPD434I "AIX Acquisition Provider" is querying for updates.
    ATKUPD435I "AIX Acquisition Provider" is acquiring update information or files.
    ATKUPD450I Reacquired information for update "U835509".
    ATKUPD436I "AIX Acquisition Provider" completed successfully.
    ATKUPD461I Processing the update information.
    ATKUPD462I Completed processing the update information.
    ATKUPD426I The update acquisition request has ended successfully.
    ATKUPD472I The check for updates task has completed. No new updates were found.
    ATKUSC209I The install needed task found updates that need to be installed for system "blueclient2":
            U836308
            U849976
            U850864
    ATKUSC210I This operation will install the updates listed above. To continue, type "1" for yes or "0" for no.
    0
    ATKUPD288I The install needed updates task has completed successfully.
    
  • 3 updates are needed for this system U836308, U849976, U850864. Theses names are not very useful so it’s possible to check which AIX version they match :
  • # smcli lsupd -P Platform=AIX -sA DisplayName | egrep "U836308|U849976|U850864"
    U836308: 6100-01-09-1015
    U850864: 6100-05-08-1207
    U849976: 6100-07-03-1207
    
  • It’s not possible to update this system in one step, ISD has to update to 6.1TL05, then to 6.1TL07, here is the upgrade to 6.1TL05 :
  • # smcli installneeded -n blueclient2 -u AIX -v
    ATKUPD489I Collecting inventory for one or more systems.
    ATKUPD421I A new update acquisition request is active.
    ATKUPD434I "AIX Acquisition Provider" is querying for updates.
    ATKUPD450I Reacquired information for update "U835509".
    ATKUPD436I "AIX Acquisition Provider" completed successfully.
    ATKUPD461I Processing the update information.
    ATKUPD462I Completed processing the update information.
    ATKUPD426I The update acquisition request has ended successfully.
    ATKUPD472I The check for updates task has completed. No new updates were found.
    ATKUSC209I The install needed task found updates that need to be installed for system "blueclient2":
            U836308
            U849976
            U850864
    ATKUSC210I This operation will install the updates listed above. To continue, type "1" for yes or "0" for no.
    1
    ATKUPD725I The update install task has started.
    ATKUPD631I The update installation staging first needs to download 1 updates, the download is being launched.
    ATKUPD421I A new update acquisition request is active.
    ATKUPD434I "AIX Acquisition Provider" is querying for updates.
    ATKUPD435I "AIX Acquisition Provider" is acquiring update information or files.
    ATKUPD451I Acquired new update "U836308".
    [..]
    ATKUPD451I Acquired new update "U831304".
    ATKUPD451I Acquired new update "U831303".
    ATKUPD451I Acquired new update "U831302".
    ATKUPD451I Acquired new update "U831301".
    ATKUPD451I Acquired new update "U831300".
    ATKUPD451I Acquired new update "U831299".
    ATKUPD451I Acquired new update "U831297".
    [..]
    ATKUPD451I Acquired new update "U814492".
    ATKUPD451I Acquired new update "U814489".
    ATKUPD451I Acquired new update "U815942".
    ATKUPD451I Acquired new update "U818231".
    ATKUPD436I "AIX Acquisition Provider" completed successfully.
    ATKUPD461I Processing the update information.
    ATKUPD462I Completed processing the update information.
    ATKUPD426I The update acquisition request has ended successfully.
    ATKUPD629I Installation staging will be performed to 1 systems.
    ATKUPD632I The Installation Staging task is starting to process system "blueclient2".
    ATKUPD911I The updates will be copied to the installation staging server "nim" to path "/export/um_lpp_source".
    ATKUPD686I The update "U849976" has been staged for installation to "/export/um_lpp_source/6100.07.03.1207/" on the managed resource "nim" successfully.
    ATKUPD686I The update "U850864" has been staged for installation to "/export/um_lpp_source/6100.05.08.1207/" on the managed resource "nim" successfully.
    ATKUPD686I The update "U836308" has been staged for installation to "/export/um_lpp_source/6100.01.09.1015/" on the managed resource "nim" successfully.
    DNZPAX050I The "6100-01-09-1015_lppsrc" lpp_source resource is created successfully.
    ATKUPD686I The update "U835281" has been staged for installation to "/export/um_lpp_source/6100.05.00.1015/" on the managed resource "nim" successfully.
    DNZPAX050I The "6100-05-00-1015_lppsrc" lpp_source resource is created successfully.
    ATKUPD686I The update "U847803" has been staged for installation to "/export/um_lpp_source/6100.07.00.1140/" on the managed resource "nim" successfully.
    DNZPAX050I The "6100-07-00-1140_lppsrc" lpp_source resource is created successfully.
    ATKUPD633I The Installation Staging task has finished processing system "blueclient2".
    ATKUPD630I The update installation staging has completed.
    ATKUPD760I Start processing update "U849976" and system "blueclient2".
    Installing U849976 on blueclient2
    ATKUPD739I Collecting inventory on system "blueclient2".
    ATKUPD572I Running compliance on system "blueclient2".
    ATKUPD727I The update install task has finished successfully.
    ATKUPD288I The install needed updates task has completed successfully.
    
  • lpp_source object have been created on NIM server, and some missing updates have been downloaded and pushed from ISD to NIM, then exported on blueclient2 :
  • # lsnim -l 6100-01-09-1015_lppsrc
    6100-01-09-1015_lppsrc:
       class       = resources
       type        = lpp_source
       arch        = power
       Rstate      = ready for use
       prev_state  = unavailable for use
       location    = /export/um_lpp_source/6100.01.09.1015
       alloc_count = 0
       server      = master
    # lsnim -l 6100-05-00-1015_lppsrc
    6100-05-00-1015_lppsrc:
       class       = resources
       type        = lpp_source
       arch        = power
       Rstate      = ready for use
       prev_state  = unavailable for use
       location    = /export/um_lpp_source/6100.05.00.1015
       alloc_count = 0
       server      = master 
    
    # exportfs
    /app/nim/os                                      -sec=sys:none,rw
    /app/nim/sp                                      -sec=sys:none,rw
    /app/nim/distrib                                 -sec=sys:none,rw
    /app/nim/tl                                      -sec=sys:none,rw
    /export/um_lpp_source/6100.07.00.1140p6100.07.03.1207 -ro,root=blueclient2,access=blueclient2
    /export/nim/scripts/blueclient2.script                -ro,root=blueclient2,access=blueclient2
    
  • On blueclient2, a nimclient custom operation has been performed, followed by a system reboot :
  • # mount
      node       mounted        mounted over    vfs       date        options
    -------- ---------------  ---------------  ------ ------------ ---------------
             /dev/hd4         /                jfs2   Jul 16 14:43 rw,log=/dev/hd8
             /dev/hd2         /usr             jfs2   Jul 16 14:43 rw,log=/dev/hd8
             /dev/hd9var      /var             jfs2   Jul 16 14:43 rw,log=/dev/hd8
             /dev/hd3         /tmp             jfs2   Jul 16 14:43 rw,log=/dev/hd8
             /dev/hd1         /home            jfs2   Jul 16 14:43 rw,log=/dev/hd8
             /dev/hd11admin   /admin           jfs2   Jul 16 14:43 rw,log=/dev/hd8
             /proc            /proc            procfs Jul 16 14:43 rw
             /dev/hd10opt     /opt             jfs2   Jul 16 14:43 rw,log=/dev/hd8
             /dev/toolsadminlv /tools/admin jfs2   Jul 16 14:43 rw,log=INLINE
    nim /export/um_lpp_source/6100.07.00.1140p6100.07.03.1207 /tmp/_nim_dir_344192/mnt0 nfs3   Jul 17 11:14 hard,intr
    
    # ps -ef | grep installp
        root 205032 401562   0 11:15:18  pts/0  0:00 grep installp
        root 237666 270508  21 11:14:38      -  0:01 /usr/sbin/installp -agXYv -e /var/adm/ras/nim.installp -f /tmp/.workdir.471266.270508_1/.genlib.installp.list.270508 -d /tmp/_nim_dir_344192/mnt0 -q -f /tmp/inuexecXTEgib
        root 270508 344192   0 11:14:28      -  0:00 /bin/ksh /usr/sbin/geninstall -Z -I -agXYv -e /var/adm/ras/nim.installp -d /tmp/_nim_dir_344192/mnt0 -f /tmp/_nim_dir_344192/updt_all.lst
        root 344192 397512   0 11:14:13      -  0:00 /bin/ksh /usr/lpp/bos.sysmgt/nim/methods/c_installp -afixes=update_all -ainstallp_flags=agXYv -alpp_source=nim:/export/um_lpp_source/6100.07.00.1140p6100.07.03.1207
        root 422064 430158   0 11:14:08      -  0:00 nimclient -o cust -a lpp_source=6100-07-00-1140p6100-07-03-1207_lppsrc -a installp_flags=agXYv -a fixes=update_all
    
    Broadcast message from root@blueclient2 (tty) at 11:35:32 ...
    
    PLEASE LOG OFF NOW ! ! !
    System maintenance in progress.
    All processes will be killed now.
    
    Broadcast message from root@blueclient2 (tty) at 11:35:32 ...
    
    ! ! ! SYSTEM BEING BROUGHT DOWN NOW ! ! !
    
    Connection to blueclient2 closed by remote host.
    Connection to blueclient2 closed.
    
    # oslevel -s
    6100-05-07-1140
    
  • Repeat this step on update from 6.1TL05 to 6.1TL07 :
  • # smcli installneeded -n blueclient2 -u AIX -v
    ATKUPD489I Collecting inventory for one or more systems.
    ATKUPD470I The check for updates task has started.
    ATKUPD421I A new update acquisition request is active.
    ATKUPD434I "AIX Acquisition Provider" is querying for updates.
    ATKUPD437I "AIX Acquisition Provider" completed successfully, but no updates that match the criteria are available to be acquired.
    ATKUPD426I The update acquisition request has ended successfully.
    ATKUPD472I The check for updates task has completed. No new updates were found.
    ATKUSC209I The install needed task found updates that need to be installed for system "blueclient2":
            U849976
            U851974
    ATKUSC210I This operation will install the updates listed above. To continue, type "1" for yes or "0" for no.
    1
    ATKUPD725I The update install task has started.
    ATKUPD487I The download task has finished successfully.
    ATKUPD629I Installation staging will be performed to 1 systems.
    ATKUPD632I The Installation Staging task is starting to process system "blueclient2".
    ATKUPD911I The updates will be copied to the installation staging server "nim" to path "/export/um_lpp_source".
    ATKUPD686I The update "U849976" has been staged for installation to "/export/um_lpp_source/6100.07.03.1207/" on the managed resource "nim" successfully.
    ATKUPD686I The update "U851974" has been staged for installation to "/export/um_lpp_source/6100.07.04.1216/" on the managed resource "nim" successfully.
    ATKUPD633I The Installation Staging task has finished processing system "blueclient2".
    ATKUPD630I The update installation staging has completed.
    ATKUPD760I Start processing update "U851974" and system "blueclient2".
    Installing U851974 on blueclient2
    ATKUPD739I Collecting inventory on system "blueclient2".
    ATKUPD572I Running compliance on system "blueclient2".
    ATKUPD727I The update install task has finished successfully.
    ATKUPD288I The install needed updates task has completed successfully.
    
  • After a system reboot, you check blueclient is in TL07 :
  • # oslevel -s
    6100-07-03-1207
    

    Is update manager really usable ?

    I’ve never use SUMA, but it seems to be hated by IBM customer, so I hope ISD can replace it. Anyway, while updating my AIX boxes using ISD update manager a ton of problems appears. Accessing Fix Central trough an enterprise proxy server is an impossible mission, can IBM please use FQDN and not IP addresses to access Fix Central in ISD ? (Have a look on /opt/ibm/director/lwi/runtime/director/eclipse/plugins/com.ibm.director.mgr.updates.server_6.3.1/ecc/config/serviceProviderIBM.xml, this file contains the list of addresses and names used for connecting to IBM for Update Manager, and SSM. Use this list to setup proxy server, yes, you’re not dreaming, with IP addresses, i’m not joking). Updates are failing most of the time because there is one fileset missing for an SP or a TL and you have to correct theses problems manually. So I think it’s not really good idea to use update manager if your AIX servers are “living” WITH and WITHOUT systems director, but if you choose to update your server using ISD remember that you have to do it ALWAYS with ISD, and it can be usable. My last word will be a very good point for ISD, I’m so tired to download update using Fix Central; it’s always a loss of time and every customer has its own method. With ISD you can now forget Fix Central url and all these proxies problems.

    If anyone can help me updating a VIO Server from 2.2.1.3 to 2.2.1.4 using ISD please leave a comment, ISD do not recognize 2.2.1.4 update as applicable to a 2.2.1.3 VIO Server, please help me !!!!

    As always, hope this can help.

One thought on “Adventures in IBM Systems Director in System P environment. Part 2 : Updating hosts with update manager

  1. Pingback: Updating TL and SP using Chef | chmod666

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>