Adventures in IBM Systems Director in System P environment. Part 3 (Part 2 update): Updating VIO Server

A few days ago I finally got the answer why my VIO Servers cannot be updated through IBM Systems Director. It took my one day and a half of work with an IBM Systems Director Guru to find a solution. A day later I have to update 4 old VIO Servers from 2.1.3.10-FP-23 to 2.2.1.4, indeed the challenge was to update these olds VIO Servers with Systems Director. Here are the steps performed, problems, and solutions that appears while updating these VIO Servers.

Manual Agent installation, discovery, access, inventory

VIO Server 2.1.3.10-FP-23 is an old VIO Server release (30 April 2010) not supported by Systems Director, so Common agent has to be installed manually. Common agent can be installed as root but as to be run as padmin user; if Common agent is running as root Systems Director can’t connect to the agent.

Copy Common agent for ISD to VIO Server and install Common Agent

Common agent can be found on Systems Director host and can be easily copied to VIO Server :

  • Copying Common agent to VIO Server, default directory for Common agent installer is : /opt/ibm/director/packaging/agent/common/aix/ :
  • # scp /opt/ibm/director/packaging/agent/common/aix/6.3/dir6_3_commonagent_aix padmin@vio2:/tmp
    
  • Install Common agent on VIO Server :
  • # oem_setup_env
    # sh /tmp/dir6_3_commonagent_aix
    ...............................................................................
    dir6_3_commonagent_aix self-extracting installation program... Please wait...
    /tmp/DirectorAgentselfextract.9044022/dirinstall.agent[3]: test: argument expected
    +=============================================================================+
    Start of product installation on vios1
    +=============================================================================+
    /var free space not enough, less than 512M, trying to increase it...[done]
    [..]
    Installation of IBM Systems DirectorPlatformAgent completed successfully.
    

Check CAS protocol

A working Common Agent with CAS protocol discovered by Systems Director is a prerequisite to push a VIO Server updates through Systems Director. Check CAS protocol is discovered by System director :

  • Discovering :
  • # smcli discover -H vios1
    
  • Accessing :
  • # smcli accesssys -u padmin vios1
    
  • Inventory :
  • # smcli collectinv -p "All Inventory" vios1
    
  • Checking CAS protocol is discovered by Systems Director :
  • # smcli lssys -A Protocols vios1
    

Prerequisites

Communication with Common agent can fail because System Director is demanding some prerequisites. I my case Common agent was not working because of these problems :

Common agent running as root

Check Common agent is running as padmin and not as root user, and is running at VIO Server boot :

# cfgsvc DIRECTOR_agent -attr RESTART_ON_REBOOT=true
# startsvc DIRECTOR_agent
Starting cimserver...
#  lssvc DIRECTOR_agent
RESTART_ON_REBOOT:TRUE
# ps -ef |grep CIM
  padmin 6684914 5505090   0 12:51:57  pts/0  0:00 grep CIM
  padmin 7602396       1   0 12:51:32      -  0:00 /opt/freeware/cimom/pegasus/bin/CIM_diagd

Timezone on Systems Director and VIO Server are identical

Check timezone are identical on VIO Server and on Systems Director (in my case Europe/Paris):

  • VIO Server :
  • # chdate -timezone Europe/Paris
    # oem_setup_env
    # grep ^TZ /etc/environment
    TZ=Europe/Paris
    
  • Systems Director :
  • # grep ^TZ /etc/environment
    TZ=Europe/Paris
    

Loopback/Localhost is present on VIO Server

If -like me- your are updating and old VIO Server check loopback is present in /etc/hosts file. It’s a well know problem identify by IBM. http://www-01.ibm.com/support/docview.wss?uid=isg3T1012169

# oem_setup_env
# hostent -a 127.0.0.0 -h "loopback localhost"
# hostent -a ::1 -h "loopback localhost"
# grep loopback /etc/hosts

DNS configuration

I highly recommend to setup your DNS resolution correctly. Systems Director has to resolve (resolution and inverse resolution) all its clients.Systems Director‘s clients have to resolve (resolution and inverse resolution) the Director itself :

  • vios1 has to resolve Systems Director :
  • # oem_setup_env
    # host -n -t A sysdir.domain.test
    sysdir.mydomain.test has address 10.10.122.104
    # host -n -t PTR 10.10.122.104
    104.122.240.10.IN-ADDR.ARPA domain name pointer sysdir.mydomain.test
    # tail -1 /etc/netsvc.conf
    hosts = local4, bind4
    
  • Systems Director has to resolve vios1 :
  • # host -n -t A vios1.mydomain.test
    vios1.mydomain.test has address 10.10.122.61
    # host -n -t PTR 10.10.122.61
    61.122.240.10.IN-ADDR.ARPA domain name pointer vios1.mydomain.test
    

Removing efix

Unfortunately IBM Systems Director cannot remove efix. You have to remove theses fixes manually before any update try. Hope this can be corrected by IBM.

  • List installed efixes to remove it later :
  • # oem_setup_env
    # emgr -l
    
    ID  STATE LABEL      INSTALL TIME      UPDATED BY ABSTRACT
    === ===== ========== ================= ========== ======================================
    1    S    IV16920s02 03/21/12 14:28:52            MSNENTDD LINK DOWN AFTER REBOOT.
    [..]
    
  • Remove installed efixes :
  • # emgr -r -L IV16920s02
    

Updating

I will not describe the update phase. You can check part 2 of this serie here. Here are a series of screenshots and output while updating systems :

  • System Director updating from 2.2.1.1 to 2.2.1.4 :


  • System Director updating from 2.1.3.10-FP-23 to 2.2.0.13 (note thay System Director has to pass two updates in this case, 2.1.3.10-FP-23 to 2.2.0.10, then 2.2.0.10 to 2.2.0.13
  • [..]
    October 3, 2012 10:44:20 AM CEST-Level:150-MEID:6248--MSG: ATKUPD632I The Installation Staging task is starting to process system "vio2".
    October 3, 2012 10:44:20 AM CEST-Level:150-MEID:6248--MSG: ATKUPD911I The updates will be copied to the installation staging server "vio2" to path "/opt/VIOS/update".
    October 3, 2012 10:57:36 AM CEST-Level:150-MEID:6248--MSG: ATKUPD686I The update "VIOS_2.2.0.13-FP24-SP03" has been staged for installation to "/opt/VIOS/update/VIOS/VIOS_2.2.0.13-FP24-SP03/" on the managed resource "vio2" successfully.
    October 3, 2012 11:25:59 AM CEST-Level:150-MEID:6248--MSG: ATKUPD686I The update "VIOS_2.2.0.10-FP24" has been staged for installation to "/opt/VIOS/update/VIOS/VIOS_2.2.0.10-FP24/" on the managed resource "vio2" successfully.
    October 3, 2012 11:25:59 AM CEST-Level:150-MEID:6248--MSG: ATKUPD633I The Installation Staging task has finished processing system "vio2".
    October 3, 2012 11:25:59 AM CEST-Level:150-MEID:0--MSG: ATKUPD630I The update installation staging has completed.
    October 3, 2012 11:25:59 AM CEST-Level:150-MEID:6248--MSG: ATKUPD760I Start processing update "VIOS_2.2.0.10-FP24" and system "vio2".
    October 3, 2012 1:31:26 PM CEST-Level:150-MEID:6248--MSG: ATKUPD764I Update "VIOS_2.2.0.10-FP24" was installed on system "vio2" successfully.
    October 3, 2012 1:31:26 PM CEST-Level:150-MEID:6248--MSG: ATKUPD760I Start processing update "VIOS_2.2.0.13-FP24-SP03" and system "vio2".
    October 3, 2012 2:36:47 PM CEST-Level:150-MEID:6248--MSG: ATKUPD764I Update "VIOS_2.2.0.13-FP24-SP03" was installed on system "vio2" successfully.
    October 3, 2012 2:37:20 PM CEST-Level:100-MEID:223519--MSG: vio2 client job status changed to "Complete".
    [..]
    October 3, 2012 2:43:00 PM CEST-Level:150-MEID:0--MSG: ATKUPD727I The update install task has finished successfully.
    

Problems

While updating VIO Server from 2.1.3.10-FP-23 to 2.2.1.4 Systems Director wasn’t able to update in 2.2.1.1. I have to update from 2.2.0.13 to 2.2.1.1 trough the old fashion way with updateios command. If anybody can give me an answer on this point it’ll be appreciated.

Hope this can help.

Adventures in IBM Systems Director in System P environment. Part 2 : Updating hosts with update manager

With the new release 2.2.1.4 of VIO Server, I have to update more than 80 VIO Servers. I gave myself a challenge, using ISD Update manager to download and install all these updates. Using Update manager seems to be easy, but configure access to the Internet through a proxy, understanding relationships between AIX hosts, VIO Server, NIM Server, and ISD Server can be painful, and it really give me an headache. It takes me more than a week to configure Update manager. The first advice I can give to you is take a look at these two awesome videos by Nigel Griffiths :

To really understand how Update manager proceeds we first have to talk about theory, then we can practice and update two AIX boxes. Yes AIX, as I am writing this article I did not succeed in updating a VIO Server from 2.2.1.3 to 2.2.1.4. Any help will be appreciated, but i’ll talk about this a bit later in conclusion.

Theory

Here are the needed steps to upgrade a system using update manager. Remember to check all steps in this order while updating a system :

  • Step 1: On the destination system who has to be updated, a common agent has to be installed.
  • Step 2: Always discover, access, and inventory the system who has to be updated.
  • Step 3: Configure this system as a nim client of the nim server using smitty nim_config_services.
  • Step 4: Request fix central to acquire udpate definitions on IBM Systems Director.
  • Step 5: Check system compliance with freshly downloaded update definitions.
  • Step 6: If an update is needed, download it on IBM Systems Director.
  • Step 7: Push downloaded update to nim server.
  • Step 8: Create nim object on nim server with the new downloaded update.
  • Step 9: Update system, reboot it if needed, and re-inventory it.

To sum up all theses steps have a look on the image below. Most of theses steps are automatised by IBM Systems Director itself, but I strongly recommand to check it manually while updating process.

Practice, using GUI interface, updating to 6.1TL05SP08

Let’s update an AIX client named blueclient2 in AIX 6.1TL05SP08, as I told you before we can check these 9 steps :

  • Step 1 & 2 : Is blueclient2, discovered, accessible, and inventoried ? Is Common Agent installed ?
  • # smcli lssys blueclient2
    blueclient2
    # smcli lssys -A Protocols -n blueclient2
    blueclient2: { 'CAS', 'CIM', 'SSH' }
    
    # smcli lssys -A AccessState -n blueclient2
    blueclient2: Unlocked
    
    # smcli lsinv -n blueclient2
    blueclient2:
        CacheMemory.ElementType = com.ibm.usmi.services.manageablecomponent.IManageableComponent
        CacheMemory.ObjectType = CacheMemory
        CacheMemory.DisplayName = L2 Cache Memory
    [..]
    
  • Step 3 : Is blue client configured as a nim client ?
  • # nimclient -l -L blueclient2
    lpp_base530                                lpp_source
    lpp_base520                                lpp_source
    lpp_base610                                lpp_source
    [..]
    
  • Step 3 : If step 3 is unsuccessfull, run nim_config_services :
  • # smitty nim_config_services
    * Communication Protocol used by client              [shell]
    
      NIM Service Handler Options
    *   Enable Cryptographic Authentication              [disable]
          for client communication?
    
        Install Secure Socket Layer Software (SSLv3)?    [no]
          Absolute path location for RPM package         [/dev/cd0]
                      -OR-
          lpp_source which contains RPM package          []
    
        Alternate Port Range for Secondary Connections
           (reserved values will be used if left blank)
          Secondary Port Number                          []
          Port Increment Range                           []
    
  • Step 4,5,6,7,8,9 are automatically performed by GUI interface.
  • On blue client, select Show and Install Updates :
  • Available updates are listed by ISD :
  • If an update is not Downloaded, you can choose to download it on ISD server, if you do not choose to download it, ISD will automatically download it for you :
  • If a restart is needed ISD will warn you, you can choose to restart system a bit later :
  • You can check process is running well by going in to task manager, and by checking steps :
  • While process is running you can have a look on NIM Server, NIM Object are created by ISD, in this example a new lpp_source object is beeing created named 6100-05-08-1207_lppsrc :
  • # lsnim -l 6100-05-08-1207_lppsrc
    6100-05-08-1207_lppsrc:
       class       = resources
       type        = lpp_source
       arch        = power
       Rstate      = ready for use
       prev_state  = unavailable for use
       location    = /export/um_lpp_source/6100.05.08.1207
       alloc_count = 1
       server      = master
    # lsnim | tail -1
    6100-05-08-1207_lppsrc                     resources       lpp_source
    
  • On blueclient2, NIM export is mounted, and a custom NIM operation is currently running :
  • # mount
      node       mounted        mounted over    vfs       date        options
    -------- ---------------  ---------------  ------ ------------ ---------------
             /dev/hd4         /                jfs2   Jul 11 16:01 rw,log=/dev/hd8
             /dev/hd2         /usr             jfs2   Jul 11 16:01 rw,log=/dev/hd8
             /dev/hd9var      /var             jfs2   Jul 11 16:01 rw,log=/dev/hd8
             /dev/hd3         /tmp             jfs2   Jul 11 16:01 rw,log=/dev/hd8
             /dev/hd1         /home            jfs2   Jul 11 16:01 rw,log=/dev/hd8
             /dev/hd11admin   /admin           jfs2   Jul 11 16:01 rw,log=/dev/hd8
             /proc            /proc            procfs Jul 11 16:01 rw
             /dev/hd10opt     /opt             jfs2   Jul 11 16:01 rw,log=/dev/hd8
             /dev/toolsadminlv /tools/admin jfs2   Jul 11 16:01 rw,log=INLINE
    nim /export/um_lpp_source/6100.05.08.1207 /tmp/_nim_dir_10682408/mnt0 nfs3   Jul 11 16:54 hard,intr
    
    # ps -ef | grep cust
        root  6357106  8192198   0 17:00:49  pts/0  0:00 grep cust
        root 11075790 11337794   0 16:54:40      -  0:00 nimclient -o cust -a lpp_source=6100-05-08-1207_lppsrc -a installp_flags=agXYv -a fixes=update_all
    
  • After updating process, blueclient2 is automatically rebooted in TL05SP08 :
  • #
    Broadcast message from root@blueclient2 (tty) at 17:04:30 ...
    
    PLEASE LOG OFF NOW ! ! !
    System maintenance in progress.
    All processes will be killed now.
    
    Broadcast message from root@blueclient2 (tty) at 17:04:30 ...
    
    ! ! ! SYSTEM BEING BROUGHT DOWN NOW ! ! !
    
    # oslevel -s
    6100-05-08-1207
    

    Practice, using command line, updating to 6.1TL07SP03

    As you know, I’m a big fan of command line, so let’s try using smcli command line interface to update our blueclient2 from AIX 6.1TL01 to 6.1TL07.

  • First step is to check needed updates for this server, using installneeded command :
  • # smcli installneeded -n blueclient2 -u AIX -v
    ATKUPD489I Collecting inventory for one or more systems.
    ATKUPD470I The check for updates task has started.
    ATKUPD421I A new update acquisition request is active.
    ATKUPD434I "AIX Acquisition Provider" is querying for updates.
    ATKUPD435I "AIX Acquisition Provider" is acquiring update information or files.
    ATKUPD450I Reacquired information for update "U835509".
    ATKUPD436I "AIX Acquisition Provider" completed successfully.
    ATKUPD461I Processing the update information.
    ATKUPD462I Completed processing the update information.
    ATKUPD426I The update acquisition request has ended successfully.
    ATKUPD472I The check for updates task has completed. No new updates were found.
    ATKUSC209I The install needed task found updates that need to be installed for system "blueclient2":
            U836308
            U849976
            U850864
    ATKUSC210I This operation will install the updates listed above. To continue, type "1" for yes or "0" for no.
    0
    ATKUPD288I The install needed updates task has completed successfully.
    
  • 3 updates are needed for this system U836308, U849976, U850864. Theses names are not very useful so it’s possible to check which AIX version they match :
  • # smcli lsupd -P Platform=AIX -sA DisplayName | egrep "U836308|U849976|U850864"
    U836308: 6100-01-09-1015
    U850864: 6100-05-08-1207
    U849976: 6100-07-03-1207
    
  • It’s not possible to update this system in one step, ISD has to update to 6.1TL05, then to 6.1TL07, here is the upgrade to 6.1TL05 :
  • # smcli installneeded -n blueclient2 -u AIX -v
    ATKUPD489I Collecting inventory for one or more systems.
    ATKUPD421I A new update acquisition request is active.
    ATKUPD434I "AIX Acquisition Provider" is querying for updates.
    ATKUPD450I Reacquired information for update "U835509".
    ATKUPD436I "AIX Acquisition Provider" completed successfully.
    ATKUPD461I Processing the update information.
    ATKUPD462I Completed processing the update information.
    ATKUPD426I The update acquisition request has ended successfully.
    ATKUPD472I The check for updates task has completed. No new updates were found.
    ATKUSC209I The install needed task found updates that need to be installed for system "blueclient2":
            U836308
            U849976
            U850864
    ATKUSC210I This operation will install the updates listed above. To continue, type "1" for yes or "0" for no.
    1
    ATKUPD725I The update install task has started.
    ATKUPD631I The update installation staging first needs to download 1 updates, the download is being launched.
    ATKUPD421I A new update acquisition request is active.
    ATKUPD434I "AIX Acquisition Provider" is querying for updates.
    ATKUPD435I "AIX Acquisition Provider" is acquiring update information or files.
    ATKUPD451I Acquired new update "U836308".
    [..]
    ATKUPD451I Acquired new update "U831304".
    ATKUPD451I Acquired new update "U831303".
    ATKUPD451I Acquired new update "U831302".
    ATKUPD451I Acquired new update "U831301".
    ATKUPD451I Acquired new update "U831300".
    ATKUPD451I Acquired new update "U831299".
    ATKUPD451I Acquired new update "U831297".
    [..]
    ATKUPD451I Acquired new update "U814492".
    ATKUPD451I Acquired new update "U814489".
    ATKUPD451I Acquired new update "U815942".
    ATKUPD451I Acquired new update "U818231".
    ATKUPD436I "AIX Acquisition Provider" completed successfully.
    ATKUPD461I Processing the update information.
    ATKUPD462I Completed processing the update information.
    ATKUPD426I The update acquisition request has ended successfully.
    ATKUPD629I Installation staging will be performed to 1 systems.
    ATKUPD632I The Installation Staging task is starting to process system "blueclient2".
    ATKUPD911I The updates will be copied to the installation staging server "nim" to path "/export/um_lpp_source".
    ATKUPD686I The update "U849976" has been staged for installation to "/export/um_lpp_source/6100.07.03.1207/" on the managed resource "nim" successfully.
    ATKUPD686I The update "U850864" has been staged for installation to "/export/um_lpp_source/6100.05.08.1207/" on the managed resource "nim" successfully.
    ATKUPD686I The update "U836308" has been staged for installation to "/export/um_lpp_source/6100.01.09.1015/" on the managed resource "nim" successfully.
    DNZPAX050I The "6100-01-09-1015_lppsrc" lpp_source resource is created successfully.
    ATKUPD686I The update "U835281" has been staged for installation to "/export/um_lpp_source/6100.05.00.1015/" on the managed resource "nim" successfully.
    DNZPAX050I The "6100-05-00-1015_lppsrc" lpp_source resource is created successfully.
    ATKUPD686I The update "U847803" has been staged for installation to "/export/um_lpp_source/6100.07.00.1140/" on the managed resource "nim" successfully.
    DNZPAX050I The "6100-07-00-1140_lppsrc" lpp_source resource is created successfully.
    ATKUPD633I The Installation Staging task has finished processing system "blueclient2".
    ATKUPD630I The update installation staging has completed.
    ATKUPD760I Start processing update "U849976" and system "blueclient2".
    Installing U849976 on blueclient2
    ATKUPD739I Collecting inventory on system "blueclient2".
    ATKUPD572I Running compliance on system "blueclient2".
    ATKUPD727I The update install task has finished successfully.
    ATKUPD288I The install needed updates task has completed successfully.
    
  • lpp_source object have been created on NIM server, and some missing updates have been downloaded and pushed from ISD to NIM, then exported on blueclient2 :
  • # lsnim -l 6100-01-09-1015_lppsrc
    6100-01-09-1015_lppsrc:
       class       = resources
       type        = lpp_source
       arch        = power
       Rstate      = ready for use
       prev_state  = unavailable for use
       location    = /export/um_lpp_source/6100.01.09.1015
       alloc_count = 0
       server      = master
    # lsnim -l 6100-05-00-1015_lppsrc
    6100-05-00-1015_lppsrc:
       class       = resources
       type        = lpp_source
       arch        = power
       Rstate      = ready for use
       prev_state  = unavailable for use
       location    = /export/um_lpp_source/6100.05.00.1015
       alloc_count = 0
       server      = master 
    
    # exportfs
    /app/nim/os                                      -sec=sys:none,rw
    /app/nim/sp                                      -sec=sys:none,rw
    /app/nim/distrib                                 -sec=sys:none,rw
    /app/nim/tl                                      -sec=sys:none,rw
    /export/um_lpp_source/6100.07.00.1140p6100.07.03.1207 -ro,root=blueclient2,access=blueclient2
    /export/nim/scripts/blueclient2.script                -ro,root=blueclient2,access=blueclient2
    
  • On blueclient2, a nimclient custom operation has been performed, followed by a system reboot :
  • # mount
      node       mounted        mounted over    vfs       date        options
    -------- ---------------  ---------------  ------ ------------ ---------------
             /dev/hd4         /                jfs2   Jul 16 14:43 rw,log=/dev/hd8
             /dev/hd2         /usr             jfs2   Jul 16 14:43 rw,log=/dev/hd8
             /dev/hd9var      /var             jfs2   Jul 16 14:43 rw,log=/dev/hd8
             /dev/hd3         /tmp             jfs2   Jul 16 14:43 rw,log=/dev/hd8
             /dev/hd1         /home            jfs2   Jul 16 14:43 rw,log=/dev/hd8
             /dev/hd11admin   /admin           jfs2   Jul 16 14:43 rw,log=/dev/hd8
             /proc            /proc            procfs Jul 16 14:43 rw
             /dev/hd10opt     /opt             jfs2   Jul 16 14:43 rw,log=/dev/hd8
             /dev/toolsadminlv /tools/admin jfs2   Jul 16 14:43 rw,log=INLINE
    nim /export/um_lpp_source/6100.07.00.1140p6100.07.03.1207 /tmp/_nim_dir_344192/mnt0 nfs3   Jul 17 11:14 hard,intr
    
    # ps -ef | grep installp
        root 205032 401562   0 11:15:18  pts/0  0:00 grep installp
        root 237666 270508  21 11:14:38      -  0:01 /usr/sbin/installp -agXYv -e /var/adm/ras/nim.installp -f /tmp/.workdir.471266.270508_1/.genlib.installp.list.270508 -d /tmp/_nim_dir_344192/mnt0 -q -f /tmp/inuexecXTEgib
        root 270508 344192   0 11:14:28      -  0:00 /bin/ksh /usr/sbin/geninstall -Z -I -agXYv -e /var/adm/ras/nim.installp -d /tmp/_nim_dir_344192/mnt0 -f /tmp/_nim_dir_344192/updt_all.lst
        root 344192 397512   0 11:14:13      -  0:00 /bin/ksh /usr/lpp/bos.sysmgt/nim/methods/c_installp -afixes=update_all -ainstallp_flags=agXYv -alpp_source=nim:/export/um_lpp_source/6100.07.00.1140p6100.07.03.1207
        root 422064 430158   0 11:14:08      -  0:00 nimclient -o cust -a lpp_source=6100-07-00-1140p6100-07-03-1207_lppsrc -a installp_flags=agXYv -a fixes=update_all
    
    Broadcast message from root@blueclient2 (tty) at 11:35:32 ...
    
    PLEASE LOG OFF NOW ! ! !
    System maintenance in progress.
    All processes will be killed now.
    
    Broadcast message from root@blueclient2 (tty) at 11:35:32 ...
    
    ! ! ! SYSTEM BEING BROUGHT DOWN NOW ! ! !
    
    Connection to blueclient2 closed by remote host.
    Connection to blueclient2 closed.
    
    # oslevel -s
    6100-05-07-1140
    
  • Repeat this step on update from 6.1TL05 to 6.1TL07 :
  • # smcli installneeded -n blueclient2 -u AIX -v
    ATKUPD489I Collecting inventory for one or more systems.
    ATKUPD470I The check for updates task has started.
    ATKUPD421I A new update acquisition request is active.
    ATKUPD434I "AIX Acquisition Provider" is querying for updates.
    ATKUPD437I "AIX Acquisition Provider" completed successfully, but no updates that match the criteria are available to be acquired.
    ATKUPD426I The update acquisition request has ended successfully.
    ATKUPD472I The check for updates task has completed. No new updates were found.
    ATKUSC209I The install needed task found updates that need to be installed for system "blueclient2":
            U849976
            U851974
    ATKUSC210I This operation will install the updates listed above. To continue, type "1" for yes or "0" for no.
    1
    ATKUPD725I The update install task has started.
    ATKUPD487I The download task has finished successfully.
    ATKUPD629I Installation staging will be performed to 1 systems.
    ATKUPD632I The Installation Staging task is starting to process system "blueclient2".
    ATKUPD911I The updates will be copied to the installation staging server "nim" to path "/export/um_lpp_source".
    ATKUPD686I The update "U849976" has been staged for installation to "/export/um_lpp_source/6100.07.03.1207/" on the managed resource "nim" successfully.
    ATKUPD686I The update "U851974" has been staged for installation to "/export/um_lpp_source/6100.07.04.1216/" on the managed resource "nim" successfully.
    ATKUPD633I The Installation Staging task has finished processing system "blueclient2".
    ATKUPD630I The update installation staging has completed.
    ATKUPD760I Start processing update "U851974" and system "blueclient2".
    Installing U851974 on blueclient2
    ATKUPD739I Collecting inventory on system "blueclient2".
    ATKUPD572I Running compliance on system "blueclient2".
    ATKUPD727I The update install task has finished successfully.
    ATKUPD288I The install needed updates task has completed successfully.
    
  • After a system reboot, you check blueclient is in TL07 :
  • # oslevel -s
    6100-07-03-1207
    

    Is update manager really usable ?

    I’ve never use SUMA, but it seems to be hated by IBM customer, so I hope ISD can replace it. Anyway, while updating my AIX boxes using ISD update manager a ton of problems appears. Accessing Fix Central trough an enterprise proxy server is an impossible mission, can IBM please use FQDN and not IP addresses to access Fix Central in ISD ? (Have a look on /opt/ibm/director/lwi/runtime/director/eclipse/plugins/com.ibm.director.mgr.updates.server_6.3.1/ecc/config/serviceProviderIBM.xml, this file contains the list of addresses and names used for connecting to IBM for Update Manager, and SSM. Use this list to setup proxy server, yes, you’re not dreaming, with IP addresses, i’m not joking). Updates are failing most of the time because there is one fileset missing for an SP or a TL and you have to correct theses problems manually. So I think it’s not really good idea to use update manager if your AIX servers are “living” WITH and WITHOUT systems director, but if you choose to update your server using ISD remember that you have to do it ALWAYS with ISD, and it can be usable. My last word will be a very good point for ISD, I’m so tired to download update using Fix Central; it’s always a loss of time and every customer has its own method. With ISD you can now forget Fix Central url and all these proxies problems.

    If anyone can help me updating a VIO Server from 2.2.1.3 to 2.2.1.4 using ISD please leave a comment, ISD do not recognize 2.2.1.4 update as applicable to a 2.2.1.3 VIO Server, please help me !!!!

    As always, hope this can help.

Adventures in IBM Systems Director in System P environment. Part 1 : Discovering, Accessing, Inventorying …. and Querying

A week ago I met an IBM Systems Director Guru. We were talking about new PureSystems products released by IBM and he says this to me “If you know how to use and manage IBM Systems Director, you’ll know how to administrate a PureSystem”. Believe it or not but I think if you want to continue to go on with IBM you’ll have to put -now or later- your efforts on IBM Systems Director. Maybe I am wrong, anyway, this tool can really makes funny things (useful in production environments), but it too can makes you mad (i’ll talk about this later). I’ll begin a serie of posts talking about IBM Systems Director which I will call ISD from now on. This one will learn you how to start with it by doing the first things you have to do on every ISD. Discovering systems, Accessing Systems, Collect Inventory on it.

Do it Always In this order

First thing you have to remember is three letters D.A.I for Do it Always In this order but also for :

  • Discovering : The first thing you have to do is to discover your systems, system discovering can be performed through IP addresses, range of IP addresses, hostname, etc. Discovering your system at first on local network can be a good thing to do. Discovering will tell you what type of agent are installed on your system.No agent (by using SSH for example, no ISD components are installed on the system), Plateform agent (using CIM protocol for basic hardware alerting and management) and Common Agent (using CAS protocol for full management of a system)
  • Access and authentication : Before starting to collect inventory on your systems, authentication and access has to be ok. Three status are differencied No access, partial access, and full access. In most case you just have to enter your root credentials, or for HMC for example, hscroot credentials.
  • Inventorying : Since you have gain access on your system it’s time to collect inventory. Collecting inventory enables you to (indeed) check a lot of information on your systems. But it also let you, to use other ISD feature, for example you can’t update a system TL before knowing what’s is it’s current TL, that’s why theses three steps are the most important in ISD. No discovering, No accessing, and no Inventory, mean no ISD at all.

And remember to always do it in this order.

About command line

As most Unix guys I love command line. For most people ISD looks like a big smoky dirty machine with an ugly web interface (I agree on this point), but like most IBM tools behind the scene there is a wonderful, powerful command line interface. I’ll try on this post to use it in most case. Use smcli lsbundle to check enabled modules and how to use it in cli

# smcli lsbundle 
ActiveEnergyManager/chconfiginfo
[..]
vsmsecurity/chvsmauth

Discovering

When starting with ISD, you’ll have no system discovered, maybe itself, but I don’t think so. Listing systems should return nothing.

# smcli lssys

Hardware Management Console discovering

Let’s discover a system with its ip address, and let’s start with an Hardware management console :

# smcli discover -i 10.10.72.33
Total discovery modules:
28

Discovery completion percentage 89%
[..]
Discovery completion percentage 100%
Discovery completed:
100%

My system is calling hmc1, it can also discover it with its hostname :

smcli discover -H hmc1
Total discovery modules:
28

Discovery completion percentage 68%
[..]
Discovery completion percentage 100%
Discovery completed:
100%

This freshly new discovered system -as you can see- is an HMC, I can check it using smcli lssys command :

# smcli lssys -t HardwareManagementConsole
hmc1

Operating System discovering

After discovering an HMC let’s trying discovering our Operating Systems on our network, you can do that selecting a range of IP addresses, in our case it’s a /24 network, so I’m ignoring network address (.0), broadcast (.255), and gateway (.254). This step can be very long depending on your subnet mask, so you can get a coffee, a beer or whatever you want during this discover.

# smcli discover -i 10.10.122.1-10.10.122.253
smcli discover -i 10.10.122.1-10.10.122.253
Total discovery modules:
28

Discovery completion percentage 4%
[..]
Discovery completion percentage 100%

Let’s list Operating Systems I’ve found with this discovering, using smcli lssys command

# smcli lssys -t OperatingSystem -sA DisplayName,IPv4Address,AccessState,Protocols |grep 10.10.122
10.10.122.20: : 10.10.122.20, { '10.10.122.20' }, Locked, { 'CAS', 'CIM', 'SSH' }
10.10.122.21: : 10.10.122.21, { '10.10.122.21' }, Locked, { 'SSH' }
10.10.122.22: : 10.10.122.22, { '10.10.122.22' }, Locked, { 'CAS', 'SSH' }
10.10.122.23: : 10.10.122.23, { '10.10.122.23' }, Locked, { 'CAS', 'CIM', 'SSH' }
10.10.122.24: : 10.10.122.24, { '10.10.122.24' }, Locked, { 'CAS', 'CIM', 'SSH' }
10.10.122.25: : 10.10.122.25, { '10.10.122.25' }, Locked, { 'CAS', 'CIM', 'SSH' }

Great ! I’ve found a bunch of Operating Systems, but their AccessState are locked, so let’s unlock a few systems, on the second part. You can notice as I told you before that my Operating Systems do not have same agents configured, some have Common Agent, others SSH, etc… In order to work properly ports have to be opened on your Operating Systems/Firewall, here is a trick to test Comment Agent port are opened

# /usr/bin/slp_query --type="*" --address=10.10.122.16
0
5
62
URL: service:management-software.IBM:platform-agent://10.10.122.16
ATTR: (vendor=IBM),(version=6.3),(uid=7F4B6213FD2F3663),(ip-address=10.10.122.16),(hostname=nim)
URL: service:management-software.IBM:platform-agent://10.10.70.16
ATTR: (vendor=IBM),(version=6.3),(uid=7F4B6213FD2F3663),(ip-address=10.10.70.16),(hostname=nim)
URL: service:TivoliCommonAgent://nim:9510
ATTR: (ca-uid=file:///var/opt/tivoli/ep/runtime/agent),(am-host=10.10.70.55),(ca-ips=10.10.122.16\2c 10.10.70.16),(ca-basic-port=9510),(ca-cert-port=9510),(ca-version=1.4.2.4),(os-uid=922AAED091B411E0AD2DE41F13F9A245)
URL: service:management-software.IBM:usma://nim
ATTR: (ip-address=10.10.122.16@10.10.70.16),(mac-address=2e.4d.42.94.7d.5),(tivguid=922AAED091B411E0AD2DE41F13F9A245),(uid=7f4b6213fd2f3663),(vendor=IBM),(System-Name=nim),(timezone-offset=120),(version=6.3.0),(port=9510),(manager=10.10.70.55)
URL: service:service-agent://10.10.122.16
ATTR: (service-type=service:management-software.IBM:usma,service:service-agent)

If this command does not show anything you’ll probably have to open some ports on your firewall, here is the list :

  • 22 TCP Inbound.
  • 427 TCP, UDP Outbound and Inbound.
  • 5988, 5989 TCP Outbound and Inbound.
  • 6988, 6989 TCP Outbound and Inbound.
  • 9510 TCP Inbound.
  • 9511-9515 TCP Outbound.

Access and authentication

In order to collect inventory on your freshly new discovered system you’ll have to unlock access, let have a look first on our HMC :

# smcli lssys -sA AccessState hmc1
hmc1: : Locked

As you can see its state is Locked, use the smcli accesssys command to unlock access to the HMC :

smcli accesssys -u hscroot hmc1
Type the password: ******
DNZCLI0727I : Waiting for request access to complete on... hmc1
Result Value: DNZCLI0734I : Request access was successful on : hmc1
# smcli lssys -sA AccessState hmc1
hmc1: : Unlocked

Great, but what if I have to unlock one billion systems ? Do I have to repeat this operation each time ? We are lucky, IBM is nice with us. Let’s unlock in one command all our AIX 7.1

# smcli accesssys -u root -w "OSTypeString='AIX' AND OSVersion='7.1'"
Type the password: ******
DNZCLI0727I : Waiting for request access to complete on... 10.10.122.104
Result Value: DNZCLI0734I : Request access was successful on : 10.10.122.104
DNZCLI0727I : Waiting for request access to complete on... 10.10.122.16
Result Value: DNZCLI0734I : Request access was successful on : 10.10.122.16
DNZCLI0727I : Waiting for request access to complete on... 10.10.122.17
Result Value: DNZCLI0734I : Request access was successful on : 10.10.122.17
DNZCLI0727I : Waiting for request access to complete on... 10.10.122.18
Result Value: DNZCLI0734I : Request access was successful on : 10.10.122.18
# smcli lssys -sA DisplayName,AccessState -w "OSTypeString='AIX' AND OSVersion='7.1'"
10.10.122.104: : 10.10.122.104, Unlocked
10.10.122.16: : 10.10.122.16, Unlocked
10.10.122.17: : 10.10.122.17, Unlocked
10.10.122.18: : 10.10.122.18, Unlocked

Let’s go on the last step, collecting inventory, if systems are discovered and accessible this last step is very easy

Collecting Inventory

Let’s continue with our method, first HMC, then Operating systems, use smcli collectinv command to collect inventory on our HMC :

# smcli collectinv -p "All Inventory" hmc1
Inventory collection percentage 0%
Inventory collection percentage 34%
Inventory collection completed:
100%

We can see that I found a bunch of object

  • P Systems (Pseries) :
  • #smcli lssys -w ServerType=HMCManagedServer
    FRM04320A
    FRM04510B
    FRM06771A
    FRM07816C
    FRM07394A
    FRM07327B
    FRM07328C
    FRM07328D
    FRM07359A
    FRM08624A
    FRM08724B
    FRM08924D
    
  • LPAR (Servers in ISD terms) :
  • # smcli lssys -t Server
    [..]
    dev023app0001
    pro000isd0100
    pro002app0001
    pro002ora0001
    [..]
    

Let’s try the same things on an Operating System object :

# smcli collectinv -p "All Inventory" 10.10.122.16
Inventory collection percentage 0%
[..]
Inventory collection percentage 98%
Inventory collection completed:
100%

Repeat this operation, or use the trick I’ve explained you before to collect inventory on all your systems.

Querying

Here are nice examples of querying :

  • How many VIOS do I have :
  • # smcli lssys -t OperatingSystem -w OSType=27
    
  • Find me all VIOS where vlan 87 can be used :
  • for i in `smcli lssys -t OperatingSystem -w OSType=27` ; do echo ==${i}== ; smcli lsinv -n ${i} -e "Service" | grep SourceTokens |grep -w 87 ; done
    

The Good, The Bad, The Ugly

Remember, these first steps are a a prerequisite for ISD in order to work properly. Next posts will cover many things such as Update Manager, and Health Monitoring with ISD. As you can see this can be a very powerful tool which have a very cool command line interface and this is good. But there are other bad things, like documentation : only one Redbook for IBM Systems Director 6.1, not up to date, it’s a shame. Online documentation is very very poor, and we didn’t find any blogs talking about this, why ???. I’ve seen many strange things while discovering, accessing and collecting inventory and I’m sick of it (example : let’s try an access on an Operating System failed, let’s smoke a cigarette, and ten minutes later access works fine)

Hope this can help