Hardware Management Console : Be autonomous with upgrade, update, and use the Integrated Management Module

I’m sure that like me you do not have a physical access to your Hardware Management Consoles, or even if you have this access, some of your HMC are so far away from your working site (even in a foreign country) that you can’t afford to physically move to this place to update it. Even worse if -like me- you are working in a big place (who says too big ?) this job is often performed by IBM Inspectors and you do not have to worry about your Hardware Management Consoles and just have to ask IBM guys for anything about HMC. For some reasons I had to update an old Hardware Management Console from v7r7.3.0 to v7r7.7.0 SP3. Everybody is confused about the differences between updating an HMC and upgrading an HMC. I know really good bloggers : Anthony English and Rob McNelly have already post about this particular subject but I have to write this post as a reminder and to clarify some points which are not tell in Anthony’s and Rob’s posts :

To finish this post I’ll talk about a feature nobody is using : the HMC is coming with an Integrated Management Module, this one allows you to have more control and to be autonomous with you HMC.

The difference between updating, upgrading and migrating

There is a lot of confusion when people are trying “update” their HMC. When do I have to update using the updhmc command, when do I have to upgrade using saveupgdata, getupdfiles and chhmc command, and finally when do I have to migrate using HMC Recovery CD/DVD ? All of this three operations are not well described by IBM. Here is what I’m doing in for each case, and this is the result of my own experience (don’t take this as an official document). Here is a little reminder it can be useful for a lot of people : an HMC version number looks like this : v7r7.7.0 SP3. v7 is the VERSION. 7.7.0 is the RELEASE and SP3 is the SERVICE PACK.

  • Updating : You have to update your HMC if your are applying a service pack, or a corrective fix update on the HMC, this operation can only be performed by the updhmc command. Use this method if fix central gives you an iso named “HMC_Update_*.iso” or a zip files named “MHxxxxx.zip”. These fixes can be applied to a minor version of the HMC.
  • Upgrading : You have to upgrade your HMC if your are moving from one minor version to another (from one release to another), for instance if your are moving from v7r7.7.0 to v7r7.8.0, this operation can be made by using HMC Recovery DVD (fix central will gives you two isos named “HMC_Recovery_*.iso”), or by using the images provided for a network upgrade (I’ll explain this in this post).
  • Migrating : You have to migrate your HMC by using HMC Recovery DVD when your are moving from one major version to another. For example when your are moving from an HMC v6 to and HMC v7 (for instance from any v6 version to v7r7.8.0). In this case you have no other choice than burning DVDs and moving in front of the HMC to perform the operation by yourself.

Upgrading

You can upgrade your HMC from its local storage by using the network images provided by IBM on a public FTP server, once connected to the public FTP server get the version you want to upgrade to and download all the files, bzImage and initrd.gz included :

# ftp://ftp.software.ibm.com/software/server/hmc/network/v7770/
# ls
FTP Listing of /software/server/hmc/network/v7770/ at ftp.software.ibm.com
[..]
Feb 26 2013 00:00      2708320 bzImage
Feb 26 2013 00:00    808497152 disk1.img
Feb 26 2013 00:00   1142493184 disk2.img
Feb 26 2013 00:00   1205121024 disk3.img
Feb 26 2013 00:00           78 hmcnetworkfiles.sum
Feb 26 2013 00:00     34160044 initrd.gz
# mget *.*

Put all the files on a server where you have an FTP server running (the HMC getupgfiles is using FTP to get the files) and download all the files with the getupgfiles command directly form the HMC (if your HMC has a direct access to the internet you can specify it the command):

hscroot@gaff:~> getupgfiles -h 192.168.0.99 -u root -d /export/HMC/network_ugrade/v7770
Enter the current password for user root:

While images are downloading the HMC is mounting a temporary filesystem called /hmcdump and put the images in it. Once the images are downloaded the filesystem /hmcdump is unmounted. You can check the download progression with a loop looking on the /hmcdump filesystem :

hscroot@gaff:~>  while true ; do date; ls -la /hmcdump; sleep 60; done
[..]
drwxr-xr-x  3 root root      4096 2013-12-24 16:26 .
drwxr-xr-x 30 root root      4096 2013-12-19 14:52 ..
-rw-r--r--  1 root hmc  824223312 2013-12-24 16:32 disk3.img
-rw-r--r--  1 root hmc         78 2013-12-24 16:26 hmcnetworkfiles.sum
drwx------  2 root root     16384 2007-12-19 03:24 lost+found
Tue Apr  1 08:10:30 CEST 2014
total 3121248
drwxr-xr-x  3 root root       4096 2013-12-24 16:52 .
drwxr-xr-x 30 root root       4096 2013-12-19 14:52 ..
-rw-r--r--  1 root hmc     2708320 2013-12-24 16:52 bzImage
-rw-r--r--  1 root hmc   808497152 2013-12-24 16:52 disk1.img
-rw-r--r--  1 root hmc  1142493184 2013-12-24 16:45 disk2.img
-rw-r--r--  1 root hmc  1205121024 2013-12-24 16:36 disk3.img
-rw-r--r--  1 root hmc          78 2013-12-24 16:26 hmcnetworkfiles.sum
-rw-r--r--  1 root hmc    34160044 2013-12-24 16:52 initrd.gz
drwx------  2 root root      16384 2007-12-19 03:24 lost+found

Please note that this filesystem is only mounted while the getupgfile command is running and can’t be mounted after the command execution … :

hscroot@gaff:~> mount /hmcdump
mount: only root can mount /dev/sda6 on /hmcdump

Before launching the upgrade save all the data needed for the upgrade to disk, close all HMC events and clear all the filesystems :

  • Save all HMC upgrade data to disk. This command is MANDATORY, it save all the partition profile data, and the user data and the whole HMC configuration, if you forget this command you have to reconfigure the HMC by hand, so be careful with this one :-) :
  • hscroot@gaff:~> saveupgdata -r disk
    
  • Close all HMC events :
  • hscroot@gaff:~> chsvcevent -o closeall
    
  • Remove all temporary HMC files from all filesystems :
  • hscroot@gaff:~> chhmcfs -o f -d 0
    

The images are now downloaded to the HMC, to upgrade the HMC you just have to tell the HMC to boot on its alternate disk and to use the files you’ve just download for the upgrade :

  • To set the alternate disk partition on the HMC as a startup device on the next HMC boot and enable the upgrade on the alternate disk use the chhmc command :
  • hscroot@gaff:~> chhmc -c altdiskboot -s enable --mode upgrade
    
  • Before rebooting check the altdiskboot attribute is set to enable :
  • hscroot@gaff:~> lshmc -r
    ssh=enable,sshprotocol=,remotewebui=enable,xntp=enable,"xntpserver=127.127.1.0,kronosnet1.fr.net.intra,kronosnet2.fr.net.intra,kronosnet3.fr.net.intra",syslogserver=,netboot=disable,altdiskboot=enable,ldap=enable,kerberos=disable,kerberos_default_realm=,kerberos_realm_kdc=,kerberos_clockskew=,kerberos_ticket_lifetime=,kerberos_keyfile_present=,"sol=disabled
    "
    
  • Reboot the HMC and wait :-) :
  • hscroot@gaff:~> hmcshutdown -t now -r
    

Depending on the HMC model and on the version of the HMC the upgrade can takes 10 minutes to 40 minutes, you’ll have to be patient and to cross your finger and pray everything is going well. But don’t worry I never had an issue with this method. Once the HMC is rebooted and upgraded, you can check that the altdiskboot attribute is now set to disable :

hscroot@gaff:~> lshmc -r
ssh=enable,sshprotocol=,remotewebui=enable,xntp=enable,"xntpserver=127.127.1.0,kronosnet1.fr.net.intra,kronosnet2.fr.net.intra,kronosnet3.fr.net.intra",syslogserver=,syslogtcpserver=,syslogtlsserver=,netboot=disable,altdiskboot=disable,ldap=enable,kerberos=disable,kerberos_default_realm=,kerberos_realm_kdc=,kerberos_clockskew=,kerberos_ticket_lifetime=,kpasswd_admin=,trace=,kerberos_keyfile_present=,legacyhmccomm=enable,sol=disabled

Updating

Once the HMC is upgraded you have to update it. Unfortunately updates files (often ISO files) are only available on fix central and not on the public FTP. Get the ISO updates file from fix central and put it on your FTP (once again) server, then use the updhmc command to update the HMC, repeat the operation for each updates, and then reboot the HMC (in the example below I’m using sftp) :

hscroot@gaff:~> updhmc -t s -i -h 192.168.0.99 -u root -f /export/HMC/v7r770/HMC_Update_V7R770_SP1.iso
Password:
iptables: Chain already exists.
ip6tables: Chain already exists.
[..]
The corrective service file was successfully applied. A mandatory reboot is required but was not specified on the command syntax.
hscroot@gaff:~> updhmc -t s -i -h 192.168.0.99 -u root -f /export/HMC/v7r770/HMC_Update_V7R770_SP2.iso
Password:

ip6tables: Chain already exists.
ACCEPT  tcp opt -- in eth3 out *  0.0.0.0/0  -> 0.0.0.0/0  tcp dpt:5989
ACCEPT  udp opt -- in eth3 out *  0.0.0.0/0  -> 0.0.0.0/0  udp dpt:657
[..]
The corrective service file was successfully applied. A mandatory reboot is required but was not specified on the command syntax.
hscroot@gaff:~> hmcshutdown -t now -r

After upgrading and updating the HMC check the version is ok with the lshmc command :

hscroot@gaff:~> lshmc -V
"version= Version: 7
 Release: 7.7.0
 Service Pack: 3
HMC Build level 20131113.1
","base_version=V7R7.7.0
"

Using and configuring the Integrated Management Module

I like to be autonomous and do things on my own. Who had never been stuck on a problem with an HMC and was forced to called an IBM inspector to reboot the HMC or even to insert a CD in the CDRom reader. A few people know this but the HMC is based on an IBM Xserie server (Who said Lenovo ?) and is shipped with an Integrated Management Module allowing you to boot, start, and stop the HMC without the need to have someone in the data-center. Unfortunately this method seems not to be supported by IBM so do it at your own risk.

Use the dedicated port for the Integrated Management Console (the Red port)

hmc_imm_port_good

From the HMC command line using the chhmc command configure the Integrated Management Module IP address :

hscroot@gaff:~> chhmc -c imm -s modify -a 10.10.20.4 -nm 255.255.255.0 -g 10.10.20.254

Restart the Integrated Management Module to commit the changes. The IMM will not be pingable before restart :

hscroot@gaff:~> chhmc -c imm -s restart

The Integrated Management Module is now pingable and you can check its configuration :

hscroot@gaff:~> lshmc -i
ipv4addr=10.10.20.4,networkmask=255.255.255.0,gateway=10.10.20.254,username=USERID,mode=Dedicated

By default the username is USERID and the password is PASSW0RD (with a zero), you can change it to fit your needs :

hscroot@gaff:~> chhmc -c imm -s modify -u immusername --passwd "abc123"

The Integrated Management Module is now configured and can be accessed from the web interface of from SSH :

hmc_imm_login

I will not detail all the actions you can do with the Integrated Management Module but here is a screen showing the Hardware Health of the HMC :

hmc_imm_hardware

One thing you can do for free (without IMM license) is to control the Power of the HMC, choosing to stop/start/restart or reboot. This feature can be very useful when the HMC is stucked :

hmc_imm_actions

If you choose to restart the HMC the Integrated Management Module will warn you before restarting :

hmc_imm_restart

You can access the HMC Integrated Management Module by using the SSH command line :

  • Use the power command to control the power of the HMC :
  • system> help power
    usage:
       power on    [-options]   - power on server
       power off   [-options]   - power off server
       power cycle [-options]   - power off, then on
       power state              - display power state
       power -rp [alwayson|alwaysoff|restore]   - host power restore policy
    options:
       -s                       - shut down OS first
       -every day               - daily or weekly on,off or cycle commands
    [Sun|Mon|Tue|Wed|Thu|Fri|Sat|Day|clear]
       -t   time                - time (hh:mm)
    additional options for on.
       -d  date                 - date (mm/dd/yyyy)
       -clear                   - clear on date
    
  • Here is an example to restart the HMC :
  • system> power cycle -s
    ok
    
  • Checking the power state :
  • system> power state
    power on
    State:Booting OS or in unsupported OS
    

The Integrated Management Module is a licensed product and unfortunately IBM does not support the Integrated Management Module on the HMC.It seems that the IMM license can’t be acquired for the HMC. I have checked on the trial licenses page and the HMC Hardware does not even exists when you have to choose the Hardware model for the trial license. This a shame because the licensed IMM allows to remote control the HMC, and to manage Virtual CDrom ….. useful for migration. So if an IBMer is reading this and have an explanation about this feel free to tell me what I’ve missed in the comments :

hmc_imm_remote_control

I hope this post will let you manage your HMC alone and to be autonomous :-)

4 thoughts on “Hardware Management Console : Be autonomous with upgrade, update, and use the Integrated Management Module

  1. Thanks for the information on how to do the HMC upgrade remotely.

    I always knew it was possible with NFS and a boot disk, but this is exactly what I have been looking for for years.

  2. Benoit, I have to thank you again. Very good topic. I just followed “Upgrading” and “Updating” chapters to upgrade an HMC from V7R7.7.0 to V7R7.9.0 SP2 and it worked well!
    Juste, too bad that this way is not verbose enough…

  3. Good information! Your posts are always very detailed and helpful.

    As far as remote access on the IMM, since all HMC’s have equivalent xSeries/Lenovo machine types (IE a 7042-CR9 is equivalent to a 5463 Model AC1), couldn’t you then get the remote access license for that machine type (which is 90Y3901 btw)?

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>