PowerVM Shared Ethernet Adapter simplification : Get rid of Control Channel Adapter

Since I started working on Virtual I/O Servers and PowerVM I’ve created many Shared Ethernet Adapters in all modes (standard, failover, or sharing). I’ve learned one important lesson “be careful when creating a Shared Ethernet Adapter“. A single mistake can cause a network outage and I’m sure that you’ve already seen someone in your team creating an ARP storm by mismatching control channel adapter or by adding a vlan that is already added on a Virtual Ethernet Adapter. Because of this kind of errors I know some customers who are trying to avoid the configuration of Shared Ethernet Adapter in failover or sharing mode to avoid any network outage. With the new version of Virtual I/O Server (starting from 2.2.2.2) network loop and ARP storms are -in most cases- detected and stopped at the Virtual I/O Server level or at the firwmare level. I always check two or three times my configuration before creating a Shared Ethernet Adapter. All these errors come -most of the time- from a lack of rigor and are in -almost- all cases due to the system administrator. With the new version of PowerVM you can now create all Shared Ethernet Adapters without specifying any control channel adapter (The Hardware Management Console and the Virtual I/O Server will do it for you). A new discovery protocol implemented on Virtual I/O Server is matching Shared Ethernet Adapters between them and will take care of creating the Control Channel vlan for you (this one will not be visible on the Virtual I/O Server). Much simpler = less errors. Here is a practical how-to :

How does it work ?

A new discovery protocol called SEA HA match partners between them by using a dedicated vlan (not configurable by the user). Here are a few things to know :

  • Multiple Shared Ethernet Adapters can share the vlan 4095 for their Control Channel link.
  • The vlan 4095 is created per Virtual Switch for this Control Channel link.
  • As always only two Shared Ethernet Adapters can be partners, the Hardware Management Console is ensuring that priority 1 and 2 are used (I’ve seen some customers using priority 3 and 4, do don’t this.)
  • Both failover and sharing mode can be used.
  • Shared Ethernet Adapters with a dedicated Control Channel Adapter, can be migrated to this configuration with a network outage, put the SEA in defined state before :

Here is any example of this configuration on a Shared Ethernet Adapter in Sharing Mode :

sea_no_ctl_chan_fig1

On the image below you can follow the steps of this new discovery protocol :

  • 1/No dedicated Control Channel Adapter in Shared Ethernet Adapter Creation. The discovery protocol will be used if you are creating a SEA in failover or sharing mode without specifying the ctl_chan attribute.
  • 2/Partners are identified by their PVID, both partners must have the same PVID.
  • 3/This PVID has to be uniq per SEA pairs.
  • 4/Additional vlans ID are compared : partners with not matching additional vlans IDs are still considered as partners if their PVID match.
  • 5/Shared Ethernet Adapter with matching additional vlan IDs and not matching PVID are not considered as partners.
  • 6/If partners are not matching their additional vlan IDs they are still considered partners but an error is logged in the errlog.

sea_no_ctl_chan_fig2

Prerequisites

Shared Ethernet Adapter without the need of a Control Channel Adapter can’t be created on all systems. At the time of writing this post only a few models of POWER7 machines (maybe POWER8) have the firmware implementing the feature. You have to check that the firmware of your machine is at least a XX780_XXX release. Be careful to check the release note of the firmware, some of the 780’s firmwares does not permit the creation of SEA without Control Channel Adapter (especially 9117-MMB) (here is an example on this page : link here, the release note says : “Support was added to the Management Console command line to allow configuring a shared control channel for multiple pairs of Shared Ethernet Adapters (SEAs). This simplifies the control channel configuration to reduce network errors when the SEAs are in fail-over mode. This feature is not supported on IBM Power 770 (9117-MMB) and IBM Power 780 (9179-MHB) systems.”). Because the Hardware Management Console is using the vlan 4095 to create the Control Channel link between Shared Ethernet Adapters it has to be aware of this feature and must ensure that the vlan 4095 is not usable or configurable by the administrator. The HMC v7R7.8.0 is aware of this that’s why the HMC must be updated at least to this level.

  • Check your machine firmware, in my case I’m working on a 9117-MMD (P7+770) with the lastest firmware available (at the time of writing this post) :
# lsattr -El sys0 -a modelname
modelname IBM,9117-MMD Machine name False
# lsmcode -A
sys0!system:AM780_056 (t) AM780_056 (p) AM780_056 (t)
  • These prerequisites can be check directly from the Hardware Management Console :
hscroot@myhmc:~> lslic -t sys -m 9117-MMD-65XXXX
lic_type=Managed System,management_status=Enabled,disabled_reason=,activated_level=56,activated_spname=FW780.10,installed_level=56,installed_spname=FW780.10,accepted_level=56,accepted_spname=FW780.10,ecnumber=01AM780,mtms=9117-MMD*658B2AD,deferred_level=None,deferred_spname=FW780.10,platform_ipl_level=56,platform_ipl_spname=FW780.10,curr_level_primary=56,curr_spname_primary=FW780.10,curr_ecnumber_primary=01AM780,curr_power_on_side_primary=temp,pend_power_on_side_primary=temp,temp_level_primary=56,temp_spname_primary=FW780.10,temp_ecnumber_primary=01AM780,perm_level_primary=56,perm_spname_primary=FW780.10,perm_ecnumber_primary=01AM780,update_control_primary=HMC,curr_level_secondary=56,curr_spname_secondary=FW780.10,curr_ecnumber_secondary=01AM780,curr_power_on_side_secondary=temp,pend_power_on_side_secondary=temp,temp_level_secondary=56,temp_spname_secondary=FW780.10,temp_ecnumber_secondary=01AM780,perm_level_secondary=56,perm_spname_secondary=FW780.10,perm_ecnumber_secondary=01AM780,update_control_secondary=HMC
  • Check your Hardware Management Console release is at least V7R7.8.0 (in my case my HMC is at the latest level available at the time of writing this post) :
hscroot@myhmc:~> lshmc -V
"version= Version: 7
 Release: 7.9.0
 Service Pack: 0
HMC Build level 20140409.1
MH01406: Required fix for HMC V7R7.9.0 (04-16-2014)
","base_version=V7R7.9.0
"

Shared Ethernet Adapter creation in sharing mode without control channel

The creation is simple, just identify your Real Adapter and your Virtual Adapter(s). Check on both Virtual I/O Server that PVID used on Virtual Adapters are the same and check priority are ok (use priority 1 on PRIMARY Virtual I/O Server and priority 2 on BACKUP Virtual I/O Server). I’m creating in this post a Shared Ethernet Adapter in Sharing Mode, steps are the same if you are creating a Shared Ethernet Adapter in auto mode.

  • Identify the Real Adapter (in my case an LACP 802.3ad adapter) :
  • padmin@vios1$ lsdev -dev ent17
    name             status      description
    ent17            Available   EtherChannel / IEEE 802.3ad Link Aggregation
    padmin@vios2$ lsdev -dev ent17
    name             status      description
    ent17            Available   EtherChannel / IEEE 802.3ad Link Aggregation
    
  • Identify the Virtual Adapters : priority 1 on PRIMARY Virtual I/O Server and priority 2 on BACKUP Virtual I/O Server (my advice is to check that additional vlan IDs are ok too) :
  • padmin@vios1$ entstat -all ent13 | grep -iE "Priority|Port VLAN ID"
      Priority: 1  Active: False
    Port VLAN ID:    15
    padmin@vios1$ entstat -all ent14 | grep -iE "Priority|Port VLAN ID"
      Priority: 1  Active: False
    Port VLAN ID:    16
    padmin@vios2$ entstat -all ent13 | grep -iE "Priority|Port VLAN ID"
      Priority: 2  Active: True
    Port VLAN ID:    15
    padmin@vios2$ entstat -all ent14 | grep -iE "Priority|Port VLAN ID"
      Priority: 2  Active: True
    Port VLAN ID:    16
    
  • Create the Shared Ethernet Adapter without specifying the ctl_chan attribute :
  • padmin@vios1$ mkvdev -sea ent17 -vadapter ent13 ent14 -default ent13 -defaultid 15 -attr ha_mode=sharing largesend=1 large_receive=yes
    ent18 Available
    padmin@vios2$ mkvdev -sea ent17 -vadapter ent13 ent14 -default ent13 -defaultid 15 -attr ha_mode=sharing largesend=1 large_receive=yes
    ent18 Available
    
  • Shared Ethernet Adapter are created! You can check that the ctl_chan attribute is empty when checking the device :
  • padmin@svios1$ lsdev -dev ent18 -attr
    attribute     value       description                                                        user_settable
    
    accounting    disabled    Enable per-client accounting of network statistics                 True
    adapter_reset yes         Reset real adapter on HA takeover                                  True
    ctl_chan                  Control Channel adapter for SEA failover                           True
    gvrp          no          Enable GARP VLAN Registration Protocol (GVRP)                      True
    ha_mode       sharing     High Availability Mode                                             True
    [..]
    pvid          15          PVID to use for the SEA device                                     True
    pvid_adapter  ent13       Default virtual adapter to use for non-VLAN-tagged packets         True
    qos_mode      disabled    N/A                                                                True
    queue_size    8192        Queue size for a SEA thread                                        True
    real_adapter  ent17       Physical adapter associated with the SEA                           True
    send_RARP     yes         Transmit Reverse ARP after HA takeover                             True
    thread        1           Thread mode enabled (1) or disabled (0)                            True
    virt_adapters ent13,ent14 List of virtual adapters associated with the SEA (comma separated) True
    
  • By using the entstat command you can check that the Control Channel exists and is using the PVID 4095 (same result on second Virtual I/O Server) :
  • padmin@vios1$ entstat -all ent18 | grep -i "Control Channel PVID"
        Control Channel PVID: 4095
    
  • Looking at the entstat output SEA are partners (one PRIMARY_SH and one BACKUP_SH :
padmin@vios1$ entstat -all ent18 | grep -i state
    State: PRIMARY_SH
padmin@vios2$  entstat -all ent18 | grep -i state
    State: BACKUP_SH

Verbose and intelligent errlog

While configuring Shared Ethernet Adapter in this mode the errlog can give you a lot of informations about your configuration. For instance if additional vlan IDs does not match betweens Virtual Adapters of a Shared Ethernet Adapter you’ll be warned by an error in the errlog. Here are a few examples :

  • Additional vlan IDs does not match between Virtual Adapters :
padmin@vios1$ errlog | more
IDENTIFIER TIMESTAMP  T C RESOURCE_NAME  DESCRIPTION
A759776F   0506205214 I H ent18          SEA HA PARTNERS VLANS MISMATCH
  • Looking on a detailed output you can get the missing vlan id :
padmin@vios1$ 
---------------------------------------------------------------------------
LABEL:          VIOS_SEAHA_DSCV_VLA
IDENTIFIER:     A759776F
Date/Time:       Tue May  6 20:52:59 2014
Sequence Number: 704
Machine Id:      00XXXXXXXX00
Node Id:         vios1
Class:           H
Type:            INFO
WPAR:            Global
Resource Name:   ent18
Resource Class:  adapter
Resource Type:   sea
Location:

Description
SEA HA PARTNERS VLANS MISMATCH

Probable Causes
VLAN MISCONFIGURATION

Failure Causes
VLAN MISCONFIGURATION

        Recommended Actions
        NONE

Detail Data
ERNUM
0000 001A
ABSTRACT
Discovered HA partner with unmatched VLANs
AREA
VLAN misconfiguration
BUILD INFO
BLD: 1309 30-10:08:58 y2013_40A0
LOCATION
Filename:sea_ha.c Function:seaha_process_dscv_init Line:6156
DATA
VLAN = 0x03E9
  • The last line is the value of the missing vlan in hexadecimal (0x03E9, 1001 converted in decimal). We can manually check that this vlan is missing on vios1 :
# echo "ibase=16; 03E9" | bc
1001
padmin@vios1$ entstat -all ent18 | grep -i "VLAN Tag IDs:"
VLAN Tag IDs:  1659
VLAN Tag IDs:  1682
VLAN Tag IDs:  1682
padmin@vios2$ entstat -all ent18 | grep -i "VLAN Tag IDs:"
VLAN Tag IDs:  1659
VLAN Tag IDs:  1001  1682
VLAN Tag IDs:  1001  1682
  • A loss of communication between SEA will also be logged in the errlog :
padmin@vios1$ errlog | more
IDENTIFIER TIMESTAMP  T C RESOURCE_NAME  DESCRIPTION
B8C78C08   0502231214 I H ent18          SEA HA PARTNER LOST
padmin@vios1$ errlog -ls | more
Location:

Description
SEA HA PARTNER LOST

Probable Causes
SEA HA PARTNER DOWN

Failure Causes
SEA HA PARTNER DOWN

        Recommended Actions
        INITIATE PARTNER DISCOVERY

Detail Data
ERNUM
0000 0019
ABSTRACT
Initiating partner discovery due to lost partner
AREA
SEA HA discovery partner lost
BUILD INFO
BLD: 1309 30-10:08:58 y2013_40A0
LOCATION
Filename:sea_ha.c Function:seaha_dscv_ka_rcv_timeout Line:2977
DATA
Partner MAC: 0x1A:0xC4:0xFD:0x72:0x9B:0x0F
  • Be careful looking at the errlog, a SEA in sharing mode will “become primary” even if it is the “backup” SEA (you have to look with errlog -ls command for the details) :
padmin@vios$ errlog | grep BECOME
E48A73A4   0506205214 I H ent18          BECOME PRIMARY
padmin@vios2$ errlog | grep BECOME
1FE2DD91   0506205314 I H ent18          BECOME PRIMARY
padmin@vios1$ errlog -ls | more
LABEL:          VIOS_SEAHA_PRIMARY
IDENTIFIER:     E48A73A4
[..]
Description
BECOME PRIMARY
[..]
padmin@vios2$ errlog -ls | more
LABEL:          VIOS_SEAHA_BACKUP
IDENTIFIER:     1FE2DD91
[..]
Description
BECOME PRIMARY
[..]
ABSTRACT
Transition from INIT to BACKUP
[..]
seahap->state= 0x00000003
Become the Backup SEA

Removing the control channel adapter from an existing Shared Ethernet Adapter

A “classic” Shared Ethernet Adapter can be modified to be usable without the need of a dedicated Control Channel Adapter. This modification require a network outage and the Shared Ethernet Adapter needs to be in defined state. I DO NOT LIKE to do administration as root on Virtual I/O Servers but I’ll do it here because of the use of the mkdev command :

  • On both Virtual I/O Servers put the Shared Ethernet Adapter in defined state :
padmin@vios1$ oem_setup_env
root@vios1# rmdev -l ent18
ent18 Defined
padmin@vios2$ oem_setup_env
root@vios2# rmdev -l ent18
ent18 Defined
  • On both Virtual I/O Servers remove the dedicated Control Channel Adapter for both Shared Ethernet Adapters :
root@vios1# lsattr -El ent18 -a ctl_chan
ctl_chan ent12 Control Channel adapter for SEA failover True
root@vios1# chdev -l ent18 -a ctl_chan=""
ent18 changed
root@vios1# lsattr -El ent18 -a ctl_chan
ctl_chan  Control Channel adapter for SEA failover True
root@vios2# lsattr -El ent18 -a ctl_chan
ctl_chan ent12 Control Channel adapter for SEA failover True
root@vios2# chdev -l ent18 -a ctl_chan=""
ent18 changed
root@vios2# lsattr -El ent18 -a ctl_chan
ctl_chan  Control Channel adapter for SEA failover True
  • Put each Shared Ethernet Adapter in available state by using the mkdev command :
root@vios1# mkdev -l ent18
ent18 Available
root@vios2# mkdev -l ent18
ent18 Available
  • Verify that the Shared Ethernet Adapter is now using vlan 4095 as Control Channel PVID :
padmin@vios1$ entstat -all ent18 | grep -i "Control Channel PVID"
    Control Channel PVID: 4095
padmin@vios2$ entstat -all ent18 | grep -i "Control Channel PVID"
    Control Channel PVID: 4095

The first step to a global PowerVM simplification

Be aware that this simplification is one of the first step of a much larger project. With the latest version of the HMC v8R80.1 a lot of new features will be available (June 2014). I can’t wait to test the “single point of management” for Virtual I/O Servers. Anyway, creating a Shared Ethernet Adapter is easier than before. Use this method to avoid human errors and misconfiguration of your Shared Ethernet Adapters. As always I hope this post will help you to understand this simplification. :-)

Hardware Management Console : Be autonomous with upgrade, update, and use the Integrated Management Module

I’m sure that like me you do not have a physical access to your Hardware Management Consoles, or even if you have this access, some of your HMC are so far away from your working site (even in a foreign country) that you can’t afford to physically move to this place to update it. Even worse if -like me- you are working in a big place (who says too big ?) this job is often performed by IBM Inspectors and you do not have to worry about your Hardware Management Consoles and just have to ask IBM guys for anything about HMC. For some reasons I had to update an old Hardware Management Console from v7r7.3.0 to v7r7.7.0 SP3. Everybody is confused about the differences between updating an HMC and upgrading an HMC. I know really good bloggers : Anthony English and Rob McNelly have already post about this particular subject but I have to write this post as a reminder and to clarify some points which are not tell in Anthony’s and Rob’s posts :

To finish this post I’ll talk about a feature nobody is using : the HMC is coming with an Integrated Management Module, this one allows you to have more control and to be autonomous with you HMC.

The difference between updating, upgrading and migrating

There is a lot of confusion when people are trying “update” their HMC. When do I have to update using the updhmc command, when do I have to upgrade using saveupgdata, getupdfiles and chhmc command, and finally when do I have to migrate using HMC Recovery CD/DVD ? All of this three operations are not well described by IBM. Here is what I’m doing in for each case, and this is the result of my own experience (don’t take this as an official document). Here is a little reminder it can be useful for a lot of people : an HMC version number looks like this : v7r7.7.0 SP3. v7 is the VERSION. 7.7.0 is the RELEASE and SP3 is the SERVICE PACK.

  • Updating : You have to update your HMC if your are applying a service pack, or a corrective fix update on the HMC, this operation can only be performed by the updhmc command. Use this method if fix central gives you an iso named “HMC_Update_*.iso” or a zip files named “MHxxxxx.zip”. These fixes can be applied to a minor version of the HMC.
  • Upgrading : You have to upgrade your HMC if your are moving from one minor version to another (from one release to another), for instance if your are moving from v7r7.7.0 to v7r7.8.0, this operation can be made by using HMC Recovery DVD (fix central will gives you two isos named “HMC_Recovery_*.iso”), or by using the images provided for a network upgrade (I’ll explain this in this post).
  • Migrating : You have to migrate your HMC by using HMC Recovery DVD when your are moving from one major version to another. For example when your are moving from an HMC v6 to and HMC v7 (for instance from any v6 version to v7r7.8.0). In this case you have no other choice than burning DVDs and moving in front of the HMC to perform the operation by yourself.

Upgrading

You can upgrade your HMC from its local storage by using the network images provided by IBM on a public FTP server, once connected to the public FTP server get the version you want to upgrade to and download all the files, bzImage and initrd.gz included :

# ftp://ftp.software.ibm.com/software/server/hmc/network/v7770/
# ls
FTP Listing of /software/server/hmc/network/v7770/ at ftp.software.ibm.com
[..]
Feb 26 2013 00:00      2708320 bzImage
Feb 26 2013 00:00    808497152 disk1.img
Feb 26 2013 00:00   1142493184 disk2.img
Feb 26 2013 00:00   1205121024 disk3.img
Feb 26 2013 00:00           78 hmcnetworkfiles.sum
Feb 26 2013 00:00     34160044 initrd.gz
# mget *.*

Put all the files on a server where you have an FTP server running (the HMC getupgfiles is using FTP to get the files) and download all the files with the getupgfiles command directly form the HMC (if your HMC has a direct access to the internet you can specify it the command):

hscroot@gaff:~> getupgfiles -h 192.168.0.99 -u root -d /export/HMC/network_ugrade/v7770
Enter the current password for user root:

While images are downloading the HMC is mounting a temporary filesystem called /hmcdump and put the images in it. Once the images are downloaded the filesystem /hmcdump is unmounted. You can check the download progression with a loop looking on the /hmcdump filesystem :

hscroot@gaff:~>  while true ; do date; ls -la /hmcdump; sleep 60; done
[..]
drwxr-xr-x  3 root root      4096 2013-12-24 16:26 .
drwxr-xr-x 30 root root      4096 2013-12-19 14:52 ..
-rw-r--r--  1 root hmc  824223312 2013-12-24 16:32 disk3.img
-rw-r--r--  1 root hmc         78 2013-12-24 16:26 hmcnetworkfiles.sum
drwx------  2 root root     16384 2007-12-19 03:24 lost+found
Tue Apr  1 08:10:30 CEST 2014
total 3121248
drwxr-xr-x  3 root root       4096 2013-12-24 16:52 .
drwxr-xr-x 30 root root       4096 2013-12-19 14:52 ..
-rw-r--r--  1 root hmc     2708320 2013-12-24 16:52 bzImage
-rw-r--r--  1 root hmc   808497152 2013-12-24 16:52 disk1.img
-rw-r--r--  1 root hmc  1142493184 2013-12-24 16:45 disk2.img
-rw-r--r--  1 root hmc  1205121024 2013-12-24 16:36 disk3.img
-rw-r--r--  1 root hmc          78 2013-12-24 16:26 hmcnetworkfiles.sum
-rw-r--r--  1 root hmc    34160044 2013-12-24 16:52 initrd.gz
drwx------  2 root root      16384 2007-12-19 03:24 lost+found

Please note that this filesystem is only mounted while the getupgfile command is running and can’t be mounted after the command execution … :

hscroot@gaff:~> mount /hmcdump
mount: only root can mount /dev/sda6 on /hmcdump

Before launching the upgrade save all the data needed for the upgrade to disk, close all HMC events and clear all the filesystems :

  • Save all HMC upgrade data to disk. This command is MANDATORY, it save all the partition profile data, and the user data and the whole HMC configuration, if you forget this command you have to reconfigure the HMC by hand, so be careful with this one :-) :
  • hscroot@gaff:~> saveupgdata -r disk
    
  • Close all HMC events :
  • hscroot@gaff:~> chsvcevent -o closeall
    
  • Remove all temporary HMC files from all filesystems :
  • hscroot@gaff:~> chhmcfs -o f -d 0
    

The images are now downloaded to the HMC, to upgrade the HMC you just have to tell the HMC to boot on its alternate disk and to use the files you’ve just download for the upgrade :

  • To set the alternate disk partition on the HMC as a startup device on the next HMC boot and enable the upgrade on the alternate disk use the chhmc command :
  • hscroot@gaff:~> chhmc -c altdiskboot -s enable --mode upgrade
    
  • Before rebooting check the altdiskboot attribute is set to enable :
  • hscroot@gaff:~> lshmc -r
    ssh=enable,sshprotocol=,remotewebui=enable,xntp=enable,"xntpserver=127.127.1.0,kronosnet1.fr.net.intra,kronosnet2.fr.net.intra,kronosnet3.fr.net.intra",syslogserver=,netboot=disable,altdiskboot=enable,ldap=enable,kerberos=disable,kerberos_default_realm=,kerberos_realm_kdc=,kerberos_clockskew=,kerberos_ticket_lifetime=,kerberos_keyfile_present=,"sol=disabled
    "
    
  • Reboot the HMC and wait :-) :
  • hscroot@gaff:~> hmcshutdown -t now -r
    

Depending on the HMC model and on the version of the HMC the upgrade can takes 10 minutes to 40 minutes, you’ll have to be patient and to cross your finger and pray everything is going well. But don’t worry I never had an issue with this method. Once the HMC is rebooted and upgraded, you can check that the altdiskboot attribute is now set to disable :

hscroot@gaff:~> lshmc -r
ssh=enable,sshprotocol=,remotewebui=enable,xntp=enable,"xntpserver=127.127.1.0,kronosnet1.fr.net.intra,kronosnet2.fr.net.intra,kronosnet3.fr.net.intra",syslogserver=,syslogtcpserver=,syslogtlsserver=,netboot=disable,altdiskboot=disable,ldap=enable,kerberos=disable,kerberos_default_realm=,kerberos_realm_kdc=,kerberos_clockskew=,kerberos_ticket_lifetime=,kpasswd_admin=,trace=,kerberos_keyfile_present=,legacyhmccomm=enable,sol=disabled

Updating

Once the HMC is upgraded you have to update it. Unfortunately updates files (often ISO files) are only available on fix central and not on the public FTP. Get the ISO updates file from fix central and put it on your FTP (once again) server, then use the updhmc command to update the HMC, repeat the operation for each updates, and then reboot the HMC (in the example below I’m using sftp) :

hscroot@gaff:~> updhmc -t s -i -h 192.168.0.99 -u root -f /export/HMC/v7r770/HMC_Update_V7R770_SP1.iso
Password:
iptables: Chain already exists.
ip6tables: Chain already exists.
[..]
The corrective service file was successfully applied. A mandatory reboot is required but was not specified on the command syntax.
hscroot@gaff:~> updhmc -t s -i -h 192.168.0.99 -u root -f /export/HMC/v7r770/HMC_Update_V7R770_SP2.iso
Password:

ip6tables: Chain already exists.
ACCEPT  tcp opt -- in eth3 out *  0.0.0.0/0  -> 0.0.0.0/0  tcp dpt:5989
ACCEPT  udp opt -- in eth3 out *  0.0.0.0/0  -> 0.0.0.0/0  udp dpt:657
[..]
The corrective service file was successfully applied. A mandatory reboot is required but was not specified on the command syntax.
hscroot@gaff:~> hmcshutdown -t now -r

After upgrading and updating the HMC check the version is ok with the lshmc command :

hscroot@gaff:~> lshmc -V
"version= Version: 7
 Release: 7.7.0
 Service Pack: 3
HMC Build level 20131113.1
","base_version=V7R7.7.0
"

Using and configuring the Integrated Management Module

I like to be autonomous and do things on my own. Who had never been stuck on a problem with an HMC and was forced to called an IBM inspector to reboot the HMC or even to insert a CD in the CDRom reader. A few people know this but the HMC is based on an IBM Xserie server (Who said Lenovo ?) and is shipped with an Integrated Management Module allowing you to boot, start, and stop the HMC without the need to have someone in the data-center. Unfortunately this method seems not to be supported by IBM so do it at your own risk.

Use the dedicated port for the Integrated Management Console (the Red port)

hmc_imm_port_good

From the HMC command line using the chhmc command configure the Integrated Management Module IP address :

hscroot@gaff:~> chhmc -c imm -s modify -a 10.10.20.4 -nm 255.255.255.0 -g 10.10.20.254

Restart the Integrated Management Module to commit the changes. The IMM will not be pingable before restart :

hscroot@gaff:~> chhmc -c imm -s restart

The Integrated Management Module is now pingable and you can check its configuration :

hscroot@gaff:~> lshmc -i
ipv4addr=10.10.20.4,networkmask=255.255.255.0,gateway=10.10.20.254,username=USERID,mode=Dedicated

By default the username is USERID and the password is PASSW0RD (with a zero), you can change it to fit your needs :

hscroot@gaff:~> chhmc -c imm -s modify -u immusername --passwd "abc123"

The Integrated Management Module is now configured and can be accessed from the web interface of from SSH :

hmc_imm_login

I will not detail all the actions you can do with the Integrated Management Module but here is a screen showing the Hardware Health of the HMC :

hmc_imm_hardware

One thing you can do for free (without IMM license) is to control the Power of the HMC, choosing to stop/start/restart or reboot. This feature can be very useful when the HMC is stucked :

hmc_imm_actions

If you choose to restart the HMC the Integrated Management Module will warn you before restarting :

hmc_imm_restart

You can access the HMC Integrated Management Module by using the SSH command line :

  • Use the power command to control the power of the HMC :
  • system> help power
    usage:
       power on    [-options]   - power on server
       power off   [-options]   - power off server
       power cycle [-options]   - power off, then on
       power state              - display power state
       power -rp [alwayson|alwaysoff|restore]   - host power restore policy
    options:
       -s                       - shut down OS first
       -every day               - daily or weekly on,off or cycle commands
    [Sun|Mon|Tue|Wed|Thu|Fri|Sat|Day|clear]
       -t   time                - time (hh:mm)
    additional options for on.
       -d  date                 - date (mm/dd/yyyy)
       -clear                   - clear on date
    
  • Here is an example to restart the HMC :
  • system> power cycle -s
    ok
    
  • Checking the power state :
  • system> power state
    power on
    State:Booting OS or in unsupported OS
    

The Integrated Management Module is a licensed product and unfortunately IBM does not support the Integrated Management Module on the HMC.It seems that the IMM license can’t be acquired for the HMC. I have checked on the trial licenses page and the HMC Hardware does not even exists when you have to choose the Hardware model for the trial license. This a shame because the licensed IMM allows to remote control the HMC, and to manage Virtual CDrom ….. useful for migration. So if an IBMer is reading this and have an explanation about this feel free to tell me what I’ve missed in the comments :

hmc_imm_remote_control

I hope this post will let you manage your HMC alone and to be autonomous :-)