Building my AIX/Power Home Lab : HMC/iSCSI/Power Saving and other tricks

I’m living in fear ! A few months ago I decided to quit my job for some reasons I can’t tell you in this blog. As many of you already know, you know exactly what you quit when you are leaving a place, but do not exactly know what you will get in the new one. Unfortunately this is the way France is doing today about jobs in general (just ask you why we do not have any Champion for Power Systems in France ?). Anyway this blog isn’t about politics but about technical stuffs. So I finally admitted that my professional path will not be a straight forward line, and with this comes the feeling that I needed to work and experiment some stuffs on my own without being at work. Here are some benefits I have to own my Power Server at home :

  • I’m the only developer of nmon2graphite. I really need a test platform to add new features and maintain nmon2graphite.
  • I can experiment things on my own without the fear to broke some production hosts, and to finally be fired. Example : I deliberately create a Shared Ethernet Adapter with wrong parameters to create a network loop (with some wireshark trace) to really understand why so many people are doing this mistake and how it really works. (I’m planning to post about this.)
  • I do not have to wait to experiment things, I’m the master/keeper in the house, no more firewall, dns problems, I can do what I want !
  • I can broke things and do it again and again. Need an IVM ? No problem less than one hour to build an IVM. Need an HMC … same thing.
  • It’s cool when some geek friends are coming home and I can say “Hey buddy, wanna play on my Power server ?”

I know this is a lost cause by I give it a try here. If some IBMers are reading this blog. My P520 Express is running a PowerVM standard edition, if someone can provides me a COD code for an enterprise edition it’ll be very cool (Am I dreaming ? Is the world really managed by money ?) . Remember that this blog is here to help the Power community, there is no business behind this blog. It’s free. I’m blogging in my free time, it’s a passion.

mypower1

It’s a long way to the promise land.

Finding a Power Server at a cheap price is not easy. My first tough was to ask people who already buy one for their own use how they did it. I decided to ask Brian Smith who owns a few Power Systems : How did you get your servers ? He kindly answers to my question and told me to check on eBay or in governmental auctions. So I started to check eBay in the beginning of May 2013. In Europe prices were too expansives for my budget (no server cheaper than 5000€) ; so I decided to check in US if prices were cheaper. For price convenience my advice is to check for a Power 520 express or a Power710 express if you can afford to spend more money. Before buying anything always :

  • Ask for the Serial Number to check activated options, go on the IBM POD for this http://www-912.ibm.com/pod/pod.
  • Always ask for the PowerVM Edition. In my case my need is PowerVM Standard or Enterprise. Go on this page to check how to identify your PowerVM edition : http://www-01.ibm.com/support/docview.wss?uid=isg3T1010860
  • Always ask for adapters and disks shipped with the server. A lot of servers are sold without any adapter.
  • Always ask for power supply with the server. A lot of servers are sold without any power.
  • If you are planning to install dual Virtual I/O Server check if a split backplane is installed with the system (in my case I have no split backplane, and I’ll tell you how to setup dual Virtual I/O Server in this case.)
  • Check the price ! Don’t get ripped off !

After three months asking questions and checking eBay everyday. I finally found something interesting. The NeuComp Company (an IBM Business Partner) is selling refurbished Power Server on eBay. I wanted to thanks Katie K. (if you are reading this Katie tell me if I can give you real name :-)) for her kindness and for answering all my questions. If you are buying from a foreign country don’t make the same mistake as me and be aware of the custom fees (in my case around 15% of the total prices), seriously DHL guys are ruthless …

poweropened

What about the Hardware Management Console ?

If you are planning to have a PowerVM Standard/Enterprise edition server it’s easier to manage it through an Hardware Management Console : two solutions :

  • Find an old hardware management console, on eBay it’s not to expansive and affordable.
  • Like Scott Vetter point in this post : The Power Systems Hardware Management Console as a virtual appliance ? it is possible to virtualize an Hardware Management Console, or an SDMC. Like him I cannot tell you here how to do it but it’s doable, and it’s not my case :-) (Don’t ask me for this one). Anyway if you want an SDMC you are totally in legality if you download and install the SDMC ova file and deploy it on a KVM or a VMware infrastructure.

Protect your ear. Check electricity bill : enable Power Save mode.

Do not forget to enable power saver mode if you are owning a Power6 or Power7 server. First : fans speeds will be decreased and the server will produce less noise. Second : you’ll notice a huge difference at the end of the month on your electricity bill if you are running the server 24 hours a day ! Notice that when the power saver mode on a managed system is enabled, the processor voltage and clock frequency are lowered to reduce the power consumption of the processors in the managed system.

  • Enabling Power Saver mode (and check with pmcycles command on a partition) :
  • # chpwrmgmt -m power520e-8203-EA4 -r sys -o enable
    root@priss.lab.chmod666.org [/root] # pmcycles -Mm
    CPU 0 runs at 3618 MHz
    CPU 1 runs at 3617 MHz
    CPU 2 runs at 3619 MHz
    CPU 3 runs at 3619 MHz
    
  • Disabling Power Save mode :
  • # chpwrmgmt -m power520e-8203-EA4 -r sys -o disable
    root@priss.lab.chmod666.org [/root] # pmcycles -Mm
    CPU 0 runs at 4201 MHz
    CPU 1 runs at 4201 MHz
    CPU 2 runs at 4202 MHz
    CPU 3 runs at 4202 MHz
    

As you can see on the output above the processor speed was decreased from 4.2GHz to 3.6GHz. I do not have any recorder but I can swear that fans are decreasing their speed at the moment the command was typed on the Hardware Management Console. Power Saving mode is not a joke it really works.

Updating/Installation

Firmware update

The server was not shipped with the latest firmware level. If you are working in a big enterprise like me this kind of operations are performed by IBM Inspectors, so the last time I updated a firmware was a couple of months ago and I did it through the IBM Systems Director, this is not the kind of operation I’m used to do, so here is a quick reminder on how to do it :

  • First of all copy all the files (xml and sdd files included) you have downloaded with the new firmware on the Hardware Management Console :
  • hscroot@gaff:~> ls /home/hscroot/fw
    01EL350_149_038.dd.xml	01EL350_149_038.pd.sdd	    01EL350_149_038.rpm
    01EL350_149_038.html	01EL350_149_038.readme.txt  01EL350_149_038.xml
    
  • You can check with the lslic command if there are “flashable” firmware in a specific directory :
  • hscroot@gaff:~> lslic -m p520express-8203-E4A-SN0666000 -r mountpoint -d /home/hscroot/fw/ -F retrievable_release concurrent_retrievable_level disruptive_retrievable_level
    None 149 149
    
  • Flash the firmware with the updlic command (note that a copy of the old microcode is backuped in the /var filesystem):
  • hscroot@gaff:~> updlic -o a -t sys -l latest -m p520express-8203-E4A-SN0666000  -r mountpoint -d /home/hscroot/fw -v
    Current profile data backup files have been copied: 
    8203-E4A*0666000: /var/hsc/profiles/0666000/backupFile_FirmwareUpdate01EL350, /var/hsc/profiles/0666000/directory/backupFile_FirmwareUpdate01EL350.dir
    
    8203-E4A*0666000: Retrieving updates.
    8203-E4A*0666000 Managed System Primary: Retrieving updates.
    8203-E4A*0666000: Installing updates.
    8203-E4A*0666000 Managed System Primary: Preparing for install.
    8203-E4A*0666000: Installing updates.
    8203-E4A*0666000 Managed System Primary: Installing updates.
    8203-E4A*0666000 Managed System Primary: Writing update files.
    8203-E4A*0666000 Managed System Primary: Writing file 80a00020.  0 bytes written.
    8203-E4A*0666000 Managed System Primary: Writing file 80a00701.  5959118 bytes written.
    8203-E4A*0666000 Managed System Primary: Writing file 80a00701.  16043962 bytes written.
    8203-E4A*0666000 Managed System Primary: Writing file 80a00711.  7858212 bytes written.
    8203-E4A*0666000 Managed System Primary: Writing file 81cf0689.  0 bytes written.
    8203-E4A*0666000 Managed System Primary: Writing file 81e00100.  2684818 bytes written.
    8203-E4A*0666000 Managed System Primary: Writing file 81e00101.  2684818 bytes written.
    8203-E4A*0666000 Managed System Primary: Writing file 81e00200.  65378 bytes written.
    8203-E4A*0666000 Managed System Primary: Writing file 81e00200.  5304258 bytes written.
    8203-E4A*0666000 Managed System Primary: Writing file 81e00400.  0 bytes written.
    8203-E4A*0666000 Managed System Primary: Writing file 81e00704.  0 bytes written.
    8203-E4A*0666000 Managed System Primary: Writing file a0e00c21.  0 bytes written.
    8203-E4A*0666000 Managed System Primary: Finished writing update files.
    8203-E4A*0666000: Activating updates.
    8203-E4A*0666000 Managed System Primary: Activating updates - Restarting Flexible Service Processor.
    8203-E4A*0666000 Managed System Primary: Flexible Service Processor Restart completed successfully.
    8203-E4A*0666000 Managed System Primary: Activation completed.
    8203-E4A*0666000: Completed All Updates.
    

Virtual I/O Server install through the Hardware Management Console.

To install my Virtual I/O Server I decided to use the new integrated Hardware Management Console feature. This feature allows you to install the Virtual I/O Server through the Hardware Management Console the first time you boot it. Unfortunately you need to have all NIM ports opened because it’s just a graphical interface to the old well known installios command based on the old nimol product and running on the HMC. What a shame I was dreaming of a full integrated installation without the need of bootp/NFS and so on. So I’m very disappointed about this new feature :-(.

  • You first have to download the Virtual I/O Server media to the Hardware Management Console, check you have enough space left on the HMC before importing images :
  • hmcimportvios

  • Wait until images are correctly uploaded to the Hardware Management Console :
  • hmcuploadimagewait

  • If images were correctly uploaded on the HMC, you’ll see it on the Virtual I/O Server image repository :
  • virtualioserverimagerepository

  • When you’ll boot your Virtual I/O Server the HMC will ask you if you want to install a Virtual I/O Server on it :
  • bootviosinstall

  • The Virtual I/O Server will boot and you’ll be asked for the IP address to use and the image you want to install, in this case set the HMC ip as NIM server (the HMC is a nimol server):
  • discoveringadapters
    hmcviosip

  • After this step the installation will begin and the output will be visible in the HMC window :
  • installhmc1

  • When the installation is finished you can check log output HMC window :
  • installhmc2
    installhmc3

Need a cheap SAN : Use iSCSI

This is the trickiest part of this post. My Power Server was not shipped with a split backplane my only solution was to create an iSCSI target and to boot my Virtual I/O Server on iSCSI, the iSCSI target partition was created on an AIX lpar and I decided to use file backed luns for backup convenience (I just have to copy the file to save my Virtual I/O Server rootvg :-))

Creating target

Filesets for iSCSI target can be found on the AIX Expansion Pack DVD. You have to install these fileset on the iSCSI target before trying to configure it.

# installp -aXYgd . devices.tmiscsw.rte 
[..]
# lslpp -Lc | grep devices.tmiscsw.rte 
devices.tmiscsw:devices.tmiscsw.rte:7.1.2.0: : :C: :iSCSI Target Software Device Driver : : : : : : :0:0:/:1241
  • To create the target follow these steps :
  • # mkdev -c driver -s node -t tmsw 
    tmsw0 Available
    # mkdev -c tmiscsi -s tmtarget -t target -p tmiscsi0 -a owner='tmsw0' -a iscsi_name="inq.priss.lab.chmod666.org"  
    target0 Available
    
  • To create lun follow these steps : In my case I choose to create file backed iSCSI lun. I use dd command to create the lun itself, and then add it to the iSCSI configuration :
  • # dd if=/dev/zero of=/luns/lun1  bs=1m count=10k
    10240+0 records in.
    10240+0 records out.
    # ls -l /luns/lun1
    -rw-r--r--    1 root     system   10737418240 Dec 14 10:22 /luns/lun1
    # mkdev -c tmiscsi -s tmdev -t lu -p 'target1' -l 'lun1' -a back_dev_type='file' -a back_dev='lun1' -a back_file='/luns/lun1'
    lun1 Available
    # lsattr -El lun1
    back_dev        lun1       Backing Device Name                           True
    back_dev_option            N/A                                           True
    back_dev_type   file       Backing Device Type                           True
    back_file       /luns/lun1 Backing File Full Path Name                   True
    dev_size                   N/A                                           True
    lun_id          0x0        Logical Unit Number ID                        False
    queue_depth     3          Maximum Number of Commands to Queue to Device True
    

Initiator configuration

  • Give a name to your initiator :
  • # chdev  -l 'iscsi0' -a initiator_name='inq.deckard'
    
  • Modify the /etc/iscsi/targets file to add the name and port of the iSCSI target, then cfgmgr :
  • # echo "192.168.0.21 3260 inq.priss.lab.chmod666.org" > /etc/iscsi/targets
    # tail -1 /etc/iscsi/targets
    192.168.0.21 3260 inq.priss.lab.chmod666.org
    # cfgmgr -vl iscsi0
    # lspv
    hdisk0          00ce7e539a9ee2fb                    rootvg          active      
    hdisk1          00ce7e53d67b71ca                    None 
    # lsdev -Cc disk | grep hdisk1 
    hdisk1 Available  Other iSCSI Disk Drive
    

Boot on iSCSI

    If like me you decided to boot on iSCSI, follow the same steps as above to configure the target. Then boot you partition on SMS menu, begin the installation. The configuration of the iSCSI initiator is performed on the BOS installation menu, follow the steps below to configure it, and then boot on iSCSI :

  • When the menu comes up, choose Configure Network Disks (iSCSI) :
  •                            Welcome to Base Operating System
                          Installation and Maintenance
    
    Type the number of your choice and press Enter. Choice is indicated by >>>.
    
    >>> 1 Start Install Now with Default Settings
        2 Change/Show Installation Settings and Install
        3 Start Maintenance Mode for System Recovery
        4 Configure Network Disks (iSCSI)
        5 Select Storage Adapters
    
        88  Help ?
        99  Previous Menu
    
    >>> Choice [1]: 4  
    
  • Then choose Configure iSCSI :
  •                                 Configure iSCSI
    
    Move cursor to desired item and press Enter.
    
      Configure iSCSI
      Network Utilities
    
  • Set your iSCSI target name, ip, port and lun number, if all is ok you’ll see the iSCSI block devices presented to your server as hdisk devices :
  •                       iSCSI Configuration -- SW Initiator
    
    Type or select values in entry fields.
    Press Enter AFTER making all desired changes.
    
                                                            [Entry Fields]
    * Ethernet Interface                                  en0
    * IP Address of Ethernet Interface                    192.168.0.6
    * Network Mask of Ethernet Interface                  255.255.255.0
    * Gateway to iSCSI Target                             192.168.0.254
    * iSCSI Target Name                                  [inq.priss]
    * IP Address of iSCSI Target                         [192.168.0.21]
    * Port Number of iSCSI Target                        [3260]
    * iSCSI Initiator Name                               [inq.deckard]
    
                                     COMMAND STATUS
    
    Command: OK            stdout: yes           stderr: no
    
    Before command completion, additional instructions may appear below.
    
    en0 changed
    inet0 changed
    iscsi0 changed
    hdisk0 Available  N/A
    hdisk1 Available  N/A
    hdisk2 Available  N/A
    hdisk3 Available  N/A
    hdisk4 Available  N/A
    
  • After this step you can choose your disks. Do it the same way as a normal installation.
  • If you are checking the bootlist on a server booting on iSCSI, check with the -v option to have a detailed output :
  • # bootlist -o -m normal
    hdisk0 
    # bootlist -o -m normal -v
    'ibm,max-boot-devices' = 0x5
    NVRAM variable: (boot-device=/lhea@23c00200/ethernet@23e00200:iscsi,ciaddr=192.168.0.6,subnet-mask=255.255.255.0,itname=inq.deckard,iport=3260,ilun=0,iname=inq.priss,siaddr=192.168.0.21,2)
    Path name: (/lhea@23c00200/ethernet@23e00200:iscsi,ciaddr=192.168.0.6,subnet-mask=255.255.255.0,itname=inq.deckard,iport=3260,ilun=0,iname=inq.priss,siaddr=192.168.0.21,2)
    match_specific_info: ut=disk/iscsi/osdisk
    hdisk0 
    
  • I had to change my iSCSI target ip address for a dumb reason, but after changing this on my lpars booting on iSCSI were not able to find their boot disk anymore. The only solution I found was to change the boot device NVRAM boot-device variable Open firmware prompt :
  • 0 > printenv 
    [..]
    boot-device              /lhea@23c00200/ethernet@23e00200:iscsi,ciaddr=8.0.0.2,subnet-mask=255.255.255.0,itname=iqn.localhost.hostid.00000000,iport=3260,ilun=0,iname=inq.priss,siaddr=8.0.0.1,2 
    [..]
    0 > setenv boot-device /lhea@23c00200/ethernet@23e00200:iscsi,ciaddr=192.168.0.6,subnet-mask=255.255.255.0,itname=inq.deckard,,iport=3260,ilun=0,iname=inq.priss,siaddr=192.168.0.21,2  ok
    0 > printenv 
    [..]
    boot-device              /lhea@23c00200/ethernet@23e00200:iscsi,ciaddr=192.168.0.6,subnet-mask=255.255.255.0,itname=inq.deckard,,iport=3260,ilun=0,iname=inq.priss,siaddr=192.168.0.21,2 
    [..]
    0 > boot -v
    iSCSI BOOT ---------------------------------------------------
    Server IP.....................192.168.0.21
    Client IP.....................192.168.0.6
    Subnet Mask...................255.255.255.0
    iSCSI Initiator...............inq.deckard
    iSCSI Target..................inq.priss
    Target Port...................3260 
    Target LUN....................0 
    
    \
    Elapsed time since release of system processors: 45 mins 24 secs
    

If you are planning to buy your own Power Server don’t hesitate to contact me if you have any question. It really gives me the opportunity to test things I was not able to do at work, PowerHA migration for 6 to 7, PowerSC, some HMC features, some NIM features (suma), the possibility are infinite. If you want to contribute to the rising of chmod666.org all over the AIX world, I’m now looking for :

  • A Cisco manageable switch with ios and 16 ports.
  • A Power7 based server (any model).
  • An old DS* array.
  • A split backplane.
  • Any old hardware.

Don’t hesitate to contact me if you are trashing old machines/switch I’m interested if you want to make me gifts :-), I can buy it if you do not want to give it :-). Once again I hope this post will help you or will make you dream. I’m living in fear it’s hard to assume :-(.