Everybody knows that I’m a huge fan of Shared Storage Pools, you can check my previous post on this subject on the blog. With the new version of 220.127.116.11 of Virtual I/O Servers Shared Storage Pool have been enhanced with some cool features : simplified management, pool mirroring, pool shrinking and the long awaited unicast mode to get rid of the multicast. This post will show you that Shared Storage Pool are now powerful and ready to be used for production server (this is my own opinion). I’ll not talk about this later but be aware that the maximum pool size is now 16TB and the pool can now serve 250 Virtual I/O Clients. Here we go :
Rolling updates are available since Virtual I/O Server 18.104.22.168 but it is the first time I am using it. This feature is not a new enhancement brought by the version 22.214.171.124 but it still good to write about it . The rolling updates allow you to update Virtual I/O Severs (members of a Shared Storage Pool) one by one without causing an outage of the entire cluster. Each Virtual I/O Server has to leave the cluster before starting the update and can rejoin the cluster after the reboot and its update. A special node called database primary node (DBN) checks every ten minutes if Virtual I/O Servers are ON_LEVEL or UP_LEVEL. Before trying new Shared Storage Pool features I had to update Virtual I/O Servers, here is a little reminder on how to do it with rolling updates :
- Who is the current database primary node in the cluster (This is the last I’ll update) :
# cluster -clustername vio-cluster -status -verbose -fmt : -field pool_name pool_state node_name node_mtm node_partition_num node_upgrade_status node_roles vio-ssp:OK:vios1.domain.test:8202-E4C02064099R:2:ON_LEVEL: vio-ssp:OK:vios2.domain.test:8202-E4C02064099R:1:ON_LEVEL: vio-ssp:OK:vios3.domain.test:8202-E4C02064011R:2:ON_LEVEL:DBN vio-ssp:OK:vios4.domain.test:8202-E4C02064011R:1:ON_LEVEL:
vio-ssp:OK:vios1.domain.test:8202-E4C02064099R:2:UP_LEVEL: vio-ssp:OK:vios2.domain.test:8202-E4C02064099R:1:UP_LEVEL: : :vios3.domain.test:8202-E4C02064011R:2:ON_LEVEL: vio-ssp:OK:vios4.domain.test:8202-E4C02064011R:1:UP_LEVEL:DBN
# clstartstop -n vio-cluster -m $(hostname) # mount nim_server:/export/nim/lpp_source/vios2231-fp27-lpp_source /mnt # updateios -dev /mnt -accept -install
# ioslevel 126.96.36.199 # clstartstop -start -n vio_cluster -m $(hostname)
# cluster -clustername vio-cluster -status -verbose -fmt : -field pool_name pool_state node_name node_mtm node_partition_num node_upgrade_status node_roles vio-ssp:OK:vios1.domain.test:8202-E4C02064099R:2:ON_LEVEL: vio-ssp:OK:vios2.domain.test:8202-E4C02064099R:1:UP_LEVEL: vio-ssp:OK:vios3.domain.test:8202-E4C02064011R:2:ON_LEVEL:DBN vio-ssp:OK:vios4.domain.test:8202-E4C02064011R:1:UP_LEVEL:
$ cluster -clustername vio-cluster -status -verbose -fmt : -field pool_name pool_state node_name node_mtm node_partition_num node_upgrade_status node_roles vio-ssp:OK:vios1.domain.test:8202-E4C02064099R:2:188.8.131.52 ON_LEVEL: vio-ssp:OK:vios2.domain.test:8202-E4C02064099R:1:184.108.40.206 ON_LEVEL:DBN vio-ssp:OK:vios3.domain.test:8202-E4C02064011R:2:220.127.116.11 ON_LEVEL: vio-ssp:OK:vios4.domain.test:8202-E4C02064011R:1:18.104.22.168 ON_LEVEL:
# crontab -l | grep sspupgrade 0,10,20,30,40,50 * * * * /usr/sbin/sspupgrade -upgrade # sspupgrade -status No Upgrade in progress
# failgrp -list The requested operation can not be performed since the software capability is currently not enabled. Please upgrade all nodes within the cluster and retry the request once the upgrade has completed successfully.
Mirroring the Shared Storage Pool with failgrp command
One of the major drawback of Shared Storage Pools was resilience. The Shared Storage Pool was not able to mirror luns from on site to one another. The failgrp command introduced in this new version is used to mirror the Shared Storage Pool across different SAN array and can easily answer to resiliency questions. In my opinion this was the missing feature needed to deploy Shared Storage Pools in a production environment. One of the major cool thing in this is that you have NOTHING to do at the lpar level. All the mirroring is performed by the Virtual I/O Server and the Shared Storage Pool themselves, no need at all to verify on each lpar mirroring of your volume groups. The single point of management for mirroring is now the failure group.
- By default after upgrading all nodes to 22.214.171.124 all luns are assigned to a failure group called Default, you can rename it if you want to. Notice that the Shared Storage Pool is not mirrored at this state :
$ failgrp -list -fmt : -header POOL_NAME:TIER_NAME:FG_NAME:FG_SIZE:FG_STATE vio-ssp:SYSTEM:Default:55744:ONLINE $ failgrp -modify -clustername vio-cluster -sp vio-ssp -fg Default -attr FG_NAME=failgrp_site1 Given attribute(s) modified successfully. $ failgrp -list -fmt : -header POOL_NAME:TIER_NAME:FG_NAME:FG_SIZE:FG_STATE vio-ssp:SYSTEM:failgrp_site1:55744:ONLINE $ cluster -clustername vio-cluster -status -verbose [..] Pool Name: vio-ssp Pool Id: 000000000AFD672900000000529641F4 Pool Mirror State: NOT_MIRRORED [..]
$ failgrp -create -clustername vio-cluster -sp vio-ssp -fg failgrp_site2: hdiskpower141 failgrp_site2 FailureGroup has been created successfully. $ failgrp -list -fmt : -header POOL_NAME:TIER_NAME:FG_NAME:FG_SIZE:FG_STATE vio-ssp:SYSTEM:failgrp_site1:223040:ONLINE vio-ssp:SYSTEM:failgrp_site2:223040:ONLINE
$ cluster -status -clustername vio-cluster -verbose Pool Name: vio-ssp Pool Id: 000000000AFD672900000000529641F4 Pool Mirror State: SYNCED
$ pv -list POOL_NAME: vio-ssp TIER_NAME: SYSTEM FG_NAME: failgrp_site1 PV_NAME SIZE(MB) STATE UDID hdiskpower156 55770 ONLINE 1D06020F6709SYMMETRIX03EMCfcp hdiskpower1 111540 ONLINE 1D06020F6909SYMMETRIX03EMCfcp POOL_NAME: vio-ssp TIER_NAME: SYSTEM FG_NAME: failgrp_site2 PV_NAME SIZE(MB) STATE UDID hdiskpower2 223080 ONLINE 1D06020F6B09SYMMETRIX03EMCfcp
Mapping made easier with the lu command
If like me you ever used a Shared Storage Pool you probably already know that mapping a device was not so easy with the mkbdsp command. Once again the new version of Virtual I/O Server is trying to simplify the logical units and backing device management with the addition of a new command called lu. Its purpose is to manage logical units creation, listing, mapping and removing in a single easy to use command, i’ll not tell you here how to use the command but here are a few nice example :
- Create a thin backing device called bd_lu01 and map it to vhost4 :
$ lu -create -clustername vio-cluster -sp vio-ssp -lu lu01 -vadapter vhost4 -vtd bd_lu01 -size 10G Lu Name:lu0011 Lu Udid:e982bf85313bcafe0af7653e8e39c3d9 Assigning logical unit 'lu01' as a backing device. VTD:bd_lu01
$ lu -map -clustername vio-cluster -sp vio-ssp -lu lu01 -vadapter vhost4 -vtd bd_lu01 Assigning logical unit 'lu01' as a backing device. VTD:bd_lu01
$ lu -list POOL_NAME: vio-ssp TIER_NAME: SYSTEM LU_NAME SIZE(MB) UNUSED(MB) UDID lu01 10240 10240 e982bf85313bcafe0af7653e8e39c3d9 [..]
Physical volume management made easier with the pv command
Before this new release Shared Storage Pool was able to replace a lun by using the chsp command. You were not able to remove a lun for the Shared Storage Pool. The new release aims to simplify the Shared Storage Pool management and a new command called pv is introduced to manage luns from a single and easy command to add, replace, remove and list luns from the Shared Storage Pool.
$ pv -add -clustername vio-cluster -sp vio-ssp -fg failgrp_site1: hdiskpower1 Given physical volume(s) have been added successfully.
$ pv -replace -clustername vio-cluster -sp vio-ssp -oldpv hdiskpower2 -newpv hdiskpower1 Current request action progress: % 5 Current request action progress: % 100 The capacity of the new device(s) is less than the capacity of the old device(s). The old device(s) cannot be replaced. $ pv -replace -clustername vio-cluster -sp vio-ssp -oldpv hdiskpower156 -newpv hdiskpower1 Current request action progress: % 5 Current request action progress: % 6 Current request action progress: % 100 Given physical volume(s) have been replaced successfully.
$ lspv | grep hdiskpower0 hdiskpower0 00f7407858d6a19d caavg_private active $ pv -replace -oldpv hdiskpower0 -newpv hdiskpower156 The specified PV is not part of the storage pool
$ pv -list -verbose -fmt : -header POOL_NAME:TIER_NAME:FG_NAME:PV_NAME:PV_SIZE:PV_STATE:PV_UDID:PV_DESC vio-ssp:SYSTEM:failgrp_site1:hdiskpower1:111540:ONLINE:1D06020F6909SYMMETRIX03EMCfcp:PowerPath Device vio-ssp:SYSTEM:failgrp_site2:hdiskpower2:223080:ONLINE:1D06020F6B09SYMMETRIX03EMCfcp:PowerPath Device
Removing a disk from the Shared Storage Pool
It is now possible to remove disk from the Shared Storage Pool, this new features is known as pool shrinking
$ pv -remove -clustername vio-cluster -sp vio-ssp -pv hdiskpower1 Given physical volume(s) have been removed successfully.
Repository disk failure and replacement
The Shared Storage pool is now able to be up without its repository disk running. Having an error on the repository disk is not a problem at all. This one can be replaced at the moment you found an error on it by the chrepos command and it’s pretty easy to do. Here is a little reminder :
- To identify repository disk you can use CAA command as root :
# /usr/lib/cluster/clras lsrepos [..] hdisk558 has a cluster repository signature. hdisk559 has a cluster repository signature. hdisk560 has a cluster repository signature. Cycled 680 disks. hdisk561 has a cluster repository signature. hdiskpower139 has a cluster repository signature. Cycled 690 disks. Found 5 cluster repository disks. [..]
# lsvg -p caavg_private caavg_private: PV_NAME PV STATE TOTAL PPs FREE PPs FREE DISTRIBUTION hdiskpower0 active 15 8 02..00..00..03..03 # lsvg -l caavg_private caavg_private: LV NAME TYPE LPs PPs PVs LV STATE MOUNT POINT caalv_private1 boot 1 1 1 closed/syncd N/A caalv_private2 boot 1 1 1 closed/syncd N/A caalv_private3 4 4 1 open/syncd N/A powerha_crlv boot 1 1 1 closed/syncd N/A # dd if=/dev/zero of=/dev/hdiskpower0 bs=1024 count=1024 1024+0 records in 1024+0 records out # lsvg -l caavg_private 0516-066 : Physical volume is not a volume group member. Check the physical volume name specified.
$ chrepos -n vio-cluster -r -hdiskpower0 chrepos: The removal of repository disks is not currently supported.
$ chrepos -r -hdiskpower0,+hdiskpower141 chrepos: Successfully modified repository disk or disks.
Cluster unicast communication by default
Using unicast in a Cluster AIX Aware cluster is a long awaited feature asked by IBM customers. As everybody knows Shared Storage Pools are based on the Cluster AIX Aware cluster and is using multicast before the 126.96.36.199 release. By updating all Virtual I/O Servers members of a Shared Storage Pool the communication_mode attribute used by CAA will now be change to unicast. This mode does not exists at all in previous version so do not try to modify it. With this new feature a new command will be available as padmin called clo. This one will lets you check and change CAA tunables, communication_mode included :
- Before updating the Virtual I/O Server to 188.8.131.52 the communication_mode tunable does not exists and you have to check it with the CAA command : clctrl (you have to be root) :
# clctrl -tune -a vio-cluster(791d19c8-5796-11e3-ac2c-5cf3fcea5648).config_timeout = 480 vio-cluster(791d19c8-5796-11e3-ac2c-5cf3fcea5648).deadman_mode = a vio-cluster(791d19c8-5796-11e3-ac2c-5cf3fcea5648).link_timeout = 0 vio-cluster(791d19c8-5796-11e3-ac2c-5cf3fcea5648).node_down_delay = 10000 vio-cluster(791d19c8-5796-11e3-ac2c-5cf3fcea5648).node_timeout = 20000 vio-cluster(791d19c8-5796-11e3-ac2c-5cf3fcea5648).packet_ttl = 32 vio-cluster(791d19c8-5796-11e3-ac2c-5cf3fcea5648).remote_hb_factor = 10 vio-cluster(791d19c8-5796-11e3-ac2c-5cf3fcea5648).repos_mode = e vio-cluster(791d19c8-5796-11e3-ac2c-5cf3fcea5648).site_merge_policy = p
$ lscluster -i | grep MULTICAST IPv4 MULTICAST ADDRESS: 184.108.40.206 IPv4 MULTICAST ADDRESS: 220.127.116.11 IPv4 MULTICAST ADDRESS: 18.104.22.168 IPv4 MULTICAST ADDRESS: 22.214.171.124
$ oem_setup_env # ls -l /usr/ios/utils/clo lrwxrwxrwx 1 root system 20 Dec 2 10:20 /usr/ios/utils/clo -> /usr/lib/cluster/clo # exit $ clo -L communication_mode NAME DEF MIN MAX UNIT SCOPE ENTITY_NAME(UUID) CUR -------------------------------------------------------------------------------- communication_mode m c vio-cluster(791d19c8-5796-11e3-ac2c-5cf3fcea5648) m --------------------------------------------------------------------------------
# clo -o communication_mode clo: 1485-506 The current cluster level does not support tunable communication_mode.
$ clo -o communication_mode vio-cluster(791d19c8-5796-11e3-ac2c-5cf3fcea5648).communication_mode = u
To sum up : in my opinion Shared Storage Pools are now production ready. I’ve never seen them in production neither in development environment but Shared Storage Pools really deserve to be used. Things are drasticly simplified and even more with this new version. Please try the new Shared Storage Pools and give me your feedbacks in the comments. Once again I hope it helps.