Shared Storage Pool : Save your time without the need of SAN team.

I’ve heard a lot of things about new Shared Storage Pool functionality but never had a chance to test it. With the upgrade of our P6 to P7 we kept 2 P520 for our test purpose, so I decided deploy Shared Storage Pool on it. Since Shared Storage Pool is running on these two pSeries I never had a single problem, I can test everything I want, I can clone Lpars, I can snaphot Lpars, I can move them with Live Partition Mobility without the help of my SAN team and many more. Using Shared Storage Pool is a real happiness, but there errors to avoid. Let’s have a look on my deployment on this post.

Best Practices on Shared Storage Pool

DNS configuration and /etc/hosts files

  • On this example Shared Storage Pool Cluster has 4 nodes (VIO Servers) (it is the maximum).
  • Each VIO Server must resolve the others, so setup DNS with commands below on each VIO Server :
  • # cfgnamesrv -add -ipaddr 10.20.2.251
    # cfgnamesrv -add -ipaddr 10.20.74.196
    # cfgnamesrv -add -dname domain.test
    # cfgnamesrv -add -slist domain.dev domain.prod
    
    # oem_setup_env
    # host -n -t A vios1
    vios1.domain.test has address 10.10.122.107
    # host -n -t A vios2
    vios2.domain.test has address 10.10.122.108
    # host -n -t A vios3
    vios3.domain.test has address 10.10.122.109
    # host -n -t A vios4
    vios4.domain.test has address 10.10.122.110
    # exit
    
  • Don’t ask me why, but each VIO Server needs entries for others VIOS participating in the SSP Cluster in its /etc/hosts file :
  • # hostmap -addr vios1.domain.test vios1 -host 10.10.122.107
    # hostmap -addr vios2.domain.test vios2 -host 10.10.122.108
    # hostmap -addr vios3.domain.test vios3 -host 10.10.122.109
    # hostmap -addr vios4.domain.test vios4 -host 10.10.122.110
    
    # hostmap -ls | grep vio
    10.10.122.107 vios1.domain.test vios1
    10.10.122.108 vios2.domain.test vios2
    10.10.122.109 vios3.domain.test vios3
    10.10.122.110 vios4.domain.test vios4
    

Network adapters configuration : Segregate Shared Storage Pool network from production network

  • Do not use 802.1q and vlan tagging interface on adapters used for SSP cluster, it is not supported.
  • Do not misunderstand the Redbook about Shared Storage Pool, you can use vlan tagging and 802.1q adapters to create an SEA, but you can’t use this SEA for Shared Storage Pool network.
  • # lsdev -dev ent5 -attr ctl_chan,virt_adapters,real_adapter
    value
    
    ent3
    ent4
    ent1
    # entstat -all ent5 | grep "VLAN Tag IDs"
    VLAN Tag IDs:  1122
    VLAN Tag IDs:  None
    
  • Use Etherchannel (if you can : perfer LACP + VPC on two different switches) to secure Shared Storage Pool network.
  • It’s not a Shared Storage Pool best practice, but secure SEA with etherchannel too.
  • Look at the image below, to understand my SSP configuration :

Shared Storage Pool configuration

Inital configuration

You’ve already configured Shared Ethernet Adapter and each VIO Server can “see” the same bunch of luns :

  • For exemple my inq output for vios1,2,3,4 is the same (lun id : 5206, 5207 and 5208) :
  • # oem_setup_env
    # /usr/lpp/EMC/Symmetrix/bin/inq.aix64_51 -showvol | egrep -i "rhdisk(5|6|7)"
    /dev/rhdisk5  :EMC     :SYMMETRIX       :5874  :67#Ru000   :   05206:    33554880
    /dev/rhdisk6  :EMC     :SYMMETRIX       :5874  :67#Rv000   :   05207:    33554880
    /dev/rhdisk7  :EMC     :SYMMETRIX       :5874  :67#Rw000   :   05208:    33554880
    # exit
    
  • Let’s create the Shared Storage Pool initial configuration on the first node (vios1). You need at least two disks, one for repository (repository contains a cluster filesystem used for SSP purpose like thin provisioning), and one for the SSP :
  • # cluster -create -clustername vio-cluster -repopvs hdisk5 -spname vio-ssp -sppvs hdisk6 hdisk7 -hostname vios1
    # lssp -clustername vio-cluster
    POOL_NAME:       vio-ssp
    POOL_SIZE:       65280
    FREE_SPACE:      64154
    TOTAL_LU_SIZE:   0
    TOTAL_LUS:       0
    POOL_TYPE:       CLPOOL
    POOL_ID:         000000000AF07A6B000000004FF6F0C4
    # cluster -list 
    CLUSTER_NAME:    vio-cluster
    CLUSTER_ID:      a549bcf4c77311e18f5400215e487480
    # cluster -status -clustername vio-cluster
    Cluster Name         State
    udvio000tst-cluster  OK
    
        Node Name        MTM           Partition Num  State  Pool State
        vios1            8203-E4A02060CE74         2  OK     OK
    
  • Use CAA (Cluster Aware AIX) commands to identify disks in the cluster (lscluster) :
  • # lscluster -d
    Storage Interface Query
    
    Cluster Name:  vio-cluster
    Cluster uuid:  a549bcf4-c773-11e1-8f54-00215e487480
    Number of nodes reporting = 1
    Number of nodes expected = 1
    Node vios1.domain.test
    Node uuid = a5315a38-c773-11e1-8f54-00215e487480
    Number of disk discovered = 3
            hdisk7
              state : UP
              uDid  : 1D0667520809SYMMETRIX03EMCfcp
              uUid  : abebe083-5751-4cec-d839-995ef356c0f5
              type  : CLUSDISK
            hdisk6
              state : UP
              uDid  : 1D0667520709SYMMETRIX03EMCfcp
              uUid  : 8fff403a-b6f5-7622-257b-ab088d488837
              type  : CLUSDISK
            hdisk5
              state : UP
              uDid  :
              uUid  : 0276300d-8d68-5c7a-3f99-a1337b10c8b6
              type  : REPDISK
    
  • More options can be used with lscluster command to identify cluster’s nodes :
  • # lscluster -c
    Cluster query for cluster vio-cluster returns:
    Cluster uuid: a549bcf4-c773-11e1-8f54-00215e487480
    Number of nodes in cluster = 1
            Cluster id for node vios1.domain.test is 1
            Primary IP address for node vios1.domain.test is 10.10.122.107
    Number of disks in cluster = 2
             for disk hdisk7 UUID = abebe083-5751-4cec-d839-995ef356c0f5 cluster_major = 0 cluster_minor = 1
             for disk hdisk6 UUID = 8fff403a-b6f5-7622-257b-ab088d488837 cluster_major = 0 cluster_minor = 2
    #cluster -i
    Network/Storage Interface Query:
    
    Cluster Name:  vio-cluster
    Cluster uuid:  a549bcf4-c773-11e1-8f54-00215e487480
    Number of nodes reporting = 1
    Number of nodes expected = 1
    
    Node vios1.domain.test
    Node uuid = a5315a38-c773-11e1-8f54-00215e487480
    Number of interfaces discovered = 2
            Interface number 1 en0
                    ifnet type = 6 ndd type = 7
                    Mac address length = 6
                    Mac address =  00:21:5E:48:74:80
                    Smoothed rrt across interface = 0
                    Mean Deviation in network rrt across interface = 0
                    Probe interval for interface = 100 ms
                    ifnet flags for interface = 0x1E080863
                    ndd flags for interface = 0x0061081B
                    Interface state  UP
                    Number of regular addresses configured on interface = 1
                    IPv4 ADDRESS: 10.10.122.107  broadcast 10.10.122.255  netmask 255.255.255.0
                    Number of cluster multicast addresses configured on interface = 1
                    IPv4 MULTICAST ADDRESS: 228.240.122.107  broadcast 0.0.0.0  netmask 0.0.0.0
            Interface number 2 dpcom
                    ifnet type = 0 ndd type = 305
                    Mac address length = 0
                    Mac address =  00:00:00:00:00:00
                    Smoothed rrt across interface = 750
                    Mean Deviation in network rrt across interface = 1500
                    Probe interval for interface = 22500 ms
                    ifnet flags for interface = 0x00000000
                    ndd flags for interface = 0x00000009
                    Interface state  UP  RESTRICTED  AIX_CONTROLLED
            Pseudo Interface
                    Interface State DOWN
    

Adding node in the pool

Using Shared Storage Pool can be really efficient if the SSP is shared by multiple VIO Servers. Only four VIO Servers can participate in an SSP. Using dual VIO Servers implies an SSP shared by two pSeries. I Hope this limitation can be raised in a future SSP version, but in our case two physicals hosts are enough.

  • Add the three additional nodes in the vio-cluster Shared Storage Pool using cluster command :
  • # cluster -addnode -clustername vio-cluster -hostname vios2.domain.test
    Partition vios2.domain.test has been added to the vio-cluster cluster.
    # cluster -addnode -clustername vio-cluster -hostname vios3.domain.test
    Partition vios3.domain.test has been added to the vio-cluster cluster.
    # cluster -addnode -clustername vio-cluster -hostname vios4.domain.test
    Partition vios4.domain.test has been added to the vio-cluster cluster.
    # lscluster -c
    Cluster query for cluster vio-cluster returns:
    Cluster uuid: a549bcf4-c773-11e1-8f54-00215e487480
    Number of nodes in cluster = 4
            Cluster id for node vios1.domain.test is 1
            Primary IP address for node vios1.domain.test is 10.10.122.107
            Cluster id for node vios2.domain.test is 2
            Primary IP address for node vios2.domain.test is 10.10.122.108
            Cluster id for node vios3.domain.test is 3
            Primary IP address for vios4.domain.test is 10.10.122.109
            Cluster id for node vios4.domain.test is 4
            Primary IP address for node vios2.domain.test is 10.10.122.110
    Number of disks in cluster = 2
             for disk hdisk6 UUID = 8fff403a-b6f5-7622-257b-ab088d488837 cluster_major = 0 cluster_minor = 2
             for disk hdisk7 UUID = abebe083-5751-4cec-d839-995ef356c0f5 cluster_major = 0 cluster_minor = 1
    Multicast address for cluster is 228.240.122.107
    
  • On each VIO Server messaging are queued in error messages :
  • # errlog | more
    EDFF8E9B   0706171512 I O StorageRM      IBM.StorageRM daemon has started.
    3B16518D   0706171512 I S ConfigRM       The node is online in the domain indicat
    4BDDFBCC   0706171512 I S ConfigRM       The operational quorum state of the acti
    A6DF45AA   0706171512 I O RMCdaemon      The daemon is started.
    A2D4EDC6   0706171512 I O RMCdaemon      The daemon is stopped.
    AFA89905   0706171512 I O cthags         Group Services daemon started
    DE84C4DB   0706171512 I O ConfigRM       IBM.ConfigRM daemon has started.
    

As you can see creating the SSP is really simple, and can be done in four commands. Using SSP is simple as creating it.

Using Share Storage Pool

Creating and assigning a multipathed virtual backing device on an lpar

  • Create the backing device on one node of the Shared Storage Pool cluster using the mkbdsp command :
  • # mkbdsp -clustername vio-cluster -sp vio-ssp 10G -bd bd_blueclient2
    Lu Name:bd_blueclient2
    Lu Udid:c74ed6736f796232558a34f3820cf7a2
    
  • On the 2 dual VIO Server hosting the lpar assign the new created backing device on vadapter using the luudid.
  • On our example an lpar named blueclient2 has two virtual scsi adapters connected on vios1 and vios2.
  • On vios1 :
  • # mkbdsp -clustername vio-cluster -sp vio-ssp -luudid c74ed6736f796232558a34f3820cf7a2 -vadapter vhost0
    Assigning file "c74ed6736f796232558a34f3820cf7a2" as a backing device.
    VTD:vtscsi4
    # lsmap -vadapter vhost0
    SVSA            Physloc                                      Client Partition ID
    --------------- -------------------------------------------- ------------------
    vhost0          U8203.E4A.060CE74-V2-C13                     0x00000001
    VTD                   vtscsi4
    Status                Available
    LUN                   0x8200000000000000
    Backing device        bd_blueclient2.c74ed6736f796232558a34f3820cf7a2
    Physloc
    Mirrored              N/A
    
  • On vios2 :
  • # mkbdsp -clustername vio-cluster -sp vio-ssp -luudid c74ed6736f796232558a34f3820cf7a2 -vadapter vhost0
    Assigning file "c74ed6736f796232558a34f3820cf7a2" as a backing device.
    VTD:vtscsi3
    # lsmap -vadapter vhost0
    SVSA            Physloc                                      Client Partition ID
    --------------- -------------------------------------------- ------------------
    vhost0          U8203.E4A.060CE74-V3-C13                     0x00000001
    VTD                   vtscsi3
    Status                Available
    LUN                   0x8300000000000000
    Backing device        bd_blueclient2.c74ed6736f796232558a34f3820cf7a2
    Physloc
    Mirrored              N/A
    
  • Install a new AIX lpar on this backing device. By default SSP luns are created with no_reserve attribute, and it’s great for LPM.
  • On blueclient2 :
  • # lsattr -El hdisk3 -a reserve_policy
    reserve_policy no_reserve Reserve Policy True
    # lspath -l hdisk3
    Enabled hdisk3 vscsi0
    Enabled hdisk3 vscsi1
    
  • luudid can be found in the ODM querying unique_id attribute (in our case c74ed6736f796232558a34f3820cf7a2) :
    # odmget -q "name=hdisk3 and attribute=unique_id" CuAt
    
    CuAt:
            name = "hdisk3"
            attribute = "unique_id"
            value = "41217C74ED6736F796232558A34F3820CF7A2103303      NVDISK03IBMvscsi"
            type = "R"
            generic = ""
            rep = "n"
            nls_index = 0
    

Listing what’s in the SSP

On each node of the SSP cluster backing devices can be listed with the lssp, here is an example :

# lssp -clustername vio-cluster -sp vio-ssp -bd
Lu Name          Size(mb)    ProvisionType    Lu Udid
bd_blueclient2   10240       THIN             c74ed6736f796232558a34f3820cf7a2
bd_blueclient21  1024        THIN             efc0011636f9147fc65f975e161a0df4
rootvg_blueclient1  16384       THIN             b9d86ae679100efc07f68c0573747c45
rootvg_blueclient2  16384       THIN             e3e9d1330bed2a7270e4c01499d55bca
rootvg_greenclient1  20480       THIN             799061a3114ef07ed6820fb8d3bd73b7
rootvg_greenclient2  16384       THIN             fce59add95a20e1fa6ace241f680f875
testvg_blueclient  4096        THIN             267384935049f3915908681fe6a520dc
Snapshot
rootvg_blueclient2_snap
rootvg_blueclient2_61TL01SP03_with_agent

As you can see all backing devices are created by default with a THIN ProvisionningType, use the thick option for a thick provisionned backing devies

# mkbdsp -clustername vio-cluster -sp vio-ssp 1G -bd a_thick_bd -thick
Lu Name:a_thick_bd
Lu Udid:33004a1426909c8641edba020efa2f3f
# lssp -clustername vio000-cluster -sp vio000-ssp -bd
Lu Name          Size(mb)    ProvisionType    Lu Udid
a_thick_bd       1024        THICK            33004a1426909c8641edba020efa2f3f
[..]

Using snapshot and rollback features

In my opinion this feature is one of the best things offered by SSP you can easily snapshot and rollback backing devices. I use this feature when I have to update an AIX box. If an error occur I can easily rollback to the latest version without being worried with alt_disk_copy/install, and with a minimum outage.

  • Create the snapshot on the rootvg backing device :
  • # snapshot -clustername vio-cluster -create rootvg_blueclient2_snap -spname vio-ssp -lu rootvg_blueclient2
    rootvg_blueclient2_snap
    # snapshot -list -clustername vio-cluster -spname vio-ssp
    Lu Name          Size(mb)    ProvisionType    Lu Udid
    rootvg_blueclient2  16384       THIN             e3e9d1330bed2a7270e4c01499d55bca
    Snapshot
    rootvg_blueclient2_snap
    
  • If an error occurred just run a rollback of this snapshot, so easy … :
  • # snapshot -rollback rootvg_blueclient2_snap  -clustername vio-cluster -spname vio-ssp -lu rootvg_blueclient2
    

Alerting

With thick provisioning we do not exactly know how Shared Storage Pool is really used. To prevent a full SSP resulting in I/O errors on client lpar. An alerting system can be configured on the SSP writing message on VIOS errlog. Here is a simple example, and a error when threshold is exceeded :

# alert -set -clustername vio000-cluster -spname vio000-ssp -type threshold -value 35
# errlog | more
0FD4CF1A   0829180812 I O VIOD_POOL      Informational Message
0FD4CF1A   0829180312 I O VIOD_POOL      Informational Message
0FD4CF1A   0829175712 I O VIOD_POOL      Informational Message

A deeper look in error message tells that threshold is exceeded :

A Storage Pool Threshold alert event occurred on pool D_E_F_A_U_L_T_061310 pool id d88404323923b54 in cluster vio-cluster cluster id a549bcf4c77311e18f5400215e487480 The alert event received is:
        Threshold Exceeded.

Storage Mobility

“Live Storage Mobilty” is not yet implemented on SSP because it needs two SSP on the same cluster. The need is for example to move backing device from one SSP to one another, resulting in things like VMware Storage Vmotion. You can’t do that right now, but you can use Storage mobility to replace a disk from the SSP, you can imagine secondary site luns always masked on all VIOS. You can replace disks one by one and moving all SSP from on site to another.

# chsp -replace -clustername vio-cluster -sp vio-ssp -oldpv hdisk6 -newpv hdisk10
Current request action progress : % 5
Current request action progress : % 10
[..]

What’s next ?

As you can see SSP are very very simple to use and to setup, I’m using two test pSeries since a couple of months running SSP and things have never be easier. I can create test box very quickly and play with it without being worried to wipe it : i’m using snapshot every days. LPM is working “out of the box” without the need of my SAN team and it’s really enjoyable, this is too one of the best SSP feature. Howerver there are some drawbacks : do not try to use SSP in a secured configuration with LinuxPPC, Linux multipathd does not do job. Replacing PV can takes time, and I can’t wait Live Storage Mobility. Just one final word : try SSP, use it and you’ll never come back to the old school vscsi way.

Hope this can help.

12 thoughts on “Shared Storage Pool : Save your time without the need of SAN team.

  1. Excellent content

    Best Practive on Shared Storage Pool – I think that should be Pactice

  2. Hey Benoit,

    Fantastic site – always used to bring myself upto speed in between contracts ;-)

    Many thanks,
    Tony B

  3. Pingback: Exploit the full potential of PowerVC by using Shared Storage Pools & Linked Clones | PowerVC secrets about Linked Clones (pooladm,mksnap,mkclone,vioservice) | chmod666

  4. Great article.
    According to picture attached, I would like know how easily replicated ssp can be used from second storage (after replication).
    If you could describe that I will be very apreciate for that.

  5. Hi Benoit,

    After more than 3 years now, what is the best way to use disks for an LPAR ? SSP or NPIV ? Or what are the (dis)advantages of each one ?

    I want to build this architecture for my customer :
    – dual VIOS on many Entry POWER8 with LPM activated
    – SSP for rootvg and other VG (tools, data, application…) by default
    – NPIV and direct-attached SAN for some critical LPAR in terms of I/O

    What do you think ? :-)

    Erwin

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>