Optimizing network. Part 1 : VIO Server load sharing

Most of our physicals network adapters are 10Gb adapters, all of these adapters are configured in SEA failover, and these SEA are configured over an Etherchannel adapter in 802.3ad (LACP) mode. Having “sleeping” adapters on the backup VIO Server is on the major drawback of this configuration. 10Gb adapters are expansive and I think It’s a shame not to use it. In most of our case Etherchannels adapters are built over two or four 10Gb physical adapters. Obviously we are NOT using 20 to 40Gb, Etherchannels adapters are used for fault tolerance. Using Shared Ethernet adapter with load sharing mode can help us to save adapters (and switch Ethernet ports), save money, save CPU on VIO Server (the two SEA are working on each VIO Server), and save bandwidth on our networks environment. Here is an example of my current configuration :

  • Shared Ethernet Adapter is build over Etherchannel :
  • # lsdev -type adapter -virtual | grep "Shared Ethernet Adapter"
    ent7             Available   Shared Ethernet Adapter
    # lsdev -type adapter -virtual | grep "EtherChannel"
    ent6             Available   EtherChannel / IEEE 802.3ad Link Aggregation
    # lsdev -dev ent7 -attr real_adapter,ha_mode
    value
    
    ent6
    auto
    
  • Etherchannel is configured in LACP (802.3ad) mode over four 10Gb physical Ethernet ports :
  • # lsdev -type adapter
    [..]
    ent0             Available   10 Gb Ethernet-SR PCI Express Dual Port Adapter (771000801410b003)
    ent1             Available   10 Gb Ethernet-SR PCI Express Dual Port Adapter (771000801410b003)
    ent2             Available   10 Gb Ethernet-SR PCI Express Dual Port Adapter (771000801410b003)
    ent3             Available   10 Gb Ethernet-SR PCI Express Dual Port Adapter (771000801410b003)
    ent4             Available   Virtual I/O Ethernet Adapter (l-lan)
    ent5             Available   Virtual I/O Ethernet Adapter (l-lan)
    ent6             Available   EtherChannel / IEEE 802.3ad Link Aggregation
    ent7             Available   Shared Ethernet Adapter
    ent8             Available   VLAN
    [..]
    # lsdev -dev ent6 -attr adapter_names,mode
    value
    
    ent0,ent1,ent2,ent3
    8023ad
    

ha_mode=auto vs ha_mode=sharing

Before trying to setup an SEA with load sharing my advice is to take a moment to understand differences between a classic SEA failover and an SEA in load sharing mode. Please click on the image below to watch it in bigger size. Here are the most important points to understand :

With the ha_mode set to auto :

  • SEA is configured over one trunk adapter (802.1q compatible), all accessible vlans are added to this trunk adapter.
  • All logical partitions are using the the same path. (yellow partitions access network through the yellow VIO Server)
  • Physical adapter on the blue VIO Server is not used until a failover.
  • All vlans are identically configured on each trunk adapter, all vlans are accessible trough the same VIO Server. (yellow one, or blue one after a failover)

  • With ha_mode set to sharing :

  • SEA is configured over two (or more) trunk adapter (802.1q compatible) (if you are using two trunk adapters on the first SEA, you have to use two trunk adapters on the second, and so on with three or four). All accessible vlans are shared among these trunk adapter. Adapters with same PVID on each VIO Server have to be configured with the exact same VID list (in our case 1122 on one trunk adapter, 1124 on the second).
  • Each logical partitions on the same vlans are using the same path, partitions on different vlans can use the path on the second VIO Server if the vlan id is present on a different trunk adapter (in our case vlans 1122 and 1124 are using two different path)
  • Each physical adapter on each VIO Server are used, in our case one for the 1222 vlan, the other for the 1124 vlan (yellow partition with VID 1224 can access network through the yellow VIO Server, blue partition with VID 1122 is going to use the blue VIO Server).
  • All vlans are identically configured on each trunk adapter WITH THE SAME PVID. Trunk adapters with PVID 999 are configured with ‘additional vlans’ ids set to 1122, Trunk adapters with PVID 998 are configured with ‘additional vlans’ ids set to 1124.
  • Creating a new load sharing SEA

    HMC side

    On each VIO Server add three new virtual adapters, one for control channel, the two others for trunk adapters (on theses ones share the vlans by looking the load on each one, HMC lsstat command can help you).

    • Control channel adapter on VIO Server 1, choose a PVID not use by one of your vlan :
    • Control channel adapter on VIO Server2, choose a PVID not use by one of your vlan :
    • First trunk Ethernet adapter on VIO Server1, choose a PVID not use by one of your vlan, check 802.1q compatible adapter box, add vlans in additional VLAN ID box (in our case we have split vlans 1122 and 1124 between the trunk adapters), check Use this adapter for Ethernet Bridging and set the priority to 1 (the priority have to be the same on the second trunk adapter) :
    • First trunk Ethernet adapter on VIO Server 2, choose a PVID not use by one of your vlan, check 802.1q compatible adapter box, add vlans in additional VLAN ID box (in our case we have split vlans 1122 and 1124 between the trunk adapters), check Use this adapter for Ethernet Bridging and set the priority to 2 (the priority have to be the same on the second trunk adapter) :
    • Second trunk Ethernet adapter on VIO Server 1, choose a PVID not use by one of your vlan, and different from the first PVID used by the first trunk adapter, check 802.1q compatible adapter box, add vlans in additional VLAN ID box (in our case we have split vlans 1122 and 1124 between the trunk adapters), check Use this adapter for Ethernet Bridging and set the priority to 1 (the priority have to be the same on the first trunk adapter) :
    • Second trunk Ethernet adapter on VIO Server 2, choose a PVID not use by one of your vlan, and different from the first PVID used by the first trunk adapter, check 802.1q compatible adapter box, add vlans in additional VLAN ID box (in our case we have split vlans 1122 and 1124 between the trunk adapters), check Use this adapter for Ethernet Bridging and set the priority to 2 (the priority have to be the same on the first trunk adapter) :

    VIO Server side

    All virtual Ethernet adapters are present on all VIO Servers :

    • ent1 is the real physical adapter.
    • ent4 is the first trunk adapter.
    • ent5 is the second trunk adapter.
    • ent6 is the control channel adapter.

    Begining with the first VIO Server, then the second VIO Server (really important), create the new Shared Ethernet Adapter :

    • First VIO Server :
    • # mkvdev -sea ent1 -vadapter ent4,ent5 -default ent4 -defaultid 999 -attr ha_mode=sharing ctl_chan=ent6
      ent7 Available
      
    • Second VIO Server :
    • # mkvdev -sea ent1 -vadapter ent4,ent5 -default ent4 -defaultid 999 -attr ha_mode=sharing ctl_chan=ent6
      ent7 Available
      

    Checking Shared Ethernet Adapter state

    You can easily check the configuration by using the lsdev command, all informations about the Shared Ethernet Adapter can be gathered using the entstat, here are a few examples :

    • Configuration checking using lsdev command :
    • lsdev -dev ent7 -attr ctl_chan,pvid_adapter,virt_adapters,real_adapter,pvid,ha_mode
      value
      
      ent6
      ent4
      ent4,ent5
      ent1
      999
      sharing
      
    • Checking state, on first VIO Server, then on the second one, mode are not primary and backup anymore, but primary_sh and backup_sh :
    • # entstat -all ent7 | grep "State" | grep -v LAN
      [..]
          State: PRIMARY_SH
      [..]
      
      # entstat -all ent7 | grep "State" | grep -v LAN
      [..]
          State: BACKUP_SH
      [..]
      
    • Checking active and inactive trunk adapter on each VIO Server. In a normal case, on each VIO Server, one adapter has to be active. (Example below : on VIO Server 1, trunk adapter with vlan id 1222 is active, and trunk adapter with vlan id 1224 is inactive. On the second one trunk adapter with vlan id 1222 is inactive, and trunk adapter with vlan id 1224 is active) :
    • # entstat -all ent7 |grep -E "Priority|Virtual Adapter|Shared Ethernet Adapter"
      Device Type: Shared Ethernet Adapter
      Statistics for adapters in the Shared Ethernet Adapter ent7
          Priority: 2
      Virtual Adapter: ent4
        Priority: 2  Active: True
      Virtual Adapter: ent5
        Priority: 2  Active: False
      
      # entstat -all ent7 |grep -E "Priority|Virtual Adapter|Shared Ethernet Adapter"
      Device Type: Shared Ethernet Adapter
      Statistics for adapters in the Shared Ethernet Adapter ent7
          Priority: 1
      Virtual Adapter: ent4
        Priority: 1  Active: False
      Virtual Adapter: ent=5
        Priority: 1  Active: True
      
    • Checking vlans configuration each VIO Server. In our case firsts adapters on each VIO Server are setup with 1122 vlan, and seconds adapters are setup with 1124 vlan (output for this command are the same on each VIO Server) :
    • # entstat -all ent7 |grep -E "Port VLAN ID|VLAN Tag IDs|VID shared|Virtual Adapter|Shared Ethernet Adapter"
      Device Type: Shared Ethernet Adapter
      Statistics for adapters in the Shared Ethernet Adapter ent7
          VID shared: 999 1122
      Virtual Adapter: ent4
      Port VLAN ID:   999
      VLAN Tag IDs:  1122
      Virtual Adapter: ent5
      Port VLAN ID:   998
      VLAN Tag IDs:  1124
      [..]
      

    Moving from a SEA failover to a load sharing SEA

    An existing Shared Ethernet Adapter can be modify from auto to sharing without an outage, however my advice is to be careful with this kind of operation. Follow the steps below in this exact order when you have to move from ha_mode=auto to ha_mode=sharing :

    • From the Hardware Management Console with a dlpar operation, add on each VIO Server a new trunk adapter, with an unused PVID, and the list of additional vlan ids (priority has to be set at the same value of the existing trunk adapter used by the Shared Ethernet Adapter).
    • Run cfgdev command on each VIO Server, one by one. You can skip this step if you are using VIO Server 2.2.2.1 or later.
    • On each VIO Server, add the new trunk adapter to the existing Shared Ethernet Adapter (example below : ent5 is the Shared Ethernet Adapter, ent4 the existing trunk adapter, and ent6 the new added trunk adapter) :
    • # chdev -dev ent5 -attr virt_adapters=ent4,ent6
      # lsdev -dev ent5 -attr virt_adapters,ha_mode
      value
      
      ent4,ent6
      auto
      
    • On each VIO Server, STARTING BY THE PRIMARY SHARED ETHERNET ADAPTER, change the ha_mode to sharing :
    • # chdev -dev ent5 -attr ha_mode=sharing
      lsdev -dev ent5 -attr virt_adapters,ha_mode
      value
      
      ent4,ent6
      sharing
      
    • If you want to migrate a vlan from one trunk adapter to another, an outage is required, remove the vlan form the first trunk adapter on each VIO Server, then add the same vlan on the second trunk adapter on each VIO Server.

    Impress your network team

    Network teams guy are my friends, they do not like to see unused and plunged switch Ethernet port. After a successful configuration of an SEA in sharing mode, call them, and test the failover (always test the SEA) :

    • In a normal situation, vlans are shared between the to Ethernet switch port :
    • After a failover, all vlans are using the same Ethernet switch port :

    Hope this can help.

    35 thoughts on “Optimizing network. Part 1 : VIO Server load sharing

    1. One quick question.
      Do I need to use different PVID for each trunk adapter or can I use 999 (in this case) instead of using 998 for the 2nd trunk adapter ?

      • You can’t use the same PVID for both trunk adapters, they have to be different. One golden rule to remember : DO NOT DUPLICATE PVID and VID on the same vswitch, doing this will result in a broadcast storm.

        Benoit.

    2. You can’t use the same PVID for both trunk adapters, they have to be different. One golden rule to remember : DO NOT DUPLICATE PVID and VID on the same vswitch, doing this will result in a broadcast storm.

      Benoit.

    3. What version of the VIO Server is required to use ha_mode=sharing? Good job; I like the part about using the dormant adapters and network ports

    4. Minor issue: The picture sea_failover_sh_rz.png does not match the text
      “All vlans are identically configured on each trunk adapter WITH THE SAME PVID. Trunk adapters with PVID 999 are configured with ‘additional vlans’ ids set to 1122, Trunk adapters with PVID 998 are configured with ‘additional vlans’ ids set to 1124.”
      In the picture, the two adapters with PVID 999 have different VID values. Same for PVID 998.

      • Thanks a lot for your help, the image is corrected now. (An erroneous version was uploaded, my bad).

    5. Very nice – but I need some help regardings the VLANs.
      Do I need a tagged switch port with tagged VLANs and as a result
      more than one VLANs / networks at all? Or are the tags discarded at switch level and the traffic
      is sent to the default VLAN of the switch port?

      • Your switch port has to be tagged with all the vlans used by this SEA configuration, it’s a load balancing by vlans and by networks. One single vlan (network) can only be served by one Virtual I/O server at a time. Vlan will be served by the second Virtual I/O server after a failover.

    6. Hi Benoit,

      Is it an obligation to use a strictly internal PVID in addition of the “IEEE 802.1q” box checked with the real VLAN ID you want to configure ?

      I would prefer to directly fill the real VLAN ID into the PVID value and do not check the “IEEE 802.1q” box… That seems more simple like that…

      • Hi Erw,

        In sharing mode the load balancing is done by VLAN, if you do not want to use a 802.1q virtual adapter you can do a SEA in sharing mode without this option but you have to setup one adapter per vlan. One of my Virtual I/O Server have more than 40 vlans defined on a SEA (2/3 adapters 802.1q), if you want to do this without 802.1q adapter this means you have to add 40 virtual adapters … pretty ugly. If you just have 2 or 4 vlans you can use your solution … but if you want to add an additionnal vlan you have to add an additionnal adapter and add it in the SEA (both SEA, both VIOS). On POWER7 a vlan can be dynamically added on an 802.1q virtual adapter. In my opinion it is much easier and much flexible to use 802.1q virtual adapter ….

        I hope this can help.

        Benoit.

    7. OK Benoit but we have to use at least 2 VEA per SEA. If not, the command exits on an error (not enough virtual adapters). Is mean that all VLAN from the first VEA will use the first VIOS and all VLAN from the second VEA will use second VIOS for example or a group of VLAN from one VEA is shared between both VIOS ?

      Thanks for all your advice and your quick answers

      • VLAN are shared by Virtual Ethernet Adapter. If you have to 2 Virtual Ethernet Adapter : first Virtual I/O Server will serve VLANs from the first Virtual Ethernet Adapter and second Virtual I/O Server will serve VLANs from second Virtual Ethernet Adapter. Before creating the SEA you can look at the traffic per VLAN and share them by use (do not set all the most used VLANs on one adapter and the others on a second one). If your SEA is already defined in ha_mode=auto you can have a look on the HMC lsstat command. This command can tell you the most use vlan per SEA.

        # lsstat -r vlan -m MANAGED_SYSTEM

        A few months ago i’ve created a scripts to graph lsstat output in rrd ; have a look a this post on DW : https://www.ibm.com/developerworks/community/forums/html/topic?id=77777777-0000-0000-0000-000014874176

        Hope this can helps.

        Benoit.

        • Thanks again Benoit.

          That wasn’t working when I have filled directly the tag ID in the PVID attribute of each virtual card. So I did the same thing as your screenshots.

          For each SEA, I used only 2 VEA. For each VEA, I fill a strictly internal VLAN for the PVID and I complete with the “IEEE 802.1q” box checked and the real tag ID.

          For the moment I have only one VLAN per VEA but with your command “entstat” I can see that both VIOS are sharing VLANs. One VEA is “True” on the first VIOS, the other is “True” on the second VIOS.

          It’s working like that. I will try to add more VLAN on all VEAs

          I will use too the command “lsstat” next time I working on this configuration.

          Thanks a lot, your topic is very helpful.

    8. Really nice article, thx a lot!

      One question:
      From other articles I found that with use of 802.1q vlan tagging, after creating a SEA they will issue the below command:
      mkvdev –vlan ent7 –tagid 433
      where ent7 is the sea and the tagid 433 is the LPAR’s PVID. Can you explain the usage of this command? Really appreciate for your help!!

      • Hi,

        This command is needed only if you want to have access to vlan 433 from the Virtual I/O Sever itself. I’m using it if I don’t have any LHEA or dedicated adapter for the administration and if I want to use the same adapter for the Shared Etherned Adapter for production and administration needed.

        As you can see on this output ent6 was created on top of the Shared Ethernet adapter
        # lsdev -virtual | grep -E “ent5|ent6″
        ent5 Available Shared Ethernet Adapter
        ent6 Available VLAN
        # lsdev -dev ent6 -attr
        attribute value description user_settable

        base_adapter ent5 VLAN Base Adapter True
        vlan_priority 0 VLAN Priority True
        vlan_tag_id 433 VLAN Tag ID True

        I’m accessing to this Virtual I/O Server trough SSH ent6 is up with an IP in vlan 433 :
        # lstcpip -interfaces

        Name Address Netmask State MAC

        en5 – – detach 00:21:5e:e2:2b:24
        et5 – – detach 00:21:5e:e2:2b:24
        en6 10.240.33.33 255.255.255.0 up 00:21:5e:e2:2b:24
        et6 – – detach 00:21:5e:e2:2b:24

        Hope it helps.

    9. Hi Benoit,
      Thanks the clear and handy steps on configuring the SEA load sharing. I just implemented a similar configuration, it works and able to failover and back.
      However, I found that, when I reconnect the 10G network after a failover test, the VIO servers can response to the change (from entstat command) within 1 or 2 seconds, but the client LPAR which using the failover VLAN will be disconnected (by ping test) and need 20 – 30 seconds to resume the connection.
      I also see that, even the VLAN which do not require to fallback, will also experience a short outage for about 5 – 6 seconds.
      Is this normal?

      Thanks.

      • Hi,
        This behavior is not normal, here is what I’m doing to resolve this problem, it works in my case, let me know if it solve yours :
        - First read the page 240 from this redbook : http://www.redbooks.ibm.com/redbooks/pdfs/sg247940.pdf
        - On the default route setup the dead gateway detection :
        # route change default -active_dgd
        - Add this line to /etc/rc.tcpip to make the change permanent.
        - Set interval of dgd to 2 seconds instead of five (reboot your server after this change) :
        # no -p -o dgd_ping_time=2
        - On network switch :
        – Disable spanning tree protocol (STP), OR
        – Enable PortFast trunk
        # switchport mode trunk eth1/1
        # spanning-tree portfast eth1/1
        – If your are using LACP 802.3ad set this one to ACTIVE on the switch side.
        - Restest a failover and let me know if this it ok for you.

        Good luck.

        Benoit.

        • Hi Benoit
          Thanks a lot for your information.

          I read p240 of the recommended Redbook which suggested to setup the dead gateway detection in the LPAR. However, in our configuration, all vlans in that network are all internal which with no default gateway available, so it seems not possible to have the DGD enable.
          Alternatively, I tried another way, when the connection is hanging after a SEA fallback, I start a ping from the LPAR to the outside. This seems working and I the path was resumed within a second! I repeated the test a few more times and it is quite consistent, if I don’t ping out from the LPAR and just ping in from outside world, I’ll have to wait for 20 – 30 second for connection to resume.

          I guess, it should have a similar behavior if I enable the DGD in other network which has a default gateway. But my question will be, if the LPAR has 5 vlan in it, is that mean I need to setup DGD for each vlan?

          Another observation is, I found that the delay resume problem only happens on the vlans which assigned to go through the secondary SEA (Priority 2). For the primary adapter vlans, they can failover and fallback with 1 or 2 seconds without ping out from the LPAR or DGD enable, after a network cable disconnection test.

          Thanks.

          Eddie.

    10. Hi,
      In the sharing mode, how about the VIOC virtual ethernet? Which PVID should it be setted up? 999 or 998? Does it still need to set the 802.1q compatible adapter box?

      • Hi,

        On the VIOC, configure one of the vlan you have set in the “additional vlan id”, do not use 999 or 998 these PVID are throwaway PVID (use them only if you switch port default vid is 999 or 998, most of the time it’s set to 1). No need to set 802.1q compatible on the lpar virtual adapter.

        Hope it helps.

        Benoit.

        • Hi Benoit,

          I still have a question. Why you use the same command for the second VIOS to creat a new SEA? In my opinion, if we want to use the second VIOS’s SEA to handle the 1122 VLAN, we should set the default adapter to ent5 and default ID to 9999. Thanks for your kindly answers.

    11. Any suggestions on creating a load sharing environment using dual VIO servers when there is only a single tagged vlan in use by all client partitions(.i.e 304)?

      • Hi,

        SEA Load sharing is designed to share multiple vlans (you need at least two vlans to use SEA Load Sharing). If you are using a single vlan and you want share the load between the two Virtual I/O Servers my advice is to create two simple Shared Ethernet Adapter (on per Virtual I/O Server : and with your vlan in Additionnal Vlan ID) in a different Virtual Switch, then create an NIB Etherchannel in the Virtual I/O Client. By alternating the primary adapter in the NIB on each Virtual I/O Client you can easily choose which Virtual I/O Server will be used per Virtual I/O Client, and by the way share the load on a lpar level …

        The document on this link : http://www.wmduszyk.com/wp-content/uploads/2012/01/PowerVM-VirtualSwitches-091010.pdf explain this solution better than me.

        If you need help regarding this solution don’t hesitate to contact me and I can help you in your configuration.

        Hope it helps,

        Benoît.

    12. Hi Benoit,
      I didn’t get why you used 999 for the defaultid while creating the SEA? In your example you used 999 as defaultid. Does this id matters or this is also a throwaway pvid. What happens if you used 1122 or some other no. instead of 999?Can you clarify this…
      mkvdev -sea ent1 -vadapter ent4,ent5 -default ent4 -defaultid 999 -attr ha_mode=sharing ctl_chan=ent6

      Thanks in Advance, :)

      • Hi Chris,

        “The SEA Default PVID is the VID used for untagged frames. All untagged packets are assigned to the SEA Default PVID value. When a tagged frame is received by a port, the tag is used. Otherwise if the frame is untagged, the value contained in the PVID is considered as a tag. This flag maps to the SEA device attribute pvid.”

        In our case frames are tagged : the vlan with ID 1122 is configured on the switch on a trunked port. If your are using vlan 1122 as defaultid the vlan will not be accessible anymore, or you have to configured it on the switch as the default id for the port where ent1 is plugged.

        Regards,

        Benoit.

    13. Benoit,
      Great article. I am supporting a previously configured SEA setup with ha_mode=auto. I want to convert to ha_mode=sharing. I have 2 trunk adapters one has PVID=1(vlans=3,28) and the other has PVID=2(vlans=187,188). Is it not best practice to use 1 and 2 or is that ok? Second question, on VIO1 the SEA defaultid=1 and on VIO2 the SEA defaultid=3. This configuration has been working but I want to make sure there will be no issues when I switch to sharing mode. Can you advise me?
      Nate

      • Hi nate,

        - I personally don’t use vlan 1, it is sometimes reserved by switch/router providerd (such as CISCO), so my advice is not to use vlan id 1. No problem at all for vlan 2.
        - The default id is only used for the untagged frames, I saw that your 2 trunk adapters have vlan id 3,28,187,188, so for this 4 vlans frames are tagged, and the default id will never be used for this 4 vlan id. So no special problem to use 3 as default id … but my advice is to use the PVID as defaultid to avoid nesting problems (in this case you have to setup you port switch with the same default id configured on your trunk adapter (PVID)). To sum up use throwaway id as default id.

        Then move to auto to sharing begining from the lowest priority, 1 then 2 :
        # chdev -dev entX -attr ha_mode=sharing

        Hope it helps.

      • Thanks ! If only IBM could hire me for such a job … it will be a pleasure :) . If ibmers are reading this feel free to contact me :) .

    14. Hi,

      I created sea using this command
      #mkvdev -sea ent12 -vadapter ent8,ent9,ent10 -default ent8 -defaultid 1 -attr ha_mode=sharing ctl_chan=ent11
      but when I check for active it says all true
      # entstat -d ent13 | grep -i active

      LACP activity: Active
      LACP activity: Active
      LACP activity: Active
      LACP activity: Active
      Priority: 1 Active: True
      Priority: 1 Active: True
      Priority: 1 Active: True
      Ideally it should be True,False,True or True , False, False and otherway on vio2.

      How to change these paramteres ?

    Leave a Reply

    Your email address will not be published. Required fields are marked *

    You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>