Use your NIM server to install Red Hat Enterprise Linux 6 for IBM Power

I’m actually working on a project including the deployement of PowerVC. Not one PowerVC but many. Like everybody know PowerVC is not running on AIX but is only -for the moment and forever- available on Red Hat Enterprise Linux 6.4. If you do not have the infrastructure or servers (check IBM Installation Toolkit for PowerLinux) to install Red Hat Enterprise Linux, you can choose to use your NIM server to do so. We will not use the NIM commands for that purpose because NIM is not designed to install Linux servers, we’ll just re-use tools and services used by NIM such as bootp and tftp to deploy our new Red Hat Enterprise Linux for Power. Here is how I did it with the errors and mistakes made while trying to do it.

Copying the Red Hat Enterprise Linux 6 DVD on the NIM Server

You’ll need to copy the whole Red Hat Enterprise Linux 6 DVD on the NIM server because you’ll somes files (initrd.img, vmlinuz) for booting the lpar with tftp and bootp, and all the RPM files for the installation itself. I personnally choose to copy the content of the DVD in /export/nim/lpp_source/nim/rhel/6.4 :

# mkdir -p /export/nim/lpp_source/rhel/6.4
# loopmount -i /export/iso/rhel-server-6.4-ppc64-dvd.iso -o "-V cdrfs -o ro" -m /mnt
# cp -ra /mnt /export/nim/lpp_source/rhel/6.4
# umount /mnt

tftpd configuration

By using the NIM server to install a Linux you’ll need to download files with tftp service. By default NIM is configured to allow dowloading files from tftp in the /tftpboot directory only. You’ll need to add (an)other location(s) for your own use, in my case I want to download yaboot.conf file initrd.img and vmlinuz files (I first wanted to add the ppc64.img file, but I had to remove it and replace it by the yaboot file -check the explication below). To allow downloading with tftp from different locations modify the /etc/tftpaccess.ctl configuration file. In my case I had to add /etc and /tftpboot/etc filesystems (I’ll will explain this later).

cat /etc/tftpaccess.ctl
# NIM access for network boot
allow:/tftpboot
allow:/tftpboot/etc
allow:/etc
allow:/export/lpp_source/rhel/6.4

Identify the file you’ll need to make your installation :

  • yaboot image and yaboot configuration file are needed to network load yaboot (you do not have to move yaboot image (it will not be modified)), just copy the yaboot.conf from the DVD to /etc (yaboot will search in /etc) :
  • # cp /export/nim/lpp_source/rhel/6.4/ppc/ppc64/yaboot.conf /etc/yaboot.conf
    
  • vmlinuz and initrd.img will be downloaded with tftp after yaboot will be loaded so move these files in /tftpboot/etc (I’ll explain this later on the yaboot configuration part) :
  • # mkdir -p /tftpboot/etc
    # cp /export/nim/lpp_source/rhel/6.4/ppc/ppc64/initrd.img /export/nim/lpp_source/rhel/6.4/ppc/ppc64/vmlinuz /tftpboot/etc 
    

bootptab modification

You cannot modifiy the /etc/bootptab file for you own use by using NIM commands so you had to do it by hand. The bootptab file will tell your client which files to download to begin the installation. Before Red Hat Enterprise Linux 6 you can use the ppc64.img to begin the installation directly by booting from this file. On Red Hat Enterprise Linux 6 you have to do it by using yaboot, so edit the /etc/bootptab file and add this parameters :

  • The name of the server, in my case : powervc.lab.chmod666.org
  • The boot file used to boot the server, in our case the yaboot file : /export/nim/lpp_source/rhel6/6.4/ppc/chrp/yaboot
  • The IP address of the server : 10.10.20.21
  • The type of network : ethernet
  • The IP address of the NIM server : 10.10.20.20
  • The gateway address :10.10.20.254
  • The netmask : 255.255.255.0

Fields are separated by ‘:’, do not forget to add the last ‘:’.

# tail -1 /etc/bootptab
powervc.lab.chmod666.org:bf=/export/nim/lpp_source/rhel6/6.4/ppc/chrp/yaboot:ip=10.10.20.21:ht=ethernet:sa=10.10.20.20:gw=10.10.20.254:sm=255.255.255.0:

It’s a matter of size : do not try to boot ppc64.img

If you already use this method to install a Red Hat Enterprise Linux 5 you probably just use the ppc64.img and do not have to be annoyed by all the yaboot configuration. You can’t do that with Red Hat Enterprise Linux 6 because the size of the ppc64.img file is greater than 32MB. tftp downloads are limited to 32MB, it is like this by design and you can verify it in the RFC. I’m not the only one who had this problem and a official bug was opened on the Red Hat Bugzilla : https://bugzilla.redhat.com/show_bug.cgi?id=613929 : Cannot install RHEL6 via TFTP netboot using ppc64.img. Both Red Hat Enterprise Linux 6 and tftp are defected by designed. If you try do so you’ll get an error while downloading the ppc64.img file “No operating system install”

Configuring yaboot

All of this comes from my own experience while trying to configure yaboot, yaboot boot process is not very well documented and I had to look in the source code to check how it works :

  1. yaboot image is loaded with tftp.
  2. yaboot will search it’s configuration file located in the /etc/, for example : /etc/00aabbccddee
  3. If yaboot does not find this file it will try fo find it’s configuration file by searching the IP address of your server in hexadecimal. If it does not find it, it’ll remove the last byte from the address, in our example /etc/0a0a1415 then /etc/0a0a14, and so on, until the first byte.
  4. If yaboot does not fine any mac address or IP address based file it will finally search for an /etc/yaboot.conf file.

I personnally chose to configure yaboot to let me install Red Hat Enterprise Linux with or without a kickstart file, so I have two blocks one called linux and the other one called linux-ks. The Linux one will let me install Linux by using the Red Hat installer and I have to answer the question for the installation. The seconde one : linux-ks let me install Linux with a kickstart based installation. In both case I have to load with tftp initrd.img and vmlinuz files, and in the kickstart case I have to download the ks.cfg file (In my case I have configured a web server to provide it but you can use AIX nfs if you want :-) ). Here is my /etc/yaboot.conf file

  • Configuration file header for /etc/yaboot.conf :
  • init-message = "\nWelcome to the 64-bit Red Hat Enterprise Linux 6.4 installer!\nHit  for boot options.\n\n"
    timeout=6000
    default=linux
    
  • linux block :
  • image=etc/vmlinuz
            label=linux
            initrd=etc/initrd.img
            read-only
    
  • linux-ks block :
  • image=etc/vmlinuz
            label=linux-ks
            initrd=etc/initrd.img
            append="ks=http://10.10.20.20:8080/rhel6/ks.cfg ksdevice=eth0 ip=10.253.112.146 netmask=255.255.255.254 gateway=10.253.112.129 noipv6"
            read-only
    

As you can see on the output above, yaboot needs initrd.img file and vmlinuz file, and it’ll search it in /tftpboot/etc do not add /tftpboot/ as the tftpboot client root is /tftpboot and you can use like me relative paths. I had to configure a web server to allow net based installation without NFS, but you can use NFS provided by the NIM server.

# tail -7 /etc/opt/freeware/apache/httpd.conf 
Alias /rhel6/ "/export/nim/lpp_source/rhel/6.4/"

    AllowOverride None
    Options ExecCGI Includes FollowSymLinks Indexes
    Order allow,deny
    Allow from all

My kickstart file is also provided by this web server :

# ls -l /export/nim/lpp_source/rhel/6.4/ks.cfg
-rw-r--r--    1 root     system         1394 Jan  2 11:06 /export/lpp_source/core/rhel/6.4/ks.cfg

Installation sequence

Here is the boot sequence after everything is configured and working as desired. Configure your IPL device as you always do, and boot it through the SMS menu :

  • First, yaboot image is downloaded with tftp :
  • TFTP BOOT ---------------------------------------------------
    Server IP.....................10.10.20.20
    Client IP.....................10.10.20.21
    Gateway IP....................10.10.20.254
    Subnet Mask...................255.255.255.0
    ( 1  ) Filename................./export/nim/lpp_source/rhel/6.4/ppc/chrp/yaboot
    TFTP Retries..................5
    Block Size....................512
    FINAL PACKET COUNT = 516
    
  • Then yaboot is searching for its configuration file, first trying by searching a configuration file named with the mac address of the machine :
  • System has 256 Mbytes in RMA
    Try to netboot
    
    claim of 0x2000000 at 0xe000000 returned 0xe000000
    
    TFTP BOOT ---------------------------------------------------
    Server IP.....................10.10.20.20
    Client IP.....................10.10.20.21
    Gateway IP....................10.10.20.254
    Subnet Mask...................255.255.255.0
    ( 1  ) Filename.................\etc\01-8a-70-76-c4-13-02
    TFTP Retries..................5
    Block Size....................512
    1
    2
    3
            !BA017021 !
    
    Error, can't read config file
    
  • If yaboot does not find this file it’ll try with the IP address of the server in hexadecimal :
  • Error, can't read config file
    
    
    claim of 0x2000000 at 0xe000000 returned 0xe000000
    
    TFTP BOOT ---------------------------------------------------
    Server IP.....................10.10.20.20
    Client IP.....................10.10.20.21
    Gateway IP....................10.10.20.254
    Subnet Mask...................255.255.255.0
    ( 1  ) Filename.................\etc\0a0a2021
    TFTP Retries..................5
    Block Size....................512
    1
    2
    3
           !BA017021 !
    
    Error, can't read config file
    
    
    claim of 0x2000000 at 0xe000000 returned 0xe000000
    
    TFTP BOOT ---------------------------------------------------
    Server IP.....................10.10.20.20
    Client IP.....................10.10.20.21
    Gateway IP....................10.10.20.254
    Subnet Mask...................255.255.255.0
    ( 1  ) Filename.................\etc\0a0a20
    TFTP Retries..................5
    Block Size....................512
    1
    2
    3
    
  • Finally yaboot will search for a file named yaboot.conf :
  • FTP BOOT ---------------------------------------------------
    Server IP.....................10.10.20.201
    Client IP.....................10.10.20.21
    Gateway IP....................10.10.20.254
    Subnet Mask...................255.255.255.0
    ( 1  ) Filename.................\etc\yaboot.conf
    TFTP Retries..................5
    Block Size....................512
    FINAL PACKET COUNT = 1
    FINAL FILE SIZE = 442  BYTES
    Config file read, 442 bytes
    
    Welcome to the 64-bit Red Hat Enterprise Linux 6.4 installer!
    Hit  for boot options.
    
  • Choose one option to boot on (in our case linux-ks) :
  • Welcome to yaboot version 1.3.14 (Red Hat 1.3.14-41.el6)
    Enter "help" to get some basic usage information
    boot:
    * linux                      linux-ks
    
    boot: linux-ks
    Please wait, loading kernel...
    
  • Kernel vmlinuz is loaded with tftp:
  • claim of 0x2000000 at 0xe000000 returned 0xe000000
    
    TFTP BOOT ---------------------------------------------------
    Server IP.....................10.10.20.20
    Client IP.....................10.10.20.21
    Gateway IP....................10.10.20.254
    Subnet Mask...................255.255.255.0
    ( 1  ) Filename.................etc\vmlinuz
    TFTP Retries..................5
    Block Size....................512
    FINAL PACKET COUNT = 32730
    FINAL FILE SIZE = 16757576  BYTES
       Elf64 kernel loaded...
    Loading ramdisk...
    
  • Ramdisk initrd.img is loaded with tftp :
  • claim of 0x2000000 at 0xe000000 returned 0xe000000
    
    TFTP BOOT ---------------------------------------------------
    Server IP.....................10.10.20.21
    Client IP.....................10.10.10.20
    Gateway IP....................10.10.20.254
    Subnet Mask...................255.255.255.0
    ( 1  ) Filename.................etc\initrd.img
    TFTP Retries..................5
    Block Size....................512
    FINAL PACKET COUNT = 54371
    
  • Finally the installation begin because with choose the linux-ks :
  • Welcome to Red Hat Enterprise Linux for ppc64
        +----------------------------¦ Retrieving +----------------------------+
        ¦                                                                      ¦
        ¦ Retrieving /install.img...                                           ¦
        ¦                                                                      ¦
        ¦                                  17%                                 ¦
        ¦                                                                      ¦
        +----------------------------------------------------------------------+
      / between elements  |  selects |  next screen
    

IBM Installation toolkit for PowerLinux

Finally if your are planning to install a lot of Linux Servers on Power Hardware IBM provide a solution to easily install and deploy Linux OS on Power Hardware. This tool is called IBM Installation Toolkit for PowerLinux and can be find here : http://www-304.ibm.com/webapp/set2/sas/f/lopdiags/installtools/home.html. You’ll need to dedicated (Linux ?) server for this product if you want to do so. I’ve never tested it but if I had to deploy more and more Linux servers I’ll give it a try, after all NIM server are used to install AIX and not Linux :-)

16 thoughts on “Use your NIM server to install Red Hat Enterprise Linux 6 for IBM Power

  1. hi chmod666

    glad to see yaboot back again, it reminds me, twelve years ago when i was a geek debian ppc users :-)

    by the way, i’m suprised about your statement : “because NIM is not designed to install Linux servers”

    I’ve never done that yet but i was pretty sure it’s possible and supported.

    There is a little bit stuff on the net about this topic, but if you have the time to validate that…
    However, we don’t know what is the future of NIM….

    http://ibmsystemsmag.com/aix/administrator/lpar/Simplifying-with-NIM/?page=2

    • Hi,

      Yes it is possible but in my opinion not designed for that purpose, you first have to replace bootp by dhcp and I do know if there are side effects to this change for standard AIX installation.

      Anyway I think the best way to go to automatize Linux On Power installations is to use the IBM installation toolkit, I’ll give it a try this week end, but I’ve never heard anybody using this tool.

      I agree with you about NIM, in my opinion it’s going to die soon.

      Happy new year ! And best regards.

      Benoît.

      • Did you try the toolkit, did it offer any better solutions?

        This was a good manual, I figured out that mac address naming, but still not able to get the install started with lpar_netboot. Manually from SMS everything works out nicely, so thanks for this.

        I really enjoiy NIM still, maybe the Shared Storage Pool cloning willl replace the installing part at some stage.

        • Hi,

          Yep, just tried the toolkit, going to write a post about it. I like the solution but I think it is ok to use it only if you are installing a lot of Linux, it has also a tools to help you to migrate from x86 to ppc.

          What’s your problem with lpar_netboot, it works from my side maybe it can help you.

          Is nim going to die ? Maybe … PowerVC does not use it, VMcontrol last version is not using it too, Redbooks are not up to date … I think things are not going well for NIM :).

          Regards,

          Benoît.

  2. Hi,

    Your post is really amazing. I followed you post step by step to perform the kickstart installation via NIM servers. Instead of using http to get the config file. I used NFS but it doesn’t work.

    It did startup the installation from Network but seem it didn’t read the kickstart config file and asked me for the input.
    (The kickstart file is working when I used cdrom to do installation).

    I am thinking if there’s something wrong with my /etc/yaboot.conf or /etc/bootptab. Could you advise? Below please find my yaboot.conf and bootptab

    Thanks for your help!

    /etc/yaboot.conf
    [root@mfcnimhk005:/mksysb/production] $ cat /etc/yaboot.conf
    init-message = “\nWelcome to the 64-bit Red Hat Enterprise Linux Server 5.10 installer!\nHit for boot options.\n\n”
    timeout=6000
    default=linux

    #image=etc/vmlinuz
    # label=linux
    # initrd=etc/ramdisk.image.gz
    # read-only
    #
    image=etc/vmlinuz
    label=linux
    initrd=etc/ramdisk.image.gz
    append=”ks=nfs:10.128.55.190:/nfs/ks_nfs.cfg ksdevice=eth2 ip=10.128.55.140 netmask=255.255.255.0 gateway=10.128.55.254″
    read-only

    /etc/bootptab

    [root@mfcnimhk005:/mksysb/production] $ tail -n1 /etc/bootptab
    linux2:bf=/nim/mksysb/rhel5.5/mnt/images/netboot/ppc64.img:ip=10.128.55.140:ht=ethernet:sa=10.128.55.190:gw=10.128.55.254:sm=255.255.255.0:

    • Hi Goran,

      – On the NIM server, first try to change this nfso attribute (check if reboot is mandatory of not) :

      # nfso -p -o nfs_use_reserved_ports=1
      

      – On your append line add proto=udp (and check nfsvers try nfsvers=3 or nfsvers=2 or nfsvers=4 depending on how you nfs server is configured on the NIM server)

      append="ks=nfs:10.128.55.190:/nfs/ks_nfs.cfg ksdevice=eth2 ip=10.128.55.140 netmask=255.255.255.0 gateway=10.128.55.254 proto=udp"
      

      Let me know if it solve you problem.

      Regards,

      Benoît.

  3. Hello,
    I’ve had this setup and working pretty much exactly as you describe with RHEL6.4 and RHEL6.5. I think I’m running into the 32MB TFTP limitation with RHEL7, however. The initrd.img file is just over 32MB, and I’m getting “Detected bad memory access” errors on the console during that transfer. Do you have any ideas for a workaround?

  4. I tried ti install RHEL_7.0_ppc64 on a p770,
    du -sm /nim/lppsource/rhel_7.0/ppc/ppc64/initrd.img
    31.37 /nim/lppsource/rhel_7.0/ppc/ppc64/initrd.img
    but also got error:
    PACKET COUNT = 51200
    Detected bad memory access to address: 00000000100001ff

  5. i tried this most of the part is working .. i think i missing something

    On the second tftpboot on bootp request it is not detecting the server and client IP address

    BOOTP: requested filename =
    BOOTP: client IP = 136.141.251.198
    BOOTP: client HW addr = 36 92 b0 0 70 2
    BOOTP: gateway IP = 136.141.251.193
    BOOTP: device /vdevice/l-lan@30000002
    BOOTP: loc-code U9133.55A.10B78CG-V7-C2-T1

    BOOTP: wait 60 seconds for Spanning Tree …

    BOOTP R = 1 BOOTP S = 2
    FILE: /nim-resources/RHEL62/bootimage/ppc/chrp/yaboot
    FINAL Packet Count = 416
    FINAL File Size = 212968 bytes.
    load-base=0x4000
    real-base=0xc00000

    Elapsed time since release of system processors: 436394 mins 44 secs

    System has 256 Mbytes in RMA
    Try to netboot
    claim of 0x2000000 at 0xe000000 returned 0xe000000

    BOOTP: chosen-network-type = ethernet,auto,none,auto
    BOOTP: server IP = 0.0.0.0
    BOOTP: requested filename = \etc1-36-92-b0-00-70-02
    BOOTP: client IP = 0.0.0.0
    BOOTP: client HW addr = 36 92 b0 0 70 2
    BOOTP: gateway IP = 0.0.0.0
    BOOTP: device /vdevice/l-lan@30000002
    BOOTP: loc-code U9133.55A.10B78CG-V7-C2-T1

    BOOTP: wait 60 seconds for Spanning Tree …

    How is thse address are getting is picked up from .. which configuration file ..

    Thanks

    • Hi,

      When yaboot is loaded for the first time this one is trying to find its configuration file. If you followed my tutorial you have create an /etc/yaboot.conf file on the nim server.
      You seems to have a problem here, what I see on your output is that :
      BOOTP: requested filename = \etc1-36-92-b0-00-70-02
      You sould have a \ between etc and the address of your machine in hexadecimal.
      Check you /etc/tftpaccess.ctl and check /etc and /tftpboot/etc are allowed.
      If you have put an /etc/yaboot.conf file in the nim, yaboot is checking the address byte by byte and if it does not find any configuration file of this kind it will finally take the yaboot.conf file.

      If you do not have any ip on this step it’s because your /etc/bootptab file has a problem. Double check it.

      Hope it helps.

      Regards.
      B.

      • this is my bootptab

        :bf=/nim-resources/RHEL62/bootimage/ppc/chrp/yaboot:ip=136.141.251.198:ht=ethernet:sa=136.141.251.245:gw=136.141.251.193:sm=255.255.255.192:

        yaboot.conf

        cat yaboot.conf
        init-message = “\nWelcome to the 64-bit Red Hat Enterprise Linux 6.2 installer!\nHit for boot options.\n\n”
        timeout=50
        default=linux

        image=etc/vmlinuz
        label=linux
        initrd=etc/initrd.img
        #append=”ks=nfs:136.141.251.245:/nfs/ks_nfs.cfg ksdevice=eth0 ip=136.141.251.198 netmask=255.255.255.192 gateway=136.141.251.193 noipv6″
        append=”repo=nfs://136.141.251.245/nim-resources/RHEL71/DVD”
        read-only

        I think i have all the information is needed there .. on first attempt its getting aall the address fine .. But when it tried to get second file .. the ip address is not carrying forward ..

        So i am not sure what else is needed for that ..

  6. Did anyone try out the IBM Installation Toolkit for Linux …. I am finding limitted info on how to use this with network boot and kickstart

    thanks

    C

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>