How to Install ngdata

Technology Summary and Contents

• CentOS 4 Linux
• VDT 1.8.1 with Globus-Base-Data-Server, GSIOpenSSH and Prima Auth Module

Review

• 11/05/08 Daniel - reviewed and tested CentOS 4, excluding pciback.
• 14/05/08 Daniel - minor changes, tested CentOS 5 via sudo, excluding pciback.

TODO

• Complete review and testing
• Simple tests to run to confirm all services (gsissh, optional ssh and gridftp) are working
• pciback needs to work easily for existing IBM gateways, possibly without reboot?
  • publish some stats, eg netperf: native to native/domU/pciback domU
• SRB installation/testing

Notes on site-specific NGData info

•  NGDataNotes

Basic Install Instructions

• Build the basic Xen guest for NGDataVdt using the procedure shown in  XenInstall and set-up networking, user authentication and user directory mounts in accordance with local conventions
• Apply for a host certificate, see:  HostCertificates
• Install key into /etc/grid-security/hostkey.pem (mode 600) and certificate into /etc/grid-security/hostcert.pem (mode 644) - both owned by root
• Login as root, set http_proxy if appropriate
  • also set in /etc/wgetrc
  • this process should work via sudo
• enable YumRepository as in: Change to ARCS repository. MIP fixes with SIP for APAC software map. Gridpulse updates
• yum install Gbuild
• NOTE: these instructions are still being tested so the 1.8.1 build script is not released yet
  • download manually:  BuildNgdataVdt181.sh
• /usr/local/bin/BuildNgdataVdt181.sh .. and answer 'y' to the cache question
  • BuildNgdataVdt161.sh was used for the previous release of this machine
• You will also need to supply the name of a local GUMS server .. e.g.: nggums.vpac.org in /etc/grid-security/prima-authz.conf under the imsContact entry.
  • TODO: automate this as for ng2
• Submit some test jobs from a Globus client machine

Security Considerations

• A machine built using these instructions has a gsissh server running on port 22, and this offers both standard ssh and gssapi authentication methods
  • the service is called gsisshd
• If appropriate for your site, you can restrict this so that only gsissh methods are offered as follows, by adding the following to: /opt/vdt/globus/etc/ssh/sshd_config

  Protocol 2
  RSAAuthentication no
  PubkeyAuthentication no
  PasswordAuthentication no
  ChallengeResponseAuthentication no
  

• You might then also want to start a normal sshd server on port 2222 (which can be firewalled to restrict access); this might be used by local administrators. You can accomplish this by adding to: /etc/ssh/sshd_config the line: Port 2222. You will then need to do: chkconfig --add sshd; service sshd start
• hosts.deny/allow can be used to simply restrict access to both services (sshd and gsisshd)

If You Need To Re-Install The VDT Components

• Login as root and do: vdt-control --force --off
• Then: mv /opt/vdt /opt/vdt.`date +%s`
• And: rm -f /etc/grid-security/prima-authz.conf
• You can then set http_proxy if appropriate and re-run the build script as outlined above

Experimental

These sections need to be reviewed and formatted correctly for the Trac wiki

Adding pciback support

• What is it?  pciback is a means to get a DomU to use a PCI interrupt rather than going via Dom0.
• Value proposition? Achieving close to wire speed from a !DomU.
• Tell the Dom0 to keep off a given PCI id by adding this in /etc/modprobe.conf in Dom0

     # These two lines are to ensure that before the BNX2 network driver is
     # loaded, pciback gets a chance to hide the devices that should be
     # directly used in Xen domUs. -JAO, 2007-04-18
     options pciback hide=(0000:09:00.0)
     install bnx2 /sbin/modprobe pciback ; /sbin/modprobe --ignore-install bnx2
  

• On our box the requisite PCI id is 09:00.0
• The Dell 1950 uses a newer Broadcomm bnx2 chipset
• The tg3 driver used by the IBM gateway box has "pci quirkiness" which is handled by the the =/etc/xen/xend-pci-quirks.sxp= file

• Confirmation from dmesg on dom0

     ...
     pciback 0000:09:00.0: seizing device
     ACPI: PCI Interrupt 0000:09:00.0[A] -> GSI 16 (level, low) -> IRQ 16
  
     ACPI: PCI interrupt for device 0000:09:00.0 disabled
     input: PC Speaker as /class/input/input4
     Broadcom NetXtreme II Gigabit Ethernet Driver bnx2 v1.4.44-1 (August 10, 2006)
     ACPI: PCI Interrupt 0000:05:00.0[A] -> GSI 16 (level, low) -> IRQ 16
     eth0: Broadcom NetXtreme II BCM5708 1000Base-T (B1) PCI-X 64-bit 133MHz found at
     mem f8000000, IRQ 16, node addr 001372fb7ab1
  

• We need to tell the !DomU about the PCI ID we've hijacked via pciback. This is done by editing the appropriate =/etc/xen/vmname.init= file i.e. add the following in the =/etc/xen/NGData= file —

      pci = [ '0000:09:00.0' ]
  

• Edit the appropriate /etc/sysconfig/network-scripts/ifcfg-eth{0,1} file and add the MAC address of the network card who's PCI id was hijacked by pciback i.e. =/etc/sysconfig/network-scripts/ifcfg-eth1=

     # Broadcom Corporation NetXtreme II BCM5708 Gigabit Ethernet
     DEVICE=eth1
     ONBOOT=yes
     HWADDR=00:13:72:fb:7a:b3
     BOOTPROTO=static
     IPADDR=192.43.239.49
     NETMASK=255.255.255.0
     TYPE=Ethernet
  

• dmesg on !DomU

      ...
     netfront: Initialising virtual ethernet driver.
     netfront: device eth0 has flipping receive path.
     ADDRCONF(NETDEV_UP): eth1: link is not ready
     bnx2: eth1 NIC Link is Up, 1000 Mbps full duplex
     ADDRCONF(NETDEV_CHANGE): eth1: link becomes ready
     eth0: no IPv6 routers present
     eth1: no IPv6 routers present
     ...