Creating Highly Available,
Redundant,
Linux Systems

The %$@(&%^)# Customer

• We want redundant control processors.

The %$@(&%^)# Customer

• We want redundant control processors.

• But we only want to see one of them.

The %$@(&%^)# Customer

• We want redundant control processors.

• But we only want to see one of them.

• Unless we want to see the others.

Watchdog Timer

• If 'the application' doesn't hit the Watchdog, the processor reboots.

Boot Linux from Disk

• Check to see if the 'important' files are there.

• A processor isn't allowed to run if its file system isn't intact.

Boot Linux from Disk

• Check to see if the 'important' files are there.

• A processor isn't allowed to run if its file system isn't intact.

• Tripwire

  http://www.tripwire.com/

• AFICK

  http://afick.sourceforge.net/

Boot Linux from Disk

• Check to see if the 'important' files are there.

• A processor isn't allowed to run if its file system isn't intact.

• Tripwire

  http://www.tripwire.com/

• AFICK

  http://afick.sourceforge.net/

  • I picked AFICK because its written in Perl.

Intact File Systems ?

• If the AFICK check determines that the 'important' files are missing or corrupted, the system re-builds itself.

• Boot Linux from FLASH

Intact File Systems ?

• If the AFICK check determines that the 'important' files are missing or corrupted, the system re-builds itself.

• Boot Linux from FLASH

  • Re-build disk partitions.

  • Rsync disk from the 'master'.

  • Re-boot from disk.

What if the disk wasn't there ?

continued...

What if the disk wasn't there ?

• Boot Linux from FLASH

Boot Linux from FLASH

• a stripped down Linux

• Busybox

  http://www.busybox.net/

• stripped down Perl

Boot Linux from FLASH

• a stripped down Linux

• Busybox

  http://www.busybox.net/

• stripped down Perl

• disk re-builder

What if the FLASH isn't there ?

UBOOT

http://uboot.sourceforge.net/

continued...

UBOOT

http://uboot.sourceforge.net/

• Uboot runs and performs a BOOTP/DHCP request.

• The 'master' answers with an IP address and a filename.

• Retrieves the file (Linux-in-FLASH) and burns it into the FLASH.

• Re-boot.

UBOOT

http://uboot.sourceforge.net/

• Uboot runs and performs a BOOTP/DHCP request.

• The 'master' answers with an IP address and a filename.

• Retrieves the file (Linux-in-FLASH) and burns it into the FLASH.

• Re-boot.

  • Hopefully from disk... but probably from FLASH.

  • Unfortunately, its not written in Perl.

Synchronizing Disks

• Rsync

  http://samba.anu.edu.au/rsync/

Synchronizing Disks

• Rsync

  http://samba.anu.edu.au/rsync/

  • Its not written in Perl, but its still good software.

Synchronizing Disks

• Rsync

  http://samba.anu.edu.au/rsync/

  • Its not written in Perl, but its still good software.

  • 'Slave' processors periodically rsync with 'masters'.

High Availability Daemons

High Availability Daemons

• HA - High-Availability Linux Project

  http://www.linux-ha.org/

High Availability Daemons

• HA - High-Availability Linux Project

  http://www.linux-ha.org/

• Written in "C"

• Configurable

• Only supports 2 nodes

High Availability Daemons (cont.)

• Bongoyo - Simple High Availability

  http://bongoyo.sourceforge.net/

High Availability Daemons (cont.)

• Bongoyo - Simple High Availability

  http://bongoyo.sourceforge.net/

• Cross platform (Perl)

• Port checker

• Configurable

  • pre/post takeover/release

• Slow switchover

smphad

• Supervisory Management Processor 'High Availability' Daemon

smphad

• Supervisory Management Processor 'High Availability' Daemon

  • "To ensure there is 'one' processor that provides all the important facilites."

smphad

• Supervisory Management Processor 'High Availability' Daemon

  • "To ensure there is 'one' processor that provides all the important facilites."

  • Written in Perl.

smphad

• Supervisory Management Processor 'High Availability' Daemon

  • "To ensure there is 'one' processor that provides all the important facilites."

  • Written in Perl.

  • Does only what 'we' need.

  • Integrated with syslog and SNMP

Masters and Slaves

• All processors have their own IP address.

• Slaves and 'masters' announce themselves (broadcast UDP).

• Slaves that don't hear 'masters' try to become the 'master'.

• Lowest IP address or highest priority becomes the 'master'.

• If a 'master' hears another 'master' it becomes a 'slave'.

Masters and Slaves (cont.)

• The 'master' gets another address.

• 'Master' addresses are aliases.

• Send a Gratuitous ARP when you become the 'master'.

• Configuration files only exist on the 'master'.

Duties of a Master

• To stay the 'master'.

• BootP/DHCP daemon

• Rsync daemon

• Send SNMP traps when 'slaves' come or go.

• Send SNMP trap when I become the 'master'.

• Communicate configuration information to 'slaves'.

Duties of a Slave

• To become a 'master'.

• Hold off when a 'master' reboots.

• Enable Rsync client.

Aliases

• Adding

  $if_socket = IO::Socket::INET->new (Proto => 'udp');
  
  $if_socket->if_addr     ("eth0:1", $master_ipaddr);
  $if_socket->if_netmask  ("eth0:1", $master_netmask);
  $if_socket->if_broadcast("eth0:1", $master_bcast);
  

• Removing

  $flags = $if_socket->if_flags ("eth0:1");           # Remove the alias I/F by clearing
  $if_socket->if_flags ("eth0:1", $flags & ~IFF_UP);  # the IFF_UP bit from its status
  

Aliases (cont.)

root@SITE_1_S_5_8:~# ifconfig
eth0      Link encap:Ethernet  HWaddr 48:00:00:00:05:12
          inet addr:20.0.2.120  Bcast:20.63.255.255  Mask:255.192.0.0
          UP BROADCAST MULTICAST  MTU:1500  Metric:1
          RX packets:990672 errors:1 dropped:0 overruns:0 frame:0
          TX packets:312456 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:100
          RX bytes:1410916961 (1345.5 Mb)  TX bytes:108021725 (103.0 Mb)
          Interrupt:10 Base address:0x2000

eth0:1    Link encap:Ethernet  HWaddr 48:00:00:00:05:12
          inet addr:20.0.2.201  Bcast:20.63.255.255  Mask:255.192.0.0
          UP BROADCAST MULTICAST  MTU:1500  Metric:1
          Interrupt:10 Base address:0x2000

Gratuitous ARP

use Net::GARP;

Net::GARP::send_garp("eth0:1");

BootStrap Sequencing

Thank You