\documentclass[10pt,oneside,twocolumn,a4paper]{article}
\pagestyle{plain}

\ifx\pdfoutput\undefined
  \usepackage[dvips]{graphicx}
\else
  \usepackage{thumbpdf}
  \usepackage[pdftex]{hyperref,graphicx}
  \hypersetup{colorlinks,citecolor=black,filecolor=black,linkcolor=black,urlcolor=black,pdftex}
  \hypersetup{pdftitle=Linux Interface Bonding}
  \hypersetup{pdfauthor=P.M.J. de Bruijn}
  \pdfcompresslevel=9
\fi

\usepackage{fancyvrb}

\usepackage[english]{babel}

\usepackage{bookman}
\usepackage{courier}


\title{\textbf{Linux Interface Bonding}}
\author{\href{mailto:keizerflipje@home.nl}{P.M.J. de Bruijn}}


\begin{document}
  \maketitle

  \section*{License}
    Copyright  2005 by \copyright \ \href{mailto:keizerflipje@home.nl}{P.M.J. de Bruijn}.
    This material may be distributed only subject to the terms and conditions set
    forth in the Open Publication License, v1.0 or later (the latest version is
    presently available at
    \href{http://www.opencontent.org/openpub/}{http://www.opencontent.org/openpub/}).

  \tableofcontents

  \section{Introduction}
    Do you have a really, really important server? Of course you do. Now
    wouldn't it be nice if we were able to protect that server from network
    failures. Network failures you ask? Well a lot can go wrong with your
    network equipment. Switches and hubs can fail. Network cards can die.
    
    Fortunately the RedHat Enterprise Linux operating system is capable of
    mitigating the effects of these failures. The solution to our headaches
    is called bonding.
    
    The Linux bonding driver also has load-balancing capabilities, but they
    are beyond the scope of this article.

  \section{Bonding, In Theory}
    The basic concept behind bonding is that two (or more) become one. You'll
    basically merge two physical NICs into a single logical NIC. The physical
    NICs will be called the slave interfaces. The logical NIC controls all
    the slave interfaces and thus is called the master interface.

    When the logical interface is brought up, the bonding driver brings up
    the first physical interface in the team. The logical interface uses it's
    own MAC address and will respond to all incoming traffic destined for
    it.

    The master interface polls the active slave interface on a regular basis,
    to see whether network connectivity is still present. If the bonding
    driver determines that active slave interface went down, it quickly
    assigns the secondary slave interface the MAC address previously used by
    the primary slave interface. The interface is then brought up, and in turn 
    becomes the active slave interface. It then begins accepting traffic.

  \section{Bonding, In Practice}

    \subsection{Hardware}
      \subsubsection{Switches}
        You can setup a NIC team to operate on a single switch, which will
        only protect you from NIC or cable failure. For maximum effectiveness
        you'll also need three (un)managed switches. One switches will act
        as a network backbone on the core layer. The other two will be the
        distribution/access layer switches. Ofcourse the backbone switch
        will still present a single point of failure into your network, this
        can be compensated for, by buying a modular switch, which can hotswap
        broken components.

        \begin{center}
          \includegraphics[width=0.75\columnwidth]{network}
        \end{center}

      \subsubsection{NICs}
        To take proper advantage of bonding you'll need a server with at least
        two NICs, preferably (but not nessecarily) the same.

        I used two Compaq NC3121 NICs, which are powered by Intel 825528B
        chips. Fortunately Intel maintains it's e100 driver well.

        Keep in mind that the point of having transparent failover is avoiding
        user complaints. When using two different NICs, you risk not knowing how
        the secondary NIC will act when called upon.

    \subsection{Software}
      \subsubsection{Distribution}
        For this article I'm using CentOS 3.4, which in turn is based on
        RedHat Enterprise Linux 3 Update 4. But the steps provided here
        should work on any RedHat like Linux distribution.

      \subsubsection{Driver}
        The first and most important thing to do is make sure you load
        the bonding driver at boot time. For this we need to modify the
        the /etc/modules.conf file, and adjust it to look like this:

        \begin{Verbatim}[frame=lines,framesep=5pt,fontsize=\footnotesize,label=modules.conf]
alias bond0 bonding
options bond0 mode=active-backup miimon=100
alias eth0 e100
alias eth1 e100
        \end{Verbatim}

        The alias statements make sure the device files are associated
        with the proper drivers. Make sure you alias the bonding driver
        before you alias another NIC driver, this is important when
        using an SNMP agent. The options statement will tell the bonding
        driver what mode we will use and at what interval (in ms) to
        monitor our NICs link status.

        For more information on the mode parameter please review the
        bonding.txt document which should be supplied with the kernel
        sources of your distribution.

      \subsubsection{Configuration}
        After the driver is properly loaded, we can start configuring
        our boot scripts.

        Each interface is configured through a corresponding ifcfg-ifname
        file located in /etc/sysconfig/network-scripts/. We'll start by
        configuring our master interface bond0:

        \begin{Verbatim}[frame=lines,framesep=7pt,xleftmargin=7pt,xrightmargin=7pt,fontsize=\footnotesize,label=ifcfg-bond0]
DEVICE=bond0
BOOTPROTO=static
ONBOOT=yes
BROADCAST=192.168.2.255
IPADDR=192.168.2.13
NETMASK=255.255.255.0
GATEWAY=192.168.2.1
USERCTL=no
TYPE=Ethernet
        \end{Verbatim}

        Next the configuration of our first physical interface:

        \begin{Verbatim}[frame=lines,framesep=5pt,fontsize=\footnotesize,label=ifcfg-eth0]
DEVICE=eth0
BOOTPROTO=none
MASTER=bond0
SLAVE=yes
USERCTL=no
TYPE=Ethernet
        \end{Verbatim}

        Our failover physical interface:

        \begin{Verbatim}[frame=lines,framesep=5pt,fontsize=\footnotesize,label=ifcfg-eth1]
DEVICE=eth1
BOOTPROTO=none
MASTER=bond0
SLAVE=yes
USERCTL=no
TYPE=Ethernet
        \end{Verbatim}

        As you might have noticed the slave interfaces have no IP addresses
        assigned to them, because they inherit the IP address of their
        master interface.

      \subsubsection{Activating}
        Everything should now be ready. You can use the normal ifup command
        to activate our newly created NIC team:

        \begin{Verbatim}[frame=lines,framesep=5pt,fontsize=\footnotesize]
#ifup bond0
ip_tables: (C) 2000-2002 Netfilter core team
Enslaving eth0 to bond0
Enslaving eth1 to bond0
e100: eth0 NIC Link is Up 100Mbps Full duplex
e100: eth1 NIC Link is Up 100Mbps Full duplex
#
        \end{Verbatim}
        
        As you can see, both eth0 and eth1 are enslaved to bond0, as they
        should be.
        
  \section{Production Use}
    \subsection{miimon}
      When using bonding in a production environment, you should carefully
      select the miimon interval. A smaller interval incurs more CPU load.

      \subsubsection{TCP}
        Most TCP based protocols should work perfectly using the 100ms
        miimon interval used in this article. Connection may stall for a
        split second when the server is changing NICs.

      \subsubsection{UDP}
        UDP based protocols are a whole different ball park. It depends
        heavily on the upper layer protocol whether loss of connectivity 
        and/or data will occur. UDP is a connectionless protocol,
        without a garantee of safe delivery. Make sure you carefully
        test all your mission critical applications based on UDP.

    \subsection{Network}
      When using the multi-tier network approach presented here, you should
      also be aware that it may take the switches some time to notice the
      changes in the network that are occuring. 
      
      If your server does failover to it's second NIC, the second switch it's 
      connected to will be briefly confused. The switch has the same MAC address
      available on it's upstream link. 
      
      The same thing goes for the backbone switch, the MAC address will be seen 
      briefly on two different downstream ports. Eventually the old port will 
      age the MAC address out of it's MAC database, and connectivy will resume. 
      Lucky for us, this all happens within a couple of ms, and nobody will
      notice, if all goes well.


  \section{Conclusion}
    Linux interface bonding allows us to provide reliable network connectivity
    to any given Linux server without incurring the cost of an add-on software
    product.

\end{document}
