Tuesday, February 3, 2015

GLBP protocol part two





GLBP Protocol  part two.


I decided to break GLBP notes on two parts. The first part of GLBP notes was about protocol functioning with some packet captures and debugging. This part will cover testing of GLBP in production.



Pic.1 Topology.






GLBP in production

As mentioned on the topology above, we have multilayer switch and default gateway configured to virtual IP of GLBP group 1 10.12.10.100. 
We will ping Loopback interface of router R2 with IP address 2.2.2.2 as a connectivity test with source IP address 10.12.10.12.

SWITCH_2#sh ip route
C       10.12.10.0 is directly connected, Vlan12
S*   0.0.0.0/0 [1/0] via 10.12.10.100

First look at GLBP configuration on GLBP routers:


R1
R1#sh run int fa 0/1

interface FastEthernet0/1
mac-address 0011.1111.1111
ip address 10.12.10.1 255.255.255.0
duplex full
speed 100
glbp 1 ip 10.12.10.100
end

R4
R4#sh run int fa 0/1

interface FastEthernet0/1
mac-address 0044.4444.4444
ip address 10.12.10.4 255.255.255.0
duplex full
speed 100
glbp 1 ip 10.12.10.100
end
R3
R4#sh run int fa 0/1

interface FastEthernet0/1
mac-address 0044.4444.4444
ip address 10.12.10.4 255.255.255.0
duplex full
speed 100
glbp 1 ip 10.12.10.100
end

R2#sh run int lo2
interface Loopback2
 ip address 2.2.2.2 255.255.255.255
end


Verify GLBP with "show" command:
R1#sh glbp
FastEthernet0/0 - Group 1
  State is Listen  <---- Local router is in the "Listen" GLBP state       
    11 state changes, last state change 11:47:14
  Virtual IP address is 10.12.10.100 
 <----- IP address protected by this router
  Hello time 3 sec, hold time 10 sec    <--- Configured  AVG timers (inherited from the AVG)
    Next hello sent in 1.532 secs
  Redirect time 10 sec, forwarder timeout 610 sec    <----- configured AVF timers ( inherited from the AVG) 
  Preemption disabled                                                                          
  Active is 10.12.10.4, priority 100 (expires in 9.552 sec)           <--- IP address of AVG
  Standby is 10.12.10.3, priority 100 (expires in 5.188 sec)             <--- IP address of ASG
  Priority 100 (default)                                                                        <--- local AVG priority
  Weighting 30, low (configured 100), thresholds: lower 40, upper 100     <---- 30 is a weight of  the local forwarder
    Track object 1 state Down decrement 70    <----here you can see tracking information related to local GLBP group
  Load balancing: round-robin
  Group members:
    0011.1111.1111 (10.12.10.1) local
    0033.3333.3333 (10.12.10.3)
    0044.4444.4444 (10.12.10.4)
  There are 3 forwarders (1 active)    <---- "1 active" means that our router is responsible for one mac address
  Forwarder 1
    State is Active                    
      21 state changes, last state change 00:49:22
    MAC address is 0007.b400.0101 (default)
    Owner ID is 0011.1111.1111
    Preemption enabled, min delay 30 sec
    Active is local, weighting 30
  Forwarder 2                      
    State is Listen               <--means that our router is watching to this address and can pick it up in case this forwarder will go down
    MAC address is 0007.b400.0102 (learnt)
    Owner ID is 0033.3333.3333
    Time to live: 604.356 sec (maximum 610 sec)
    Preemption enabled, min delay 30 sec
    Active is 10.12.10.3 (primary), weighting 200 (expires in 4.352 sec)  
  Forwarder 3
    State is Listen
    MAC address is 0007.b400.0103 (learnt)
    Owner ID is 0044.4444.4444
    Time to live: 608.712 sec (maximum 610 sec)
    Preemption enabled, min delay 30 sec   <---After 30 seconds. Forwarder 3 will pick up our virtual mac-address in case our router will go down
    Active is 10.12.10.4 (primary), weighting 100 (expires in 8.708 sec)



ARP table on multilayer switch:
SWITCH_2#sh arp | i 10.12.10.1|Add
Protocol  Address          Age (min)  Hardware Addr   Type   Interface
Internet  10.12.10.12             -   0012.d994.fdc4  ARPA   Vlan12

Let's ping IP address of R2:
SWITCH_2#ping 2.2.2.2
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 2.2.2.2, timeout is 2 seconds:
.!!!!
Success rate is 80 percent (4/5), round-trip min/avg/max = 25/46/93 ms

We can see arp entry in arp table:
SWITCH_2#sh arp | i 10.12.10.1|Add
Protocol  Address          Age (min)  Hardware Addr   Type   Interface
Internet  10.12.10.12             -   0012.d994.fdc4  ARPA   Vlan12
Internet  10.12.10.100            0   0007.b400.0103  ARPA   Vlan12
SWITCH_2#


Now we will manually "shut" and "no shut"  interface Vlan 12 (which IP is the source IP address for ICMP request) and look at ARP table:

SWITCH_2#sh arp | i Add|10.12.10.1
Protocol  Address          Age (min)  Hardware Addr   Type   Interface
Internet  10.12.10.12             -   0012.d994.fdc4  ARPA   Vlan12
Internet  10.12.10.100           73   0007.b400.0103  ARPA   Vlan12
SWITCH_2(config)#int vl 12
SWITCH_2(config-if)#shut
SWITCH_2(config-if)#no shut

*Apr 19 01:31:09.367: %LINK-5-CHANGED: Interface Vlan12, changed state to administratively down
*Apr 19 01:31:09.376: %LINEPROTO-5-UPDOWN: Line protocol on Interface Vlan12, changed state to down
*Apr 19 01:31:11.439: %LINK-3-UPDOWN: Interface Vlan12, changed state to up
*Apr 19 01:31:11.448: %LINEPROTO-5-UPDOWN: Line protocol on Interface Vlan12, changed state to up

SWITCH_2(config-if)#do sh arp | i Add|10.12.10.1
Protocol  Address          Age (min)  Hardware Addr   Type   Interface
Internet  10.12.10.12             -   0012.d994.fdc4  ARPA   Vlan12
Internet  10.12.10.100            0   0007.b400.0102  ARPA   Vlan12
SWITCH_2(config-if)#shut
SWITCH_2(config-if)#no shut
SWITCH_2(config-if)#do sh arp | i Add|10.12.10.1
Protocol  Address          Age (min)  Hardware Addr   Type   Interface
Internet  10.12.10.12             -   0012.d994.fdc4  ARPA   Vlan12
Internet  10.12.10.100            0   0007.b400.0101  ARPA   Vlan12     
<---- Mac address changes every time
SWITCH_2(config-if)#shut
SWITCH_2(config-if)#no shut
SWITCH_2(config-if)#do sh arp | i Add|10.12.10.1
Protocol  Address          Age (min)  Hardware Addr   Type   Interface
Internet  10.12.10.12             -   0012.d994.fdc4  ARPA   Vlan12
Internet  10.12.10.100            0   0007.b400.0102  ARPA   Vlan12
We have different IP  virtual mac address every time when SVI_12 Up/Down. Round robin scheme in work.


GLBP forwarding preemption

Now we will manually shut interface to current forwarder  :
First look at current ARP entry for IP address 10.12.10.100

SWITCH_2#sh arp | i Addr|10.12.10.10
Protocol  Address          Age (min)  Hardware Addr   Type   Interface
Internet  10.12.10.100            0   0007.b400.0101  ARPA   Vlan12
SWITCH_2#

0007.b400.0101 is the mac address of R4:
R4(config-if)#do sh glbp bri
Interface   Grp  Fwd Pri State    Address         Active router   Standby router
Fa0/1       1    -   100 Standby  10.12.10.100    10.12.10.3      local
Fa0/1       1    1   -   Active   0007.b400.0101  local           -
Fa0/1       1    2   -   Listen   0007.b400.0102  10.12.10.3      -
Fa0/1       1    3   -   Listen   0007.b400.0103  10.12.10.1      -

 
Now let's shut R4's GLBP interface configured and look who will preempt is's virtual mac address:
Turn on GLBP debugging
R4(config-if)#do deb glbp terse
GLBP:
  GLBP Errors debugging is on
  GLBP Events debugging is on
    (protocol, redundancy, track)
  GLBP Packets debugging is on
    (Request, Reply)





Do the ping test :
Type escape sequence to abort.
Sending 100, 100-byte ICMP Echos to 2.2.2.2, timeout is 2 seconds:
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!....!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
Success rate is 96 percent (96/100), round-trip min/avg/max = 16/30/75 ms
SWITCH_2#


R4's interface "shutdown":
R4(config)#int fa 0/1
R4(config-if)#
R4(config-if)#
R4(config-if)#shut
R4(config-if)#

*Jan 23 11:15:27.459: GLBP: Fa0/1 Interface down
*Jan 23 11:15:27.463: GLBP: Fa0/1 1.1 Active: e/Forwarder disabled
*Jan 23 11:15:27.463: GLBP: Fa0/1 1.1 Active -> Init           <--GLBP stops      
*Jan 23 11:15:27.463: %GLBP-6-FWDSTATECHANGE: FastEthernet0/1 Grp 1 Fwd 1 state Active -> Init
*Jan 23 11:15:27.475: GLBP: Fa0/1 1.2 Listen: e/Forwarder disabled
*Jan 23 11:15:27.475: GLBP: Fa0/1 1.2 Listen -> Init
*Jan 23 11:15:27.475: GLBP: Fa0/1 1.3 Listen: e/Forwarder disabled
*Jan 23 11:15:27.475: GLBP: Fa0/1 1.3 Listen -> Init
*Jan 23 11:15:27.479: GLBP: Fa0/1 1 Standby: e/GLBP disabled
*Jan 23 11:15:27.479: GLBP: Fa0/1 1 Active router IP is unknown, was 10.12.10.3  
<--R4 lost GLBP active router
*Jan 23 11:15:27.479: GLBP: Fa0/1 1 Standby router is unknown, was local   <-- R4 was standby router
*Jan 23 11:15:27.479: GLBP: Fa0/1 1 Standby -> Init
*Jan 23 11:15:29.459: %LINK-5-CHANGED: Interface FastEthernet0/1, changed state to administratively down
*Jan 23 11:15:30.459: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet0/1, changed state to down




Look at debugging on other routers in this GLBP group R1 and R3:

R1#
*Jan 23 11:25:39.711: GLBP: Fa0/1 1 Listen: f/Standby timer expired (10.12.10.4)
*Jan 23 11:25:39.711: GLBP: Fa0/1 1 Standby router is unknown, was 10.12.10.4
 <--we lost R4 as a GLBP neighbor
*Jan 23 11:25:39.711: GLBP: Fa0/1 1 Listen -> Speak
*Jan 23 11:25:40.099: GLBP: Fa0/1 1.1 Ignoring Hello (135/10.12.10.3 < 167/10.12.10.4)
*Jan 23 11:25:40.891: GLBP: Fa0/1 1.1 Ignoring Hello (135/10.12.10.3 < 167/10.12.10.4) 
<---Ignore this GLBP hellos because R4 is no longer a GLBP neighbor
*Jan 23 11:25:41.439: GLBP: Fa0/1 1.1 Listen: g/Active timer expired
*Jan 23 11:25:41.439: GLBP: Fa0/1 1.1 Listen -> Active
*Jan 23 11:25:41.439: %GLBP-6-FWDSTATECHANGE: FastEthernet0/1 Grp 1 Fwd 1 state Listen -> Active 
<--Became an active forwarder for R4 
*Jan 23 11:25:43.907: GLBP: Fa0/1 1.1 Active: i/Hello rcvd from higher pri Active router (135/10.12.10.3)  <--R3 is more prefered as backup forwarder for R4's virtual mac address
*Jan 23 11:25:43.911: GLBP: Fa0/1 1.1 Active -> Listen 
*Jan 23 11:25:43.911: %GLBP-6-FWDSTATECHANGE: FastEthernet0/1 Grp 1 Fwd 1 state Active -> Listen
*Jan 23 11:25:49.727: GLBP: Fa0/1 1 Speak: f/Standby timer expired (unknown)
*Jan 23 11:25:49.727: GLBP: Fa0/1 1 Standby router is local
*Jan 23 11:25:49.727: GLBP: Fa0/1 1 Speak -> Standby  <-- became the GLBP standby router


R3#
*Jan 23 11:23:39.283: GLBP: Fa0/1 1 Standby router is unknown, was 10.12.10.4
*Jan 23 11:23:39.635: GLBP: Fa0/1 1.1 Listen: g/Active timer expired
*Jan 23 11:23:39.635: GLBP: Fa0/1 1.1 Listen -> Active
*Jan 23 11:23:39.635: %GLBP-6-FWDSTATECHANGE: FastEthernet0/1 Grp 1 Fwd 1 state Listen -> Active
*Jan 23 11:23:41.031: GLBP: Fa0/1 1.1 Active: j/Hello rcvd from lower pri Active router (135/10.12.10.1)
*Jan 23 11:23:42.139: GLBP: Fa0/1 1.1 Active: j/Hello rcvd from lower pri Active router (135/10.12.10.1)
*Jan 23 11:23:49.327: GLBP: Fa0/1 1 Standby router is 10.12.10.1
R3#sh glbp brief
Interface   Grp  Fwd Pri State    Address         Active router   Standby router
Fa0/1       1    -   100 Active   10.12.10.100    local           10.12.10.1
Fa0/1       1    1   -   Active   0007.b400.0101  local           -  
<- two active virtual mac addresses
Fa0/1       1    2   -   Active   0007.b400.0102  local           -
Fa0/1       1    3   -   Listen   0007.b400.0103  10.12.10.1      -  <- it is R1 


Now we will shut R3's interface and remain only one GLBP gateway (simulate R3 crash):


R3(config)#int fa 0/1
R3(config-if)#shut

*Jan 23 11:54:03.623: %GLBP-6-STATECHANGE: FastEthernet0/1 Grp 1 state Active -> Init
*Jan 23 11:54:05.599: %LINK-5-CHANGED: Interface FastEthernet0/1, changed state to administratively down
*Jan 23 11:54:06.599: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet0/1, changed state to down

We can see above, that R1  became active GLBP forwarder or all 3 routers:
R1#
*Jan 23 11:56:12.823: GLBP: Fa0/1 1.1 Listen: g/Active timer expired
*Jan 23 11:56:12.823: GLBP: Fa0/1 1.1 Listen -> Active
*Jan 23 11:56:12.823: %GLBP-6-FWDSTATECHANGE: FastEthernet0/1 Grp 1 Fwd 1 state Listen -> Active
*Jan 23 11:56:13.015: GLBP: Fa0/1 1.2 Listen: g/Active timer expired
*Jan 23 11:56:13.015: GLBP: Fa0/1 1.2 Listen -> Active
*Jan 23 11:56:13.015: %GLBP-6-FWDSTATECHANGE: FastEthernet0/1 Grp 1 Fwd 2 state Listen -> Active
*Jan 23 11:56:13.535: GLBP: Fa0/1 1 Standby: g/Active timer expired (10.12.10.3)
*Jan 23 11:56:13.535: GLBP: Fa0/1 1 Active router IP is local, was 10.12.10.3
*Jan 23 11:56:13.535: GLBP: Fa0/1 1 Standby router is unknown, was local
*Jan 23 11:56:13.535: GLBP: Fa0/1 1 Standby -> Active  
<---R1 is the AVG now
*Jan 23 11:56:13.539: %GLBP-6-STATECHANGE: FastEthernet0/1 Grp 1 state Standby -> Active


Verify GLBP:
R1#sh glbp brief
Interface   Grp  Fwd Pri State    Address         Active router   Standby router
Fa0/1       1    -   100 Active   10.12.10.100    local           unknown
Fa0/1       1    1   -   Active   0007.b400.0101  local           -
Fa0/1       1    2   -   Active   0007.b400.0102  local           -
Fa0/1       1    3   -   Active   0007.b400.0103  local           -

R1#sh glbp
FastEthernet0/1 - Group 1
  State is Active
    10 state changes, last state change 00:04:10
  Virtual IP address is 10.12.10.100
  Hello time 3 sec, hold time 10 sec
    Next hello sent in 2.656 secs
  Redirect time 600 sec, forwarder timeout 14400 sec
  Preemption disabled
  Active is local
  Standby is unknown
  Priority 100 (default)
  Weighting 100 (default 100), thresholds: lower 1, upper 100
  Load balancing: round-robin
  Group members:
    0011.1111.1111 (10.12.10.1) local
  There are 3 forwarders (3 active)
  Forwarder 1
    State is Active
      9 state changes, last state change 00:04:11
    MAC address is 0007.b400.0101 (learnt)
    Owner ID is 0044.4444.4444
    Redirection disabled
    Time to live: 12305.408 sec (maximum 12567 sec)
    Preemption enabled, min delay 30 sec
    Active is local, weighting 100
  Forwarder 2
    State is Active
      1 state change, last state change 00:04:11
    MAC address is 0007.b400.0102 (learnt)
    Owner ID is 0033.3333.3333
    Redirection enabled, 337.824 sec remaining (maximum 600 sec)
    Time to live: 14137.824 sec (maximum 14400 sec)
    Preemption enabled, min delay 30 sec
    Active is local, weighting 100
  Forwarder 3
    State is Active
      7 state changes, last state change 00:58:27
    MAC address is 0007.b400.0103 (default)
    Owner ID is 0011.1111.1111
    Redirection enabled
    Preemption enabled, min delay 30 sec
    Active is local, weighting 100



We can see above after forwarder timeout expired GLBP Active gateway stops protection:

R3(config-if)#
*Jan 22 05:16:27.011: %GLBP-6-FWDSTATECHANGE: FastEthernet0/1 Grp 1 Fwd 3 state Active -> Disabled
R3(config-if)#
*Jan 22 05:17:06.031: %GLBP-6-FWDSTATECHANGE: FastEthernet0/1 Grp 1 Fwd 1 state Active -> Disabled
R3(config-if)#