Chapter 1 Need for Overlays in Massive Scale Data Centers 1 Evolution of the Data Center 1 Changing Requirements of Data Centers 4 Data Center Architectures 6 CLOS 8 Fat-Tree 9 Single Fabric 9 Need for Overlays 10 Summary 15 References 15
Chapter 2 Introduction to Overlay Technologies 19 Overlay Technologies Overview 20 Cisco FabricPath 22 FabricPath Requirements 22 FabricPath Benefits 23 FabricPath Architecture 24 FabricPath Encapsulation 24 FabricPath Data Plane Operation 25 TRILL 26 TRILL Requirements 27 TRILL Frame Format 28 TRILL Data Plane Operation 28 Locator ID/Separator Protocol 30 LISP Frame Format 30 LISP Routing 30 VXLAN 32 VXLAN Frame Format 33 VXLAN Data Path Operation 34 NVGRE 35 NVGRE Frame Format 36 NVGRE Data Path Operation 36 Overlay Transport Virtualization 38 OTV Frame Format 39 OTV Operation 40 Provider Backbone Bridges (PBB) 41 Shortest Path Bridging 43 Shortest Path Bridging MAC 43 Shortest Path Bridging VID 45 Summary 47 References 47
Chapter 3 IS-IS 49 Introduction to IS-IS 49 Concepts 50 Neighbor Discovery 51 Topology Exchange 51 Flooding 51 Route Computation 52 Link State Protocol Scaling 52 Link State Protocol in a Local Area Network 53 IS-IS Architecture Details 55 TRILL and FabricPath Specific Changes in IS-IS 56 Overview of TRILL and FabricPath 57 IS-IS Frame Formats 58 Router Capability TLV 59 Multitopology-Aware Port Capability TLV 59 TRILL IS-IS Neighbor Discovery 59 TRILL HELLOs 60 P2P HELLOs 63 TRILL Neighbor TLV 64 Router Capability Sub-TLVs 64 Multitopology-Aware Port Capability Sub-TLVs 64 Area Address TLV 67 Protocols Supported TLV 67 TRILL and FabricPath Topology Exchange 67 Flooding 69 Nickname or SwitchID Resolution 70 Shortest Path Computation 71 Distribution Trees Computation 71 Pruning the Distribution Tree 74 ESADI 77 MAC Reachability TLV 78 Fine Grained Labeling 79 Pseudo Node 81 Multi Topology Routing 83 Summary 84 References 84 Additional Resources 84
Chapter 4 FabricPath 85 FabricPath Overview 86 FabricPath Architecture 87 Core and Edge 88 Addressing Concepts 89 VLANs 89 vPC+ 89 FabricPath Encapsulation 91 FabricPath Control Plane Protocols 93 IGMP Snooping in FabricPath Multicast Networks 96 FabricPath Dynamic Resource Allocation Protocol 97 Allocation of Resources by DRAP 97 FabricPath MAC Address Learning 98 Control Plane Learning 98 Data Plane Learning 98 FabricPath STP Interaction 102 Topology Change Notifications Forwarding 105 FabricPath Packet Forwarding 106 Broadcast: ARP Request 108 Unicast: ARP Reply 111 Unicast: Data 113 IP Multicast Forwarding 116 FabricPath Basic Configuration 119 FabricPath Benefits 121 Summary 122 References 122
Chapter 5 TRILL 123 Need for TRILL 124 Spanning Tree in Layer 2 Networks 124 Issues with Spanning Tree Protocol 126 Virtual Switching System 127 Giant Virtual Switch 128 Flat Network 129 Layer 3 Network 130 Concepts and Terminologies 130 RBridge 131 Appointed Forwarder 132 Addressing Concepts 133 TRILL Frame Format 134 TRILL Control Plane 136 Unicast 136 Multicast 137 Pruning 139 TRILL Data Plane 141 Unicast 141 Ingress RBridge Processing 141 Processing of TRILL Packets 143 Multidestination 143 Ingress Processing 144 Core and Egress Processing 146 Egress Processing 146 MAC Address Learning in TRILL-Based Networks 147 Dynamic Learning 147 Learning Through Control Protocols 147 Work in Progress 148 Multitopology Routing 148 Fine-Grained Labeling 149 Ingress RBridge 152 Core RBridge 152 Egress RBridge 152 Pseudo Node 152 Choosing a Pseudo Nickname 154 Multiple Distribution Trees per Member RBridge 156 Synchronization of MAC Addresses 158 Case Studies 159 Bidirectional Packet Flow 159 Traffic from H1 to H2 160 Traffic from H2 to H1 164 Packet Flow for Pseudo Node 167 Packet Originating from Host H1 170 Reverse Traffic from Host H2 to H1 172 Summary 174 References 174 Additional Resources 175
Chapter 6 VXLAN 177 VXLAN Overview 177 Advent of VXLAN 178 VXLAN Architecture 179 VXLAN Header Format 181 VXLAN Packet Forwarding 182 Broadcast: ARP Request 183 Unicast: ARP Reply 184 Unicast: Data 186 Unknown Unicast 187 VM Mobility Case 188 IPv6 Data Forwarding 190 NS Request and NA Response 191 VXLAN Gateway 192 Inter-VXLAN Communication 196 Layer 3 Multicast 198 Multicast-Less VXLAN 200 Floodless VXLAN Forwarding 203 VXLAN as a Network Overlay 205 Other VXLAN Considerations 207 VXLAN Basic Configuration 208 VXLAN Gateway Configuration 210 Summary 211 References 211
Chapter 7 FabricPath Deployment, Migration, and Troubleshooting 213 vPC 214 vPC Overview 214 vPC Terminology 215 vPC Benefits 216 vPC Deployment Scenarios 217 Double-Sided vPC 218 vPC Operations 219 vPC Traffic Flow 224 Cisco Fabric Services over Ethernet 225 vPC ARP Sync 225 vPC Peer Gateway 225 vPC Verification 227 vPC+ 231 vPC+ Overview 231 vPC+ Basics 232 vPC+ Basic Packet Flow 236 Active/Active HSRP Forwarding 238 FabricPath Topologies 241 Migration to FabricPath Network 242 Conversion from Classical Layer 2 to FabricPath Network 242 Conversion of vPC to vPC+ (Classical Ethernet to FabricPath) 244 Configuring vPC+ on Secondary Switch 246 Configuring vPC+ on Primary Switch 249 Conversion of Access Switch (Sw3) Connecting to Secondary (Sw2) to FabricPath 251 Converting Access Switch Sw3 Uplink Connecting to Sw1 to FabricPath 254 Monitoring and Troubleshooting in FabricPath Networks 257 Loopback Message 258 Path Trace Message 259 Multicast Trace Message 259 FabricPath OAM Configuration Model 261 Summary 270 References 270
Chapter 8 TRILL Deployment, Migration, and Troubleshooting 271 Introduction 271 TRILL Deployment 271 TRILL Between Access and Distribution 274 TRILL Core 274 Layer 2 Bridging Case 276 Layer 3 Routing Cases 277 Expanding the POD 285 TRILL Everywhere 286 Meshed Distribution with No Core 287 Link Aggregation or Pseudo-Node Deployments 287 CLOS Network Model 289 Migration Toward TRILL 290 TRILL and Spanning Tree 291 Appointed Forwarder Solution 292 Spanning Tree Solution 293 Bottom-up Migration Toward TRILL 296 Top-down Migration Toward TRILL 298 Monitoring and Troubleshooting in TRILL Networks 299 OAM Packet Format 300 Connectivity Verification 302 Path Tracing 303 TRILL Configuration Model 304 Summary 304 References 305
Chapter 9 Multi-Overlay Deployments 307 Overview 307 Case Study 1: TRILL or FabricPath Network with VXLAN to Virtualized Servers 309 Case Study 2: Data Center Interconnect Using OTV 315 Case Study 3: Interconnecting TRILL or FabricPath Data Centers Using OTV 321 Merging TRILL or FabricPath Networks 321 Independent TRILL or FabricPath Networks 323 Interconnection of TRILL and FabricPath Data Centers 325 Packet Flow 325 Summary 327 References 328
Sanjay Hooda, CCIE No. 11737, is currently a principal engineer at Cisco, where he works with embedded systems and helps define new product architectures. His current passion is to design the next-generation campus architecture, and he is focused on simplifying
the design and deployment of wired and wireless infrastructure. Over the last 17 years, Sanjay's experience spans various areas including high availability; messaging in large-scale distributed systems; Supervisory Control and Data Acquisition (SCADA); large-scale software projects; and enterprise campus and LAN,WAN, and data center network design.
Shyam Kapadia, Ph.D., is currently a technical leader in the Data Center Group at Cisco. He graduated from the University of Southern California with Ph.D. and master's degrees in computer science in 2006. His research interests broadly lie in the area of networking systems including wired, wireless, ad-hoc, vehicular, and sensor networks. He has co-authored several conference and journal publications in these areas including a book chapter in the relatively nascent area of intermittently connected wireless networks (http://anrg.usc.edu/~kapadia/publications.html). At Cisco, for the first few years, he was an integral part of the team that delivered the next-generation Catalyst 6500 Sup 2T platform. During the past few years, he has been intrinsically involved in developing solutions for data center environments with more than 25 submitted patents in this area. Over the past 12 years, Shyam has been the speakers chair for a premiere Open Source conference, Southern California Linux Exposition (SCALE), hosted in the Los Angeles area. In his spare time, he loves watching international movies and is passionate about sports like cricket, basketball, and American football.
Padmanabhan Krishnan is a software engineer in the Data Center Group at Cisco. He joined Cisco 7 years ago and has more than 12 years of experience in various areas of networking and telecommunication. He obtained his master's degree in computer science from the University of Missouri, Kansas City, and his bachelor's degree in engineering from Madras University, India. His research work for the master's degree included Diffserv, MPLS traffic engineering, and QOS routing/Connection Admission Control in ad-hoc wireless networks. Padmanabhan has worked in many overlay technologies in Cisco such as 802.1ah, TRILL, FabricPath, and VPLS. He was responsible for the design and development of the core infrastructure used by the forwarding drivers and many Layer 2 features in the next generation Catalyst 6500 Sup 2T Platform. Prior to joining Cisco, Padmanabhan worked in ATM signaling and DVB-RCS, an interactive on-demand multimedia satellite communication system specification.