Fat-Tree Networks and Data Center Networking

14-848: CLOUD INFRASTRUCTURE DATA CENTER NETWORKING LECTURE 5 * FALL 2019 (KESDEN)

REMEMBER SOCRATIVE https://api.socrative.com/rc/Nfu6Lp

FAT-TREE NETWORKS More throughput at higher levels, more even across levels Not easy to do since buying more powerful switches is harder To extent possible, more cost per unit capacity Not possible beyond a modest point This is somewhat necessarily the case as, if bigger switches were more readily available and economical, they d be used at the bottom, and we d be back where we started.

FAT-TREES WITH SKINNY SWITCHES: GOALS Use all commodity switches Full throughput from host-to-host Compatible with usual TCP/IP stack Better energy efficiency per unit throughput from more smaller switches than fewer bigger switches

FAT TREE (K=4) Note the replacement of aggregation layer switches with 2 layers of K/2 K-port switches (K/2)2 core routers (K/2)2 servers per pod K-port switches support K3/4 servers

FAT TREE DETAILS K-ary fat free: three layers (core, aggregation, edge) Each pod consists of (K/2)2 servers and 2 layers of K/2 K-port switches. Each edge switch connects (K/2) servers to (K/2) aggregator switches Each aggregator switch connects (K/2) edge and (K/2) core switches (K/2)2 core switches, each ultimately connecting to K pods Providing K different roots, not 1. Trick is to pick different ones K-port switches support K3/4 servers/host: (K/2 hosts/switch * K/2 switches per pod * K pods)

USING MULTIPLE PATHS Must pick different paths ( path diversity ) or will have a hotspot Unless sessions use the same path, reordering will be a problem and need to be resolved with buffering higher up Static paths may not respond to actual, dynamic workloads Can be done at different levels. Higher levels, e.g. transport, are more flexible, but likely more effort and slower Lower levels are likely less adaptive, but simpler and faster. Ability to weight or remove paths can aid fault tolerance

PORTLAND SOLUTION Use commodity switches and off-load services into software on commodity server Start With Fat Tree for a topology without hot spots Use layer-2 to avoid routing, forwarding, and related complexity Separate host identifier from host location IP addresses identify host, but not location, just and ID Use Pseudo MAC Address to identify location at Level-2

PORTLAND ADDRESSES Normally MAC addresses are arbitrary no clue about location IP normally is hierarchical, but here we are using it only as a host identifier If MAC addresses are not tied to location, switch tables grow linearly with growth of network, i.e. O(n) PortLand uses hierarchical MAC addresses, called Pseudo MAC or PMAC addresses to provide for switch location <pod:position:port:vmid> <16,8,8,16> bits

PORTLAND PMAC ADDRESSES 1 0 3 2 Position 0 0 1 1 0 1 PMAC: <pod.position.port.vmid> 48 bits: <16-bits.8-bits.8-bits.16-bits>

0 1 PORTLAND PMAC ADDRESSES 3 2 <pod:position:port:vmid> <16,8,8,16> bits

NAME RESOLUTION: MACPMACIP End hosts continue to use Actual MAC (AMAC) addresses Switches convert PMAC<->AMAC for the host Edge switch responsible for creating PMAC:AMAC mapping and telling Fabric Manager Software on commodity server, can be replicated, etc. Simplicity is a virtue. Mappings timed out of Fabric Manager s cache, if not used. ARPs are for PMACs First ask fabric manager which keeps cache. Then, if needed, broadcast.

VM MIGRATION Flat address space. IP address unchanged after migration, higher level doesn t see state change After migration IP<->PMAC changes, as PMAC is location dependent VM sends gratuitous ARP with new mapping. Fabric Manager receives ARP and sends invalidation to old switch Old switch sets flow table to software, causing ARP to be sent to any stray packets Forwarding the packet is optional, as retransmit (if reliable) will fix delivery

LOCATION DISCOVERY: CONFIGURING SWITCH IDS Humans = Not right Answer Discovery = Right Answer Send messages to neighbors Get Tree Level Hosts don t reply, so edge only hears back from above Aggregate hears back from both levels Core hears back only from aggregate Contact Fabric Managerwith tree level to get ID Fabric Manager is service running on commodity host Assigns ID Soft state

NO LOOPS, NO SPANNING TREES Forwarding can only go up the tree. Cycles not possible.

FAILURE Keep-alives like the link discovery messages Miss a keep alive? Tattle to the Fabric Manager Fabric manager tells effected switches, which adjust own tables. O(N) vs O(N2) for traditional routing algorithms (Fabric Manager tells every switch vs every switch tells every switch)

MISC NETWORK MANAGEMENT: VIRTUAL LANS (14-740 REVIEW)

LOOKING BACK Connectivity Hosts can talk! No possibility of loops Efficiency Much less memory needed in switches, O(N) fault handlingh Self configuring Discovery protocol + ARP Robust Failure handling coordinated by FM VMs and Migration Each has own IP address, each has own MAC address Commodity hardware Nothing magic.

REVIEW: LOCAL AREA NETWORKS (LANS) Originally described a network that was local in space, i.e. a small network These almost universally used one shared communication fabric As distinguished from Wide Area Network (WAN), which often described (and still described) networks over larger distances, often times connected by leased lines, e.g. service from a telco. For decades LAN has been used to describe a network that has one shared communications fabric Regardless of size

REVIEW: HOW BIG CAN A LAN BE? Originally, the size of a LAN was very limited by utilization Broadcasting + limited network time means they top out at ~30% utilization Modern switches limit broadcasting and enable more stations. But, there is still a collision domain, so scaling still has limits There are other reasons for multiple LANs Administrative domains Security concerns Different technologies

REVIEW: NETS AND SUBNETS: ALTERNATIVES TO LARGE LANS Benefits: Hierarchical addresses enable greater scale Technology specific link-layer and protocols hidden underneath enabling the joining of different technologies Different administrative domains can provide own services, firewall policies, etc Costs: Big one is IP address space fragmentation Administrative costs

VLANS PROVIDE BIG BENEFITS Large flat network Limit traffic to VLAN Limits scope of broadcasts and thereby impact upon congestion Limits scope of transmissions, and thereby who can learn ID of hosts and about their traffic Can cross geographic boundaries, extending VLANs beyond traditional LAN limits

VLANS: BIG PICTURE BENEFIT Build out one large switched network Configure it to act like any number of LANs But without any of the geographic limitations

VLANS: PORT-BASED Static VLAN: VLAN=Group of Ports Port = switches wire connection Two VLANs configured on a 16-port switch How do the VLANs communicate with each other? Courtesy Bill Nace, 14-740

VLANS: TRUNKED SWITCHES Trunked connection: port belongs to all VLANs all frames at that port are forwarded to all VLANs But, how does the receiving side know which VLAN a particular frame belongs to? Courtesy Bill Nace, 14-740

802.1Q TAGGED ETHERNET VLAN identifier added to Ethernet frame 4-byte VLAN tag Includes 12-bit VLAN identifier Sending switch adds tag, receiving switch parses and removes tag Courtesy Bill Nace, 14-740