Understanding Latency Factors in SDN Control Plane Implementation

Measuring Control Plane Latency in SDN-enabled Switches Keqiang He, Junaid Khalid, Aaron Gember-Jacobson, Sourav Das, Chaithan Prakash, Aditya Akella, Li Erran Li and Marina Thottan 1

Latency in SDN app app app Centralized Controller Control plane OpenFlow msgs Data plane Time taken to install 100 rules ? Some XXX msecs?? Can be as large as 10 secs!!! 2

Do SDN apps care about latency? Intra-DC Traffic Engineering MicroTE[CoNEXT 11] routes predictable traffic on short time scales of 1-2 sec Fast Failover Reroute the affected flows quickly in face of failures Longer update time increases congestion and drops Mobility SoftCell[CoNEXT 13] advocates SDN in cellular networks Routes setup must complete within ~30-40 ms Latency can significantly undermine the effectiveness of many SDN apps 3

Factors contributing to latency? Control Software Design and Distributed Controllers Speed of Control Programs and Network Latency Not received much attention Latency in Network Switches 4

Our Work Two contributions: Systematic experiments to quantify control plane latencies in production switches Latency in network switches Factors that affect latency and low level explanations 5

Elements of Latency inbound latency Controller I3 I2 CPU board Inbound Latency Switch ASIC SDK OF Agent I1 I1: Send to ASIC SDK I2: Send to OF Agent I3: Send to Controller DMA Memory CPU PCI Switch Fabric No Forwarding Engine Hardware Tables Match Lookup PHY PHY Packet 6

Elements of Latency outbound latency Controller O1 CPU board Switch Outbound Latency O1: Parse OF Msg O2: Software schedules the rule O3: Reordering of rules in table O4: Rule is updated in table ASIC SDK OF Agent DMA Memory CPU O2 O4 PCI Switch Fabric Forwarding Engine Hardwar e Tables O3 Lookup PHY PHY 7

Measurement Scope Inbound Latency Increases with flow arrival rate Increases with interference from outbound msgs Higher CPU usage for higher arrival rates Outbound Latency Insertion Modification Deletion Please see our paper for details 8

Measurement Methodology Switch CPU RAM OF Flow table size Data Plane Version Vendor A 2 Ghz 2GB 1.0 4096 40*10G+4*40 Vendor B-1.0 1.0 896 1 Ghz 1GB 1.3 1792 (ACL) 14*10G+4*40G Vendor B-1.3 Vendor C ? ? 1.0 750 48*10G+4*40G 9

Insertion Latency Measurement Per rule insertion latency = t1 t0 Insert B rules in a burst (back-to-back) SDN Controller eth0 Control Channel t0 t1 Flows OUT Flows IN eth2 Libpcap Pktgen eth1 1Gbps of 64B Ethernet packets OpenFlow Switch Pre-install 1 default drop all rule with low priority 10

Insertion Latency Priority Effects Affected by priority patterns on all the switches we measured Per rule insertion always takes order of msec 30 10 30 10 25 8 25 8 insertion delay (ms) insertion delay (ms) insertion delay (ms) insertion delay (ms) 20 6 20 6 15 4 15 10 4 10 5 2 5 2 0 0 0 0 0 0 20 40 60 80 100 100 200 300 400 500 600 700 800 0 0 20 40 60 80 100 100 200 300 400 500 600 700 800 rule # rule # rule # rule # (b) Burst size 100, incr. priority (b) Burst size 800, decr. priority (a) Burst size 100, same priority (a) Burst size 800, same priority Vendor B-1.0 switch Vendor A switch 11

Insertion Latency Table occupancy Effects Affected by the types of rules in the table table 100 table 400 table 100 table 400 20000 20000 avg completion time (ms) 18000 avg completion time (ms) 18000 TCAM Organization, Rule Priority & Table Occupancy 16000 16000 14000 14000 12000 12000 10000 10000 8000 8000 Switch Software Overhead 6000 6000 4000 4000 2000 2000 0 0 0 100 200 300 400 500 600 0 100 200 300 400 500 600 burst size burst size (b) high priority rules into a table with low priority rules (a) low priority rules into a table with high priority rules Vendor B-1.0 switch 12

Modification Latency Measurement Per rule modification latency = t1 t0 Modify B rules in a burst (back-to-back) SDN Controller eth0 Control Channel t0 t1 Flows OUT Flows IN eth2 Libpcap Pktgen eth1 1Gbps of 64B Ethernet packets OpenFlow Switch Pre-install B dropping rules 13

Modification Latency Same as insertion latency on vendor A 2X as insertion latency on vendor B-1.3

Modification Latency Much higher on vendor B-1.0 and vendor C Not affected by rule priority but affected by table occupancy 160 160 140 140 modification delay (ms) modification dely (ms) 120 120 Poorly optimized switch software 100 100 80 80 60 60 40 40 20 20 0 0 0 20 40 60 80 100 0 50 100 150 200 rule # rule # (b) 200 rules in table (a) 100 rules in the table Vendor B-1.0 switch 15

Deletion Latency Measurement Per rule deletion latency = t1 t0 Delete B rules in a burst (back-to-back) SDN Controller eth0 Control Channel t0 t1 Flows OUT Flows IN eth2 Libpcap Pktgen eth1 1Gbps of 64B Ethernet packets OpenFlow Switch Pre-install B dropping rules & 1 pass all rule with low priority 16

Deletion Latency Higher than insertion latency for all the switches we measured Not affected by rule priority but affected by table occupancy 14 14 12 deletion delay (ms) 12 deletion delay (ms) 10 10 8 8 Deletion is incurring TCAM Reorganization 6 6 4 4 2 2 0 0 0 20 40 60 80 100 0 50 100 rule # 150 200 rule # (a) 100 rules in table (b) 200 rules in table Vendor A switch 17

Recommendations for SDN app designers Insert rules in a hardware-friendly order Consider parallel rule updates if possible Consider the switch heterogeneity Avoid explicit rule deletion if timeout can be applied 18

Summary Latency in SDN is critical to many applications Long latency can undermine many SDN app s effectiveness greatly Assumption: Latency is small or constant Latency is high and variable Varies with Platforms, Type of operations, Rule priorities, Table occupancy, Concurrent operations Key Factors: TCAM Organization, Switch CPU and inefficientSoftware Implementation Need careful design of future switch silicon and software in order to fully utilize the power of SDN! 19

20

Inbound Latency Increases with flow arrival rate CPU Usage is higher for higher flow arrival rates Flow Arrival Rate (packets/sec) Mean Delay per packet_in (msec) CPU Usage 100 3.32 15.7 % 200 8.33 26.5 % vendor A switch 21

Inbound Latency Increases with interference from outbound msgs 300 300 250 250 inbound delay (ms) inbound delay (ms) Low Power CPU 200 200 150 150 100 100 Software Inefficiency 50 50 0 0 0 200 400 600 800 1000 0 200 400 600 800 1000 flow # flow # (b) w/o flow_mod/pkt_out (a) with flow_mod/pkt_out vendor A switch. Flow Arrival Rate = 200/s 22

How accurate? 500 flows 1Gbps 64B Ethernet packet Inter-packet gap of a flow = 256 us Solution: Either increase the packet rate or reduce the number of flows or measure the accumulated latency for many flows 23