Dynamic Routing Architecture for Storage Stack Optimization

sRoute: Treating the Storage Stack Like a Network Authors: Iona Stefanovici, Bianca Schroeder, Greg O Shea, and Eno Thereska Presented by: Arnav Mishra 1

Agenda Introduction/Background 1. Routing Types & Challenges 2. Design 3. sSwitches a) Controllers b) Fault Tolerance c) Implementation 4. Control Applications 5. Tail Latency a) Replica Set b) File Cache c) Conclusion 6. Discussion Points 7. 2

Introduction Current Process: IO routing doesn t exist in storage stack Endpoints are predetermined, rigid structure Solution: sRoute = architecture for routing abstraction from storage stack Uses centralized control panel + sSwitches to route IO requests 10x better latency for tail IO 60% better throughput for customized replication protocol 2x better throughput for customized caching 3

High Level Ideas/Summary Bringing dynamic routing to IO Components Control Panel: Sets forwarding rules and sSwitch locations sSwitches: Execute forwarding rules on IO as it comes through Modifies endpoints and paths for improved IO Performance Sends Reads/Writes to less loaded servers 4

Background IO Routing is nearly impossible with current systems: pre-determined destinations, static movement channels Goal: Dynamically re-route read/write Send write to less loaded servers Send read to latest data Challenges Keeping consistency with changing endpoints for read/write Minimizing number of times IO has to traverse sSwitches IOFlow Currently: Bypasses stages on IO Path Cannot change endpoints/path 5

Types of IO Routing Endpoint From Source P to File X routed to File Y Improves tail latency, copy-on-write, file versioning, and data re-encoding Dynamically changes endpoint of new data Waypoint From Sources P and R to File X routed via special stage W Inject specialized waypoint routing to selectively route IO to correct stages Scatter From Source P and R to subset of data replicas Enables specialized replication and erasure coding policies Dynamically chooses endpoint to read/write from 6

Visual Representation of Routing 7

Routing Challenges Prevent system-wide configuration updates from making system instability sSwitches can t send IO through incorrect paths Some applications require reads to follow writes Must keep track of last data s location/path Some file systems require knowing destination endpoint of IO Must maintain file system functionality and semantics in storage stack Must maintain a low overhead in adding dynamically changing direction to IO 8

Notice the sSwitchs control over IO direction in the new architecture Design Tenants runs vms/containers with IOFlow sRoute built on top of IOFlow sSwitches used via 4 API calls: Insert, Delete, Quiesce, and Drain Control Panel gives sSwitches locations and forwarding rules 9

sSwitches Forwards IO via rules from Control Panel Rule = IO Header + Action/Delegate Function IO Packets matched with IO Header First matched rule is executed Responses don t have to be routed same was as requests Takes care of file connections, handling namespace conflicts Open/Create are expensive operations: tradeoff = open all files at start and have empty files vs open when IO begins and have expensive first IO 10

sSwitch API Calls 11

Controllers Tenant policies changed into sSwitch API calls and sent as forwarding rules Identifies locations for sSwitches (hypervisors/storage servers/etc.) Control Delegates Restricted Control Panel at sSwitches Increases efficiency by making control decisions Can only make API calls, no changes to IO Header/Data (security issue) 12

Consistency Per-IO IO flows through either old rules or updated rules, not hybrid mix Taken care of by quiesce and drain calls to queue Per-Flow Read requests sometimes need data from last write request Not an issue if both read/write come from same source Solutions: Central controller updates forwarding rules after each write (inefficient) Insert sSwitches with delegate functions near sources 13

Fault Tolerance Loss of controller can t affect data integrity Would cause slower IO Controllers can choose to replicate forwarding rules Rules created at delegates either sent to central controller or forwarded as a header on IOs If sSwitch on Kernel fails, entire server could fail due to added code 14

Limitations sRoute lacks verification tools Currently possible for controller to route IO to arbitrary, incorrect locations causing data loss Storage controller doesn t have access to network Storage Controller and Network Controller could send data to different places 15

Routing Implementation sSwitch in kernel (in C) and in user-level (in C#) total of 25 kLOC Within Server s IO Stack: Uses filter driver architecture of Windows Kernel attaches control code to beginning of filter driver Bypassing is done by returning early from driver Across Remote Servers: Kernel sSwitch sends IO to user-level sSwitch Transmits to remote location using TCP/RDMA sSwitch on remote server intercepts arriving packet 16

sSwitch Implementation sSwitch = stage that is identified by host name and driver name Driver identified by device driver name, device name, altitude sSwitch stores files in sroute-folder directory to decrease namespace conflicts Limitations: sSwitches can t intercept individual IO to memory mapped files Does not support byte-range file locking for multiple accesses to same file Intercepting IO response in Windows is costly 17

Control Applications Testbed 12 servers, running 16 Intel Xeon 2.4 GHz cores Run Windows Server 2012 Act as Hyper-V or Storage Servers 3 Workloads Unmodified SQL server with small IOs Public IO trace from Exchange email server IoMeter 18

Tail Latency Control IOs often arrive in bursts; load balancing must be implemented Endpoint Routing Detects extra load by comparing average receive rate with current receive rate Forward reads to less loaded servers where data is up to date If no writes happen, everything is still latest, send to Vmmax When spikes end: Trigger data reclaim to point to original VHDmax Rules are deleted, sSwitches removed CPU overhead < 1% Much better latency at Smax, little change at Smin 19

IOs Arrive in Bursts Notice the peak in the request rate. 20

Latency Improvement with Tail Latency Control Notice the huge decrease in latency for high # of IOs with sRoute 21

Replica Set Control Write-Set/Read-Set = Num of servers to contact for a read/write request Workload = Read-Only read-set should be only replicas of primary server Writes in Workload all requests happen at primary server which writes to write set After 30 seconds if no writes: sSwitch inserted to send reads to randomly chosen servers using control delegate If write received, reset to write stream Latency overhead for first write can be reduced with higher time interval before assuming read-only 22

Latency Improvement with Tail Latency Control Notice the throughput jumps whenever write requests end Notice the latency spike when the first write comes in to notify the controller 23

File Cache Accessing data in cache >3x faster than accessing disks Filter rules gain explicit control over customizing cache size, evictions, and write policies IO is cached if header matches installed filter rule Cache isolation: Controller sets sizes for each cache, sSwitches isolate them sRoute can forward to or bypass caches sRoute improved throughput 57%, performance 2x in test Throughput increases when both workloads run because they are isolated Caches don t overwrite each other (one process won t take cache from another) With IoMeter, controller gives it of cache 40% cache hit ratio 24

Conclusion Routing is inherently programmable and dynamic Substituting hard-coded implementation for common routing core improves performance 25

Piazza Discussion Points No detailed discussion of consistency concerns with path/endpoint changes Agreed, something that they mention needs to be researched further Control Plane as single point of failure Slide 14: Losing control panel won t make system worse than current line Added overhead from managing hops/security code/etc. CPU added overhead = 1% (no further discussion) Installed priorities could conflict with network priorities desirable to have a control plane that understands both the network and storage (203) No discussion of exponential growth of routing rules No mentions of this Does the overhead make this better for scenarios with a lot of IO Slide 20, latency improved at all levels 26

Questions? 27