Workload-Driven Analysis of File Systems in Shared Multi-Tier Data Centers

NETWORK BASED COMPUTING LABORATORY Workload-driven Analysis of File Systems in Shared Multi-Tier Data-Centers over InfiniBand K. Vaidyanathan P. Balaji H. W. Jin D.K. Panda Network-Based Computing Laboratory Department of Computer Science and Engineering The Ohio State University

NETWORK BASED COMPUTING LABORATORY Presentation Outline Introduction and Background Characterization of local and network- based file systems Multi File System for Data-Centers Experimental Results Conclusions

NETWORK BASED COMPUTING LABORATORY Introduction Exponential growth of Internet Primary means of electronic interaction Online book-stores, World-cup scores, Stock markets Ex. Google, Amazon, etc Highly Scalable and Available Web-Services Performance is critical for such Services Utilizing Clusters for Web-Services? [shah01] High Performance-to-cost ratio Has been proposed by Industry and Research Environments [shah01]: CSP: A Novel System Architecture for Scalable Internet and Communication Services. H. V. Shah, D. B. Minturn, A. Foong, G. L. McAlpine, R. S. Madukkarumukumana and G. J. Regnier In USITS 2001

NETWORK BASED COMPUTING LABORATORY Cluster-Based Data-Centers Web Server (Apache) Proxy Server Clients Storage WAN Application Server (PHP) Database Server (MySQL) Nodes are logically partitioned provides specific services (serving static and dynamic content) Use high speed interconnects like InfiniBand, Myrinet, etc. Requests get forwarded through multiple tiers Replication of content on all nodes

NETWORK BASED COMPUTING LABORATORY Shared Cluster-Based Data-Centers A B C A B C A B C } } } Web Server Proxy Server Clients Website A Storage Website B Website C WAN Application Server Database Server Hosting several unrelated services on a single data-center Currently used by several ISPs and Web Service Providers (IBM, HP) Replication of content Amount of data replicated increases linearly with the number of web- sites hosted

NETWORK BASED COMPUTING LABORATORY Issues in Shared Cluster-Based Data-Centers File System Caches being shared across multiple web-sites Under-utilization of aggregate cache of all nodes Web-site Content Replication of content on all nodes if we use local file system Need to fetch the document via network if we use network file system, however no replication required Can we adapt the file system to avoid these?

NETWORK BASED COMPUTING LABORATORY File System Interactions Network-based File Systems Web Server Local file system Proxy Server Local file system SAN SAN Local file system Database Server Application Server Data-Center Interaction File System Interaction

NETWORK BASED COMPUTING LABORATORY Existing File Systems Metadata Manager Meta Data Local file system Web Server I/O(OST) Node Data SAN compute node compute node compute node compute node I/O(OST) Node Data Server-side Cache Client-side Cache Network-based File System: Parallel Virtual File System (PVFS) and Lustre (supports client-side caching) Local File System: ext3fs and memory file system (ramfs)

NETWORK BASED COMPUTING LABORATORY Presentation Outline Introduction and Background Characterization of local and network- based file systems Multi File System for Data-Centers Experimental Analysis Conclusions

NETWORK BASED COMPUTING LABORATORY Characterization of local and network-based File Systems Network Traffic Requirements Aggregate Cache Cache Pollution Effects

NETWORK BASED COMPUTING LABORATORY Network Traffic Requirements Absolute Network Traffic generated Static Content Dynamic Content Network Utilization Large/Small burst (static or dynamic content) Overhead of Metadata Operations

NETWORK BASED COMPUTING LABORATORY Aggregate Cache in Data-Centers Local File Systems use only single node s cache Small files get huge benefits, if in memory. Otherwise, we pay a penalty of accessing the disk Large Files may not fit in memory and also have high penalties in accessing the disk Network File Systems use aggregate cache from all nodes Large Files, if striped, can reside in file system cache on multiple nodes Small files also get benefits due to aggregate cache

NETWORK BASED COMPUTING LABORATORY Cache Pollution Effects Working set frequently accessed documents; usually fits in memory Shared Data-Centers Multiple web-sites share the file system cache; each website has lesser amount of file system cache to utilize Bursts of requests/accesses to one web-site may result in cache pollution May result in drastic drop in the number of cache hits

NETWORK BASED COMPUTING LABORATORY Presentation Outline Introduction and Background Characterization of local and network- based file systems Multi File System for Data-Centers Experimental Results Conclusions

NETWORK BASED COMPUTING LABORATORY Multi File System for Data-Centers Characterization ext3fs ramfs pvfs lustre Network Traffic generated Min Min More traffic Min Use of Aggregate Cache No No Yes Yes Cache pollution effects Yes No Yes Yes Metadata overhead No No Yes Yes

NETWORK BASED COMPUTING LABORATORY Multi File System for Data-Centers A combination of file systems for different environments Memory file system and local file system (ext3fs) for workloads with high temporal locality Memory file system and network file system (pvfs/lustre) for workloads with low temporal locality

NETWORK BASED COMPUTING LABORATORY Presentation Outline Introduction and Background Characterization of local and network- based file systems with data-centers Multi File System for Data-Centers Experimental Results Conclusions

NETWORK BASED COMPUTING LABORATORY Experimental Test-bed Cluster 1 with: 8 SuperMicro SUPER X5DL8-GG nodes; Dual Intel Xeon 3.0 GHz processors 512 KB L2 Cache, 2 GB memory; PCI-X 64 bit 133 MHz Cluster 2 with: 8 SuperMicro SUPER P4DL6 nodes; Dual Intel Xeon 2.4 GHz processors 512 KB L2 Cache, 512 MB memory; PCI-X 64 bit 133 MHz Mellanox MT23108 Dual Port 4x HCAs; MT43132 24-port switch Apache 2.0.48 Web and PHP 4.3.7 Servers; MySQL 4.0.12, PVFS 1.6.2, Lustre 1.0.4

NETWORK BASED COMPUTING LABORATORY Workloads Zipf workloads: the relative probability of a request for the ith most popular document is proportional to 1/i with 1 High Temporal locality (constant ) Low Temporal locality (varying ) TPC-W traces according to the specifications Class Class 0 Class 1 Class 2 Class 3 Class 4 File Sizes 1K 250K 1K 1MB 1K 4MB 1K 16MB 1K 64MB Size 25 MB 100 MB 450 MB 2 GB 6 GB

NETWORK BASED COMPUTING LABORATORY Experimental Analysis (Outline) Basic Performance of different file systems Network Traffic Requirements Impact of Aggregate Cache Cache Pollution Effects Multi File System for Data-Centers

NETWORK BASED COMPUTING LABORATORY Basic Performance Latency ext3fs (usecs) 4K ramfs (usecs) 4K pvfs (usecs) lustre (usecs) 1M 1M 4K 1M 4K 1M Open & Close overhead 6 6 6 6 1060 1060 876 876 Read Latency (cache) 4 1602 4 1578 680 13825 1998 7.7 Read Latency (no cache) 1500 76312 1400 2379 9600 44108 3000 50713 Network File Systems incur high overhead for metadataoperations (open() and close()) Lustre supports client-side cache For large files, network-based file system does better than local file system due to striping of the file

NETWORK BASED COMPUTING LABORATORY Network Traffic Requirements 800000 800000 #packets sent/received #packets sent/received 600000 600000 400000 400000 200000 200000 0 0 Zipf Class 0 Zipf Class 1 Zipf Class 2 Zipf Class 3 TPCW Class 0 TPCW Class 1 TPCW Class 2 TPCW Class 3 ext3fs pvfs lustre ext3fs pvfs lustre Absolute Network Traffic Generated: Increases proportionally compared to the local file system for PVFS For Lustre, the traffic is close to that of the local file system For dynamic content, the network traffic does not increase with increase in database size

NETWORK BASED COMPUTING LABORATORY Impact of Caching and Metadata operations 14000 250 12000 200 10000 ext3fs ramfs pvfs lustre ext3fs ramfs pvfs lustre 150 8000 TPS TPS 6000 100 4000 50 2000 0 0 TPCW Class 0 TPCW Class 1 TPCW Class 2 TPCW Class 3 Zipf Class 0 Zipf Class 1 Zipf Class 2 Zipf Class 3 Local File Systems are better for workloads with high temporal locality Surprisingly Lustre performs comparable with local file systems

NETWORK BASED COMPUTING LABORATORY Impact of Aggregate Cache 100 80 ext3fs pvfs lustre 60 TPS 40 20 0 = 0.8 = 0.75 = 0.7 = 0.65 = 0.6 = 0.55 = 0.5 = 0.4 = 0.3 Workload with varying temporal locality Aggregate Cache improves data-center performance for network-based file systems

NETWORK BASED COMPUTING LABORATORY Cache Pollution Effects in Shared Data-Centers Percentage of Cached/NonCached Content 100% 80% 60% NonCached Cached 40% 20% 0% Shared Shared Shared Shared Shared Single Single Single Single Single Zipf Class 0 Zipf Class 1 Zipf Class 2 Zipf Class 3 Zipf Class 4 Small Workloads, web-sites are not affected Large Workloads, cache pollution affects multiple web-sites Placing files on memory file system might avoid the cache pollution effects

NETWORK BASED COMPUTING LABORATORY Multi File System Data-Centers 50% 60% Performance Improvement Performance Improvement 50% 40% 40% Zipf Class 0 Zipf Class 1 Zipf Class 2 TPCW Class 0 TPCW Class 1 TPCW Class 2 30% 30% 20% 20% 10% 10% 0% 0% Low Load Medium Load Heavy Load Low Load Medium Load Heavy Load Performance benefits for static content is close to 48% Performance benefits for dynamic content is close to 41%

NETWORK BASED COMPUTING LABORATORY Multi File System Data-Centers 20 18 16 14 12 TPS 10 8 6 4 2 0 = 0.75 = 0 .6 5 = 0 .5 5 = 0 .4 5 Workload with varying temporal locality pvfs pvfs with ramfs Benefits are two folds: Avoidance of Cache Pollution Reduced overhead of open() and close() operations for small files

NETWORK BASED COMPUTING LABORATORY Conclusions & Future Work Fragmentation of resources in shared data-Centers Under-utilization of file system cache in clusters Cache Pollution affects performance Studied the impact of file systems in terms of network traffic, aggregate cache and cache pollution effects Proposed a Multi File System approach to utilize the benefits from each file system Combination of Network and Memory File System for static content with low temporal locality Memory File System and local file system for static content with high temporal locality and dynamic content Propose to perform dynamic reconfiguration based on each node s memory cache and provide prioritization and QoS

NETWORK BASED COMPUTING LABORATORY Web Pointers Network Based Computing Laboratory NOWLAB http://www.cse.ohio-state.edu/~panda http://nowlab.cse.ohio-state.edu {vaidyana,balaji,jinhy,panda}@cse.ohio-state.edu