Programming Models, Operating Systems, and Resource Allocation in EECS 582
Explore key concepts in programming models, operating systems, and resource allocation as covered in EECS 582. Learn about MapReduce, Spark RDDs, Borg, Mesos, Omega, GFS, and more, to understand scalability, fault tolerance, scheduling, and allocation strategies in large-scale systems.
Download Presentation
Please find below an Image/Link to download the presentation.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author. If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.
You are allowed to download the files provided on this website for personal or commercial use, subject to the condition that they are used lawfully. All files are the property of their respective owners.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author.
E N D
Presentation Transcript
EECS 582 Final Review Mosharaf Chowdhury EECS 582 F16 1
Stats on the 11 Papers Weve Reviewed 3.5 3 2.5 2 Score 1.5 1 0.5 0 EECS 582 F16 2
Programming Models 3.5 3 2.5 2 Score 1.5 1 0.5 0 EECS 582 F16 3
Programming Models MapReduce Exposes scalability and fault tolerance with little programming experience Doesn t work for well for iterative algorithms Spark RDDs suits iterative workloads well Lineage for fault tolerance allows avoiding checkpointing Ease of usability EECS 582 F16 4
Operating Systems 3.5 3 2.5 2 Score 1.5 1 0.5 0 EECS 582 F16 5
Operating Systems Borg Classifies jobs into long and short with different priorities, preempting if required Hides details of allocation and failures from programmers Centralized schedulers can be scalable Mesos Two-level scheduling with resource offers Frameworks can choose to accept or reject offers Failure handling is left to the apps EECS 582 F16 6
Resource Allocation 3.5 3 2.5 2 Score 1.5 1 0.5 0 EECS 582 F16 7
Resource Allocation Omega Schedule mix of batch and interactive jobs with good placement Optimistic concurrency control targeted toward larger clusters Shared-state scheduler DRF Generalization of max-min allocation to multiple resources and heterogeneous clusters Many properties to maximize utilization and fairness without cheating EECS 582 F16 8
File System 3.5 3 2.5 2 Score 1.5 1 0.5 0 EECS 582 F16 9
File System GFS Workload-guided design: appends and large reads with small number of huge files Centralized design with replication for fault tolerance FDS Data and compute are NOT collocated Exploits full bisection bandwidth networks Stores everything has blobs to maximize sequential I/O EECS 582 F16 10
Memory Management 3.5 3 2.5 2 Score 1.5 1 0.5 0 EECS 582 F16 11
Memory Management PACMan Coordinated caching for DFSes All-or-nothing property dictates two eviction policies Prefers small jobs EC-Cache Alternative to replication that erasure codes instead Improves performance and tail latency by exploiting parallel I/O and better load balancing by splitting individual objects EECS 582 F16 12
Networking 3.5 3 2.5 2 Score 1.5 1 0.5 0 EECS 582 F16 13
Networking Fat Tree DC network topology to provide full bisection bandwidth by arranging commodity switches into multiple stages Approximates Clos topology Global scheduling to minimize congestions (Hedera) Varys Coflow abstraction to exploit application-level algorithm Heuristics to improve order and allocate rates using all-or-nothing Introduced the concurrent open shop scheduling with coupled resources EECS 582 F16 14
Final Poster and Paper Posters are a good way to interact with others and get feedback Mileage may vary, but its important to be able to talk about what you do Research paper The key part Should be written similar to the papers you ve read As if you d submit it to a workshop with ~3 more months of work or to a conference after ~6 more months of work How to Write a Great Research Paper by Simon Peyton Jones 9/7/16 EECS 582 F16 15
Rough Outline [8 Pages w/o References] Abstract Introduction (Highlight the importance and give intuition of solution) Motivation (Use data and simple examples) Overview (Summarize your overall solution so that readers can follow later) Core Idea (Main contribution w/ challenges and how you address them) Implementation (Discuss non-obvious parts of your implementation) Evaluation (Convince readers that it works and when it fails) Related Work (Let readers know that you know your competition!) Discussion (Know your limitations and possible workarounds) Conclusion (Summarize and point out future work) 9/7/16 EECS 582 F16 16
