Grid Architecture and Tools: Key Concepts Revealed

GRID AND CLOUD COMPUTING Grid Architecture and Tools Courtesy: Dr Gnanasekaran Thangavel http://web.uettaxila.edu.pk/CMS/FALL2017/teGNCCms/

UNIT 2 OVERVIEW Grid Architecture and Tools: The Hourglass model Layered Architecture Simulation Tools Clouds and Grids Service Oriented Architecture 4/4/2025 2

Grid Architecture 4/4/2025 3

The Hourglass Model Focus on architecture issues Propose set of core services as basic infrastructure Used to construct high-level, domain-specific solutions (diverse) Design principles Keep participation cost low Enable local control Support for adaptation IP hourglass model A p p l i c a t i o n s Diverse global services tcp-ip Core services Local OS 4/4/2025 4

The architecture consists of four layers. The lowest fabric layer provides interfaces to local resources at a specific site. These interfaces are tailored to allow sharing of resources within a virtual organization. The connectivity layer consists of small, standard collection of communication protocols for supporting grid transactions that span the usage of multiple resources. In addition, the connectivity layer will contain security protocols to authenticate users and resources. The next layer in the hierarchy is the collective layer. It deals with handling access to multiple resources and typically consists of services for resource discovery, allocation and scheduling of tasks onto multiple resources, data replication, and so on. The application layer consists of the applications that operate within a virtual organization and which make use of the grid computing environment. 4/4/2025 5

Layered Grid Architecture (By Analogy to Internet Architecture) Application Internet Protocol Architecture Coordinating multiple resources : ubiquitous infrastructure services, app-specific distributed services Collective Application Sharing single resources : negotiating access, controlling use Resource Talking to things : communication (Internet protocols) & security Connectivity Transport Internet Controlling things locally : Access to, & control of, resources Fabric Link 4/4/2025 6

Example: Data Grid Architecture App Discipline-Specific Data Grid Application Coherency control, replica selection, task management, virtual data catalog, virtual data code catalog, Collective (App) Replica catalog, replica management, co-allocation, certificate authorities, metadata catalogs, Collective (Generic) Access to data, access to computers, access to network performance data, Resource Communication, service discovery (DNS), authentication, authorization, delegation Connect Fabric Storage systems, clusters, networks, network caches, 4/4/2025 7

Simulation tools GridSim Job scheduling http://sourceforge.net/projects/gridsim SimGrid Single client multiserver scheduling Bricks Scheduling GangSim- Ganglia VO OptoSim Data Grid Simulations G3S Grid Security Services Simulator security services 4/4/2025 8

Simulation tools GridSim is a Java-based toolkit for modeling, and simulation of distributed resource management and scheduling for conventional Grid environment. GridSim is based on SimJava, a general purpose discrete- event simulation package implemented in Java. All components in GridSim communicate with each other through message passing operations defined by SimJava. http://www.gridbus.org/gridsim http://www.buyya.com/gridsim/ 4/4/2025 9

Salient features of the GridSim It allows modeling of heterogeneous types of resources. Resources can be modeled operating under space- or time- shared mode. Resource capability can be defined (in the form of MIPS (Million Instructions Per Second) benchmark. Resources can be located in any time zone. Weekends and holidays can be mapped depending on resource s local time to model non-Grid (local) workload. Resources can be booked for advance reservation. Applications with different parallel application models can be simulated. 4/4/2025 10

Salient features of the GridSim Application tasks can be heterogeneous and they can be CPU or I/O intensive. There is no limit on the number of application jobs that can be submitted to a resource. Multiple user entities can submit tasks for execution simultaneously in the same resource, which may be time- shared or space-shared. This feature helps in building schedulers that can use different market-driven economic models for selecting services competitively. Network speed between resources can be specified. It supports simulation of both static and dynamic schedulers. Statistics of all or selected operations can be recorded and they can be analyzed using GridSim statistics analysis methods. 4/4/2025 11

A Modular Architecture for GridSim Platform and Components. Application, User, Grid Scenario s input and Results Output Appn Conf Res Conf User Req Grid Sc Grid Resource Brokers or Schedulers GridSim Toolkit Appn modeling Res entity Info serv Job mgmt Res alloc Statis Resource Modeling and Simulation Load Netw Reservation Single CPU Clusters SMPs Basic Discrete Event Simulation Infrastructure SimJava Distributed SimJava Virtual Machine PCs Workstation SMPs Clusters Distributed Resources 4/4/2025 12

Cloud: Definition A large-scale distributed computing paradigm that is driven by economies of scale, in which a pool of abstracted, virtualized, dynamically-scalable, managed computing power, storage, platforms, and services are delivered on demand to external customers over the Internet. (According to Foster, Zhao, Raicu and Lu, Cloud Computing and Grid Computing 360-Degree Compared, 2008) 13

CLOUDS

Cloud Computing Just a new name for Grid? Yes No . Nevertheless Yes!!! 15

Cloud: just a new name for Grid? YES: Reduce the cost of computing Increase reliability Increase flexibility (third party) 16

Cloud: just a new name for Grid? NO: Great increase demand for computing (clusters, high speed networks) Billions of dollars being spent by Amazon, Google, Microsoft to create real commercial large-scale systems with hundreds of thousands of computers www.top500.org shows computers with 100,000+ computers Analysis of massive data 17

Cloud: just a new name for Grid? Nevertheless YES: Problems are the same in clouds and grids Common need to manage large facilities Define methods to discover, request and use resources Implement highly parallel computations 18

Clouds: key points of the definition Differences related to traditional distributed paradigms: Massively scalable Can be encapsulated as an abstract entity that delivers different levels of service Services can be dynamically configured (via virtualization or other approaches) and delivered on demand 19

Clouds: reasons for interest Rapid decrease in hardware cost, increase in computing power (multi-cores etc.) and storage capacity Exponentially growing data size Widespread adoption of Services Computing and Web 2.0 apps 20

Clouds: yet about definition The interesting thing about Cloud Computing is that we ve redefined Cloud Computing to include everything that we already do. . . . I don t understand what we would do differently in the light of Cloud Computing other than change the wording of some of our ads. Larry Ellison (Oracle CEO), quoted in the Wall Street Journal, September 26, 2008 21

Clouds: yet about definition A lot of people are jumping on the [cloud] bandwagon, but I have not heard two people say the same thing about it. There are multiple definitions out there of the cloud. Andy Isherwood (HP VP of sales), quoted in ZDnet News, December 11, 2008 22

Clouds: yet about definition It s stupidity. It s worse than stupidity: it s a marketing hype campaign. Somebody is saying this is inevitable and whenever you hear somebody saying that, it s very likely to be a set of businesses campaigning to make it true. Richard Stallman (known for his advocacy of free software), quoted in The Guardian, September 29, 2008 23

Clouds: yet about definition From a hardware point of view, three aspects are new in Cloud Computing: 1. The illusion of infinite computing resources available on demand, thereby eliminating the need for Cloud Computing users to plan far ahead for provisioning; 2. The elimination of an up-front commitment by Cloud users, thereby allowing companies to start small and increase hardware resources only when there is an increase in their needs; and 3. The ability to pay for use of computing resources on a short- term basis as needed (e.g., processors by the hour and storage by the day) and release them as needed, thereby rewarding conservation by letting machines and storage go when they are no longer useful. 24

Clouds: relation with other paradigms 25

Web 2.0, Clouds, and Internet of Things HTC HTC: High : High - - Throughput Computing Throughput Computing Peer HPC HPC: High : High - - Performance Computing Performance Computing P2P: Peer to P2P: Peer to Peer MPP MPP: Massively Parallel Processors : Massively Parallel Processors 26 4/4/2025

27 What is a Service Oriented Architecture?

28 What is a Service Oriented Architecture (SOA)? A method of design, deployment, and management of both applications and the software infrastructure where: All software is organized into business services that are network accessible and executable. Service interfaces are based on public standards for interoperability.

29 Key Characteristics of SOA Quality of service, security and performance are specified. Software infrastructure is responsible for managing. Services are cataloged and discoverable. Data are cataloged and discoverable. Protocols use only industry standards.

30 What is a Service ? A Service is a reusable component. A Service changes business data from one state to another. A Service is the only way how data is accessed. If you can describe a component in WSDL, it is a Service.

31 Why Getting SOA Will be Difficult Managing for Projects: Software: 1 - 4 years Hardware: 3 - 5 years; Communications: 1 - 3 years; Project Managers: 2 - 4 years; Reliable funding: 1 - 4 years; User turnover: 30%/year; Managing for SOA: Data: forever. Infrastructure: 10+ years.

32 Why Managing Business Systems is Difficult? 40 Million lines of code in Windows XP is unknowable. Testing application (3 Million lines) requires >1015 tests. Probability correct data entry for a supply item is <65%.. Output / Office Worker: >30 e-messages /day.

33 How to View Organization for SOA PERSO N AL LEV EL Private Applications and Files Privacy and I ndividual Security Barrier VARIETY HERE G raphic InfoW indow, Personal Tools, Inquiry Languages Custom ized Applications, Prototyping Tools, Local Applications and Files LO CAL LEV EL A pplications Security Barrier Applications D evelopm ent & M aintenance APPLICATIO N LEV EL Business Security Barrier Service B OSD Service A BU SIN ESS LEV EL Process Security Barrier Functional Process A Functional Process B Functional Process C Functional Process D PRO CESS LEV EL Corporate Policy, Corporate Standards, Reference M odels, D ata M anagem ent and Tools, Integrated System s Configuration D ata Base, Shared Com puting and Telecom m unications EN TERPRISE LEV EL STABILITY HERE Industry Standards, Com m ercial O ff-the-Shelf Products and Services G LO BAL LEV EL

34 SOA Must Reflect Timing Private Applications and Files PERSONAL Graphic InfoWindow, Personal Tools, Inquiry Languages Customized Applications, Prototyping Tools, Local Applications and Files SHORT TERM ADAPTABILITY & TECHNOLOGY SIMPLICITY LOCAL Applications Development & Maintenance APPLICATION Infrastructure Support Business A Business B BUSINESS Functional Process A Functional Process B Functional Process C Functional Process D LONG TERM STABILITY & TECHNOLOGY COMPLEXITY PROCESS Corporate Policy, Corporate Standards, Reference Models, Data Management and Tools, Integrated Systems Configuration Data Base, Shared Computing and Telecommunications, Security and Survivability ENTERPRISE Industry Standards, Commercial Off-the-Shelf Products and Services GLOBAL

35 SOA Must Reflect Conflicting Interests Personal Local Organizations Missions Enterprise

36 Organization of Infrastructure Services Infrastructure Services (Enterprise Information) Data Services Security Services Computing Services Communication Services Application Services

37 Organization of Data Services Data Services Discovery Services Management Services Collaboration Services Semantic Services Interoperability Services

38 Data Interoperability Policies Data is an enterprise resource. Single-point entry of unique data. Enterprise certification of all data definitions. Data stewardship defines data custodians. Zero defects at point of entry. De-conflict data at source, not at higher levels. Data aggregations from sources data, not from reports.

39 Organization of Security Services Security Services Transfer Services Protection Services Certification Services Authentication Services Systems Assurance

40 Security Services = Information Assurance Conduct Attack/Event Response Ensure timely detection and appropriate response to attacks. Manage measures required to minimize the network s vulnerability. Secure Information Exchanges Secure information exchanges that occur on the network with a level of protection that is matched to the risk of compromise. Provide Authorization and Non-Repudiation Services Identify and confirm a user's authorization to access the network.

41 Organization of Computing Services Computing Services Control & Quality Computing Facilities Resource Planning Configuration Services Financial Management

42 Computing Services Provide Adaptable Hosting Environments Global facilities for hosting to the edge . Virtual environments for data centers. Distributed Computing Infrastructure Data storage, and shared spaces for information sharing. Shared Computing Infrastructure Resources Access shared resources regardless of physical access device.

43 Organization of Communication Services Communication Services Spectrum Management Connectivity Arrangements Continuity of Services Resource Management Interoperability Services

44 Communication Services Provide Information Transport Transport information, data and services anywhere. Ensures transport between end-user devices and servers. Expand the infrastructure for on-demand capacity.

45 Organization of Application Services Application Services Code Binding Services Maintenance Management Experimental Services Component Repository Portals

46 Application Services and Tools Provide Common End User Interface Tools Application generators, test suites, error identification, application components and standard utilities. Common end-user Interface Tools. E-mail, collaboration tools, information dashboards, Intranet portals, etc.

47 A Few Key SOA Protocols Universal Description, Discovery, and Integration, UDDI defines the publication and discovery of web service implementations. The Web Services Description Language, WSDL, is an XML-based language that defines Web Services. SOAP is the Service Oriented Architecture Protocol. It is a key SOA in which a network node (the client) sends a request to another node (the server). The Lightweight Directory Access Protocol, or LDAP is protocol for querying and modifying directory services. Extract, Transform, and Load, ETL, is a process of moving data from a legacy system and loading it into a SOA application.

References 1. Kai Hwang, Geoffery C. Fox and Jack J. Dongarra, Distributed and Cloud Computing: Clusters, Grids, Clouds and the Future of Internet , First Edition, Morgan Kaufman Publisher, an Imprint of Elsevier, 2012. Distributed Computing. http://distributedcomputing.info/index.html Jie Wu, Distributed System Design, CRC Press, 1999. Distributed Computing, Wikipedia http://en.wikipedia.org/wiki/Distributed_computing www.psgtech.edu/yrgcc/attach/GridComputing-an%20introduction.ppt www.cse.unr.edu/~mgunes/cpe401/cpe401sp12/lect15_cloud.ppt csnotes.upm.edu.my/kelasmaya/web.nsf/.../$FILE/Distributed%20Computing.ppt www.strassmann.com/pubs/gmu/2007-11-slides.ppt 2. 3. 4. 5. 6. 7. 8. 4/4/2025 48

Assignment #1 Visit the links for GridSim information on slide 9 to: Write a report on GridSim, its main features Selected any of the related publications about GridSim and summarise in you report. 4/4/2025 49

Thank You Questions and Comments? http://web.uettaxila.edu.pk/CMS/FALL2017/teGNCCms/ 4/4/2025 50