Data Modeling for Microservices with Cassandra and Spark

This presentation covers data modeling for microservices using Cassandra and Spark. Topics include IT transformation, creating data architecture, metadata usage for diagnostics, challenges faced, IT capabilities, distribution systems, reservation systems, new data platforms, and distribution platform architecture.

• Data Modeling
• Microservices
• Cassandra
• Spark
• IT Transformation

lflin Follow

Uploaded on Mar 08, 2025 | 1 Views

Download Presentation

Please find below an Image/Link to download the presentation.

The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author.If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.

You are allowed to download the files provided on this website for personal or commercial use, subject to the condition that they are used lawfully. All files are the property of their respective owners.

The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author.

Presentation Transcript

1. Data modeling for microservices with Cassandra and Spark Jeff Jeff Carpenter, Choice Carpenter, Choice Hotels International Hotels International Page 1 Page 1 Strata + Hadoop World NYC Sept 26-29, 2016 Strata + Hadoop World NYC Sept 26-29, 2016

2. Agenda 1 2 3 4 5 IT Transformation Distribution and Analytics Creating a Data Architecture Data Modeling for Microservices Using Metadata for Diagnostics and Analytics Challenges Page 2 Strata + Hadoop World NYC Sept 26-29, 2016

3. IT Capabilities Guest Franchise Relations Distribution Corporate IT This talk Hotel Manage- ment Business Intelligence Page 3 Strata + Hadoop World NYC Sept 26-29, 2016

4. Distribution - Central Reservation System Guest Domain Franchisee Domain Customer & Loyalty Billing Hotel Management Domain External Channels Property Systems Distribution Domain Business Intelligence Domain CRS Web and Mobile Reporting & Analytics Page 4 Strata + Hadoop World NYC Sept 26-29, 2016

5. Current Reservation System By The Numbers 25 years 6,000 hotels 50 4,000 1 instance distribution channels transactions / second Page 5 Strata + Hadoop World NYC Sept 26-29, 2016

6. New Systems: Distribution and Data Platforms Realtime data Distribution Platform Data Platform History See: Choice Hotels's journey to better understand its customers through self-service analytics This Talk: how we model data and use the self-service platform Page 6 Strata + Hadoop World NYC Sept 26-29, 2016

7. Distribution Platform - Architecture Tenets Cloud-native Microservices Open Source Infrastructure Extensibility Stable, Scalable, Secure Page 7 Strata + Hadoop World NYC Sept 26-29, 2016

8. What is a Microservice? (one definition) Composing Service Entity Service Client AMQ REST API Events Persistence Message Driven Service DB Data Ownership Page 8 Strata + Hadoop World NYC Sept 26-29, 2016

9. How can we design our data architecture & models to be Scalable? Extensible? Maintainable? Analytics-ready? Page 9 Strata + Hadoop World NYC Sept 26-29, 2016 Strata + Hadoop World NYC Sept 26-29, 2016

10. Our Data Stack Metrics Reporting & Analytics Logging Long Term Storage Non- relational storage Page 10 Strata + Hadoop World NYC Sept 26-29, 2016

11. Data Modeling Then and Now Conceptual Data Model Identifying domains and relationships Services own data Logical Data Model SOA and Canonical Data Model Identifying data types and relationships Data Dictionary Physical Models Isolated Systems Java APIs RESTful APIs (JSON) Events (JSON) Cassandra Schemas Page 11 Strata + Hadoop World NYC Sept 26-29, 2016

12. Conceptual Data Model - Domains hotels rates inventory offers reservations Page 12 Strata + Hadoop World NYC Sept 26-29, 2016

13. Conceptual Data Model Domain Relationships Guest Domain Distribution Domain guest hotels offers loyalty inventory Hotel Management Domain reservations rates stay Page 13 Strata + Hadoop World NYC Sept 26-29, 2016

14. Logical Data Model Identifying Types Rates Domain Rate Rate Product Product Rate Plan Rate Plan id code hotelId effectiveDates Conditions id ratePlanId productId hotelId dateSpan id code hotelId features Rate Plan Service Price Price condition amount Rate Service Composite Rate Service Page 14 Strata + Hadoop World NYC Sept 26-29, 2016

15. Standardizing Common Data Types Instead of a Canonical Data Model, we standardize basic building blocks Feature, Category, Brand Geospatial Financial Time Contact information Address Address lines[] city subdivision country postalCode Page 15 Strata + Hadoop World NYC Sept 26-29, 2016

16. Data Types Microservice Identification Internal / External Client Apps Data Maintenance Apps Reservation Service Hotel Service Rates Service Inventory Service Offer Service Reservation Domain Hotel Domain Rates Domain Inventory Domain Offer Domain Page 16 Strata + Hadoop World NYC Sept 26-29, 2016

17. Physical Data Models Physical Models RESTful APIs (JSON) Events (JSON) Cassandra Schemas Java APIs JSON = primary definition of the data type owned by each service Page 17 Strata + Hadoop World NYC Sept 26-29, 2016

18. Key Data Types RESTful Resource Paths Hotel Service /hotels Offer Service /offers Rates Service /rates Reservation Service /reservations Inventory Service /inventory Page 18 Strata + Hadoop World NYC Sept 26-29, 2016

19. Java and RESTful APIs common pattern REST REST GET /types/<id> GET /types?<query parameters> POST /types/ (JSON body) PUT /types/ (JSON body) DELETE /types/<id> Java API Java API Type getTypeById() Type[] searchType(TypeSearchCriteria) createType(Type) updateType(Type) deleteType(TypeId) Page 19 Strata + Hadoop World NYC Sept 26-29, 2016

20. Cassandra Data Modeling (an idealized view) Page 20 Strata + Hadoop World NYC Sept 26-29, 2016

21. Cassandra Data Modeling Access Patterns Page 21 Strata + Hadoop World NYC Sept 26-29, 2016

22. Cassandra Data Modeling Chebotko Diagrams Page 22 Strata + Hadoop World NYC Sept 26-29, 2016

23. Cassandra Data Modeling - Physical Page 23 Strata + Hadoop World NYC Sept 26-29, 2016

24. Cassandra Data Modeling - Schemas CREATE KEYSPACE hotel WITH replication = {'class': 'SimpleStrategy', 'replication_factor' : 3}; CREATE TABLE hotel.hotels_by_poi ( poi_name text, hotel_id text, name text, phone text, address frozen<address>, PRIMARY KEY ((poi_name), hotel_id) ) WITH CLUSTERING ORDER BY ( hotel_id ASC) ; CREATE TYPE hotel.address ( street text, city text, state_or_province text, postal_code text, country text ); Page 24 Strata + Hadoop World NYC Sept 26-29, 2016

25. And now Back to reality Page 25 Strata + Hadoop World NYC Sept 26-29, 2016

26. Access Patterns and Denormalization Hotels by amenity Keyspace hotel Hotels by this Locate hotel by identifier hotels_by_id Find hotels by city, state, country hotels_by_brand Find hotels by brand Hotels by that hotels_by_postal_code Find hotels within X miles of point Y Find hotels by postal code Hotels by something else Page 26 Strata + Hadoop World NYC Sept 26-29, 2016

27. Metadata Events Incoming Request AMQ Service Logs Request Request Context Requestor Tracking ID Token Locale Context ELK Stack Page 27 Strata + Hadoop World NYC Sept 26-29, 2016

28. Asynchronous events { "type" : "UPDATE", "trackingId" : "0da7b794-f2c3- ", "requestor": "Legacy CRS", "newEntity" : { "hotelId": "AZ123", "productId": "NSK", "date": "2016-05-20", "consumedCount": "22", "totalCount": "25 }, "oldEntity" : { "hotelId": "AZ123", "productId": "NSK", "date": "2016-05-20", "consumedCount": "20", "totalCount": "25 } } Sample Inventory Event Request Request Context Requestor Tracking ID Token Locale Context Event Event Type Request Context Old entity New entity Create Update Delete Entity (old/new) Entity (old/new) Id Page 28 Strata + Hadoop World NYC Sept 26-29, 2016

29. Putting It Together Diagnostics Incoming Request Service Logs Metrics History Data ELK Stack Metrics Store Data Platform node C* node node node Page 29 Strata + Hadoop World NYC Sept 26-29, 2016

30. Putting It Together Long Term Storage ELK Stack Metrics Store Data Platform node C* node node node Long Term Storage Page 30 Strata + Hadoop World NYC Sept 26-29, 2016

31. Separating Active and History Data Now Yesterday s data is ancient history Rate + Inventory Data Time Page 31 Strata + Hadoop World NYC Sept 26-29, 2016

32. History architecture History capture Service AMQ Kafka node Other subscribers Spark S3 node node History retrieval node Customer Service Apps History Service Impala Data Platform - Cloudera Page 32 Strata + Hadoop World NYC Sept 26-29, 2016

33. Microservice Data Challenges No Joins? Data Maintenance Data Integrity Cascading Deletes Transactions Page 33 Strata + Hadoop World NYC Sept 26-29, 2016

34. Distributed Transactions, Anyone? Commit the contract Reservation Service Booking Client reservations Reserve the inventory Inventory Service Data Maintenance Apps Data synchronization inventory Page 34 Strata + Hadoop World NYC Sept 26-29, 2016

35. Alternatives to Distributed Transactions Strong consistency Approach Approach C* Lightweight Transaction Example Example Scope Scope Updating inventory counts Data Tier Writing to multiple denormalized hotel tables Data synchronization, reservation processing C* Logged Batch Data Tier Retrying failed calls Service Compensating transactions Verifying reservation processing System Eventual consistency Strata + Hadoop World NYC Sept 26-29, 2016 Page 35

36. Final Thoughts Data Models > Microservices Events = Streams Use Metadata Everywhere Page 36 Strata + Hadoop World NYC Sept 26-29, 2016

37. Now Available! Cassandra: The Definitive Guide, 2nd Edition Completely reworked for Cassandra 3.X: Data modeling in CQL SASI indexes Materialized views Lightweight transactions DataStax drivers New chapters on security, deployment, and integration Strata + Hadoop World NYC Sept 26-29, 2016 Page 37

38. Contact Info @jscarp @choicehotels careers.choicehotels.com jeffreyscarpenter Page 38 Strata + Hadoop World NYC Sept 26-29, 2016