Developing Reserver Class for Background Tasks in Ceph System
This project focuses on enhancing the background task management in Ceph by developing a new reserver class. The motivation stems from the need to efficiently handle scrubbing, backfill, and recovery tasks with predefined priorities. The proposed solution involves splitting the Async Reserver class into Local and Remote types, each equipped with new scheduling algorithms to optimize resource utilization. The system's current flow includes requesting and scheduling reservations on OSDs, ensuring efficient task execution. Additionally, detailed states in Ceph during backfilling and the main queues in the Async Reserver are explored. Overall, the aim is to reduce response time and improve task management in Ceph's background operations.
Download Presentation
Please find below an Image/Link to download the presentation.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author.If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.
You are allowed to download the files provided on this website for personal or commercial use, subject to the condition that they are used lawfully. All files are the property of their respective owners.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author.
E N D
Presentation Transcript
Reserver Class(es) for Background Tasks in Ceph Ankit Jain, Prerak Mall, Sukanya Venkataraman
Problem and Motivation Problem Statement: Develop a new reserver class for background work in Ceph Motivation: 3 types of background work - Scrubbing, Backfill, Recovery which are separate, performed according to hard-coded priorities (with preemption) Efficiently and effectively use cluster resources to reduce the response time to background work
Quick Overview of Current System OSDs have 2 reserver objects Local Request background tasks to be carried out Remote Carry out background tasks Placement Groups governed by a state machine that request/cancel local/remote reservations based on states Scheduling of scrubbing separate from recovery/backfill Scheduling based on multi-level priority queue with preemption
Our Solution Split Async Reserver class into 2 type Local, Remote New scheduling algorithms Local Reserver: Uniformly distributed, OSD states aware, load distributed Remote Reserver: OSD states aware, load distributed, Network Efficient
States in Ceph (Backfilling) Local Backfill Reserved Request Backfill WaitLocalBackfill Reserved Activating WaitRemoteBackfill Reserved Backfilled Recovered All Backfill Reserved Failure Backfilling Not Backfilling 1. UnfoundBackfill 2. Defer Backfill 3. RemoteReservation RevokedTooFull
The details Ceph Current Flow Request Reservation (Primary) Schedule Reservation (Local) Establish connection with Target OSD Primary OSD s Local Reserver Finisher Placement Group AsyncReserver Schedule Reservation (Target OSD) Request Reservation (Target OSD) Target OSD s Remote Reserver Carry Out Task (Target OSD)
AsyncReserver 3 main queues in_progress Current tasks that are being carried out by the Finisher preempt_queue Tasks in the in_progress queue that can be preempted all_queues Tasks waiting to be either scheduled for a remote reservation request, or scheduled to be performed Scheduling algorithm Same for local, remote reservations Schedule based on priority If number of tasks in in_progress exceeds maximum background tasks allowed in an OSD (configurable), preempt tasks from preempt_queue
Our Solution Split the AsyncReserver into AsyncReserverLocal and AsyncReserverRemote Local Reserver scheduling algorithms Select what tasks to schedule on which OSDs Uniform Distribution: Maintain a Least Recently Used OSD queue to uniformly distribute load amongst OSDs. Assume no knowledge of state of other OSDs OSD load-based distribution: Take the least loaded OSD amongst all target OSDs (average out incase of multiple targets) Remote Reserver scheduling algorithms Select what tasks to schedule from which OSDs Uniform Selection: Maintain a Least Recently Used OSD queue to uniformly select tasks from different OSDs. Favor Long Connections: Select tasks from those OSDs which already have an established connection
Simulation (in Python) RR (Remote Reserver) Success Failure RR req Local Reserver Request Reservation Failure Success RR TPool RR req Failure RR req Task 2 Failure Local Reserver Request Reservation RR TPool Workload Thread Pool RR req Failure RR req Failure Local Reserver Request Reservation RR TPool RR req Failure Success
Integration with Ceph Current Implementation : /src/common/AsyncResever.h Replaced original AsyncReserver with AsyncReserverLocal and AsyncReserverRemote Defined new API to select reservation for uniform load in LocalAsyncReserver Main Functionality in do_queues()
Current do_queues() implementation AsyncReserver Added uniform_load API AsyncReserverLocal
Summary Local, remote reservers have different kinds of tasks and behaviours, hence need different algorithms Scheduling based on overall state of the OSDs in the cluster reduces response time
Conclusion This is not yet extensively tested for various workloads in Ceph Improvements/Alternatives: Dynamically change priorities Dynamically change maximum allowed background tasks per OSD Integrate replica deletion with backfilling currently separate, even though related
