Bottom Half Processing in Operating Systems

Top Half / Bottom Half Processing David Ferry, Chris Gill, Brian Kocoloski, James Orr CSE 422S - Operating Systems Organization Washington University in St. Louis St. Louis, MO 63130 1

Bottom Halves Defer Work Modern interrupt handlers are split into top half (fast) and bottom half (slow) components Top half: Does minimum work to service hardware Sets up future execution of bottom half Clears interrupt line Bottom half: Performs deferred processing Example: Network card top half clears card buffer while bottom half processes and routes packets 2 CSE 422S Operating Systems Organization

Advantages of Bottom Halves 1. Allow the kernel to run time-sensitive work before bottom half processing occupies resources 2. Allow the deferred work itself to be pre- emptible, because it runs with interrupts enabled Though some interrupt handlers may run with all interrupts disabled (IRQF_DISABLED in older versions) Others only disable the current interrupt line, which also could still be problematic All other devices that use that interrupt line are blocked e.g., devices on PCI buses can share interrupt lines 3 CSE 422S Operating Systems Organization

Interrupt context Note: there is a key distinction between interrupt context and execution with interrupts disabled Top halves run in interrupt context and with interrupts disabled Bottom halves can run in interrupt context, but run with interrupts enabled Why is this distinction important? Disabling interrupts prevents the kernel from running higher priority work i.e., it creates non-determinism from the kernel s perspective 4 CSE 422S Operating Systems Organization

When does bottom half work run? At some point in the future Bottom halves give the kernel the opportunity to decide whether or not now is a good time to do this work That means: The kernel could run a bottom half immediately The kernel could do higher priority work first You thus have no guarantee on when deferred work will finish (or even when it will start) 5 CSE 422S Operating Systems Organization

Bottom Half Mechanisms Softirqs (interrupt context) Statically defined Same softirq handler can execute concurrently on different processors, so may be more difficult to program correctly Offer most concurrency at the expense of being harder to program - locking must be done by driver Tasklets (interrupt context) Run on top of softirqs Should be your default choice over softirqs Tasklet handlers of same type do not execute concurrently Work Queues (process context) Defers work to generic kernel threads (kworker) It s ok for the work function to sleep, block, etc. Replaces having to make your own deferred kernel thread 6 CSE 422S Operating Systems Organization

Softirqs Most complex bottom half mechanism Only used in a small set of situations When the work to do is substantial and can benefit from concurrency i.e., when you want to run your bottom half work concurrently on multiple cores Rarely used directly by programmers Defined at compile time, code can t register dynamically Need to watch out for data races (can execute concurrently) Stored in a fixed-sized array of pointers to functors (struct with action handler function pointer as a member) 7 CSE 422S Operating Systems Organization

When do softirqs run? Consider the example of a network interrupt handler Some packets come in, the top-half copies them to DRAM, lowers the interrupt, and then schedules the bottom-half softirq to run (NET_RX_SOFTIRQ) Q: When will the bottom half run? A: it depends Return from hardware interrupt code path (i.e., immediately) In the ksoftirqd kernel thread (i.e, once the kernel schedules a kernel thread to execute it) In any code that explicitly checks for the existence of softirq work 8 CSE 422S Operating Systems Organization

Tasklets Tasklets may be easier to use than (non-tasklet) softirqs Can be registered dynamically (by code) or at compile time Tasklets are (implemented atop) softirqs Can register as HI_SOFTIRQ (run at high priority) or as TASKLET_SOFTIRQ (run at normal priority) Tasklet handlers of same type can t run concurrently Can share data (across processors) w/out locking (still can t block) But, don t need to worry about data races within a handler type 9 CSE 422S Operating Systems Organization

Review: when do Softirqs (and therefore, tasklets) run? 3 ways the kernel checks for pending softirqs (LKD pp. 138): In the return from hardware interrupt code path In the ksoftirqd kernel thread In any code that explicitly checks for and executes softirqs, such as the networking subsystem 10 CSE 422S Operating Systems Organization

Work Queues Work is deferred to run in kernel threads Execute in process context (not in interrupt context) So, can sleep, lock, block, etc. Thus, any work that must sleep/block/etc. should run there Data structure for work queues One is defined for each type of worker thread Per CPU: work items, a thread task structure, and a spin lock Work structure packages up data and processing Work is implemented as pointer to function that is run While work list isn t empty, thread calls each entry s function When work list is empty, thread goes to sleep This approach is suitable for many deferred work situations 11 CSE 422S Operating Systems Organization

Which bottom half should you use? Does your work absolutely need to do operations that may block? E.g., large memory allocations If so, use work queues Is your work very time sensitive, or does it need concurrency? If so, use softirqs You need to modify the kernel source to use this operation i.e., no kernel modules Otherwise, use tasklets 12 CSE 422S Operating Systems Organization