Timing Forensics and Control in Advanced Operating Systems

Timing Forensics and Control Marion Sudvarg, Chris Gill, David Ferry, Brian Kocoloski, Son Dinh CSE 522S Advanced Operating Systems Washington University in St. Louis St. Louis, MO 63130 1

Observing and Measuring Process Timings CSE 522S Advanced Operating Systems 2

Benchmarks vs. Microbenchmarks Benchmarks focus on overall system behavior Might measure timing, resource usage, etc. Often target some resource or behavior (e.g., compute-bound versus I/O bound execution) Can help to identify system bottlenecks Microbenchmarks focus on fine-grain behavior Test a particular control flow/code path/mechanism Use sequence of microbenchmarks to construct timelines of events Attempt to isolate a particular policy/mechanism Give basic units of measure to reason about CSE 522S Advanced Operating Systems 3

Synchronous vs. Asynchronous Timing Activities within a thread are synchronous Easy to measure by taking before/after timestamps E.g., latency to issue a request to create a thread (Don t conflate latency with execution time!) Asynchronous w.r.t. kernel, other threads Need multiple timestamps in multiple places Causal nature of mechanisms must be considered Sometimes, only relativistic measures are possible But, with a common clock, also detailed timing Need a clock that is synchronized across cores CSE 522S Advanced Operating Systems CSE 522S Advanced Operating Systems 4

Asynchrony-Safe Microbenchmarks Must use independent memory locations E.g., preallocated array of timestamps, with each index used independently by various activities Timestamping must be quick and regular The overhead should should be as deterministic as possible, so as not to change timing behavior too much (e.g. don t allocate dynamic memory) TS[0] TS[1] TS[2] TS[3] TS[4] TS[5] CSE 522S Advanced Operating Systems 5

CSE 422S Review: Ftrace the Function Tracer Tracing beyond system calls; many features: Event tracepoints (scheduler, interrupts, etc.) Trace any kernel function Call graphs Kernel stack size Latency tracing How long interrupts disabled How long preemption disabled Has a user interface called trace-cmd Very nice graphical trace browser called Kernelshark CSE 522S Advanced Operating Systems 6

CSE 422S Review: KernelShark Provides a graphical browser for ftrace output Shows a detailed timeline for each CPU core Powerful filter and zoom interface features https://kernelshark.org/Documentation.html CSE 522S Advanced Operating Systems 7

CSE 422S Review: Ftrace Internals When tracing is enabled, the kernel maintains: Per-CPU ring buffer for holding events Per-CPU kernel thread that empties ring buffer If readers can t keep up, data is lost (dropped) Tracepoints in kernel: Kernel maintains list of tracepoint locations Locations normally converted to no-ops (ftrace_make_nop()) Trace code is runtime-patched into kernel code when activated (ftrace_make_call()) CSE 522S Advanced Operating Systems 8

CSE 422S Review: Completely Fair Scheduler (CFS) Goal: All tasks receive a weighted proportion of processor time. On a system with N tasks, each task should be promised 1/N processor time i.e. completely fair Allows interactive tasks to run at high priority while sharing CPU equally between CPU bound tasks. Does not require OS to determine a priori if a process is I/O or CPU bound Fundamental idea: Abandons notion of fixed timeslice (and varying fairness), for fixed fairness (and varying timeslice) CSE 522S Advanced Operating Systems 9

Control of Process Timing CSE 522S Advanced Operating Systems 10

Explicit Control of CPU Time Resource limits (discussed last time) provide mechanisms for explicit control of CPU time RLIMIT_CPU: limit (seconds) on the amount of CPU time the process can consume RLIMIT_RTTIME: limit (microseconds) on the amount of CPU time a real-time process may consume before making a blocking system call For either resource limit: When soft limit is reached, process is sent SIGXCPU When hard limit is reached, process is sent SIGKILL getrusage() provides process introspection of CPU time usage via rusage structure: struct timeval ru_utime: user CPU time used struct timeval ru_stime: system CPU time used CSE 522S Advanced Operating Systems 11

Enter the Forkbomb Again! CFS will give a proportional share to each forked process they can hog the CPU! The fork() system call resets child resource usage statistics bypass resource limits! What is a flexible, principled mechanism to constrain resource usage? Answer: CPU cgroups CSE 522S Advanced Operating Systems 12

CPU Control Groups Enable observation and control over CPU usage by a group of processes Can either weight or limit absolute bandwidth of a group All times listed in interface files are in microseconds Note cgroups v2 does not yet support realtime processes Interfaces include: Interface File Description cpu.stat Shows current CPU usage, including statistics for CPU throttling and bursting induced by processes in the cgroup cpu.weight Similar to a nice value, weights execution under CFS, but applied to the group as a whole cpu.max Applies a bandwidth limit to execution time in a given period cpu.uclamp.min Requested minimum CPU utilization, affects throttling and bursting decisions by the schedutil cpufreq governor cpu.uclamp.max Requested maximum limit on CPU utilization Complete list at: https://docs.kernel.org/admin-guide/cgroup-v2.html#cpu CSE 522S Advanced Operating Systems 13

CPUset Control Groups Provides a way to pin a group of processes to a set of CPUs Optimize process placement on large NUMA systems Constrain a container to a subset of CPU cores Hierarchical: cannot have CPUs not granted to parent CPUset cgroup Provides these interfaces: cpuset.cpus: Requested CPUs cpuset.mems: Requested NUMA memory nodes cpuset.[cpus|mems].effective: Read-only allowed set for the cgroup. cpuset.[cpus|mems] parent.cpuset[cpus|mems].effective cpuset.cpus.partition: Default value member. If marked root, the cgroup is granted exclusive access to the CPUs in cpuset.cpus CSE 522S Advanced Operating Systems 14

Todays Studio Integrate CPU and CPUset cgroups into your simple container environment Observe and control processes CPU usage Use ftrace and KernelShark to observe how cgroups moderate process behavior Constrain your container to a set of CPU cores CSE 522S Advanced Operating Systems 15

Todays Readings LSP pages 177-204: An overview of Linux process management and scheduling The CPU and CPUSet Controller sections of the Control Group v2 pages in the Linux Kernel Documentation The KernelShark Documentation (Optional) LPI ch. 36: Covers process resource usage and limits CSE 522S Advanced Operating Systems 16

Project Project proposals due tomorrow night (Friday, Feb 10) at 11:59pm One submission per group Feedback returned in one week (Friday, Feb 17 at 11:59pm) Proposals can be revised and resubmitted based on feedback CSE 522S Advanced Operating Systems 17