Resource Limits and Control in Operating Systems

Control Groups and Subsystem Observability Marion Sudvarg, Chris Gill, Brian Kocoloski, Son Dinh CSE 522S Advanced Operating Systems Washington University in St. Louis St. Louis, MO 63130 1

So far, we have discussed Observability Hardware performance monitoring File event monitoring (inotify) proc filesystem top, htop, ps Isolation chroot Namespaces Capabilities These provide mechanisms to build and monitor containers But what about control? How do we control and limit use of system resources? Today: Resource limits Introduction to control groups (cgroups) Observing subsystem-wide resource use CSE 522S Advanced Operating Systems 2

Resource Limits Processes on a system must share limited resources CPU bandwidth, memory, disk/storage space, etc. View own resource usage with getrusage() A Resource Limitconstrains a process s usage of a specific resource setrlimit(resource, const *{soft limit, hard limit}) getrlimit(resource, *{soft limit, hard limit}) Soft Limit: the effective (current) limit Hard Limit: an enforced ceiling on the soft limit Requires CAP_SYS_RESOURCE to adjust A process can only set its own rlimit Useful to constrain a process after exec() Not useful for constraining other existing processes CSE 522S Advanced Operating Systems 3

Inspecting Resource Limits RLIMIT_CPU RLIMIT_FSIZE RLIMIT_DATA RLIMIT_STACK RLIMIT_CORE RLIMIT_RSS RLIMIT_NPROC RLIMIT_NOFILE RLIMIT_MEMLOCK RLIMIT_AS RLIMIT_LOCKS RLIMIT_SIGPENDING RLIMIT_MSGQUEUE RLIMIT_NICE RLIMIT_RTPRIO RLIMIT_RTTIME CSE 522S Advanced Operating Systems 4

Resource Limit Example: Constraining Memory Usage What if we want to constrain a process to 128MB of RAM? const rlim_t lim = 0x8000000; const struct rlimit rl = {lim, lim}; setrlimit(RLIMIT_RSS, &rl); const char * path = ./myprogram ; execl(path, path); RLIMIT_RSS: Resident set size, i.e. size of main memory (not in swap) Now, ./myprogram cannot use more than 128MB of main memory This constraints it from interfering with the system Or does it? CSE 522S Advanced Operating Systems 5

Counterexample: Anomalous Process Behavior A classical adversarial attack is the fork bomb A planted process repeatedly forks children (who recursively repeatedly fork children, etc.) Eventually uses up system resources, resulting in a denial of service for other users (possibly freezing the system) However, anomalous isn t always adversarial A process forks children to retry failed transactions or to run redundant transactions for fault-tolerance Misconfiguration or unanticipated environmental conditions may lead to a feedback loop resulting in a process cascade Never attribute to malice what can be adequately explained by someone else s feature Terry Tidwell, PhD 2011, Washington University CSE 522S Advanced Operating Systems 6

Limitations of Resource Limits If a process forks, every child has the same resource constraints as the parent Allows resource limits to be circumvented! A process with n children now uses (n+1)*rlimit units of the resource What about RLIMIT_NPROC? Limits the number of processes for a user RLIMIT_NPROC*RLIMIT_RSS: hard limit on a user s RAM usage But, this lacks flexibility What if a user wants to run a single memory-intensive process? What if a user wants to run more than RLIMIT_NPROC processes, but each uses very little memory? How can we constrain resource usage in a flexible, principled fashion? Answer: Control Groups CSE 522S Advanced Operating Systems 7

Linux Control Groups Constrain resource usage for hierarchical groups of processes using various controllers: Memory (today s focus) CPUs I/O Network Number of processes Performance monitoring events A forked child process is added to its parent s cgroup The hierarchy is reflected as a pseudo- filesystem Everything is a file Limits set through reading/writing file interfaces Enables file event multiplexing techniques for monitoring /sys/fs/cgroup/unified Implicitly contains all system processes not in a child cgroup / containers cg1 c1 c2 c3 cgroups pseudo-filesystem CSE 522S Advanced Operating Systems 8

cgroups v1 vs v2 cgroups come in two flavors: v1 and v2 Going forward, we will talk about v2 unless specified otherwise v2 was released to reduce complexity compared to v1 v2 provides a unified hierarchy for all cgroups controllers v1 allowed separate pseudo-filesystems to be mounted for different controllers v2 allows different controllers to be active at different levels of the hierarchy v2requires that only leaf node cgroups have processes (besides the root cgroup) v1 allowed different threads under the same process to be in different cgroups v2 restricts this behavior in a way that preserves process/thread hierarchy v2 allows improved notification for empty cgroups with no processes v1 allowed each cgroup to specify a program to launch when it empties v2 uses a status file instead allows a single program to multiplex across cgroups v1 is still available for backward compatibility Originally, v2 did not implement all v1 controllers Now supports most of the same features CSE 522S Advanced Operating Systems 9

Memory Control Groups Enable observation and control over memory use of a group of processes Includes kernel memory for the processes, e.g. dentries and inodes Memory limits enforced by Out Of Memory (OOM) killer kernel thread Remember, only leaf nodes have processes: interfaces in a non-leaf cgroup control all descendent cgroups in hierarchy Interfaces include: Interface File Description memory.current Shows current memory usage memory.stat A read-only key-value file with details of memory allocation memory.high A value that defines a threshold above which processes are throttled and memory is aggressively reclaimed memory.max Above this threshold, OOM killer is invoked on processes for which memory can t be reclaimed, terminating them if necessary memory.events A read-only file with key-value pairs tracking events triggered by limits Complete list at: https://docs.kernel.org/admin-guide/cgroup-v2.html#memory CSE 522S Advanced Operating Systems 10

Cgroups Delegation A subtree of the cgroups hierarchy can be delegated to a nonprivileged user Here, uid 1000 has control over user1000_containers They can t modify the user1000_containers interfaces subtree resource usage remains constrained They can create and manage child cgroups They can move processes within the subtree cgroups namespaces can define delegation boundaries DELEGATION / user1000_containers cg1 c1 c2 c3 cgroups pseudo-filesystem CSE 522S Advanced Operating Systems 11

Cgroups Namespaces Constrain processes to viewing a portion of the namespace hierarchy: /sys/fs/cgroup/unified/containers/c1 /c1 Highly useful in container environments! Prevents information leaks Easier container migration Prevent container processes from accessing ancestor cgroup directories CSE 522S Advanced Operating Systems 12

Reading Assignments LSP ch. 9: Covers userspace interaction with Linux kernel memory management mechanisms LSP pp. 204-209: Covers resource limits We provide a condensed PDF focusing on relevant sections of the man 7 cgroups man page man 7 cgroups_namespaces: Coverage of the cgroups namespace type The Memory Controller section of the Control Group v2 pages in the Linux Kernel Documentation (stop reading at the IO header) (Optional) DKR pp. 256-260: A brief overview of Docker s interaction with cgroups CSE 522S Advanced Operating Systems 13

Todays Studio Configure your Raspberry Pi to mount the cgroups v2 hierarchy Experiment with using the memory controller to set and detect limits and events First to enforce a hard limit on memory usage Then to enforce a soft limit, using inotify to monitor for events triggered when this threshold is passed Integrate cgroups into your simple container environment Observe and control your container s memory usage Use cgroupnamespaces to isolate your container s view of the cgroups hierarchy CSE 522S Advanced Operating Systems 14