Process Observability, Isolation, and Performance Monitoring Techniques
Explore the concepts of process observability, isolation, and performance monitoring in operating systems. Learn about tools and techniques like strace, ptrace, cgroups, and the /proc filesystem that provide insights into process behavior and system performance.
Download Presentation
Please find below an Image/Link to download the presentation.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author. If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.
You are allowed to download the files provided on this website for personal or commercial use, subject to the condition that they are used lawfully. All files are the property of their respective owners.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author.
E N D
Presentation Transcript
Process Observability and Isolation Chris Gill, Brian Kocoloski, Marion Sudvarg CSE 522S Advanced Operating Systems Washington University in St. Louis St. Louis, MO 63130 1
Process Observability Generally when inspecting an application, you re interested in correctness or performance Correctness (covered in 422) strace, ptrace, ftrace Debuggers, e.g. the GNU debugger (gdb) Performance /proc filesystem Utilities built on top of it (e.g. ps, top, htop) CSE 522S Advanced Operating Systems 2
Performance and Isolation Processes are not fully isolated! Misbehavior by one process can cause system-wide performance issues: Extensive CPU consumption Memory leaks causing high memory utilization Later we will discuss resource isolation mechanisms to constrain interference Namespaces Resource limits cgroups Virtualization Today we will discuss introspective techniques to monitor performance CSE 522S Advanced Operating Systems 3
The /proc Pseudo-Filesystem The /proc filesystem provides introspective capabilities: Interface to internal data structures in the kernel Mostly read-only, but some files allow changing kernel settings at runtime Each process has a subdirectory (its PID) E.g., /proc/2350/ contains entries with information about PID 2350 Each process has same layout underneath Command line, status, etc. /proc 2349 2350 status cmdline status cmdline CSE 522S Advanced Operating Systems 4
/proc Filesystem Information In Linux, /proc/<pid> provides quite a bit of useful information about each process cmdline: command that originated the process cwd: symlink to current working directory of the process environ: names/values of environment variables exe: symlink to original executable file fd: directory containing symlink for each open file desc. maps: file containing info about memory maps mem: binary image representing the virtual memory status: information about run state and memory usage <many others > CSE 522S Advanced Operating Systems 5
/proc Performance Introspection Does the process have a lot of open files? fd: directory containing symlink for each open file desc. fdinfo: directory containing an informational file for each open file desc pos: read/write offset flags: open flags inotify: inotify watch details Is the process allowed to run on every CPU? status: information about run state and memory usage Cpus_allowed: cpumask of allowed cpus Cpus_allowed_list: human-readable list CSE 522S Advanced Operating Systems 6
/proc Performance Introspection Does the process have a lot of virtual memory areas? maps: file containing info about memory maps Is the process using a lot of memory status: information about run state and memory usage VmSize: amount of allocated memory VmRSS: resident set size (amount of memory that has been paged in) CSE 522S Advanced Operating Systems 7
Linux top and htop Utilities Utilities that sit on top of /proc List running processes, report key statistics Process id, resource usage, etc. Updated with reasonable frequency and detail New htop utility adds some bells and whistles Color coding, can kill processes from within it Install on RPi3 via sudo apt-get install htop Can display information selectively (see man pages) E.g., -u switch lets you see only one users processes CSE 522S Advanced Operating Systems 8
Linux ps Utility Unlike top and htop, ps prints a list of processes once then exits By default, selects all processes with the same effective user ID as the current user Common command: ps aux a: Show processes from all users x: Show processes not attached to a terminal u: Display user-oriented format (shows additional fields) Can be piped into grep for filtering CSE 522S Advanced Operating Systems 9
Personal anecdote I was recently working on my laptop, and it was getting really hot, fan was running at very high frequency, etc. I figured it was due to a bunch of web browser tabs, so I closed them all but it didn t help So I ran htop and sorted by CPU utilization there was a rogue OpenPGP process running and consuming 100% utilization Killed it by hitting <F9> and sending SIGKILL CSE 522S Advanced Operating Systems 10
Introspection for Correctness Say you debug a process and want to modify its memory /proc/<pid>/mem: binary image representing the virtual memory If you want to modify a variable integer x to have the value 2: int page_num = address / PAGE_SIZE; int pgoff = address - page_num * PAGE_SIZE; int * address = 0xffa0910; //Address of x int fd = open("/proc/pid/mem"); void * addr = mmap(NULL, PROT_WRITE, ..., fd, page_num); *((int *)addr + pgoff) = 2; CSE 522S Advanced Operating Systems 11
/proc Filesystem Considerations The /proc filesystem allows introspection of all processes on the system! How can processes be isolated so they are not aware of each other? Answer: Namespaces! (next time) A process can be isolated to its own PID space, and its own view of a subset of the filesystem As we ve seen, the /proc filesystem can be used to configure resource limits (e.g. for inotify) It can also display process resource limits and control group information (discussed later) CSE 522S Advanced Operating Systems 12
Studio Exercises Today Refresh your understanding of gdb and gain additional experience: Set dynamic watchpoints based on pointers in your program Set dynamic watchpoints with C-style commands to gdb Fix corrupted process memory during runtime debugging Explore the proc pseudo-filesystem and utilities that read from it ps htop (Optionally) explore the use of /proc/<pid>/mem to modify active process memory CSE 522S Advanced Operating Systems 13
