Linux File I/O: Understanding Buffering and Syscalls

Buffer and File I/O Chris Gill, Marion Sudvarg, and James Orr CSE 422S - Operating Systems Organization Washington University in St. Louis St. Louis, MO 63130 1

Everything is a File in Linux Anything you can get a file descriptor for Regular files on disk, pipes, FIFOs, sockets Many operations (especially read, write, and close) may be applicable to any of those abstractions Standard streams pre-assigned file descriptors STDIN_FILENO (0), STDOUT_FILENO (1), STDERR_FILENO (2) Each process has its own set of file descriptors for the things it has open CSE 422S Operating Systems Organization 2

File Input/Output (I/O) Syscalls Some low level syscalls apply broadly E.g., open, read, write, close Or readv/writev for scatter/gather I/O (more later) Others do more file-specific things Use lseek to reset current file offset (read/write pointer) Use fcntl to check or modify access mode/status Use dup or dup2 to duplicate open file descriptors Use truncate or ftruncate to resize a file Use pread/pwrite to read/write at a specified file offset Use preadv/pwritev for scatter/gather read/write at a specified file offset CSE 422S Operating Systems Organization 3

File I/O Library Functions Operate on a file stream pointer (FILE *) instead of an (integer) file descriptor Can convert between them using fileno and fdopen Implements portable higher-level I/O operations atop the (Linux) I/O syscalls Use fopen/fclose to open/close a file stream Use fscanf/fprintf for formatted file I/O Use fgets/fputs to read/write into/from a char * buffer Use getline to read a line into a char * buffer Provides I/O operations to/from char * buffers Use sscanf/sprintf for formatted buffer I/O Buffer sizing matters, sprintf null-terminates CSE 422S Operating Systems Organization 4

File Input/Output Buffering For performance reasons, file I/O syscalls may buffer data temporarily, then flush later E.g., move data from user space to kernel space memory and then allow write syscall to return (flush data from kernel memory to disk later) Can use fsync or fdatasync to force flush to disk Also, stdio library functions may buffer data May not perform read or write immediately Use setvbuf to specify buffering behavior Use fflush or fclose to flush stream to kernel CSE 422S Operating Systems Organization 5

Scatter/Gather I/O Atomic input and output with io vectors (iovec) Arrays of pointers to (and sizes of) memory buffers Copying pointers usually costs less than copying data Scatter-read using readv Moves data from a file into a set of memory buffers Gather-write using writev Moves data from a set of memory buffers into a file iovec iov1[2] 4 7 the slithy buffered data and t was brillig iovec iov2[3] 6 8 4 CSE 522S Advanced Operating Systems 6

Advice to the Kernel for File I/O Similar to memory page access advice to the kernel If no advice (normal behavior), the kernel still tries to optimize by doing a small amount of read-ahead Also can advise whether or not access is intended and if so whether in random or sequential order New posix_fadvise call replaces readahead call Both can alert the kernel a file range will be accessed Kernel can exploit advice it is given to improve I/O scheduling, pre-fetch, etc. to improve I/O performance CSE 522S Advanced Operating Systems 7

Studio Exercises Today Gain experience with key I/O syscalls Opening and closing regular files Reading data from them and writing data to them Gain experience with key I/O library functions Formatted input and output operations Reading from and writing to buffers and files Managing data structures in memory while reading and writing data between them and files on disk CSE 422S Operating Systems Organization 8