Master Linux Fundamentals for Effective Server Management

Linux Fundamentals Be comfortable with Linux

Agenda Connectivity to the servers Profile and Environment Variables Manipulating Files and Managing Directories File and Directory Permissions Important commands Basics of vi editor Regular Expressions Piping output between commands Standard Input, Output and Error Standard Directories Shell Scripting Scheduling Jobs

What is daemon process?

What is default port number for SSH?

Connectivity to the servers We typically use SSH or SSH based tools to connect to remote servers Also we might have to connect from one linux server to another linux server How SSH works? SSH password less login

Setting up password less login On the machine from where you want to connect to remote machines Generate keys for each category of servers (ssh-keygen) Enable passphrase for all critical categories such as production Run ssh-copy-id (use -i <COMPLETE_PATH_OF_KEY_FILE> - if it is not default) Run ssh to connect to remote machine (use -i <COMPLETE_PATH_OF_KEY_FILE> - if it is not default) Default key file names are id_rsa for private key and id_rsa.pub for pulic key

Troubleshooting connectivity issues Troubleshooting ssh (using telnet or nc) Common Issues DNS alias might have typos DNS alias might not be mapped to correct IP address DNS alias or IP address might be blocked by firewall SSH service might be down File permissions On the source machine .ssh should be 700 Private key should be 600 On the target machine .ssh should be 700 Also authorized_keys should be 600

Other uses of SSH Copying files (back and forth between servers) scp rsync (very powerful and important)

Profile and Environment Variables Command to see environment variables env Profile will set the defaults for a session (.profile or .bash_profile) History of commands will be preserved in hidden file ends with history We can use export to set environment variables such as *_HOME or update PATH for a given session If we add same export commands to .*profile files, they will be available for all sessions

Manipulating Files Creating files Empty file or update properties touch Use cat command to append or redirect output to a file Using editors such as vi Using output of a command while creating files Reading files view can be used to open files in read only mode Read complete file cat Preview data from file use head or tail Copying or moving/renaming files Copy file cp Move or rename file mv Moving only mapping between file name and location and hence it is faster Deleting files Delete file rm

ManagingDirectories Creating Directories mkdir Reading contents of directories ls, -l list, -t sort by time, -r sort in reverse find - very important command Copying or moving/renaming directories Copy cp -rf Move or rename mv -f Deleting directories Delete rm -rf

File and Directory permissions Understanding permissions owner, group and other Changing permission chmod Changing owner chown Better control ACLs (Access control lists)

Important commands File details Getting help help man Listing files ls Searching files find

Important commands Manipulating data Filter for content from a file grep and egrep Sorting data sort Get unique records uniq (data need to be pre-sorted to eliminate duplicates) Delimit structured data cut

Important Commands System Details Get details about CPUs lscpu Get details of memory free Get details of storage Details for each mount point df Details for usage of a directory du IP Addresses configured ip addr (OS specific)

Important Commands management To get load average uptime To get details about current usage of resources top To get process details ps command

Important Commands manipulating dates Format options date +%a date +%A date +%b date +%B date +%d date +%D date +%F date +%H date +%I date +%j date +%m date +%M date +%S date +%N Purpose of Option Displays Weekday name in short (like Mon, Tue, Wed) Displays Weekday name in full short (like Monday, Tuesday) Displays Month name in short (like Jan, Feb, Mar ) Displays Month name in full short (like January, February) Displays Day of month (e.g., 01) Displays Current Date; shown in MM/DD/YY Displays Date; shown in YYYY-MM-DD Displays hour in (00..23) format Displays hour (01..12) format Displays day of year (001..366) Displays month (01..12) Displays minute (00..59) Displays second (00..60) Displays nanoseconds (000000000..999999999) Displays time; shown as HH:MM:SS Note: Hours in 24 Format Displays day of week (1..7); 1 is Monday Displays week number of year, with Sunday as first day of week (00..53) 5 Displays full year i.e. YYYY alphabetic time zone abbreviation (e.g., EDT) Output Thu Thursday Feb February 7 02/07/13 07/02/13 23 11 38 2 44 17 573587606 date +%T date +%u date +%U date +%Y date +%Z 23:44:17 4 2013 IS

Important Commands miscellaneous To get list of environment variables env Present working directory pwd To get list of active users who Current user whoami Fully qualified host name hostname -f Changing password passwd (do not change password on the labs) Killing a process kill Submitting a process to the background nohup

Basics of vi editor Insert and Command mode Press esc to go to command mode (default) Press i for insert mode or a for append mode Navigating Left (h), right (l), up (k) and down (j) (one at a time) Beginning of line (^) and end of line ($) very important Page by page Manipulating Deleting one character x and X Replacing r (one character), R (series of characters as we type) Copying yy (yank) Deleting bulk dd (one line), dw (one word) Pasting P

Basics of vi editor Searching Opening and managing multiple files Saving (esc and then :x or :wq) Discard (esc and then :!q)

Regular Expressions Used to filter output of commands Very wide usage Simple commands such as ls Search commands such as find

Regular Expressions Regular Expressions are used in string processing There is significant overlap between regex and pattern matching ^ is used for beginning of the line and $ is used for end of the line Click here for the cheatsheet.

Piping output between commands Let us do some examples where data need to be piped to other commands Typically the output of one command will be passed as input to another command Filtering the output Sorting Getting unique records And more xargs is a special command to build execution pipelines. Output will be passed as arguments rather than input to xargs. We will use additional commands after xargs for which output will be passed as argument Find files and delete

Standard Input, Output and Error Standard Input 0 Standard Output 1 Standard Error 2

Standard Directories /etc /var/log /opt /usr And more

Shell Scripting It is nothing but running series of commands One need to have mastery over commands to excel in shell scripting

Basic constructs if condition Reading input and processing using while loop for loop Checking if file exists

Shell Scripting and piped commands Examples Get count by status and save output to a file Run find command on root file system and Save errors to find_errors.log Valid output to finding_files.out Find all the files which are greater than 10 MB in /data Find all the files that are modified in past 24 hours Save errors to finding-modified-files.err Save output to finding-modified-files.out Top 5 directories consuming space in /tmp Top 5 users consuming space in /tmp Number of sessions that are active per user Top 5 users by number of sessions that are currently active Get number of records from list of tables using sqoop eval (sqoop eval template will be provided)

Scheduling Jobs Using cron