Posted onEdited onSymbols count in article: 2.3kReading time ≈2 mins.
An h1 header
Paragraphs are separated by a blank line.
2nd paragraph. Italic, bold, and monospace. Itemized lists look like:
this one
that one
the other one
Note that — not considering the asterisk — the actual text content starts at 4-columns in.
Block quotes are written like so.
They can span multiple paragraphs, if you like.
Use 3 dashes for an em-dash. Use 2 dashes for ranges (ex., “it’s all in chapters 12–14”). Three dots … will be converted to an ellipsis. Unicode is supported. ☺
An h2 header
Here’s a numbered list:
first item
second item
third item
Note again how the actual text starts at 4 columns in (4 characters from the left side). Here’s a code sample:
# Let me re-iterate ...
for i in 1 .. 10 { do-something(i) }
As you probably guessed, indented 4 spaces. By the way, instead of indenting the block, you can use delimited blocks, if you like:
1 2 3
define foobar() { print "Welcome to flavor country!"; }
(which makes copying & pasting easier). You can optionally mark the delimited block for Pandoc to syntax highlight it:
1 2 3 4 5 6
import time # Quick, count to ten! for i inrange(10): # (but not *too* quick) time.sleep(0.5) print i
An h3 header
Now a nested list:
First, get these ingredients:
carrots
celery
lentils
Boil some water.
Dump everything in the pot and follow this algorithm:
find wooden spoon
uncover pot
stir
cover pot
balance wooden spoon precariously on pot handle
wait 10 minutes
goto first step (or shut off burner when done)
Do not bump wooden spoon or it will fall.
Notice again how text always lines up on 4-space indents (including that last line which continues item 3 above).
% Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 43698 0 43698 0 0 148k 0 --:--:-- --:--:-- --:--:-- 217k
7634
Unix Philosophy
We use pipes (the | character) to chain the output of one program as the input of another
.center[]
Unix Philosophy
Everything is a file.
A text file is a file
A binary file is a file
A directory is a file
The keyboard is a file
The monitor is a file
All I/O devices are files
Files don’t need extensions. File types are determined using MIME types. MIME types are embedded in the file header (inode to be precise).
Some common extensions and their corresponding MIME types are listed below:
Understanding the Unix Philosophy is key to effectively using the terminal. Consider the two equivalent ways of viewing a file with less.
1 2
$ less file $ cat file | less
To understand the what’s going on, consider the following:
I/O is a file
stdin, stdout, and stderr are files
stdout of one program can be piped to stdin of another program using |
Understanding Filesystems
Have you noticed that directories take up 4096 bytes of memory?
Files on a Linux system have an inode table. The inode table consists of:
file size
ids
file mode
timestamps
symbolic link count
pointers to the disk
A directory is just a special file containing the inode numbers of the files inside it as an array. The size of this array is usually less 4096 bytes.
Filesystems in Linux are usually block aligned. 4096 bytes is a common block size, and since the directory size is less than 4096 bytes, it’s aligned at the 4096 byte boundary, making it’s size 4096 bytes.
Understanding Filesystems
Multiple pointers to the inode number may exist. These pointers are called hard links. All hard links are equivalent.
The alternative, is to have symbolic links, which point to the original file path, and not the inode number itself.
.center[]
class: inverse, center, middle
Unix Patterns
Wildcards
Wildcards are patterns for searching files
* represents zero or more characters
? represents a single character
[] represents a range of characters
1 2 3 4 5 6 7 8 9 10 11 12 13 14
$ ls blue blah blarg bar baz file example new.txt old.txt foo1 foo2 foo3
$ ls b* blue blah blarg bar baz
$ ls ba? bar baz
$ ls *.txt new.txt old.txt
$ ls *[0-9]* foo1 foo2 foo3
Under the hood, the shell auto completes wildcards before executing the command.
Permissions
Linux has 3 permissions:
r - read (view the file)
w - write (edit the file)
x - execute (run the file)
The permissions are granted for 3 groups:
u - owner - person who owns the file (usually the person who created it)
g - group - accessible to everyone in the group
o - others - accessible to everyone
Permissions
Use ls -l to view the permissions on a file:
1 2 3 4
$ touch file && mkdirdir $ ls -l -rw-rw-r-- 1 siddhu siddhu 0 May 13 02:03 file drwxrwxr-x 2 siddhu siddhu 4096 May 13 02:04 dir
The format is:
1 2 3 4 5
d r w x r w x r w x
^ ^ ^ ^ | | | | directory owner group others
A dash (-) indicates the absense of that field
rwx indicates read/write/execute privilege on the file
Permissions
Use chmod to alter the permissions of a file:
1 2 3 4 5
$ touch file $ chmod g+w file # Give (+) group (g) write (w) privilege $ chmod o-r file # Revoke (-) read (r) privilege from others (o) $ chmod a-x file # Revoke execute (x) privilege from all groups (a) $ chmod u+r file # Give read privilege to owner (u)
To set rwx permissions of all groups simultaneously, use binary representation:
| feeds the output of one command, as the input of the next
> writes the stdout of one command to a file
>> appends the stdout of one command to a file
2> writes the stderr of one command to a file
Processes
Threads:
The number of threads is limited by the hardware
The kernel can use special threads called kernel threads
Kernel threads run on top of hardware threads
POSIX allows multiple p-threads to run on a single thread
pthreads run on top of kernel threads, they add an address space
Userspace threads are called green threads
Compilers may allow multiple green threads to run on a single p-thread
Processes:
A process is a program which may use one or more threads
Processes are expensive, and should not be spawned unnecessarily
Processes are isolated by process isolation
Processes communicate using inter-process communication
Processes
Threads may run in userspace, or kernel space
.center[]
Processes
The OS manages theads, offloading to different hardware using different techniques. You can help the OS by specifying your execution model as a programmer.
If you want to output process info to stdout, use ps [aux]. It gives a lot of output, so we use grep to filter the results.
Processes
To kill an existing process, use kill [signal] <PID>
If kill doesn’t destroy the process by itself, you can pass the signal -9 which destroys the process at the kernel level.
1 2 3 4 5 6 7 8 9 10 11 12 13
$ ps aux | grep 'firefox' ryan 6978 8.8 23.5 2344096 945452 ? Sl 08:03 49:53 /usr/lib64/firefox/firefox
$ kill 6978
$ ps aux | grep 'firefox' ryan 6978 8.8 23.5 2344096 945452 ? Sl 08:03 49:53 /usr/lib64/firefox/firefox
$ kill -9 6978
$ ps aux | grep 'firefox'
$
Processes
A process can run as a foreground or background job.
To list the current background jobs, use the jobs command. Pressing Ctrl-Z while running a program sends it to the background. To bring a job back to the foreground, use fg <job-number>
1 2 3 4 5 6 7 8 9 10 11 12
$ sleep 15 & [1] 21637
$ sleep 10 # (you press CTRL + z, notice the prompt comes back.)
cat is used for concatenating files, but abused for printing files to stdout.
1 2 3 4 5 6 7 8 9 10 11 12 13 14
$ cat sample Fred apples 20 Susy oranges 5 Mark watermellons 12 Robert pears 4 Terry oranges 9 Lisa peaches 7 Susy oranges 12 Mark grapes 39 Anne mangoes 7 Greg pineapples 3 Oliver rockmellons 2 Betty limes 14 Julie bananas 30
head
head prints out the first n lines of it’s input. If no value for n is given, it defaults to printing 10 lines.
head -n file
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16
$ head sample Fred apples 20 Susy oranges 5 Mark watermellons 12 Robert pears 4 Terry oranges 9 Lisa peaches 7 Susy oranges 12 Mark grapes 39 Anne mangoes 7 Greg pineapples 3
$ head -3 sample Fred apples 20 Susy oranges 5 Mark watermellons 12
tail
tail prints out the last n lines of it’s input. If no value for n is given, it defaults to printing 10 lines.
tail -n file
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16
$ tail sample obert pears 4 Terry oranges 9 Lisa peaches 7 Susy oranges 12 Mark grapes 39 Anne mangoes 7 Greg pineapples 3 Oliver rockmellons 2 Betty limes 14 Julie bananas 30
sort is a utility to sort the input alphabetically (though, other sorting mechanisms are available)
sort -options file
1 2 3 4 5 6 7 8 9 10 11 12 13 14
$ sort sample Anne mangoes 7 Betty limes 14 Fred apples 20 Greg pineapples 3 Julie bananas 30 Lisa peaches 7 Mark grapes 39 Mark watermellons 12 Oliver rockmellons 2 Robert pears 4 Susy oranges 12 Susy oranges 5 Terry oranges 9
nl
nl numbers the lines in the input
nl -options file
1 2 3 4 5 6 7 8 9 10 11 12 13 14
$ nl -w 1 -s '. ' sample 1. Fred apples 20 2. Susy oranges 5 3. Mark watermellons 12 4. Robert pears 4 5. Terry oranges 9 6. Lisa peaches 7 7. Susy oranges 12 8. Mark grapes 39 9. Anne mangoes 7 10. Greg pineapples 3 11. Oliver rockmellons 2 12. Betty limes 14 13. Julie bananas 30
wc
wc counts the number of words (or lines/characters) in the input
cut is a neat utility for grabbing cloumns from a file
cut -options path
1 2 3 4 5 6 7 8 9 10 11 12 13 14
$ cut -f 1,3 -d ' ' sample Fred 20 Susy 5 Mark 12 Robert 4 Terry 9 Lisa 7 Susy 12 Mark 39 Anne 7 Greg 3 Oliver 2 Betty 14 Julie 30
Common options:
-f for fields
-d for delimiters
sed
sed is a stream editor, which should be used to search and replace text
sed <expression> file
1 2 3 4 5 6 7 8 9 10 11 12 13 14
$ sed 's/oranges/bananas/g' sample Fred apples 20 Susy bananas 5 Mark watermellons 12 Robert pears 4 Terry bananas 9 Lisa peaches 7 Susy bananas 12 Mark grapes 39 Anne mangoes 7 Greg pineapples 3 Oliver rockmellons 2 Betty limes 14 Julie bananas 30
awk
awk is a programming language for text manipulation
tac is just cat in reverse. It prints the last line first, and first line last.
tac file
1 2 3 4 5 6 7 8 9 10 11 12 13 14
$ tac sample Julie bananas 30 Betty limes 14 Oliver rockmellons 2 Greg pineapples 3 Anne mangoes 7 Mark grapes 39 Susy oranges 12 Lisa peaches 7 Terry oranges 9 Robert pears 4 Mark watermellons 12 Susy oranges 5 Fred apples 20
class: inverse, center, middle
Scripting
Scripting
A script is a document containing actions the shell can perform.
Scripts are typically written using sh, but recently python has been replacing shell scripts.
1 2 3 4 5
#!/bin/bash # declare STRING variable STRING="Hello World" # print variable on a screen echo$STRING
1 2 3 4 5
#!/bin/python # declare STRING variable STRING="Hello World" # print variable on a screen print STRING
Set execute privileges before running:
1 2 3
$ chmod +x hello_world $ ./hello_world Hello World
class: inverse, center, middle
Development Environments
Isolated Development Environments
Inspired by functional programming, isolated development environments prevent mutating the global system by creating isolated containers/shells.
Why?
Packages are installed in multiple locations (/bin, /sbin, /usr/bin, etc.)
A version control system maintains a database containing different versions of the file. Useful for seeing what changed, and rolling back to older revisions.
.right-column[]
class: inverse, center, middle
Advanced Navigation
Being more productive
.center[]
Special Variables
$1, $2, $3, ... are the positional parameters.
"$@" is an array-like construct of all positional parameters, {$1, $2, $3 …}.
"$*" is the IFS expansion of all positional parameters, $1 $2 $3 ….
$# is the number of positional parameters.
$- current options set for the shell.
$$ pid of the current shell (not subshell).
$_ most recent parameter (or the abs path of the command to start the current shell immediately after startup).
$IFS is the (input) field separator.
$? is the most recent foreground pipeline exit status.
$! is the PID of the most recent background command.
$0 is the name of the shell or shell script.
History Completion
!! executes the previous command
1 2 3 4 5 6 7 8
$ echo"Hello, World" Hello, World
$ !! echo"Hello, World" Hello, World
$
History Completion
!-n executes the n-th previous command
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17
$ echo"World 1" World 1
$ echo"World 2" World 2
$ echo"World 3" World 3
$ echo"World 4" World 4
$ !-3 echo"World 2" World 2
$
History Completion
!programName executes the previous command that starts with programName
1 2 3 4 5 6 7 8 9 10
$ ps | grep zsh 28618 pts/1 00:00:00 zsh
$ echo"Hello, World"
$ !ps ps | grep zsh 28618 pts/1 00:00:00 zsh
$
History Completion
!?text executes the previous command that contains the string text
1 2 3 4 5 6 7 8 9 10 11
$ ls /etc/cron.daily/passwd /etc/cron.daily/passwd
$ echo"Hello, World" Hello, World
$ !?cron ls /etc/cron.daily/passwd /etc/cron.daily/passwd
$
History Completion
^str1^str2^ replaces the string str1 in the previous command with str2
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17
$ ls -l /etc/cron.daily/passwd -rwxr-xr-x 1 root root 249 Feb 16 2014 /etc/cron.daily/passwd
]
]
]
]
]
]
]
]
]
]
]
]
]
]
]
]
]
]