通過《 The Linux Command Line》入門linux命令行

原文發於我的獨立博客：通過《The Linux Command Line》入門linux命令行

此書網站：The Linux Command Line ，它是免費的。

它有中文版，而且還有一個好聽的名字：快樂的 Linux 命令行

學習有兩種方法，一種是系統的學，一種是根據實際需求來學。兩種各有優缺點，前一種，知識不會有缺漏，對日後的融會貫通和觸類旁通有好處，但是這樣慢，而且有可能不如針對性學習的精神集中。後一種，只找需求的學，能夠任務完成就行，這樣快，它的缺點就是前一種的優點。顯然通看整本書屬於第一種。

作者寫的很好，娓娓道來，講的很清楚，就像一本故事書一樣。我很喜歡這樣的書。作者比較幽默，一些用詞看了會讓人發笑。

這本書的內容很淺，真的只是入門。

正如作者所說，這是一場linux的journey(旅行)，就把看書當旅行吧。

以下是隨手記的，只適合自己看。

命令

• date
• cal-calendar
• df-current amount of free space on your disk drives
• free-display the amount of free memory
• exit-closing the terminal emulator window

2 - Navigation

• pwd - Print name of current working directory
• cd - Change directory

cd             home directory
cd -           previous working directory
cd ~user_name  home directory of user
.              點號代表當前文件夾

• ls - List directory contents

ls -a   all files
ls -l   long format
ls -lt  sort the result by the file's modification time
ls -lt --reverse

• file – Determine file type

• less – View file contents

  和cat的區別？參考

4 - Manipulating Files And Directories

• cp

  如果文件已經存在，直接覆蓋，沒有提示，要提示用-i

• mv

  move (rename) files

• mkdir

mkdir dir1 dir2

• rm – Remove Files And Directories

rm -r playground    # 遞歸刪除

• ln – Create Links

ln myfile hard       # hard link
ln -s myfile soft    # symbolic link

軟鏈接”和“硬鏈接”的區別：硬鏈接指向磁盤區塊，軟鏈接指向絕對路徑。

hard links的缺點：

1.cannot span physical devices.

2.cannot reference directories, only files.

5 - Working With Commands

• type – Indicate how a command name is interpreted

tsc@tsc:~$ type ls
ls is aliased to `ls --color=auto'  # alias

tsc@tsc:~$ type cd
cd is a shell builtin               # builtin

tsc@tsc:~$ type cp
cp is /bin/cp                       # executable program

• which – Display An Executable’s Location

tsc@tsc:~$ which cp
/bin/cp

只用於exe

• help – Get help for shell builtins

  注意是shell builtin

tsc@tsc:/bin$ help cd
cd: cd [-L|[-P [-e]] [-@]] [dir]
    Change the shell working directory.

理解註釋：[]是可選，|是互斥。

• --help – Display Usage Information

tsc@tsc:~$ mkdir --help

• man – Display a command’s manual page

沒有example

tsc@tsc:~$ man ls

• apropos – Display a list of appropriate commands
• info – Display a command’s info entry
• whatis – Display a very brief description of a command
• alias – Create an alias for a command

起別名前先檢查是否存在，用type：

tsc@tsc:~$ type test
test is a shell builtin

使用：

alias name='string' # name後不能有空格
tsc@tsc:~$ alias foo='cd /usr; ls; cd -'
tsc@tsc:~$ foo

終端關閉後作用就消失了。

去除：

tsc@tsc:~$ unalias foo

• trick:多條命令於一行

command1; command2; command3...
tsc@tsc:~$ cd /usr; ls

• man .info .–help的用法和區別

  help最簡單，info最詳細，man在兩者之間。

6 – Redirection

• cat - Concatenate files

cat file1 # 輸出文件到屏幕，沒有分頁
cat > file.txt # cat沒有參數時接受stdin
this is test.
               # ctrl+d 結束
# < 接受文件輸入
tsc@tsc:~$ cat < lazy_dog.txt

• sort - Sort lines of text
• uniq - Report or omit repeated lines

# -d 顯示重複的
tsc@tsc:~$ ls /bin /usr/bin | sort | uniq -d | less

• grep - Print lines matching a pattern

“global regular expression print”,find功能。

tsc@tsc:~$ ls /bin /usr/bin | sort | uniq | grep zip

• wc - Print newline, word, and byte counts for each file

聯合使用看條目個數：

tsc@tsc:~$ ls /bin /usr/bin | sort | uniq | wc -l

• head - Output the first part of a file
• tail - Output the last part of a file
• tee - Read from standard input and write to standard output and files

這個名字很有意思，tee有三通管的意思，配合pipe使用，它的作用是從stdin讀入，複製到stdout和文件。

# 一方面輸出到ls.txt，一方面傳給grep
tsc@tsc:~$ ls /usr/bin/ | tee ls.txt | grep zip

• I/O redirection 的作用

  I/O redirection allows us to change where output goes and where input comes from.

• >重定向

tsc@tsc:~$ ls -l /usr/bin/ > ls-output.txt
tsc@tsc:~$ > ls-output2.txt  # 創建一個空文件

要注意重複>時，原來的文件會被清空。

• >> 追加

tsc@tsc:~$ ls -l /usr/bin >> ls-output2.txt

文件不存在會新建。

• 輸出error

tsc@tsc:~$ ls -l /bin/usr 2> ls-error.txt

我的問題：在不知道有沒有error的情況下，>和2>要如何同時執行？

• 都輸出到一個文件

有兩種方法：

# 1.老版本
# 順序很重要
tsc@tsc:~$ ls -l /bin/usr > ls-output.txt 2>&1

# 2.新，用&>
tsc@tsc:~$ ls -l /bin/usr &> ls-output.txt
tsc@tsc:~$ ls -l /bin/usr &>> ls-output.txt # 追加

• Pipelines

tsc@tsc:~$ ls -l /usr/bin | less

千萬不要把>用成|，不然有些後果很嚴重。

• filters

tsc@tsc:~$ ls /bin /usr/bin | sort | less

7 – Seeing The World As The Shell Sees It

這節講expansion， 就是*、~之類的，本質就是變量。

• echo – Display a line of text

It prints out its text arguments on standard output

tsc@tsc:~$ echo this is a test
this is a test

• Tilde Expansion

tsc@tsc:~/playground$ echo ~
/home/tsc

• Arithmetic Expansion

形式：

$((expression))

tsc@tsc:~/playground$ echo $((2+2))
4

• brace expansion

  括號中間不能有空格。

  用途：批量創建有順序的文件。

• Parameter Expansion

• Command Substitution

  command的結果也可以作爲expansion

echo $(ls)

• Double Quotes

  “$”, “\” (backslash), and “`” (backquote) 失效，但是parameter expansion, arithmetic expansion, and command substitution 仍有效。

注意區別：

echo $(cal)
echo "$(cal)"

空格、tab、換行符都被用來分割，雙引號則抑制。所以前一個命令不會換行，後一個會。

• Single Quotes

  抑制所有expansions

• escape character

和雙引號一起使用，抑制部分符號

8 – Advanced Keyboard Tricks

這節講鍵盤技巧，提高效率。主要是command line和history的技巧。command line的：移動，刪除。history的上下一個命令。

• clear – Clear the screen

• history – Display the contents of the history list

• Cursor Movement

• completion

  補全

• history

  history expansion-使用history中1209行的命令：

tsc@tsc:~$ !1209

• History Commands快捷鍵

9 – Permissions

思考：多用戶爲什麼存在？想想計算中心。

多用戶存在要解決的問題：

1.一個用戶的錯誤操作不能使計算機crash

2.一個用戶不能影響其他用戶

• 幾個相關文件夾

/etc/passwd
User accounts .

/etc/group
groups

/etc/shadow
user's password

• rwx

Owner Group World

• id – Display user identity

$ id
uid=1000(tsc) gid=1000(tsc) groups=1000(tsc),4(adm),24(cdrom),27(sudo),30(dip),46(plugdev),113(lpadmin),128(sambashare)

# root
# id
uid=0(root) gid=0(root) groups=0(root)

• chmod – Change File Mode
• File Modes中八進制的妙用

$ chmod 600 foo.txt

• umask – Set Default Permissions

• su – Run A Shell With Substitute User And Group IDs

  默認是superuser

$ su -
exit

• sudo – Execute A Command As Another User

  糾正自己的認識，su,sudo是改變用戶的，不一定是super，默認纔是。

• chown – Change File Owner And Group

• chgrp – Change Group Ownership

• passwd - change user password

10 – Processes

進程

Processes are how Linux organizes the different programs waiting for their turn at the CPU.

• ps - report a snapshot of the current processes.

• top - display Linux processes

• & - Putting A Process In The Background

• jobs - list the jobs that have been launched from our terminal.

• fg - Returning A Process To The Foreground

  爲什麼要回到前臺來？後臺的程序不受terminal的控制。

• Ctrl-z - Stopping (Pausing) A Process

  注意和ctrl-c的區別，c是結束。

• bg - resume the program’s execution in the background

• kill - “kill” processes

• killall - kill processes by name

• halt

• poweroff

• reboot

• shutdown

Part 2 – Configuration And The Environment

11 – The Environment

什麼是environment？

configuration的作用：store program settings

• printenv - print all or part of environment

# 所有environment
tsc@tsc:~$ printenv | less

# 某一個environment
tsc@tsc:~$ printenv USER

tsc@tsc:~$ echo $HOME

printenv顯示的一些常見變量：

SHELL
PATH   # 所有的，最新的

• set

  will show both the shell and environment variables

• printenv和set的區別

  printenv只顯示environment variables，set顯示shell和environment variables.

• 追加

PATH=$PATH:$HOME/bin

• export

  tells the shell to make the contents of PATH available to child processes of this shell.

• 啓動順序

Login Shell Sessions ：

/etc/profile     # global
~/.bash_profile
~/.bash_login    # 如果上面那個不存在,讀這個
~/.profile       # 如果上面那個不存在,讀這個

.profile文件中有PATH，並且將$HOME/bin添加上了，所以啓動系統後$HOME/bin中的命令是直接可以用的。$HOME是用戶目錄。

Non-Login Shell Sessions ：

/etc/bash.bashrc # global
~/.bashrc        # user's

• Login和Non-Login的區別？

  參考，

• PATH的作用

  命令從中找。

• 改環境變量時改哪個文件？

.profile：

add directories

define additional environment variables

.bashrc：

everything else

• Text Editors

有兩種：

1.graphical ：gedit

2.text based ：nano, vi, emacs

• nano

ctrl-x : exit
ctrl-o : save, write out
#      : comments

• 使.bashrc生效

首先要知道原理，啓動流程，在session一開始纔會read .bashrc，所以需要重啓terminal纔會生效。當然，可以強行重讀.bashrc，用source命令：

source .bashrc

12 – A Gentle Introduction To vi

• why vi?

  always available

  lightweight and fast

• vi 操作

:q
:q!   # 強制退出
i     # insert
esc   # to command mode
:w    # save ,有:的命令叫ex command

# move
h, j, k, l
ctrl-f/b
numberG
gg # 第一個字符
G last line of the file
0 (zero) 行首
^ To the first non-whitespace character on the current line.
$ end of current line
w beginning of next word or punctuation
W ignoring punctuation
b beginning of previous word or punctuation
B ignoring punctuation


# edit
u    # undo last change
i    # insert
# append
a    # append, i是前插
A    # 直接到最後append
# opening a line
o    # 新建一行在下面
O    # 新建一行在上面

# delete
x    # current character
3x
dd   # current line,d有cut的作用
dw   # 
dW   # ignoring punctuation
d$   # current cursor to end
d0   # current cursor to beginning
dG   # current cursor to end of file
d20G # current cursor to 20 line，不是要刪除20行
5dd  # 5行
# Cutting, Copying, And Pasting
p    # 小寫粘貼在後一行（dd），普通在光標前
P    # 大寫粘貼在前一行（dd），普通在光標後

p    # paste after cursor
P    # upper case, paste before cursor
  # copy
yy   # copy current line
5yy  # copy 5 line
yW   # ignoring punctuation
yw   # 
y$   # current cursor to end of line
y0   # current cursor to beginning of line
y^   # current cursor to first non-whitespace char
yG  # current cursor to last line of file
y20G # current cursor to 20th line

# redo,undo
u  # undo
ctrl+r # redo

J    # join line

# search, replace
fa   # 行內搜索a，按;繼續搜索
/    # 文件內搜索，n繼續
:%s/line/Line/g    # 替換
:%s/line/Line/gc   # 有提醒的替換
%代表對所有內容進行操作
:1,2s/line/Line/gc # 對1-2行進行操作

Ctrl-e, Ctrl-y # Scroll down and scroll up

The leading tilde characters (”~”) indicate that no text exists on that line.

• modal editor

  command mode

• Switching case of characters 大小寫轉換

You can change the case of text:

Toggle case “HellO” to “hELLo” with g~ then a movement.

Uppercase “HellO” to “HELLO” with gU then a movement.

Lowercase “HellO” to “hello” with gu then a movement.

Alternatively, you can visually select text then press ~to toggle case, or U to convert to uppercase, or u to convert to lowercase.

• 常規操作

# 到行尾
$  反：0
A  反：I
Word movement: w, e, b

Linux vi and vim editor: Tutorial and advanced features

13 – Customizing The Prompt

• PS1-prompt string one

就是這個：

tsc@tsc:~$

更改它：

tsc@tsc:~$ PS1="<\u@\h \w>\$ "
<tsc@tsc ~>$ 
\u   # username of the current user.
\h   # Hostname of the local machine minus the trailing domain name.
\w   # Name of the current working directory.
\$   # This displays a “$” character unless we have superuser privileges. In that case, it displays a “#” instead.

改顏色：

通過escape code 實現，escape code 的開頭是八進制的033 ，例子：

\033[0;30m
\033是開頭
0   是optional character attribute，0代表text color是normal，1是bold
30m 是instruction
tsc@tsc:~$ PS1="\[\033[0;31m\]<\u@\h \w>\$\[\033[0m\] "
\033[0m   # 代表結束脩改

Part 3 – Common Tasks And Essential Tools

要知道package是什麼，如何install, remove, update

14 – Package Management

linux的哪個發行版纔是最好的？考慮的因素不是界面，而是：

1.packaging system

2.vitality of the distribution’s support community

• packaging 的兩大陣營

Debian “.deb” camp

Red Hat “.rpm” camp

Packaging System     Distributions (Partial Listing)
Debian Style (.deb)  Debian, Ubuntu, Linux Mint, Raspbian
Red Hat Style (.rpm) Fedora, CentOS, Red Hat Enterprise Linux, OpenSUSE

• package files

• shared libraries

  軟件都是有依賴的，被依賴的就放在這裏

  dependency

• package tool

  分爲high, low-level兩種，

  high:metadata search, dependency

  low-level: installing, removing

  Red Hat系列使用同一low-lvel program(rpm), high-level則不同，yum使用者：Fedora, Red Hat Enter, CentOS.

Distributions	Low-Level Tools	High-Level Tools
Debian-Style	dpkg	apt-get, aptitude
Fedora, Red Hat, Enterprise Linux, CentOS	rpm	yum

- 查找和安裝package的方法
- What is the difference between apt-get update and upgrade?

15 – Storage Media

• 什麼是mounting?

attaching the device to the file system tree.

• mount – Mount a file system

沒有參數時，顯示已經mounted的文件系統。

如何看？device on mount_point type file_system_type (options).

• free - Display amount of free and used memory in the system

• partition number 是什麼？

• Unable to enumerate USB device under virtualbox

  usb3.0插的3.0的口上在vm中讀不出來，改插在2.0的口上就可以，真是。。。

• 磁盤、分區及Linux文件系統 [Disk, Partition, Linux File System]

  inode的作用：實現文件存儲的關鍵。

  ​

• tail的妙用

# ubuntu
tail -f /var/log/syslog

實時觀察系統在做什麼。

• partition layout 是什麼？
• IDE、SATA、SCSI、SAS、FC、SSD硬盤類型介紹
• fsck - check and repair a Linux filesystem

16 – Networking

• IP Addresses, Host Names, and Domain Names

  明白host name與domain name的區別。

• ping, ping6 - send ICMP ECHO_REQUEST to network hosts

• traceroute - print the route packets trace to network host

• ip - show / manipulate routing, devices, policy routing and tunnels

  DHCP (Dynamic Host Configuration Protocol)

• netstat - Print network connections, routing tables, interface statistics, masquerade connections, and multicast memberships

• ftp — Internet file transfer program

  ftp是明文傳輸。

• Wget - The non-interactive network downloader.

• ssh

  rlogin and telnet 和ftp有一樣的缺陷。

  ssh解決了兩大問題：First, it authenticates that the remote host is who it says it is (thus preventing so-called “man in the
  middle” attacks), and second, it encrypts all of the communications between the local and
  remote hosts.

ssh root@ip -p 端口號

• windows下的ssh軟件

  PuTTY

17 – Searching For Files

找文件。

• locate - find files by name

  locate查找的數據庫是如何更新的？由updatedb 程序更新，可以手動更新。

• locate vs find: usage, pros and cons of each other

• find - search for files in a directory hierarchy

$ find ~ -type f | wc
# 文件名有jpg，大小大於1M
$ find ~ -type f -name "*.jpg" -size +1M | wc -l

• test

$ find ~ \( -type f -not -perm 0600 \) -or \( -type d -not -perm 0700 \)

順序很重要。

• action

把查找的結果作爲命令的輸入參數

$ find ~ -type f -name 'foo*' -exec ls -l '{}' ';'
$ find ~ -type f -name 'foo*' -exec ls -l '{}' +

$ find ~ -type f -name 'foo*' -print | xargs ls -l

{} is a symbolic representation of the current pathname

semicolon is a required delimiter indicating the end of the command.

+：the system only has to execute the ls command once

• 文件名中空格的處理

$ find ~ -type f -name '*.jpg' -print0 | xargs --null ls -l

• touch - change file timestamps

  文件不存在時會創建。

• stat - display file or file system status

  which is a kind of souped-up version of ls.

18 – Archiving And Backup

管理大量的文件-壓縮。

• compression archiving區別？

本節講3個內容：

compression

archiving

synchronization

• gzip, gunzip, zcat - compress or expand files

gzip壓縮，gunzip解壓。

gzip會替換原來的文件。

• zcat - equivalent to gunzip with the -c option
• zless
• bzip2, bunzip2 - a block-sorting file compressor
• linux壓縮和打包是兩個要領，分別對應zip和tar，window下面是不是沒有區分？
• tar -

tape archive

.tar or .tgz, which indicate a “plain” tar archive and a gzipped archive, respectively.

# c create
$ tar cf playground.tar playground
# t list
$ tar tf playground.tar
# x extract
$ tar xf ../playground.tar
# r append

# extract a single file
$ tar xf archive.tar pathname

注意區分mode與option，c是mode，f是option。

mode要先寫。

• incremental backups - 增量式備份
• linux shell 重定向 減號-的作用

studout或者stdin就可以用減號（-）來替代

• 不解壓縮直接查看.gz和.bz2命令

使用zcat可以查看*.gz文件內容，使用bzcat可以直接查看*.bz2文件.

• 用ssh從遠程服務器複製文件到本地

$ ssh [e-mail] -p 端口號 'tar cf - filename' | tar xf -
[e-mail]'s password: 

先tar，傳回來再導出。

• zip - 既壓縮也歸檔
• zip,unzip的作用是交換indows文件，linux用tar,gzip
• rsync - 備份與同步

rsync options source destination
$ rsync -av playground foo

19 – Regular Expressions

後面幾章都是關於文字處理的。先講regular expression.

• metacharacters & literal characters

metacharacters是^ $ . [ ] { } - ? * + ( ) | \ ，其餘就是literal characters。

• The Any Character

就是點號，the dot or period character

• anchors

The caret (^) (只匹配行首出來的)and dollar sign ($) （只匹配行尾出現的）

$ grep -h '^zip' dirlist*.txt
$ grep -h 'zip$' dirlist*.txt
$ grep -h '^zip$' dirlist*.txt

• 點號不是任意長度嗎？加上^$就代表一個？

$ grep -h '.zip' dirlist*.txt
$ grep -i '^..j.r$' /usr/share/dict/words

• Bracket Expressions And Character Classes

[] 用來只匹配一個字符。

$ grep -h '[bg]zip' dirlist*.txt
^ : negation,只在第一個位置時有效
$ grep -h '[^bg]zip' dirlist*.txt # 注意出現的位置,表示取反
- : range
$ grep -h '^[A-Za-z0-9]' dirlist*.txt

# 要匹配-（短橫線），就把它放到第一位
$ grep -h '[A-Z]' dirlist*.txt
$ grep -h '[-AZ]' dirlist*.txt

• POSIX Character Classes

出現的原因：posix和ASCII不一樣。

• alternation

[]是單個，它(|)是string

$ echo "AAA" | grep -E 'AAA|BBB|CCC'
$ grep -Eh '^(bz|gz|zip)' dirlist*.txt

• Quantifiers

匹配的個數

? - Match An Element Zero Or One Time ,一個字符出現的次數，和word裏面的?不同，word裏面代表任意一個字符，這裏的?不能單獨使用，要跟在一個字符後面。

$ echo "[phone]" | grep -E '^\(?[0-9][0-9][0-9]\)? [0-9][0-9][0-9]-[0-9][0-9][0-9][0-9]$'

* - Match An Element Zero Or More Times

$ echo "This works." | grep -E '[[:upper:]][[:upper:][:lower:] ]*\.'

和?一樣，前面要有字符。

+ - Match An Element One Or More Times

至少出現一次。

$ echo "This that" | grep -E '^([[:alpha:]]+ ?)+$'

{ } - Match An Element A Specific Number Of Times

$ echo "[phone]" | grep -E '^\(?[0-9]{3}\)? [0-9]{3}-[0-9]{4}$'

• find使用reg

$ find . -regex '.*[^-_./0-9a-zA-Z].*'
#                   反             0-無窮個字符

• locate使用reg

$ locate --regex 'bin/(bz|gz|zip)'

• less和vi中也可以使用

注意它們的區別，在vi中reg是basic

# less
/^\([0-9]{3}\) [0-9]{3}-[0-9]{4}

# vi
/([0-9]\{3\}) [0-9]\{3\}-[0-9]\{4\}

20 – Text Processing

真是一切皆文件。

• cat的一些option

$ cat -A foo.txt # 顯示所有符號，包括控制符
$ cat -ns foo.txt # n顯示行號，s抑制多餘的空行

• sort

$ sort > foo.txt
c
b
a

# 多個文件
$ sort file1.txt file2.txt file3.txt > final_sorted_list.txt

du - estimate file space usage

$ du -s /usr/share/* | head # 顯示前10個
$ du -s /usr/share/* | sort -nr | head # 大到小排序後顯示前10個

按指定列排序(k)：

$ ls -l /usr/bin | sort -nr -k 5 | head

多個key排序：

$ sort --key=1,1 --key=2n distros.txt
# 一個field內多個排序：
$ sort -k 3.7nbr -k 3.1nbr -k 3.4nbr distros.txt

指定分割符：

$ sort -t ':' -k 7 /etc/passwd | head

• uniq

$ sort foo.txt | uniq

• cut

# 取第3列
$ cut -f 3 distros.txt

每個field的長度可能不同，如果要取所有行中時間中的年，就不可能用字符位置去表示。而先把field取出來，就規整了。

$ cut -f 3 distros.txt | cut -c 7-10

• expand - 把tab換爲等量空格

$ expand distros.txt | cut -c 23-

• 改cut的默認分割符，用d option

$ cut -d ':' -f 1 /etc/passwd | head

• paste

adds one or more columns of text to a file

$ sort -k 3.7nbr -k 3.1nbr -k 3.4nbr distros.txt > distros-by-date.txt
$ cut -f 1,2 distros-by-date.txt > distros-versions.txt
$ cut -f 3 distros-by-date.txt > distros-dates.txt
$ paste distros-dates.txt distros-versions.txt

• join

It joins data from multiple files based on a shared key field.

$ join distros-key-names.txt distros-key-vernums.txt | head

• comm

$ comm file1.txt file2.txt
# -n 抑制列
$ comm -12 file1.txt file2.txt

• diff

$ diff file1.txt file2.txt
# context format 看兩個各自都有什麼，沒有什麼
$ diff -c file1.txt file2.txt
# unified format 看從舊到新的變化
$ diff -u file1.txt file2.txt

• patch

apply changes to text files

用來更新文件，比如代碼。

$ diff -Naur file1.txt file2.txt > patchfile.txt
$ patch < patchfile.txt

• tr

轉換

$ echo "lowercase letters" | tr a-z A-Z
LOWERCASE LETTERS
#　多對一的轉換
$ echo "lowercase letters" | tr [:lower:] A
# squeeze
$ echo "aaabbbccc" | tr -s ab
abccc

• sed

stream editor

# search-and-replace
$ echo "front" | sed 's/front/back/'
# 誰跟在command後面就是delimiter
$ echo "front" | sed 's_front_back_'
# 指定要處理的行數
$ echo "front" | sed '1s/front/back/'

# p：打印出來
# -n：  not to print every line by default
$ sed -n '1,5p' distros.txt
$ sed -n '/SUSE/p' distros.txt
$ sed -n '/SUSE/!p' distros.txt

# 一般出輸出到屏幕，用i原地
$ sed -i 's/lazy/laxy/; s/jumped/jimped/' foo.txt

• reg的back references

$ sed 's/\([0-9]\{2\}\)\/\([0-9]\{2\}\)\/\([0-9]\{4\}\)$/\3-\1-\2/' distros.txt

• 全部替換

$ echo "aaabbbccc" | sed 's/b/B/'
$ echo "aaabbbccc" | sed 's/b/B/g'

• sed用script
• aspell

拼寫檢查

$ aspell check foo.txt

# -H (HTML) checking-mode option
$ aspell -H check foo.txt

改完了會產生一個bak文件，存在原來的文本。

21 – Formatting Output

講文本輸出格式。

• nl

顯示行號

$ nl distros.txt | head
# 和cat -n 一樣

$ sort -k 1,1 -k 2n distros.txt | sed -f distros-nl.sed | nl

• fold – Wrap Each Line To A Specified Length

$ echo "The quick brown fox jumped over the lazy dog." | fold -w 12
# 不打斷單詞
$ echo "The quick brown fox jumped over the lazy dog." | fold -w 12 -s

• fmt – A Simple Text Formatter

it fills and joins lines in text while preserving blank lines and indentation.

$ fmt -w 50 fmt-info.txt | head
$ fmt -cw 50 fmt-info.txt
# 一行如果不足50個字符，會把第二行的補過來
# 只對'# '開頭的行操作
$ fmt -w 50 -p '# ' fmt-code.txt

• pr – Format Text For Printing

used to paginate text.

# l:page length
# w:width
$ pr -l 15 -w 65 distros.txt

首尾會有空行。

• printf – Format And Print Data

“print formatted”

使用：printf “format” arguments

$ printf "I formatted the string: %s\n" foo
$ printf "%d, %f, %o, %s, %x, %X\n" 380 380 380 380 380 380

Multiple optional components ：

%[flags][width][.precision]conversion_specification

• Document Formatting Systems

tex後來取代了troff.

• groff

$ zcat /usr/share/man/man1/ls.1.gz | head
$ man ls | head
$ zcat /usr/share/man/man1/ls.1.gz | groff -mandoc -T ascii | head

# 存爲postscript到桌面
$ zcat /usr/share/man/man1/ls.1.gz | groff -mandoc > ~/Desktop/foo.ps

# ps轉pdf
$ ps2pdf ~/Desktop/foo.ps ~/Desktop/ls.pdf

• tbl

22 – Printing

介紹了ps的由來。

CUPS (Common Unix Printing System) ：provides print drivers and print-job management

Ghostscript： a PostScript interpreter, acts as a RIP.

• pr – Convert Text Files For Printing

pr is used to adjust text to fit on a specific page size, with optional page headers and margins.

# 3列
$ ls /usr/bin | pr -3 -w 65 | head

CUPS 有兩種printing，lpr, lp.

• lpr – Print Files (Berkeley Style)

$ ls /usr/bin | pr -3 | lpr
lpr: Error - no default destination available.
# 查看printer
$ lpstat -a
lpstat: No destinations added.

• lp – Print Files (System V Style)
• a2ps

“Anything to PostScript.”

$ ls /usr/bin | pr -3 -t | a2ps -o ~/Desktop/ls.ps -L 66 -R

• lpstat – Display Print System Status

# 查詢printer狀態
$ lpstat -a
$ lpstat -a

• lpq – Display Printer Queue Status

$ lpq
PDF is ready
no entries

• lprm / cancel – Cancel Print Jobs

23 – Compiling Programs

compile

linker，解決共享問題，用共享庫。

$ which gcc
/usr/bin/gcc

• 用ftp下載source code

$ mkdir src
$ cd src
$ ftp ftp.gnu.org
Name (ftp.gnu.org:me): anonymous
ftp> cd gnu/diction
ftp> ls
ftp> get diction-1.11.tar.gz
ftp> bye

$ tar xzf diction-1.11.tar.gz
# 先查看是不是在一個文件夾裏，以免解壓散開
$ tar tzvf diction-1.11.tar.gz | less

共享的頭文件在/usr/include ，The header files in this directory were installed when we installed the compiler.

• Building The Program

一般需要兩步。

# 由於script不在默認的目錄，需要./告訴shell它在當前目錄
$ ./configure
config.status: creating Makefile
config.status: creating diction.1
config.status: creating diction.texi
config.status: creating diction.spec
config.status: creating style.1
config.status: creating test/rundiction
config.status: creating config.h

Makefile is a configuration file that instructs the make program exactly how to build the program. Without it, make will refuse to run.

$ make

• make只complie需要complie的文件，省時間。
• Installing The Program

$ sudo make install

Part 4 – Writing Shell Scripts

24 – Writing Your First Script

• script開頭的#!

shebang，指明interpreter，每個script開頭都應該加這行。

• 讓script可以執行

$ chmod 755 hello_world

owner的權限是x就可以了。755所有人可以執行，700只有owner可以執行。必須是可讀的。

• 執行

$ ./hello_world

必須要加上./ ,因爲文件位置的問題。自動識別的命令的目錄都寫在了PATH中，可以查看它：

$ echo $PATH

• 把某個路徑添加到PATH

在.bashrc （它在~中）中添加：

# 把~/bin添加在PATH的前面
export PATH=~/bin:"$PATH"

# 在當前terminal中生效
$ . .bashrc
# 或者
$ source .bashrc

• script放哪？

 ~/bin ： 自己用
 /usr/local/bin ： 同一個系統多人用
 /usr/local/sbin： 系統管理員用
 /usr/local：   locally supplied software，

• line-continuations

就是\

• vi中對script的配置

:syntax on
:set hlsearch
:set tabstop=4
:set autoindent

永久：
寫在~/.vimrc，沒有就創建

25 – Starting A Project

shell是動態語言

uppercase letters to designate constants and lower case
letters for true variables.

$ vim ~/bin/sys_info_page
$ chmod 755 ~/bin/sys_info_page
$ sys_info_page
$ sys_info_page > sys_info_page.html
$ firefox sys_info_page.html

• Variables And Constants

uppercase letters to designate constants and lower case letters for true variables.

Note that in an assignment, there must be no spaces between the variable name, the equals sign, and the value.

a=z                  # Assign the string "z" to variable a.
b="a string"         # Embedded spaces must be within quotes.
c="a string and $b"  # Other expansions such as variables can be 
                     # expanded into the assignment.
d="$(ls -l foo.txt)" # Results of a command.
e=$((5 * 7))         # Arithmetic expansion.
f="\t\ta string\n"   # Escape sequences such as tabs and newlines.

大括號的作用

$ filename="myfile"
$ touch "$filename"
$ mv "$filename" "$filename1"
mv: cannot move 'myfile' to '': No such file or directory

$ mv "$filename" "${filename}1"

• here document

command << token
text
token

# shell example
cat << _EOF_
<HTML>
    <HEAD>
        <TITLE>$TITLE</TITLE>
    </HEAD>
</HTML>
_EOF_

# terminal example
$ cat << _EOF_
> $foo
> "$foo"
> '$foo'
> \$foo
> _EOF_

26 – Top-Down Design

講程序設計的思想。

• shell functions

兩種形式：

function name {
commands
return
}

and the simpler (and generally preferred) form:

name () {
commands
return
}

in order for function calls to be recognized as shell functions and not interpreted as the names of external programs, shell function definitions must appear in the script before they are called.

• local variables

要加local

funct_1 () {
    local foo # variable foo local to funct_1
    foo=1
    echo "funct_1: foo = $foo"
}

27 – Flow Control: Branching With if

shell的if語句挺複雜的。

if commands; then
    commands
[elif commands; then
    commands...]
[else
    commands]
fi

x=5
if [ "$x" -eq 5 ]; then
    echo "x equals 5."
else
    echo "x does not equal 5."
fi  

• exit status

程序結果後返回給系統的。查詢：

$ ls -d /usr/bin
$ echo $?

true - do nothing, successfully

false - do nothing, unsuccessfully

• test

語法：

test expression
and the more popular:
[ expression ]

expression是true時返回0，否則返回1

test和[ 本質上是一樣的。

使用：

#!/bin/bash

# test-file: Evaluate the status of a file

FILE=~/.bashrc

if [ -e "$FILE" ]; then
    if [ -f "$FILE" ]; then
        echo "$FILE is a regular file."
    fi
    if [ -d "$FILE" ]; then
        echo "$FILE is a directory."
    fi
    if [ -r "$FILE" ]; then
        echo "$FILE is readable."
    fi
    if [ -w "$FILE" ]; then
        echo "$FILE is writable."
    fi
    if [ -x "$FILE" ]; then
        echo "$FILE is executable/searchable."
    fi
else
    echo "$FILE does not exist"
    exit 1
fi

exit  # ?

When a script “runs off the end” (reaches end of file), it terminates with an exit status of the last command executed.

• String Expressions

#!/bin/bash
# test-string: evaluate the value of a string
ANSWER=maybe
if [ -z "$ANSWER" ]; then
    echo "There is no answer." >&2
    exit 1
fi
if [ "$ANSWER" = "yes" ]; then
    echo "The answer is YES."
elif [ "$ANSWER" = "no" ]; then
    echo "The answer is NO."
elif [ "$ANSWER" = "maybe" ]; then
    echo "The answer is MAYBE."
else
    echo "The answer is UNKNOWN."
fi

• Integer Expressions
• test的選項真多！！！
• test用regex

[[]]

# 檢驗是不是數
if [[ "$INT" =~ ^-?[0-9]+$ ]]; then

• (( )) - Designed For Integers

$ if ((1)); then echo "It is true."; fi
It is true.

if ((INT == 0)); 
if ((INT < 0)); 
if (( ((INT % 2)) == 0)); 

• Combining Expressions

Operation   test   [[ ]] and (( ))
AND         -a           &&
OR          -o           ||
NOT         !            !

例子：

# [[]]
if [[ "$INT" -ge "$MIN_VAL" && "$INT" -le "$MAX_VAL" ]];
# test
if [ "$INT" -ge "$MIN_VAL" -a "$INT" -le "$MAX_VAL" ];

• Control Operators: Another Way To Branch

The && (AND) and || (OR)

$ mkdir temp && cd temp
$ [[ -d temp ]] || mkdir temp

• []和[[]]的區別

[[]]和[]一樣，不過有兩個新特性，：

string1 =~ regex

== operator

• [[]]和表達式之間要有空格

[[ "$count" -gt 5 ]]
[[ "$count" -gt 5]] # 會報錯，執行起來會很可怕

28 – Reading Keyboard Input

• read – Read Values From Standard Input

read a single line of standard input.

read [-options] [variable...]

# -n option,suppresses the trailing newline on output
echo -n "Please enter an integer -> "
read int

read var1 var2 var3 var4 var5 # 多個

# -p prompt
read -p "Enter one or more values > "
# -t seconds
# -s Silent mode. 
if read -t 10 -sp "Enter secret passphrase > " secret_pass; 

• IFS

Internal Field Separator

file_info=$(grep "^$user_name:" $FILE)

# 寫在一行，IFS只改變跟在後面的 command
IFS=":" read user pw uid gid name home shell <<< "$file_info"

• here string

The <<< operator indicates a here string.

• You Can’t Pipe read

就像函數與子函數，凡是這一類都不能pip。

• Validating Input
• Menus

就是提示和if。

• 一個程序中出現多個exit是個不好的設計

那應該如何做？

29 – Flow Control: Looping With while / until

有3各loop

• while

語法：
while commands; do commands; done

#!/bin/bash
# while-count: display a series of numbers
count=1
while [[ "$count" -le 5 ]]; do
    echo "$count"
    count=$((count + 1))
done
echo "Finished."

• Breaking Out Of A Loop

break

continue

• until

和while相反

count=1
until [[ "$count" -gt 5 ]]; do
    echo "$count
    count=$((count + 1))
done
echo "Finished.

• read file with loop

#!/bin/bash

# while-read

while read distro version release; do
    printf "distro: %s\tversion: %s\treleased: %s\n" \
        "$distro" \
        "$version" \
        "$release"
done < distros.txt

30 – Troubleshooting

講錯誤的處理。

• 引號不對稱

• 特定符號缺失

• 意外的expansion

  是否報錯取決於expansion的值。

• 雙引號的作用

# number爲空時會報錯
number=
if [ $number = 1 ]; then
# 這樣就不會了  “”=1
[ "$number" = 1 ]

Multi-word string也應該用雙引號括起來。

所以，總是用雙引號把變量括起來。

• logical error
• 給出錯誤信息是比較好的做法
• unix文件名稱什麼字符都可以用，有兩個例外

/ , null character(a zero byte)

• the POSIX Portable Filename Character Set

uppercase letters A-Z, lowercase letters a-z, numerals 0-9, period (.), hyphen (-), and underscore(_).

• 檢查input
• 要會寫測試

利用echo輸出

在更改的地方加註釋，方便恢復。

• test case

case要反映edge and corner

• debug技術

commenting out :部分註釋

tracing :在一些關鍵點增加輸出，查看是否執行到這裏了

可以在script的第一行加入x option:

#!/bin/bash -x

也可以用set設置部分tracing

set -x # turn on
...
set +x # turn off

• 輸出中間過程中變量的值
• 推薦《The Art of Unix Programming》

31 – Flow Control: Branching With case

• case的用法

  1 #!/bin/bash
  2 
  3 # case-menu
  4 
  5 clear
  6 echo "
  7 please select:
  8 
  9 1. display system information
 10 2. display disk space
 11 3. display home space utilization
 12 0. quit
 13 "
 14 read -p "enter selection [0-3] > "
 15 
 16 case "$REPLY" in
 17     0)  echo "program terminated"
 18         exit
 19         ;;
 20     1)  echo "hostname: $HOSTNAME"
 21         uptime
 22         ;;
 23     2)  df -h
 24         ;;
 25     3) if [[ "$(id -u)" -eq 0 ]]; then
 26             echo "home space utilization (all users)"
 27             du -sh /home/*
 28         else
 29             echo "home space utilization ($USER)"
 30             du -sh "$HOME"
 31         fi
 32         ;;
 33     *)  echo "invalid entry" >&2
 34         exit 1
 35         ;;
 36 esac

32 – Positional Parameters

講處理command的option

• positional parameter

$0是當前命令的路徑。

注意它的含義：

Note that $0 always contains the full pathname of the first item on the command line (i.e., the name of the program) and does not contain the name of the shell function as we might expect.

是整個路徑，不是命令的名字。

大於9時用大括號括起來。

• 參數個數：$#
• shift

讓$1下次變成$2，所有參數都移動，$#減1

• basename

去除文件名前面的字符，只要文件名。

• positional parameter也可以用在function中

• Difference between echo -e “” and echo $“”

  e的意思就是擴展，支持轉義，e和$支持的符號部分不相同。

• $*和$@的區別

不加雙引號時是一樣的，遇到空格就拆分，加雙引號時不一樣，$*會將所有參數放到一個字符串中，$@會將每個輸入的參數分別當作參數，也就是不區分輸入時的空格。

$@經常用。

33 – Flow Control: Looping With for

講for

# 兩種風格
for variable [in words]; do
    commands
done

# c語言的
for (( expression1; expression2; expression3 )); do
    commands
done

例子：

[me@linuxbox ~]$ for i in A B C D; do echo $i; done

Create list:

itscs-MacBook-Pro:~ itsc$ for i in {A..D}; do echo $i; done
# 文件列表 pathname expansion
itscs-MacBook-Pro:learnCommandLine itsc$ for i in dis*.txt; do echo "$i"; done

34 – Strings And Numbers

• 一般情況下將變量括起來，防止被拆分
• parameter expansion是什麼？

變量是其中的一種

• 大括號的作用

$a
${a}
a="foo"
echo "${a}_file" # 和其它字符相連時防止形成不存在的變量

• 處理空的和不存在的字符

parameter爲空或者不存在就用word，存在就用它自己。
${parameter:-word}

和上面基本一樣，區別：要賦值。
${parameter:=word}

unset和empty就發發送word到error
${parameter:?word}

${parameter:+word}

• String operation

${#parameter} # 換成長度

$ foo="This string is long."
$ echo "'$foo' is ${#foo} characters long.
'This string is long.' is 20 characters long

${parameter:offset}
${parameter:offset:length}
[me@linuxbox ~]$ foo="This string is long."
[me@linuxbox ~]$ echo ${foo:5}
string is long.
[me@linuxbox ~]$ echo ${foo:5:6}
string

刪除一部分
${parameter#pattern}
${parameter##pattern}
[me@linuxbox ~]$ foo=file.txt.zip
[me@linuxbox ~]$ echo ${foo#*.}
txt.zip
[me@linuxbox ~]$ echo ${foo##*.}
zip

反着刪除
${parameter%pattern}
${parameter%%pattern}
[me@linuxbox ~]$ foo=file.txt.zip
[me@linuxbox ~]$ echo ${foo%.*}
file.txt
[me@linuxbox ~]$ echo ${foo%%.*}
file

替換
${parameter/pattern/string}
${parameter//pattern/string}
${parameter/#pattern/string}
${parameter/%pattern/string}

• 可以用expansion來提高script的效率
• 大小寫轉換

可以用來做什麼？比如數據庫的查找，匹配的時候把輸入和數據庫中的都統一大小寫。

• declare
• shell不能進行浮點運算

itscs-MacBook-Pro:learnCommandLine itsc$ echo $((3.3+4.2))
-bash: 3.3+4.2: syntax error: invalid arithmetic operator (error token is ".3+4.2")

解決：perl, awk.書裏用最簡單的bc

爲什麼連浮點運算都不支持？真是麻煩。

35 – Arrays

• scalar variables

Single value

• shell的array只能一維

[me@linuxbox ~]$ a[1]=foo
[me@linuxbox ~]$ echo ${a[1]}
foo
[me@linuxbox ~]$ declare -a a

# assign
name[subscript]=value
name=(value1 value2 ...)
[me@linuxbox ~]$ days=(Sun Mon Tue Wed Thu Fri Sat)
[me@linuxbox ~]$ days=([0]=Sun [1]=Mon [2]=Tue [3]=Wed [4]=Thu [5]=Fri [6]=Sat)

• 遍歷array

[me@linuxbox ~]$ animals=("a dog" "a cat" "a fish")
[me@linuxbox ~]$ for i in ${animals[*]}; do echo $i; done
[me@linuxbox ~]$ for i in ${animals[@]}; do echo $i; done
[me@linuxbox ~]$ for i in "${animals[*]}"; do echo $i; done
[me@linuxbox ~]$ for i in "${animals[@]}"; do echo $i; done

• bash的array不一定是要連續的

所以需要有方法知道哪些位置上有值

[me@linuxbox ~]$ foo=([2]=a [4]=b [6]=c)
[me@linuxbox ~]$ for i in "${foo[@]}"; do echo $i; done
a
b
c
[me@linuxbox ~]$ for i in "${!foo[@]}"; do echo $i; done
2
4
6

• array的追加

$ foo=(a b c)
$ foo[100]=e
$ echo ${foo[@]}
a b c e
$ foo+=(k l)
$ echo ${foo[@]}
a b c e k l
$ for i in "${foo[@]}"; do echo $i; done
a
b
c
e
k
l
$ for i in "${!foo[@]}"; do echo $i; done
0
1
2
100
101
102

subscript不是連續的

• associative arrays

下標可以是字符

36 – Exotica

講一些不常用的，在特定場景下使用的

• Group Commands And Subshells

Group command:
{ command1; command2; [command3; ...] }
Subshell:
(command1; command2; [command3;...])

它們是做什麼的？

manage redirection

{ ls -l; echo "Listing of foo.txt"; cat foo.txt; } > output.txt
(ls -l; echo "Listing of foo.txt"; cat foo.txt) > output.txt

結合pip

{ ls -l; echo "Listing of foo.txt"; cat foo.txt; } | lpr

• 注意大括號的使用

due to the way bash implements
group commands, the braces must be separated from the commands by a space and the
last command must be terminated with either a semicolon or a newline prior to the closing brace.

• group和subshell的區別

subshell和它的名字一樣，返回時enviroment會丟失，所以一般情況下用group.

echo "foo" | read
echo $REPLY # 這就是subshell的例子，reply是空的

commands in pipelines are always executed in subshells

• process substitution

用來解決subshell的問題

read < <(echo "foo")
echo $REPLY

• asynchronous execution
• 什麼是異步？

什麼是阻塞，非阻塞，同步，異步？

舉個打電話的例子：

阻塞/同步：打一個電話一直到有人接爲止
非阻塞：打一個電話沒人接，每隔10分鐘再打一次，知道有人接爲止
異步：打一個電話沒人接，轉到語音郵箱留言（註冊），然後等待對方回電（call back)

看起來異步是最高效，充分利用資源，可以想像整個系統能支持大規模併發。但問題是調試很麻煩，不知道什麼時候call back。

• Named pipes

create a connection between two processes and can be
used just like other types of files.

named pipe是一個文件起着pipe的作用

[me@linuxbox ~]$ mkfifo pipe1
[me@linuxbox ~]$ ls -l pipe1
prw-r--r-- 1 me me 0 2009-07-17 06:41 pipe1

# terminal 1
[me@linuxbox ~]$ ls -l > pipe1

# terminal2
[me@linuxbox ~]$ cat < pipe1

問題

• [[:upper:]]哪一節的？

• 每個文件夾下的.和..的作用？爲什麼必須要有它們？

  它們代表當前和前一級目錄，這個大家都知道，可以爲什麼要有它們？設計原理是什麼？

  參考：SO，只說了有，沒有說什麼爲什麼。

• shell中各種括號[]、[[]]的作用與區別？

  參考。

需要了解的知識

• linux文件系統

磁盤、分區及Linux文件系統 [Disk, Partition, Linux File System]

• 硬盤接口

IDE、SATA、SCSI、SAS、FC、SSD硬盤類型介紹

• Understand synopsis in manpage

  可以直接在命令man man中看。

• vi color

ENABLING VI SYNTAX COLORS

cd ~
vi .vimrc
syntax on
colorscheme desert

• vi indent

Redefine tab as 4 spaces

添加到vimrc中即永久：

set tabstop=4

• 標準錯誤輸出>&2輸出到哪了？

  屏幕。

• vim color

/usr/share/vim/vim74/colors/

我覺得還不錯的：

koehler

industry

murphy

ron

• ls如何按文件類型排序？

How to make ls sort by file extension and then name?

ls -lX

• 平鋪式窗口管理器-Awesome
• vi comment

Reveal the line numbers:

:set number

then

:5,$s/^/#/g     comment

or this:

:5,$s/^/#/      uncomment

• vim如何複製到clipboard?

Vim: copy selection to OS X clipboard still fail to work

To check if your Vim has clipboard support: vim --version | grep -i clipboard. You should see +clipboard. If you see -clipboard your Vim was compiled without clipboard support…

itscs-MacBook-Pro:learnCommandLine itsc$ vim --version | grep -i clipboard
-clipboard       +job             +path_extra      +user_commands
+eval            -mouse_dec       +statusline      -xterm_clipboard

我的是減號，說明不支持。

Getting Vim with +clipboard support

要先裝Homebrew，裝完後的vim在 /usr/local/bin/vim,原來的在/usr/bin，也存在，只不過PATH中，local的在前面，命令會先用它。

• 單引號、雙引號區別

echo $1
echo "$1"
echo '$1'

$ ./longest words
words
words
$1

雙引號與無引號在這裏是一樣的，雙引號的作用體現在字符串爲空的時候，爲空的時候是空字符串，可以判斷，而無引號就麻煩了，比如等號，一邊爲空會報錯，說等號是個二元操作符兩邊都要有值。單引號的作用就是禁止轉義，全部爲字符串。

• linux中shell變量#, @,0, 1,$2的含義解釋

$#
添加到Shell的參數個數

$! shell parameter, which will always contain the process ID of the
last job put into the background.

網站

The Bash Hackers Wiki