linux系統分析工具續-SystemTap和火焰圖（Flame Graph）

本文爲網上各位大神文章的綜合簡單實踐篇，參考文章較多，有些總結性東西，自認暫無法詳細寫出，建議讀文中列出的參考文檔，相信會受益頗多。下面開始吧（本文出自 “cclo的博客” 博客，請務必保留此出處http://xuclv.blog.51cto.com/5503169/1184517）

SystemTap簡介：

  SystemTap provides free software (GPL) infrastructure to simplify the gathering of information about the running Linux system. This assists diagnosis of a performance or functional problem. SystemTap eliminates the need for the developer to go through the tedious and disruptive instrument, recompile, install, and reboot sequence that may be otherwise required to collect data.
  SystemTap provides a simple command line interface and scripting language for writing instrumentation for a live running system. We are publishing samples, as well as enlarging the internal "tapset" script library to aid reuse and abstraction.

詳情參考：

http://sourceware.org/systemtap/wiki

http://www.ibm.com/developerworks/cn/linux/l-systemtap/

http://blog.yufeng.info/archives/855

火焰圖簡介：

  通過性能分析來確定什麼原因導致CPU繁忙是日常工作中長做的事情，這往往會涉及到棧性能分析。通過定期採樣的方式來確定哪些代碼是導致CPU繁忙的原因是一個比較粗糙的方法。一個更好的方式是創建一個定時中斷來收集程序運行時的計數，函數地址，甚至整個堆棧回溯，最後打印爲我們可閱讀的報告.我們常用的性能分析工具有oprofile,gprof,dtracesystemtap 等
  Flame Graph：火焰圖，是一個把採樣所得到的堆棧跟蹤可視化展示的工具。它是基於上面提到的性能分析工具的結果，Flame graph本身並不具備性能檢測的能力。

參考：

https://github.com/brendangregg/FlameGraph

http://dtrace.org/blogs/brendan/2011/12/16/flame-graphs/

http://dtrace.org/blogs/brendan/2012/03/17/linux-kernel-performance-flame-graphs/

systemtap安裝及生成nginx的火焰圖：

  OS：ubuntu12.10 				#支持uprobes機制
  systemtap：2.1					#盡少的bug

1> 安裝systemtap，這裏源碼編譯

  1:安裝編譯環境：
   $ sudo apt-get install build-essential

  2:安裝systemtap
   $ wget http://sourceware.org/systemtap/ftp/releases/systemtap-2.1.tar.gz
   $ tar zxvf systemtap-2.1.tar.gz
   $ sudo ./configure -prefix=/opt/systemtap -disable-docs -disable-publican -disable-refdocs
   如報以下錯：
   configure: error: missing elfutils development headers/libraries (install elfutils-devel, libebl-dev, libdw-dev and/or libebl-devel
   $ sudo apt-get install libdw-dev      #其實缺的該是libdw1這個包，按提示安裝libdw-dev罷了，另三個包都木有
   $ sudo make
   $ sudo make install

  3:創建個軟鏈接：
   $ sudo ln -s /opt/systemtap/bin/stap /usr/sbin/stap

2> 安裝debug packages (Kernel debug info packages on Ubuntu, which will aid in providing information for bugs)

  1：$ uname -r
     3.5.0-17-generic

  2:在http://ddebs.ubuntu.com/pool/main/l/linux/網址找到對應內核的的debug packages，下載並安裝之
    $ sudo dpkg -i linux-p_w_picpath-3.5.0-17-generic-dbgsym_3.5.0-17.28_amd64.ddeb

如沒有找到自己系統對於的debug packages，建一個，方法如下：(How do I build a debuginfo kernel if one isn't available?)（官網原文，本文不涉及）
    $ cd $HOME
    $ sudo apt-get install dpkg-dev debhelper gawk
    $ mkdir tmp
    $ cd tmp
    $ sudo apt-get build-dep --no-install-recommends linux-p_w_picpath-$(uname -r)
    $ apt-get source linux-p_w_picpath-$(uname -r)
    $ cd linux-2.6.31 (this is currently the kernel version of 9.10)
    $ fakeroot debian/rules clean
    $ AUTOBUILD=1 fakeroot debian/rules binary-generic skipdbg=false
    $ sudo dpkg -i ../linux-p_w_picpath-debug-2.6.31-19-generic_2.6.31-19.56_amd64.ddeb

3> 測試systemtap，出現hello world，安裝完成

   $ sudo stap -e 'probe kernel.function("sys_open") {log("hello world") exit()}'

4> systemtap安裝參考：

http://sourceware.org/systemtap/wiki/SystemtapOnUbuntu

https://wiki.ubuntu.com/Kernel/Systemtap

http://www.dcshi.com/?p=124

5> 安裝LNMP（非必須，安裝nginx即可）

   $ sudo apt-get install nginx mysql-server mysql-client php5 php5-fpm php5-mysql

   $ sudo vim /etc/nginx/sites-enabled/default     #啓用下面幾行

location ~ \.php$ { 
        fastcgi_split_path_info ^(.+\.php)(/.+)$; 
#       # NOTE: You should have "cgi.fix_pathinfo = 0;"in php.ini 
# 
#       # With php5-cgi alone: 
#       fastcgi_pass 127.0.0.1:9000; 
#       # With php5-fpm: 
        fastcgi_pass unix:/var/run/php5-fpm.sock; 
        fastcgi_index index.php; 
        include fastcgi_params; 
}

   $ sudo vim /usr/share/nginx/www/index.php       #隨便建個測試頁

<html><body><h1>It is my works!!</h1></body></html> 
<?php 
$link=mysql_connect('localhost','root','root'); 
if ($link) 
    echo "Success"; 
else
    echo "Failure"; 
mysql_close(); 
phpinfo(); 
?>

   訪問http://192.168.1.94/	                  #訪問確保正常

6> 用systemtap進行nginx的分析，並用flameGraph畫出相應的火焰圖。

   1：編寫systemtap腳本: vim ngx.stp

global s; 
global quit = 0; 
probe timer.profile { 
    if (pid() == target()) { 
        if (quit) { 
            foreach (i in s-) { 
                print_ustack(i); 
                printf("\t%d\n", @count(s[i])); 
            } 
            exit() 
        } else { 
            s[ubacktrace()] <<< 1; 
        } 
    } 
} 
probe timer.s(20) { 
    quit = 1 
}

   2:執行如下命令：

   $ sudo stap --ldd -d /usr/sbin/nginx --all-modules -D MAXMAPENTRIES=256 -D MAXACTION=20000 -D MAXTRACE=100 -D MAXSTRINGLEN=4096 -D MAXBACKTRACE=100 -x 2082 ngx.stp --vp 0001 > ngx.out
      可能需修改的地方：
      /usr/sbin/nginx 	             #nginx的安裝位置，一般是此
      -D MAXMAPENTRIES=256 	     #本機是個虛擬機，只給了其512M內存，爲了保證內存不溢出，設爲256
      -x 2082			     #指定其中一個nginx worker進程的pid
   在執行以上命令的同時保證對nginx的壓力測試，另開終端執行：
   $ ab -n 900000 -c 50 http://192.168.1.94/index.php

   $ perl stackcollapse-stap.pl ngx.out > ngx.out2

   $ perl flamegraph.pl ngx.out2 > ngx.svg
                                   #以上2個perl腳本在https://github.com/brendangregg/FlameGraph

7> 訪問192.168.1.94/ngx.svg        #火焰圖出爐了

   每個框代表一個棧裏的一個函數
   Y軸代表棧深度（棧楨數）。最頂端的框顯示正在運行的函數，這之下的框都是調用者。在下面的函數是上面函數的父函數
   X軸代表採樣總量。從左到右並不代表時間變化，從左到右也不具備順序性
   框的寬度代表佔用CPU總時間。寬的框代表的函數可能比窄的運行慢，或者被調用了更多次數。框的顏色深淺也沒有任何意義
   如果是多線程同時採樣，採樣總數會超過總時間

8> 可能遇到的問題：

  1：如果有類似如下行，執行 $ sudo rm .systemtap -rf
   Pass 4: using cached /home/ubuntu/.systemtap/cache/24/stap_2479e8647ccf262def735ebf0eeb57c5_5866.ko

  2：如果有類似如下行，那應該有指望
   Pass 4: compiled C into "stap_2711449662ecf03b09d98a743c2122d0_5946.ko" in 24060usr/6960sys/103610rea

  3：如果有類似如下行，可以考慮忽略下
   WARNING: Missing unwind data for module, rerun with 'stap -d stap_2711449662ecf03b09d98a743c2122d0_3437'

  4：如果火焰圖只出現大量的init函數，說明完全不正確，暫不知道怎麼解決，nginx如果是apt-get安裝，嘗試執行如下行，可能有用
   $ sudo apt-get install nginx-full-dbg

  5：WARNING: missing unwind/symbol data for module 'kernel' :user-space facilities not available without kernel CONFIG_UTRACE 
   這個錯誤是說你的 kernel 沒有提供 utrace/uprobes 用戶態支持。兩種解法：
   1. 自己給Ubuntu自帶的老kernel應用utrace補丁，並重新編譯它。chaoslawful老師寫過一篇博客分享過ubuntu上的步驟：http://chaoslawful.iteye.com/blog/1463564
   2. 將kernel升級到官方最新的3.5或以上的版本。最新的kernel默認包含了uprobes機制，不再需要utrace補丁了（這是爲什麼用ubuntu12.10的原因）

9> 火焰圖生成參考的文檔：

http://www.dcshi.com/?p=287

https://groups.google.com/forum/#!msg/openresty/u-puKWWONMk/bxsyQdWMkJIJ

http://idning.github.io/systemtaprst.html

附：

$ stap -h
Systemtap translator/driver (version 2.1/0.153, non-git sources)
Copyright (C) 2005-2013 Red Hat, Inc. and others
This is free software; see the source for copying conditions.
enabled features: TR1_UNORDERED_MAP NLS

Usage: stap [options] FILE         Run script in file.
   or: stap [options] -            Run script on stdin.
   or: stap [options] -e SCRIPT    Run given script.
   or: stap [options] -l PROBE     List matching probes.
   or: stap [options] -L PROBE     List matching probes and local variables.

Options (in /home/ubuntu/.systemtap/rc and on command line):
   --         end of translator options, script options follow
   -h --help  show help
   -V --version  show version
   -p NUM     stop after pass NUM 1-5, instead of 5			
              (parse, elaborate, translate, compile, run)
   -v         add verbosity to all passes
   --vp {N}+  add per-pass verbosity [00000]
   -k         keep temporary directory
   -u         unoptimized translation 
   -w         suppress warnings 
   -W         turn warnings into errors 
   -g         guru mode 
   -P         prologue-searching for function probes 
   -b         bulk (percpu file) mode 
   -s NUM     buffer size in megabytes, instead of 0
   -I DIR     look in DIR for additional .stp script files, in addition to
              /opt/systemtap/share/systemtap/tapset
   -D NM=VAL  emit macro definition into generated C code
   -B NM=VAL  pass option to kbuild make
   --modinfo NM=VAL
              include a MODULE_INFO(NM,VAL) in the generated C code
   -G VAR=VAL set global variable to value
   -R DIR     look in DIR for runtime, instead of
              /opt/systemtap/share/systemtap/runtime
   -r DIR     cross-compile to kernel with given build tree; or else
   -r RELEASE cross-compile to kernel /lib/modules/RELEASE/build, instead of
              /lib/modules/3.5.0-17-generic/build
   -a ARCH    cross-compile to given architecture, instead of x86_64
   -m MODULE  set probe module name, instead of 
              stap_4736
   -o FILE    send script output to file, instead of stdout. This supports
              strftime(3) formats for FILE
   -c CMD     start the probes, run CMD, and exit when it finishes
   -x PID     sets target() to PID
   -F         run as on-file flight recorder with -o.
              run as on-memory flight recorder without -o.
   -S size[,n] set maximum of the size and the number of files.
   -d OBJECT  add unwind/symbol data for OBJECT file
   --ldd      add unwind/symbol data for all referenced object files.
   --all-modules
              add unwind/symbol data for all loaded kernel objects.
   -t         collect probe timing information
   --runtime=MODE
              set the pass-5 runtime mode, instead of kernel
   --privilege=PRIVILEGE_LEVEL
              check the script for constructs not allowed at the given privilege level
   --unprivileged
              equivalent to --privilege=stapusr
   --compatible=VERSION
              suppress incompatible language/tapset changes beyond VERSION,
              instead of 2.1
   --check-version
              displays warnings where a syntax element may be 
              version dependent
   --skip-badvars
              substitute zero for bad context $variables
   --suppress-handler-errors
              catch all runtime errors, quietly skip probe handlers
   --use-server[=SERVER-SPEC]
              specify systemtap compile-servers
   --list-servers[=PROPERTIES]
              report on the status of the specified compile-servers:
              all,specified,online,trusted,signer,compatible
   --remote=HOSTNAME
              run pass 5 on the specified ssh host.
              may be repeated for targeting multiple hosts.
   --remote-prefix
              prefix each line of remote output with a host index.
   --tmpdir=NAME
              specify name of temporary directory to be used.
   --download-debuginfo[=OPTION]
              automatically download debuginfo using ABRT.
              yes,no,ask,<timeout value>
   --dump-probe-types
              show a list of available probe types.
   --sysroot=DIR
              specify sysroot directory where target files (executables,
              libraries, etc.) are located.
   --sysenv=VAR=VALUE
              provide an alternate value for an environment variable
              where the value on a remote system differs.  Path
              variables (e.g. PATH, LD_LIBRARY_PATH) are assumed to be
              relative to the sysroot.
   --suppress-time-limits
              disable -DSTP_NO_OVERLOAD -DMAXACTION and -DMAXTRYACTION limits

報告完畢！！！

linux系統分析工具續-SystemTap和火焰圖（Flame Graph）

如何使用 JS 判斷用戶是否處於活躍狀態

通過HPA+CronHPA組合應對業務複雜彈性伸縮場景

Linux系統分析工具之vmstat,free,ping（三）

Linux系統分析工具之nicstat，dstat（四）

（總結）Nginx配置文件nginx.conf中文詳解

linux系統分析工具之Blktrace（十）

MongoDB journaling 工作原理

https://yachay.unat.edu.pe/blog/index.php?comment_area=format_blog&comment_component=blog&comment_co

linux以太網驅動總結