bcc-tools工具之offcputime【轉】

轉自：https://www.cnblogs.com/haoxing990/p/12203997.html

先了解什麼是off-cpu

• On-CPU: where threads are spending time running on-CPU.
• Off-CPU: where time is spent waiting while blocked on I/O, locks, timers, paging/swapping, etc.

從上面的意思基本上了解offcputime的意思是什麼了:用於測量某一進程被阻塞的時間。

老樣子，還是從help開始說起：

usage: offcputime [-h] [-p PID | -t TID | -u | -k] [-U | -K] [-d] [-f]
                  [--stack-storage-size STACK_STORAGE_SIZE]
                  [-m MIN_BLOCK_TIME] [-M MAX_BLOCK_TIME] [--state STATE]
                  [duration]

Summarize off-CPU time by stack trace

positional arguments:
  duration              duration of trace, in seconds

optional arguments:
  -h, --help            show this help message and exit
  -p PID, --pid PID     trace this PID only    #僅僅跟蹤某一進程阻塞時間
  -t TID, --tid TID     trace this TID only　　 #僅僅跟蹤某一線程阻塞時間
  -u, --user-threads-only
                        user threads only (no kernel threads) #僅僅跟蹤用戶態而非內核態線程阻塞時間
  -k, --kernel-threads-only
                        kernel threads only (no user threads) #僅僅跟蹤內核態線程阻塞時間
  -U, --user-stacks-only
                        show stacks from user space only (no kernel space  #僅僅顯示用戶態調用棧關係
                        stacks)
  -K, --kernel-stacks-only                                     #僅僅顯示內核態調用棧關係
                        show stacks from kernel space only (no user space
                        stacks)
  -d, --delimited       insert delimiter between kernel/user stacks  #將內核態和用戶態調用棧分割開來
  -f, --folded          output folded format                   # 採用摺疊模式輸出d
  --stack-storage-size STACK_STORAGE_SIZE　　　　　　　　　　　　  # 設置棧跟蹤過程存儲空間大小
                        the number of unique stack traces that can be stored
                        and displayed (default 1024)
  -m MIN_BLOCK_TIME, --min-block-time MIN_BLOCK_TIME           #只打印阻塞時間不小於xxx us的進程情況       
                        the amount of time in microseconds over which we store  
                        traces (default 1)
  -M MAX_BLOCK_TIME, --max-block-time MAX_BLOCK_TIME           #只打印阻塞時間不大於xxx us的進程情況
                        the amount of time in microseconds under which we
                        store traces (default U64_MAX)
  --state STATE         filter on this thread state bitmask (eg, 2 ==   #追蹤進程爲某種狀態的阻塞情況，例如不可中斷
                        TASK_UNINTERRUPTIBLE) see include/linux/sched.h

./offcputime -K

顯示所有進程內核態棧調用情況

schedule
    schedule_timeout
    io_schedule_timeout
    bit_wait_io
    __wait_on_bit
    wait_on_page_bit_killable
    __lock_page_or_retry
    filemap_fault
    __do_fault
    handle_mm_fault
    __do_page_fault
    do_page_fault
    page_fault
    chmod
        13  表示阻塞時間爲13us

    schedule
    rcu_nocb_kthread
    kthread
    ret_from_fork
    ddebug_tables
    rcuos/0
        22 表示阻塞時間爲22us

./offcputime -K -f 5

採用摺疊方式顯示棧調用流程

bash;entry_SYSCALL_64_fastpath;sys_read;vfs_read;__vfs_read;tty_read;n_tty_read;call_rwsem_down_read_failed;rwsem_down_read_failed;schedule 8
yes;entry_SYSCALL_64_fastpath;sys_write;vfs_write;__vfs_write;tty_write;n_tty_write;call_rwsem_down_read_failed;rwsem_down_read_failed;schedule 14
run;page_fault;do_page_fault;__do_page_fault;handle_mm_fault;__do_fault;filemap_fault;__lock_page_or_retry;wait_on_page_bit_killable;__wait_on_bit;bit_wait_io;io_schedule_timeout;schedule_timeout;schedule 33
rcuos/4;ret_from_fork;kthread;rcu_nocb_kthread;schedule 45
bash;entry_SYSCALL_64_fastpath;sys_read;vfs_read;__vfs_read;pipe_read;pipe_wait;schedule 88
run;page_fault;do_page_fault;__do_page_fault;handle_mm_fault;__do_fault;filemap_fault;__lock_page_or_retry;wait_on_page_bit_killable;__wait_on_bit;bit_wait_io;io_schedule_timeout;schedule_timeout;schedule 108
jbd2/xvda1-8;mb_cache_list;ret_from_fork;kthread;kjournald2;jbd2_journal_commit_transaction;__wait_on_buffer;out_of_line_wait_on_bit;__wait_on_bit;bit_wait_io;io_schedule_timeout;schedule_timeout;schedule 828
jbd2/xvda1-8;mb_cache_list;ret_from_fork;kthread;kjournald2;jbd2_journal_commit_transaction;__wait_on_buffer;out_of_line_wait_on_bit;__wait_on_bit;bit_wait_io;io_schedule_timeout;schedule_timeout;schedule 6201
supervise;entry_SYSCALL_64_fastpath;sys_rename;dput;__dentry_kill;iput;evict;ext4_evict_inode;truncate_inode_pages_final;truncate_inode_pages_range;wait_on_page_bit;__wait_on_bit;bit_wait_io;io_schedule_timeout;schedule_timeout;schedule 41049
run;entry_SYSCALL_64_fastpath;sys_wait4;do_wait;schedule 120709
bash;entry_SYSCALL_64_fastpath;sys_wait4;do_wait;schedule 699320 us

更多例子

examples:
    ./offcputime             # trace off-CPU stack time until Ctrl-C
    ./offcputime 5           # trace for 5 seconds only
    ./offcputime -f 5        # 5 seconds, and output in folded format
    ./offcputime -m 1000     # trace only events that last more than 1000 usec
    ./offcputime -M 10000    # trace only events that last less than 10000 usec
    ./offcputime -p 185      # only trace threads for PID 185
    ./offcputime -t 188      # only trace thread 188
    ./offcputime -u          # only trace user threads (no kernel)
    ./offcputime -k          # only trace kernel threads (no user)
    ./offcputime -U          # only show user space stacks (no kernel)
    ./offcputime -K          # only show kernel space stacks (no user)

 

是時候好好總結下自己走過的路。