基於 dd 的塊設備讀寫性能測試

概述

本文檔說明測試塊設備（block device）讀寫性能的方法。塊設備包含如 SATA、USB、SD、eMMC、Nand 等。

測試技術要點

1. 讀寫使用 dd 命令，基於塊設備在 /dev/ 下的設備文件進行操作，不依賴文件系統。
2. 從 /dev/urandom 獲取隨機數進行讀寫測試。
3. 考慮硬件框圖、數據流向。直接從 DDR 內存讀取數據寫入到待測設備，直接從待測設備讀取數據然後存儲到 DDR 內存。不能從待測設備讀取再寫入待測設備，把待測設備的讀與寫糾纏一起。
4. 在 tmpfs /dev/shm/ 路徑下暫存數據，即使用 DDR 內存。
5. 清除緩存數據：echo 3 > /proc/sys/vm/drop_caches 。確保數據確實寫入到、讀取自待測設備，避免後續的 cmp 比較只是比較緩存中的數據。
6. 注意 dd 的 bs 設置會影響總體測試數據。512 KB 以上會達到最快。讀寫模式使用 conv=sync 確保數據完成寫入、讀取。
7. 注意操作系統不能從待測設備啓動，避免操作系統其他數據佔用待測設備的通信帶寬。
8. 統計 CPU 負載。計算不同數據量測試的平均值、方差。

測試操作方法

確認 /dev/shm、待測設備的可用空間

TARGET# df -h

配置最大測試數據量

測試程序 blk_dd_rw_diagnose.sh 的 MAX_COUNT 配置必須小於 /dev/shm 可用空間的一半，同時必須小於待測設備的可用空間，單位 MB。

執行測試

TARGET# ./blk_dd_rw_diagnose.sh -n /dev/mmcblk1p3

檢查測試結果

測試將生成兩個文件 write_dia.log read_dia.log，注意是否有報錯讀寫數據不一致，留意測試的平均值、方差與 CPU 佔用率。方差大說明本輪測試的多個不同數據量的測試結果波動性大。

參考代碼

#! /bin/bash
# Description:
#   Diagnose data read/write speed of block device using dd command.

# NOTE: run df -h to see max size of /dev/shm, two files saving here.
readonly MAX_COUNT=200      # size = count * 1 MB. Max size is half of /dev/shm.
readonly COUNT_STEP=10      # Incresing step.
readonly BLOCK_SIZE=1048576 # 1 MB
readonly RANDOM_SRC_FILE="/dev/shm/random_file.test"
readonly READ_FILE="/dev/shm/read_file.test"
readonly WRITE_DIA_FILE='write_dia.log'
readonly READ_DIA_FILE='read_dia.log'

usage() {
cat <<-EOF >&2
    usage: ./${0##*/} [-m mount point] [-n device node]
        -h Print this usage
        e.g. ./${0##*/} -m /run/media/sda1
        e.g. ./${0##*/} -n /dev/sda1
EOF
exit 0
}

########## MAIN ##########
DEV_NODE=""
MNT_POINT=""
while getopts :n:m:h arg
do case $arg in
        n)      DEV_NODE="$OPTARG";;
        m)      MNT_POINT="$OPTARG";;
        h)      usage;;
        :)      echo "$0: Must supply an argument to -$OPTARG." >&2
                exit 1
                ;;
        \?)     echo "Invalid Option -$OPTARG ignored." >&2
                usage
                exit 1
                ;;
esac
done

if [[ ! -z ${DEV_NODE} ]];then
    DST_FILE=${DEV_NODE}
elif [[ ! -z ${MNT_POINT} ]];then
    DST_FILE="${MNT_POINT}/dst_file"
else
    echo "ERROR! Yout need to specify at least one of -n or -m."
    usage
    exit 1
fi

echo "MB  secs  MB/s  CPU" > ${WRITE_DIA_FILE}
echo "MB  secs  MB/s  CPU" > ${READ_DIA_FILE}

cnt=${COUNT_STEP}
while [[ ${cnt} -le ${MAX_COUNT} ]];do
    echo "Testing ${cnt} MB data write and read..."

    # Generate random data file.
    echo "Generating ${cnt} MB random data file with /dev/urandom ..."
    dd if=/dev/urandom of=${RANDOM_SRC_FILE} bs=${BLOCK_SIZE} count=${cnt} 2> /dev/null
    sync
    echo 3 > /proc/sys/vm/drop_caches

    # Write out data to block device.
    time -v dd if=${RANDOM_SRC_FILE} of=${DST_FILE} bs=${BLOCK_SIZE} count=${cnt} conv=sync 2> /tmp/tmp.out$$
    sync
    echo 3 > /proc/sys/vm/drop_caches
    min=$(cat /tmp/tmp.out$$ | grep Elapsed | awk '{print $8}' | awk -F 'm' '{print $1}')
    sec=$(cat /tmp/tmp.out$$ | grep Elapsed | awk '{print $9}' | awk -F 's' '{print $1}')
    cpu=$(cat /tmp/tmp.out$$ | grep CPU | awk '{print $7}')
    cos=$( echo "scale=2; ${min} * 60 + ${sec}" | bc )
    speed=$( echo "scale=2; ${cnt} / ${cos}" | bc )
    echo "Write -> size:${cnt} MB  time:${cos} seconds  speed:${speed} MB/s  CPU:${cpu}"
    echo "${cnt}  ${cos}  ${speed}  ${cpu}" >> ${WRITE_DIA_FILE}

    # Read data from block device.
    time -v dd if=${DST_FILE} of=${READ_FILE} bs=${BLOCK_SIZE} count=${cnt} conv=sync 2> /tmp/tmp.out$$
    sync
    echo 3 > /proc/sys/vm/drop_caches
    min=$(cat /tmp/tmp.out$$ | grep Elapsed | awk '{print $8}' | awk -F 'm' '{print $1}')
    sec=$(cat /tmp/tmp.out$$ | grep Elapsed | awk '{print $9}' | awk -F 's' '{print $1}')
    cpu=$(cat /tmp/tmp.out$$ | grep CPU | awk '{print $7}')
    cos=$( echo "scale=2; ${min} * 60 + ${sec}" | bc )
    speed=$( echo "scale=2; ${cnt} / ${cos}" | bc )
    echo "Read  -> size:${cnt} MB  time:${cos} seconds  speed:${speed} MB/s  CPU:${cpu}"
    echo "${cnt}  ${cos}  ${speed}  ${cpu}" >> ${READ_DIA_FILE}

    cmp ${RANDOM_SRC_FILE} ${READ_FILE}
    if [ $? -ne 0 ]; then
        err="Read data and write data are not the same!"
        echo ${err}
        sed -i "$ s/$/  ERROR! ${err}/" ${WRITE_DIA_FILE}   # Append to last line.
        sed -i "$ s/$/  ERROR! ${err}/" ${READ_DIA_FILE}
    else
        echo "The read data is the same as the write data."
    fi
    sync

    cnt=$(( ${cnt} + ${COUNT_STEP} ))
done

w_ave_sig=$(tail -n+2 ${WRITE_DIA_FILE} | awk '{a[++i]=$3;}END{for(i in a)sum+=a[i];ave=sum/NR;for(i in a)tmp+=(a[i]-ave)*(a[i]-ave);print ave,tmp/NR}')
r_ave_sig=$(tail -n+2 ${READ_DIA_FILE} | awk '{a[++i]=$3;}END{for(i in a)sum+=a[i];ave=sum/NR;for(i in a)tmp+=(a[i]-ave)*(a[i]-ave);print ave,tmp/NR}')
echo "Write speed average and sigma: ${w_ave_sig}"
echo "Read speed average and sigma : ${r_ave_sig}"
echo -e "\nWrite speed average and sigma: ${w_ave_sig}" >> ${WRITE_DIA_FILE}
echo -e "\nRead speed average and sigma : ${r_ave_sig}"  >> ${READ_DIA_FILE}

rm -f ${RANDOM_SRC_FILE}
rm -f ${READ_FILE}
if [[ ! -z ${MNT_POINT} ]];then
    rm -f ${DST_FILE}
fi
echo "Diagnose done! Please check log files: ${WRITE_DIA_FILE} ${READ_DIA_FILE}"

2019年7月31日