利用hadoop streaming對tensorflow模型進行分佈式預測

0. python打包並上傳到hdfs

# 安裝Python
mkdir Python
export PYTHON_ROOT=~/Python
tar -xvf Python-3.6.8.tgz
pushd Python-3.6.8
./configure --prefix="{PYTHON_ROOT}" --enable-unicode=cs4
make && make install
popd
rm -rf Python-3.6.8.tgz

# Python打包
export PYTHON_ROOT=~/Python
pushd "${PYTHON_ROOT}"
zip -r Python.zip *
popd

# 推送到hdfs
hadoop fs -put Python.zip /usr/dm/tools/

1. 模型本地分發

     以Keras保存的.h5模型爲例.

#!/bin/bash
# 輸入輸出
INPUT_DIR="hdfs://ns3-backup/usr/dm/data/predict_feature/dt=20191201/part_*"
output_DIR="hdfs://ns3-backup/usr/dm/data/predict_feature_results/dt=20191201/"

# Python包
PYTHON="hdfs://ns3-backup/usr/dm/tools/Python.zip#Python"

# 模型位置
MODEL="./models.h5"

# hadoop stream命令
hadoop jar /usr/local/hadoop-2.7.3/share/hadoop/tools/lib/hadoop-streaming-2.7.3.jar \
    -archives ${PYTHON} \
    -input ${INPUT_DIR} \
    -output ${OUTPUT_DIR} \
    -mapper "Python/bin/python3 mapper.py" \
    -mapper "Python/bin/python3 reducer.py ${MODEL}" \
    -jobconf mapred.map.tasks=1000 \
    -jobconf mapred.reduce.tasks=1000 \
    -jobconf mapred.job.name="predict" \
    -jobconf mapreduce.map.memory.mb=4096 \
    -jobconf mapreduce.reduce.memory.mb=4096 \
    -inputformat com.sina.hadoop.rcfile.RCFileAsTextInputFormat \  # 輸入爲RCFile格式
    -file mapper.py \
    -file reducer.py \
    -file ${MODEL}
    

2. 模型從HDFS分發

     還是以Keras保存的.h5模型爲例.

#!/bin/bash
# 輸入輸出
INPUT_DIR="hdfs://ns3-backup/usr/dm/data/predict_feature/dt=20191201/part_*"
output_DIR="hdfs://ns3-backup/usr/dm/data/predict_feature_results/dt=20191201/"

# Python包
PYTHON="hdfs://ns3-backup/usr/dm/tools/Python.zip#Python"

# 模型位置
MODEL="hdfs://ns3-backup/usr/dm/model/models.h5#models"

# hadoop stream命令
hadoop jar /usr/local/hadoop-2.7.3/share/hadoop/tools/lib/hadoop-streaming-2.7.3.jar \
    -archives ${PYTHON} \
    -input ${INPUT_DIR} \
    -output ${OUTPUT_DIR} \
    -mapper "Python/bin/python3 mapper.py" \
    -mapper "Python/bin/python3 reducer.py models" \
    -jobconf mapred.map.tasks=1000 \
    -jobconf mapred.reduce.tasks=1000 \
    -jobconf mapred.job.name="predict" \
    -jobconf mapreduce.map.memory.mb=4096 \
    -jobconf mapreduce.reduce.memory.mb=4096 \
    -inputformat com.sina.hadoop.rcfile.RCFileAsTextInputFormat \  # 輸入爲RCFile格式
    -file mapper.py \
    -file reducer.py \
    -cacheFile ${MODEL}
    

3. 帶多個目錄的模型從HDFS分發

   從本地使用 -file分發模型和從HDFS使用-cacheFile分發模型最終都是在同一級目錄下使用。使用tf.saved_model.load讀取tpb模型時存在需要讀取model下的saved_model.pb和assets及variables兩個目錄，模型結果如下：

models -| saved_model.pb 
       -| aseets    -| key 
       -| variables -| variables.data-00000-of-00001
                    -| variables.index

我們可以使用與python用法一樣的壓縮方式進行加載.

模型打包：

cd model && zip -r models.zip *
hadoop fs -put models.zip /usr/dm/data/models

hadoop streaming命令如下：

#!/bin/bash
# 輸入輸出
INPUT_DIR="hdfs://ns3-backup/usr/dm/data/predict_feature/dt=20191201/part_*"
output_DIR="hdfs://ns3-backup/usr/dm/data/predict_feature_results/dt=20191201/"

# Python包
PYTHON="hdfs://ns3-backup/usr/dm/tools/Python.zip#Python"

# 模型位置
MODEL="hdfs://ns3-backup/usr/dm/model/models.zip#models"

# hadoop stream命令
hadoop jar /usr/local/hadoop-2.7.3/share/hadoop/tools/lib/hadoop-streaming-2.7.3.jar \
    -archives ${PYTHON}, ${MODEL} \
    -input ${INPUT_DIR} \
    -output ${OUTPUT_DIR} \
    -mapper "Python/bin/python3 mapper.py" \
    -mapper "Python/bin/python3 reducer.py models" \
    -jobconf mapred.map.tasks=1000 \
    -jobconf mapred.reduce.tasks=1000 \
    -jobconf mapred.job.name="predict" \
    -jobconf mapreduce.map.memory.mb=4096 \
    -jobconf mapreduce.reduce.memory.mb=4096 \
    -inputformat com.sina.hadoop.rcfile.RCFileAsTextInputFormat \  # 輸入爲RCFile格式
    -file mapper.py \
    -file reducer.py