Pipe in PySpark

官網的文檔聲明，pipe管道將pyspark的RDD對象fork給外部進程，然後再返回RDD對象

>>> sc.parallelize(['1', '2', '', '3']).pipe('cat').collect()
[u'1', u'2', u'', u'3']

---

python代碼如下，外部進程爲cat命令

#!/usr/bin/python
# -*- coding: utf-8 -*-
import sys
import os
from pyspark import SparkContext
from pyspark import SparkConf

if __name__ == '__main__':
    sc = SparkContext()
    data  = ["hi","hello","how","are","you"]
    dataRDD = sc.parallelize(data)
    pipeRDD = dataRDD.pipe('cat').collect()
    print pipeRDD

---

再提交任務即測試成功

spark-submit –master=yarn-client xx.py