python3操作pyspark

首先centos默認的是py2，我這裏改的py3，安裝完spark後，要安裝python環境
接下來我們測試一下
在/usr/local/spark-2.4.5-bin-hadoop2.7/bin下執行

spark-submit test.py

test.py

from pyspark import SparkConf,SparkContext
#import logging
#logging.basicConfig(level=logging.ERROR, format=' %(asctime)s - %(levelname)s -%(message)s')

conf=SparkConf().setMaster('local').setAppName('word_count')
sc = SparkContext(conf=conf)
d = ['a b c d', 'b c d e', 'c d e f']
d_rdd = sc.parallelize(d)
rdd_res = d_rdd.flatMap(lambda x: x.split(' ')).map(lambda word: (word, 1)).reduceByKey(lambda a, b: a+b)
print(rdd_res.collect())

結果

[('a', 1), ('b', 2), ('c', 3), ('d', 3), ('e', 2), ('f', 1)]

表示我們安裝成功