13.IPython Notebook 在Hadoop YARN-client模式运行验证

前置准备Hadoop集群搭建

1.启动集群

   start-all.sh

2.切换至ipynotebook工作目录

   cd ~/pythonwork/ipynotebook

3.Hadoop YARN-client模式运行

   PYSPARK_DRIVER_PYTHON=ipython PYSPARK_DRIVER_PYTHON_OPTS="notebook" HADOOP_CONF_DIR=/usr/local/hadoop/etc/hadoop  pyspark --master yarn  --deploy-mode client

  1.Master yarn-client is deprecated since 2.0. Please use master "yarn" with specified deploy mode instead