英雄惜英雄-当Spark遇上Zeppelin之实战案例

{"type":"doc","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"我们在之前的文章中提到过","attrs":{}},{"type":"link","attrs":{"href":"https://mp.weixin.qq.com/s?__biz=MzU3MzgwNTU2Mg==&mid=2247495608&idx=1&sn=dfb63a8e4af3820746a7903664b83f58&chksm=fd3ea92dca49203bb4461d89a3fbf2c6007888c47ef8fbb3c8b2640eec7b45f9f6b7d71d9967&token=808170609&lang=zh_CN#rd","title":""},"content":[{"type":"text","text":"《大数据可视化从未如此简单 - Apache Zepplien全面介绍》","attrs":{}}]},{"type":"text","text":"一文中介绍了 Zeppelin 的主要功能和特点，并且最后还用一个案例介绍了这个框架的使用。这节课我们用两个直观的小案例来介绍 Zepplin 和 Spark 如何配合使用。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"到目前为止，Apache Spark 已经支持三种集群管理器类型（Standalone，Apache Mesos 和 Hadoop YARN ）。本文中我们根据官网文档使用 Docker 脚本构建一个Spark standalone mode ( Spark独立模式 )的环境来使用。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":4},"content":[{"type":"text","text":"Spark独立模式环境搭建","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Spark standalone 是Spark附带的简单集群管理器，可以轻松设置集群。您可以通过以下步骤简单地设置 Spark独立环境。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"注意","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"由于 Apache Zeppelin 和 Spark 为其 Web UI 使用相同的 8080 端口，因此您可能需要在 conf / zeppelin-site.xml 中更改 zeppelin.server.port 。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":" 1. 构建 Docker 文件","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"您可以在脚本 / docker / spark-cluster-managers 下找到 docker 脚本文件。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"codeblock","attrs":{"lang":""},"content":[{"type":"text","text":"cd $ZEPPELIN_HOME/scripts/docker/spark-cluster-managers/spark_standalone\ndocker build -t \"spark_standalone\" .","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"2. 启动Docker","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"codeblock","attrs":{"lang":""},"content":[{"type":"text","text":"docker run -it \\\n-p 8080:8080 \\\n-p 7077:7077 \\\n-p 8888:8888 \\\n-p 8081:8081 \\\n-h sparkmaster \\\n--name spark_standalone \\\nspark_standalone bash;","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在这里运行 docker 容器的 sparkmaster 主机名应该在 /etc/hosts 中绑定映射关系。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"3. 在Zeppelin中配置Spark解释器","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"将 Spark master 设置为 spark://< hostname >:7077 在 Zeppelin 的解释器设置页面上。","attrs":{}}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/2a/2a14feb46a0af6228878a72f6e64b7b5.png","alt":"file","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"4. 用Spark解释器运行Zeppelin","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在 Zeppelin 中运行带有 Spark 解释器的单个段落后，浏览 https://< hostname>：8080，并检查 Spark 集群是否运行正常。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/41/417440ad06f4bcd753bef18dae731b7e.png","alt":"file","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"然后我们可以用以下命令简单地验证 Spark 在 Docker 中是否运行良好。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"codeblock","attrs":{"lang":""},"content":[{"type":"text","text":"ps -ef | grep spark","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":4},"content":[{"type":"text","text":"Spark on Zepplin读取本地文件","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"假设我们本地有一个名为bank.csv的文件，样例数据如下：","attrs":{}}]},{"type":"codeblock","attrs":{"lang":""},"content":[{"type":"text","text":"age:Integer, job:String, marital : String, education : String, balance : Integer\n20;teacher;single;本科;20000\n25;plumber;single;本科;10000\n21;doctor;single;本科;25000\n23;singer;single;本科;20000\n...","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"首先，将csv格式的数据转换成RDD Bank对象，运行以下脚本。这也将使用filter功能过滤掉一些数据。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"codeblock","attrs":{"lang":""},"content":[{"type":"text","text":"val bankText = sc.textFile(\"yourPath/bank/bank-full.csv\")\ncase class Bank(age:Integer, job:String, marital : String, education : String, balance : Integer)\n\n// split each line, filter out header (starts with \"age\"), and map it into Bank case class\nval bank = bankText.map(s=>s.split(\";\")).filter(s=>s(0)!=\"\\\"age\\\"\").map(\n s=>Bank(s(0).toInt,\n s(1).replaceAll(\"\\\"\", \"\"),\n s(2).replaceAll(\"\\\"\", \"\"),\n s(3).replaceAll(\"\\\"\", \"\"),\n s(5).replaceAll(\"\\\"\", \"\").toInt\n )\n)\n// convert to DataFrame and create temporal table\nbank.toDF().registerTempTable(\"bank\")","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"如果想使用图形化看到年龄分布，可以运行如下sql：","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"%sql ","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"select age, count(1) from bank where age < 30 group by age order by age","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/ad/ad251591f6f751418210550ed2865499.png","alt":"file","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"您可以输入框通过更换设置年龄条件30用${maxAge=30}。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"%sql ","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"select age, count(1) from bank where age < ${maxAge=30} group by age order by age","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"如果希望看到有一定婚姻状况的年龄分布，并添加组合框来选择婚姻状况：","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"%sql ","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"select age, count(1) from bank where marital=\"${marital=single,single|divorced|married}\" group by age order by age","attrs":{}}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/f9/f9f148c1af2215ba8df29abf13b751a2.png","alt":"file","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Zeppelin支持画图，功能简单但强大，可同时输出表格、柱状图、折线图、饼状图、折线图、点图。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"下面将各年龄的用户数用画出来，画图的实现可以将结果组织成下面这种格式：","attrs":{}}]},{"type":"codeblock","attrs":{"lang":""},"content":[{"type":"text","text":"println(“%table column_1\\tcolumn_2\\n”+value_1\\tvalue_2\\n+…)","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/0e/0e6ddf8dcd5e3924d547922491e87461.png","alt":"file","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"最后，我们甚至可以直接将运算结果存入 Mysql 中：","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"codeblock","attrs":{"lang":""},"content":[{"type":"text","text":"%spark\ndf3.write.mode(\"overwrite\").format(\"jdbc\").option(\"driver\",\"com.mysql.jdbc.Driver\").option(\"user\",\"root\").option(\"password\",\"password\").option(\"url\",\"jdbc:mysql://localhost:3306/spark_demo\").option(\"dbtable\",\"record\").save()","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":4},"content":[{"type":"text","text":"Spark on Zepplin读取HDFS文件","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"首先我们需要配置HDFS文件系统解释器，我们需要进行如下的配置。在笔记本中，要启用HDFS解释器，可以单击齿轮图标并选择HDFS。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/11/11b526258377571d3ae33d2df23a9d12.png","alt":"file","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"然后我们就可以愉快的使用Zepplin读取HDFS文件了：","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"例如：下面先读取HDFS文件，该文件为JSON文件，读取出来之后取出第一列然后以Parquet的格式保存到HDFS上：","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/d9/d967dd3c96c90117a60bf843d629680c.png","alt":"file","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":4},"content":[{"type":"text","text":"Spark on Zepplin读取流数据","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"我们可以参考官网中，读取Twitter实时流的案例：","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"codeblock","attrs":{"lang":""},"content":[{"type":"text","text":"import org.apache.spark.streaming._\nimport org.apache.spark.streaming.twitter._\nimport org.apache.spark.storage.StorageLevel\nimport scala.io.Source\nimport scala.collection.mutable.HashMap\nimport java.io.File\nimport org.apache.log4j.Logger\nimport org.apache.log4j.Level\nimport sys.process.stringSeqToProcess\n\n/** Configures the Oauth Credentials for accessing Twitter */\ndef configureTwitterCredentials(apiKey: String, apiSecret: String, accessToken: String, accessTokenSecret: String) {\n val configs = new HashMap[String, String] ++= Seq(\n \"apiKey\" -> apiKey, \"apiSecret\" -> apiSecret, \"accessToken\" -> accessToken, \"accessTokenSecret\" -> accessTokenSecret)\n println(\"Configuring Twitter OAuth\")\n configs.foreach{ case(key, value) =>\n if (value.trim.isEmpty) {\n throw new Exception(\"Error setting authentication - value for \" + key + \" not set\")\n }\n val fullKey = \"twitter4j.oauth.\" + key.replace(\"api\", \"consumer\")\n System.setProperty(fullKey, value.trim)\n println(\"\\tProperty \" + fullKey + \" set as [\" + value.trim + \"]\")\n }\n println()\n}\n\n// Configure Twitter credentials\nval apiKey = \"xxxxxxxxxxxxxxxxxxxxxxxxx\"\nval apiSecret = \"xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx\"\nval accessToken = \"xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx\"\nval accessTokenSecret = \"xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx\"\nconfigureTwitterCredentials(apiKey, apiSecret, accessToken, accessTokenSecret)\n\nimport org.apache.spark.streaming.twitter._\nval ssc = new StreamingContext(sc, Seconds(2))\nval tweets = TwitterUtils.createStream(ssc, None)\nval twt = tweets.window(Seconds(60))\n\ncase class Tweet(createdAt:Long, text:String)\ntwt.map(status=>\n Tweet(status.getCreatedAt().getTime()/1000, status.getText())\n).foreachRDD(rdd=>\n // Below line works only in spark 1.3.0.\n // For spark 1.1.x and spark 1.2.x,\n // use rdd.registerTempTable(\"tweets\") instead.\n rdd.toDF().registerAsTable(\"tweets\")\n)\n\ntwt.print\n\nssc.start()","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"同理，Zepplin也可以读取Kafka中的数据，注册成表然后进行各种运算。我们参考一个Zepplin版本的WordCount实现：","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"codeblock","attrs":{"lang":""},"content":[{"type":"text","text":"%spark\nimport _root_.kafka.serializer.DefaultDecoder\nimport _root_.kafka.serializer.StringDecoder\nimport org.apache.spark.streaming.kafka.KafkaUtils\nimport org.apache.spark.storage.StorageLevel\nimport org.apache.spark.streaming._\n \n// prevent INFO logging from pollution output\nsc.setLogLevel(\"INFO\")\n \n// creating the StreamingContext with 5 seconds interval\nval ssc = new StreamingContext(sc, Seconds(5))\n \nval kafkaConf = Map(\n \"metadata.broker.list\" -> \"localhost:9092\",\n \"zookeeper.connect\" -> \"localhost:2181\",\n \"group.id\" -> \"kafka-streaming-example\",\n \"zookeeper.connection.timeout.ms\" -> \"1000\"\n)\n \nval lines = KafkaUtils.createStream[Array[Byte], String, DefaultDecoder, StringDecoder](\n ssc,\n kafkaConf,\n Map(\"test\" -> 1), // subscripe to topic and partition 1\n StorageLevel.MEMORY_ONLY\n)\n \nval words = lines.flatMap{ case(x, y) => y.split(\" \")}\n\nimport spark.implicits._\n\nval w=words.map(x=> (x,1L)).reduceByKey(_+_)\nw.foreachRDD(rdd => rdd.toDF.registerTempTable(\"counts\"))\nssc.start()","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"从上面的temporary table counts 中查询每小批量的数据中top 10 的单词值。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"codeblock","attrs":{"lang":""},"content":[{"type":"text","text":"select * from counts order by _2 desc limit 10","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/37/3720d879c64abe2fdae0a1fae3dc2e4f.png","alt":"file","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"怎么样？是不是很强大？推荐大家可以自己试试看。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"原文链接：","attrs":{}},{"type":"link","attrs":{"href":"https://mp.weixin.qq.com/s?__biz=MzU3MzgwNTU2Mg==&mid=2247497261&idx=1&sn=a7d01f2d015b3d012e97230dde3d7f7f&chksm=fd3eb0b8ca4939aeb885cd64372b3e221d49e2bb6e99b5a1ea109d0673e413114b4bfebcb4df&token=808170609&lang=zh_CN#rd","title":""},"content":[{"type":"text","text":"英雄惜英雄-当Spark遇上Zeppelin之实战案例","attrs":{}}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"blockquote","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"欢迎关注，","attrs":{}},{"type":"link","attrs":{"href":"https://shimo.im/docs/jdPhrtFwVCAMkoWv","title":""},"content":[{"type":"text","text":"《大数据成神之路》","attrs":{}}]},{"type":"text","text":"系列文章","attrs":{}}]}],"attrs":{}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}}]}