原创 Spark repartition VS coalesce
Spark repartition VS coalesce demo val data=sc.parallelize(Array("a","b","c","d","e","a","b","e","b","f"),2) scala>
2020-06-11 12:24:43
原创 scala入門系列(六) - 集合 Array Map List
array object ArrayApp { def main(args: Array[String]): Unit = { //定長 val str = new Array[String](5) p
2020-06-11 12:24:43
原创 Spark調優指南(三)-調整數據結構
Tuning Data Structures 調整數據結構 The first way to reduce memory consumption is to avoid the Java features that add ove
2020-06-11 12:24:43
原创 Spark 調優指南(一)-數據序列化
Spark 調優指南(一)-數據序列化 官網介紹 Data Serialization 數據序列化 Serialization plays an important role in the performance of any d
2020-06-11 12:24:43
1
原创 Spark調優指南(四)-內存管理
Spark調優指南(四)-內存管理 官網介紹 Memory Management Overview 內存管理 Memory usage in Spark largely falls under one of two categor
2020-06-11 12:24:43
原创 scala入門系列(八) case class
object CaseClassApp { def main(args: Array[String]): Unit = { // val user=User(“s”) val user=new User(“s”) user(“b”
2020-06-11 12:24:43
原创 scala入門系列(七) String常見用法
字符串常見用法 object StringApp { def main(args: Array[String]): Unit = { val str = "aaa" println("str:"+str)
2020-06-11 12:24:43
原创 cdh Hbase安裝部署
wget http://archive.cloudera.com/cdh5/cdh/5/hbase-1.2.0-cdh5.7.0.tar.gz tar -zxvf hbase-1.2.0-cdh5.7.0.tar.gz -C /h
2020-06-11 12:24:33
原创 Spark 算子 reduceByKey vs groupByKey 對比
Spark 算子 reduceByKey vs groupByKey 對比 案例 準備測試數據 val data=sc.parallelize(Array("a","b","c","d","e","a","b","e","b"
2020-06-11 12:24:33
原创 Spark調優指南(二)-內存調優
Spark調優指南(二)-內存調優 官網介紹 Memory Tuning There are three considerations in tuning memory usage: the amount of memory u
2020-06-11 12:24:33
原创 maxwell安裝部署
部署 maxwell wget https://github.com/zendesk/maxwell/releases/download/v1.14.1/maxwell-1.14.1.tar.gz tar -zxvf m
2020-06-11 12:24:33
1
原创 flume配置 離線hdfs實時kafka
flume數據數據 一路到hdfs 供離線分析，一路經kafka消息中間件 配置案例 flume-hdfs-kafka.sources = r1 flume-hdfs-kafka.channels = c1 flume-hdfs-
2020-06-11 12:24:33
原创 scala入門系列(三) - 函數 map filter left right flod flatten flatmap
object FunApp { def main(args: Array[String]): Unit = { val l = List(1,2,3,4,5,6,7,8) //map操作的是l的每一個元素
2020-06-11 12:24:33
原创 maven 添加resource打包
maven打包需要配置如下信息，否則resoure是不打緊jar的 <resources> <resource> <directory>src/main/resource</directory>
2020-06-11 12:24:33
原创 二分查找算法
** * 二分查找 */ public class BinarySearch { public int directBinarySearch(int[] array,int elem){ int low
2019-07-30 09:19:52