idea裏搭建spark、mapreduce開發環境

1、安裝scala的idea插件：

file —— settings —— plugins ，輸入scala，搜索插件下載安裝，注意版本：

2、配置scala的SDK：

先下載解壓scala，直接從linux服務器端解壓一個就行

file —— project structure —— library，配置之後，new就可以看到scala class了：

配置spark和scala的環境變量：

分別下載hadoop，spark和scala解壓，增加環境變量：

3、新建maven項目：

file —— new project —— maven ，

有2個xml配置文件如下：

（1）pom.xml

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <groupId>Learn-BigData</groupId>
    <artifactId>bigdata</artifactId>
    <version>1.0-SNAPSHOT</version>

    <properties>
        <maven.compiler.source>1.8</maven.compiler.source>
        <maven.compiler.target>1.8</maven.compiler.target>
        <scala.version>2.11.8</scala.version>
        <spark.version>2.4.0</spark.version>
        <hadoop.version>2.8.5</hadoop.version>
        <encoding>UTF-8</encoding>
    </properties>

    <repositories>
        <repository>
            <id>nexus-aliyun</id>
            <name>Nexus aliyun</name>
            <url>http://maven.aliyun.com/nexus/content/groups/public</url>
        </repository>
    </repositories>

    <dependencies>
        <!-- 導入scala的依賴 -->
        <dependency>
            <groupId>org.scala-lang</groupId>
            <artifactId>scala-library</artifactId>
            <version>${scala.version}</version>
            <scope>compile</scope>
        </dependency>

        <!-- 導入spark的依賴 -->
        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-core_2.11</artifactId>
            <version>${spark.version}</version>
            <scope>compile</scope>
        </dependency>

        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-sql_2.11</artifactId>
            <version>${spark.version}</version>
            <scope>compile</scope>
        </dependency>

        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-hive_2.11</artifactId>
            <version>${spark.version}</version>
            <scope>compile</scope>
        </dependency>

        <!-- 指定hadoop-client API的版本 -->
        <dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-client</artifactId>
            <version>${hadoop.version}</version>
            <scope>compile</scope>
        </dependency>

        <dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-common</artifactId>
            <version>${hadoop.version}</version>
            <scope>compile</scope>
        </dependency>

        <dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-hdfs</artifactId>
            <version>${hadoop.version}</version>
            <scope>compile</scope>
        </dependency>
        <dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-mapreduce-client-common</artifactId>
            <version>${hadoop.version}</version>
            <scope>compile</scope>
        </dependency>
        <dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-mapreduce-client-core</artifactId>
            <version>${hadoop.version}</version>
            <scope>compile</scope>
        </dependency>

        <dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-mapreduce-client-jobclient</artifactId>
            <version>${hadoop.version}</version>
            <scope>compile</scope>
        </dependency>

        <dependency>
            <groupId>commons-cli</groupId>
            <artifactId>commons-cli</artifactId>
            <version>1.3.1</version>
            <scope>compile</scope>
        </dependency>

        <!-- https://mvnrepository.com/artifact/org.apache.spark/spark-streaming -->
        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-streaming_2.11</artifactId>
            <version>${spark.version}</version>
            <scope>compile</scope>
        </dependency>

        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-mllib_2.11</artifactId>
            <version>${spark.version}</version>
            <scope>compile</scope>
        </dependency>

        <dependency>
            <groupId>commons-configuration</groupId>
            <artifactId>commons-configuration</artifactId>
            <version>1.6</version>
            <scope>compile</scope>
        </dependency>

    </dependencies>

    <build>
        <pluginManagement>
            <plugins>
                <!-- 編譯scala的插件 -->
                <plugin>
                    <groupId>net.alchim31.maven</groupId>
                    <artifactId>scala-maven-plugin</artifactId>
                    <version>3.2.2</version>
                </plugin>
                <!-- 編譯java的插件 -->
                <plugin>
                    <groupId>org.apache.maven.plugins</groupId>
                    <artifactId>maven-compiler-plugin</artifactId>
                    <version>3.5.1</version>
                </plugin>
            </plugins>
        </pluginManagement>
        <plugins>
            <plugin>
                <groupId>net.alchim31.maven</groupId>
                <artifactId>scala-maven-plugin</artifactId>
                <executions>
                    <execution>
                        <id>scala-compile-first</id>
                        <phase>process-resources</phase>
                        <goals>
                            <goal>add-source</goal>
                            <goal>compile</goal>
                        </goals>
                    </execution>
                    <execution>
                        <id>scala-test-compile</id>
                        <phase>process-test-resources</phase>
                        <goals>
                            <goal>testCompile</goal>
                        </goals>
                    </execution>
                </executions>
            </plugin>

            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-compiler-plugin</artifactId>
                <executions>
                    <execution>
                        <phase>compile</phase>
                        <goals>
                            <goal>compile</goal>
                        </goals>
                    </execution>
                </executions>
            </plugin>


            <!-- 打jar插件 -->
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-shade-plugin</artifactId>
                <version>2.4.3</version>

                <configuration>
                    <createDependencyReducedPom>false</createDependencyReducedPom>
                </configuration>

                <executions>
                    <execution>
                        <phase>package</phase>
                        <goals>
                            <goal>shade</goal>
                        </goals>
                        <configuration>
                            <filters>
                                <filter>
                                    <artifact>*:*</artifact>
                                    <excludes>
                                        <exclude>META-INF/*.SF</exclude>
                                        <exclude>META-INF/*.DSA</exclude>
                                        <exclude>META-INF/*.RSA</exclude>
                                    </excludes>
                                </filter>
                            </filters>
                        </configuration>
                    </execution>
                </executions>
            </plugin>
        </plugins>
    </build>


</project>

（2）dependency-reduced-pom.xml，這個文件是打包時生成的，沒啥用。

4、編寫JavaWordCount

package cn.edu360.spark;

import org.apache.spark.SparkConf;
import org.apache.spark.api.java.JavaPairRDD;
import org.apache.spark.api.java.JavaRDD;
import org.apache.spark.api.java.JavaSparkContext;
import org.apache.spark.api.java.function.FlatMapFunction;
import org.apache.spark.api.java.function.Function2;
import org.apache.spark.api.java.function.PairFunction;
import scala.Tuple2;

import java.util.Arrays;
import java.util.Iterator;


public class JavaWordCount {

    public static void main(String[] args) {

        SparkConf conf = new SparkConf().setAppName("JavaWordCount");
        //創建sparkContext
        JavaSparkContext jsc = new JavaSparkContext(conf);
        //指定以後從哪裏讀取數據
        JavaRDD<String> lines = jsc.textFile(args[0]);
        //切分壓平
        JavaRDD<String> words = lines.flatMap(new FlatMapFunction<String, String>() {
            @Override
            public Iterator<String> call(String line) throws Exception {
                return Arrays.asList(line.split(" ")).iterator();
            }
        });

        //將單詞和一組合在一起
        JavaPairRDD<String, Integer> wordAndOne = words.mapToPair(new PairFunction<String, String, Integer>() {
            @Override
            public Tuple2<String, Integer> call(String word) throws Exception {
                return new Tuple2<>(word, 1);
            }
        });

        //聚合
        JavaPairRDD<String, Integer> reduced = wordAndOne.reduceByKey(new Function2<Integer, Integer, Integer>() {
            @Override
            public Integer call(Integer v1, Integer v2) throws Exception {
                return v1 + v2;
            }
        });

        //調換順序
        JavaPairRDD<Integer, String> swaped = reduced.mapToPair(new PairFunction<Tuple2<String, Integer>, Integer, String>() {
            @Override
            public Tuple2<Integer, String> call(Tuple2<String, Integer> tp) throws Exception {
                //return new Tuple2<>(tp._2, tp._1);
                return tp.swap();
            }
        });

        //排序
        JavaPairRDD<Integer, String> sorted = swaped.sortByKey(false);

        //調整順序
        JavaPairRDD<String, Integer> result = sorted.mapToPair(new PairFunction<Tuple2<Integer, String>, String, Integer>() {
            @Override
            public Tuple2<String, Integer> call(Tuple2<Integer, String> tp) throws Exception {
                return tp.swap();
            }
        });

        //將數據保存到hdfs
        result.saveAsTextFile(args[1]);

        //釋放資源
        jsc.stop();
        
    }
}

5、打包：雙擊package打包：

view -- tool windows -- maven project，沒有出現如下目錄時，點擊+號，去選中pom.xml文件：

6、打包成功後，選擇這個：

7、進入上傳到linux服務器上：

進入spark的安裝目錄的bin目錄，執行以下代碼：
spark-submit --master spark://hdp-01:7077 --class cn.edu360.spark.JavaWordCount /root/learn_dh/original-SparkTest-1.0-SNAPSHOT.jar hdfs://hdp-01:9000/spark/input/test.txt hdfs://hdp-01:9000/spark/output/wc1005

命令解釋：

1、--master spark://hdp-01:7077 ，指定spark集羣的master

2、--class cn.edu360.spark.JavaWordCount，指定java類名全路徑

3、/root/learn_dh/original-SparkTest-1.0-SNAPSHOT.jar，指定這個jar包在linux服務器上jar的絕對路徑

4、hdfs://hdp-01:9000/spark/input/test.txt ，HDFS上輸入文件的路徑

5、hdfs://hdp-01:9000/spark/output/wc1005，HDFS上輸出文件路徑。（這路徑不能是已經存在的，否則會報錯）

可以在http://hdp-01:8080/這裏查看執行情況。

8、本地運行spark程序，則：

setMaster爲local，

本地運行時，要配置輸入輸出文件的路徑：

9、idea打開項目時，需要選中到src這一級目錄，否則，打開後看不見項目結構圖，這個坑的很啊：

例如，直接選中項目根目錄打開是這樣的，初學者注意下，有點莫名其妙的。

10、本地運行mapreduce和spark程序：

配置好上面的pom文件之後，不用再按照網上說的添加spark和hadoop的jar包，spark設置setMaster("local")就可以運行。

二步設置：

1、設置運行環境，edit configuration：

2、

新建maven項目，配置好pom後，不用再添加其他spark和hadoop的jar包，否則容易報莫名其妙的錯誤，估計是依賴衝突造成的。

在windows下配置好scala、hadoop、spark的環境變量之後：

在cmd下，輸入，scala、spark-shell，可直接在本地編寫scala、spark程序

idea裏搭建spark、mapreduce開發環境

1、安裝scala的idea插件：

2、配置scala的SDK：

3、新建maven項目：

4、編寫JavaWordCount

5、打包：雙擊package打包：

6、打包成功後，選擇這個：

7、進入上傳到linux服務器上：

8、本地運行spark程序，則：

電腦窗口跑到屏幕外拉不回來

win10更新後VMware Workstation Pro 無法運行

批量給pdf添加目錄-親測可用

sqoop1和sqoop2安裝配置

hive報錯：permission denied by sticky bit無法刪除文件解決辦法

Mac下配置sublime實現LaTeX

https://yachay.unat.edu.pe/blog/index.php?comment_area=format_blog&comment_component=blog&comment_co

linux以太網驅動總結