軟件版本：
CDH：5.8.0；Hadoop：2.6.0 ； Spark：1.6.0； Hive：1.1.0；JDK：1.7 ； SDK：2.10.6（Scala）
工程下載：https://github.com/fansy1990/spark_hive_source_destination/releases/tag/V1.1
目標：
在Spark加載PMML文件處理數據（參考：http://blog.csdn.net/fansy1990/article/details/53293024）及Spark讀寫Hive（http://blog.csdn.net/fansy1990/article/details/53401102）的基礎上，整合這兩個操作，即使用Spark讀取Hive表數據，然後加載PMML文件到模型，使用模型對讀取對Hive數據進行處理，得到新的數據，寫入到新的Hive表。
實現：
1. 工程pom文件，工程pom文件添加了spark、spark-hive以及pmml的依賴支持，如下：
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>cdh5.7.3</groupId>
<artifactId>spark_hive</artifactId>
<version>1.0-SNAPSHOT</version>
<inceptionYear>2008</inceptionYear>
<properties>
<scala.version>2.10.6</scala.version>
<spark.version>1.6.0-cdh5.7.3</spark.version>
</properties>
<repositories>
<repository>
<id>cloudera</id>
<url>https://repository.cloudera.com/artifactory/cloudera-repos/</url>
</repository>
</repositories>
<dependencies>
<!-- Spark -->
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-core_2.10</artifactId>
<version>${spark.version}</version>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-mllib_2.10</artifactId>
<version>${spark.version}</version>
<exclusions>
<exclusion>
<groupId>org.jpmml</groupId>
<artifactId>pmml-model</artifactId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>org.jpmml</groupId>
<artifactId>pmml-evaluator</artifactId>
<version>1.2.15</version>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-hive_2.10</artifactId>
<version>${spark.version}</version>
</dependency>
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>4.10</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.specs</groupId>
<artifactId>specs</artifactId>
<version>1.2.5</version>
<scope>test</scope>
</dependency>
</dependencies>
<build>
<sourceDirectory>src/main/scala</sourceDirectory>
<testSourceDirectory>src/test/scala</testSourceDirectory>
<plugins>
<plugin>
<groupId>org.scala-tools</groupId>
<artifactId>maven-scala-plugin</artifactId>
<version>2.15.2</version>
<executions>
<execution>
<goals>
<goal>compile</goal>
<goal>testCompile</goal>
</goals>
</execution>
</executions>
<configuration>
<scalaVersion>${scala.version}</scalaVersion>
<args>
<arg>-target:jvm-1.7</arg>
</args>
</configuration>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
<version>2.4.2</version>
<executions>
<execution>
<phase>package</phase>
<goals>
<goal>shade</goal>
</goals>
<configuration>
<createDependencyReducedPom>false</createDependencyReducedPom>
<finalName>example-${project.version}</finalName>
<artifactSet>
<excludes>
<exclude>oro*</exclude>
<exclude>org.apache.*:*</exclude>
<exclude>junit:junit</exclude>
<exclude>org.sc*</exclude>
<exclude>org.sp*</exclude>
<exclude>org.sl*</exclude>
<exclude>org.r*</exclude>
<exclude>org.c*</exclude>
<exclude>org.t*</exclude>
<exclude>org.e*</exclude>
<exclude>org.u*</exclude>
<exclude>org.x*</exclude>
<exclude>org.js*</exclude>
<exclude>org.jo*</exclude>
<exclude>org.f*</exclude>
<exclude>org.m*</exclude>
<exclude>org.o*</exclude>
<exclude>*:xml-apis</exclude>
<exclude>log4j*</exclude>
<exclude>org.antlr*</exclude>
<exclude>org.datanucleus*</exclude>
<exclude>net*</exclude>
<exclude>commons*</exclude>
<exclude>com.j*</exclude>
<exclude>com.x*</exclude>
<exclude>com.n*</exclude>
<exclude>com.i*</exclude>
<exclude>com.t*</exclude>
<exclude>com.c*</exclude>
<exclude>com.gi*</exclude>
<exclude>com.google.code*</exclude>
<exclude>com.google.p*</exclude>
<exclude>com.f*</exclude>
<exclude>com.su*</exclude>
<exclude>com.a*</exclude>
<exclude>com.e*</exclude>
<exclude>javax*</exclude>
<exclude>s*</exclude>
<exclude>i*</exclude>
<exclude>j*</exclude>
<exclude>a*</exclude>
<exclude>x*</exclude>
</excludes>
</artifactSet>
<relocations>
<relocation>
<pattern>com.google.common</pattern>
<shadedPattern>com.shaded.google.common</shadedPattern>
</relocation>
<relocation>
<pattern>org.dmg.pmml</pattern>
<shadedPattern>org.shaded.dmg.pmml</shadedPattern>
</relocation>
<relocation>
<pattern>org.jpmml.agent</pattern>
<shadedPattern>org.shaded.jpmml.agent</shadedPattern>
</relocation>
<relocation>
<pattern>org.jpmml.model</pattern>
<shadedPattern>org.shaded.jpmml.model</shadedPattern>
</relocation>
<relocation>
<pattern>org.jpmml.schema</pattern>
<shadedPattern>org.shaded.jpmml.schema</shadedPattern>
</relocation>
</relocations>
</configuration>
</execution>
</executions>
</plugin>
</plugins>
</build>
</project>
2. 測試環境構建：
1）首先是生成pmml文件，這個文件已經由其他程序生成，如下：
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<PMML version="4.2" xmlns="http://www.dmg.org/PMML-4_2">
<Header description="linear SVM">
<Application name="Apache Spark MLlib"/>
<Timestamp>2016-11-16T22:17:47</Timestamp>
</Header>
<DataDictionary numberOfFields="4">
<DataField name="field_0" optype="continuous" dataType="double"/>
<DataField name="field_1" optype="continuous" dataType="double"/>
<DataField name="field_2" optype="continuous" dataType="double"/>
<DataField name="target" optype="categorical" dataType="string"/>
</DataDictionary>
<RegressionModel modelName="linear SVM" functionName="classification" normalizationMethod="none">
<MiningSchema>
<MiningField name="field_0" usageType="active"/>
<MiningField name="field_1" usageType="active"/>
<MiningField name="field_2" usageType="active"/>
<MiningField name="target" usageType="target"/>
</MiningSchema>
<RegressionTable intercept="0.0" targetCategory="1">
<NumericPredictor name="field_0" coefficient="-0.36682158807862086"/>
<NumericPredictor name="field_1" coefficient="3.8787681305811765"/>
<NumericPredictor name="field_2" coefficient="-1.6134308474471166"/>
</RegressionTable>
<RegressionTable intercept="0.0" targetCategory="0"/>
</RegressionModel>
</PMML> 2）準備hive表及表數據：
-- field_0,field_1,field_2
-- 98,97,96
create table svm (
field_0 double ,
field_1 double,
field_2 double
)
ROW FORMAT DELIMITED
FIELDS TERMINATED BY ','
STORED AS TEXTFILE;
-- import data , get ride of first line
load data inpath 'svm.data' into table svm;
3) 編譯及打包：下載工程後，先執行build-》Make project，看到target目錄下生成classes文件，如下：
因爲使用了java和scala混合編程，所以這裏需要先編譯，然後在執行maven的package命令，這樣的到的jar包纔會包含pmml-spark的相關class文件，並且由於這裏沒有引入pmml-spark的依賴，只是把其源碼放到這裏而已，所以需要這樣做，打包後，得到target目錄下的所需jar包；
4）調用：
直接使用spark-submit的方式進行調用，其命令如下：
spark-submit --class pmml.SparkReadWriteHiveWithPMML --master yarn --deploy-mode cluster --jars /usr/lib/hive/lib/datanucleus-core-3.2.10.jar --files /usr/lib/hive/conf/hive-site.xml example-1.0-SNAPSHOT.jar svm tmp4 /tmp/svm.pmml如果輸出表存在，那麼會報錯（如tmp4存在）：
首先是yarn任務，如下：
接着是hive中的表，如下：
從hive表中可以看到數據被pmml模型正確的預測得到了。
總結：
1. Hive表如果使用分區表情況會比較複雜，暫時沒有驗證過；
2. 如果hive表已經存在，那麼會出現異常，是否可以考慮在代碼中把輸出的表刪掉？
3. pmml-spark依賴是否可以直接寫入pom文件，而不需要把源碼放入工程？
如果您覺得lz的文章還行，並且您願意動動手指，可以爲我投上您的寶貴一票！謝謝！
http://blog.csdn.net/vote/candidate.html?username=fansy1990