CDH 6.3Unable to instantiate SparkSession with Hive support because Hive classes are not found.解決辦法

系統環境

CentOS 7.7.1908
CDH 6.3.1

問題現象

執行spark-submit，具體命令如下：

spark-submit --class xxx  --master yarn --deploy-mode cluster --driver-memory 4g --executor-memory 2g --executor-cores 1 --queue thequeue  /tmp/xxx.jar

console輸出如下：

2020-06-25 17:59:56 WARN  NativeCodeLoader:60 - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2020-06-25 17:59:56 INFO  RMProxy:133 - Connecting to ResourceManager at master01-prod.cdh/10.0.101.86:8032
2020-06-25 17:59:56 INFO  Client:57 - Requesting a new application from cluster with 3 NodeManagers
2020-06-25 17:59:56 INFO  Configuration:2662 - resource-types.xml not found
2020-06-25 17:59:56 INFO  ResourceUtils:419 - Unable to find 'resource-types.xml'.
2020-06-25 17:59:56 INFO  Client:57 - Verifying our application has not requested more than the maximum memory capability of the cluster (13458 MB per container)
2020-06-25 17:59:56 INFO  Client:57 - Will allocate AM container, with 4505 MB memory including 409 MB overhead
2020-06-25 17:59:56 INFO  Client:57 - Setting up container launch context for our AM
2020-06-25 17:59:56 INFO  Client:57 - Setting up the launch environment for our AM container
2020-06-25 17:59:56 INFO  Client:57 - Preparing resources for our AM container
2020-06-25 17:59:56 WARN  Client:69 - Neither spark.yarn.jars nor spark.yarn.archive is set, falling back to uploading libraries under SPARK_HOME.
2020-06-25 17:59:57 INFO  Client:57 - Uploading resource file:/tmp/spark-44972a1d-fb2d-443b-b6c8-d94ae7d4dc83/__spark_libs__3873912406489385752.zip -> hdfs://master01-prod.cdh:8020/user/root/.sparkStaging/application_1591413191159_0063/__spark_libs__3873912406489385752.zip
2020-06-25 17:59:58 INFO  Client:57 - Uploading resource file:/tmp/dopEtlDev.jar -> hdfs://master01-prod.cdh:8020/user/root/.sparkStaging/application_1591413191159_0063/dopEtlDev.jar
2020-06-25 17:59:58 INFO  Client:57 - Uploading resource file:/tmp/spark-44972a1d-fb2d-443b-b6c8-d94ae7d4dc83/__spark_conf__8206675857340680660.zip -> hdfs://master01-prod.cdh:8020/user/root/.sparkStaging/application_1591413191159_0063/__spark_conf__.zip
2020-06-25 17:59:58 INFO  SecurityManager:57 - Changing view acls to: root
2020-06-25 17:59:58 INFO  SecurityManager:57 - Changing modify acls to: root
2020-06-25 17:59:58 INFO  SecurityManager:57 - Changing view acls groups to:
2020-06-25 17:59:58 INFO  SecurityManager:57 - Changing modify acls groups to:
2020-06-25 17:59:58 INFO  SecurityManager:57 - SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(root); groups with view permissions: Set(); users  with modify permissions: Set(root); groups with modify permissions: Set()
2020-06-25 17:59:58 INFO  HiveConf:188 - Found configuration file jar:file:/tmp/dopEtlDev.jar!/hive-site.xml
2020-06-25 17:59:58 INFO  YARNHadoopDelegationTokenManager:57 - Attempting to load user's ticket cache.
2020-06-25 17:59:58 INFO  Client:57 - Submitting application application_1591413191159_0063 to ResourceManager
2020-06-25 17:59:58 INFO  YarnClientImpl:310 - Submitted application application_1591413191159_0063
2020-06-25 17:59:59 INFO  Client:57 - Application report for application_1591413191159_0063 (state: ACCEPTED)
2020-06-25 17:59:59 INFO  Client:57 -
	 client token: N/A
	 diagnostics: AM container is launched, waiting for AM container to Register with RM
	 ApplicationMaster host: N/A
	 ApplicationMaster RPC port: -1
	 queue: root.thequeue
	 start time: 1593079198932
	 final status: UNDEFINED
	 tracking URL: http://master01-prod.cdh:8088/proxy/application_1591413191159_0063/
	 user: root
2020-06-25 18:00:00 INFO  Client:57 - Application report for application_1591413191159_0063 (state: ACCEPTED)
2020-06-25 18:00:01 INFO  Client:57 - Application report for application_1591413191159_0063 (state: ACCEPTED)
2020-06-25 18:00:02 INFO  Client:57 - Application report for application_1591413191159_0063 (state: ACCEPTED)
2020-06-25 18:00:03 INFO  Client:57 - Application report for application_1591413191159_0063 (state: ACCEPTED)
2020-06-25 18:00:04 INFO  Client:57 - Application report for application_1591413191159_0063 (state: ACCEPTED)
2020-06-25 18:00:05 INFO  Client:57 - Application report for application_1591413191159_0063 (state: ACCEPTED)
2020-06-25 18:00:06 INFO  Client:57 - Application report for application_1591413191159_0063 (state: FAILED)
2020-06-25 18:00:06 INFO  Client:57 -
	 client token: N/A
	 diagnostics: Application application_1591413191159_0063 failed 2 times due to AM Container for appattempt_1591413191159_0063_000002 exited with  exitCode: 13
Failing this attempt.Diagnostics: [2020-06-25 18:00:06.105]Exception from container-launch.
Container id: container_1591413191159_0063_02_000001
Exit code: 13

[2020-06-25 18:00:06.106]Container exited with a non-zero exit code 13. Error file: prelaunch.err.
Last 4096 bytes of prelaunch.err :
Last 4096 bytes of stderr :
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/data4/yarn/nm/usercache/root/filecache/57/__spark_libs__3873912406489385752.zip/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-6.3.1-1.cdh6.3.1.p0.1470567/jars/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]


[2020-06-25 18:00:06.106]Container exited with a non-zero exit code 13. Error file: prelaunch.err.
Last 4096 bytes of prelaunch.err :
Last 4096 bytes of stderr :
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/data4/yarn/nm/usercache/root/filecache/57/__spark_libs__3873912406489385752.zip/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-6.3.1-1.cdh6.3.1.p0.1470567/jars/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]


For more detailed output, check the application tracking page: http://master01-prod.cdh:8088/cluster/app/application_1591413191159_0063 Then click on links to logs of each attempt.
. Failing the application.
	 ApplicationMaster host: N/A
	 ApplicationMaster RPC port: -1
	 queue: root.thequeue
	 start time: 1593079198932
	 final status: FAILED
	 tracking URL: http://master01-prod.cdh:8088/cluster/app/application_1591413191159_0063
	 user: root
2020-06-25 18:00:06 ERROR Client:73 - Application diagnostics message: Application application_1591413191159_0063 failed 2 times due to AM Container for appattempt_1591413191159_0063_000002 exited with  exitCode: 13
Failing this attempt.Diagnostics: [2020-06-25 18:00:06.105]Exception from container-launch.
Container id: container_1591413191159_0063_02_000001
Exit code: 13

[2020-06-25 18:00:06.106]Container exited with a non-zero exit code 13. Error file: prelaunch.err.
Last 4096 bytes of prelaunch.err :
Last 4096 bytes of stderr :
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/data4/yarn/nm/usercache/root/filecache/57/__spark_libs__3873912406489385752.zip/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-6.3.1-1.cdh6.3.1.p0.1470567/jars/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]


[2020-06-25 18:00:06.106]Container exited with a non-zero exit code 13. Error file: prelaunch.err.
Last 4096 bytes of prelaunch.err :
Last 4096 bytes of stderr :
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/data4/yarn/nm/usercache/root/filecache/57/__spark_libs__3873912406489385752.zip/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-6.3.1-1.cdh6.3.1.p0.1470567/jars/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]


For more detailed output, check the application tracking page: http://master01-prod.cdh:8088/cluster/app/application_1591413191159_0063 Then click on links to logs of each attempt.
. Failing the application.
Exception in thread "main" org.apache.spark.SparkException: Application application_1591413191159_0063 finished with failed status
	at org.apache.spark.deploy.yarn.Client.run(Client.scala:1158)
	at org.apache.spark.deploy.yarn.YarnClusterApplication.start(Client.scala:1606)
	at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:851)
	at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:167)
	at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:195)
	at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86)
	at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:926)
	at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:935)
	at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
2020-06-25 18:00:06 INFO  ShutdownHookManager:57 - Shutdown hook called
2020-06-25 18:00:06 INFO  ShutdownHookManager:57 - Deleting directory /tmp/spark-15956e0f-d264-4042-8968-1a3ba0b696a9
2020-06-25 18:00:06 INFO  ShutdownHookManager:57 - Deleting directory /tmp/spark-44972a1d-fb2d-443b-b6c8-d94ae7d4dc83

解決方案

1.查看yarn日誌
訪問：http://master01-prod.cdh:19888/jobhistory/logs//slave01-prod.cdh:8041/container_1591413191159_0057_02_000001/container_1591413191159_0057_02_000001/root

具體日誌如下：

Log Type: prelaunch.err

Log Upload Time: Thu Jun 25 16:40:56 +0800 2020

Log Length: 0


Log Type: prelaunch.out

Log Upload Time: Thu Jun 25 16:40:56 +0800 2020

Log Length: 70

Setting up env variables
Setting up job resources
Launching container

Log Type: stderr

Log Upload Time: Thu Jun 25 16:40:56 +0800 2020

Log Length: 544

SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/data2/yarn/nm/usercache/root/filecache/42/__spark_libs__2999614218847688045.zip/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-6.3.1-1.cdh6.3.1.p0.1470567/jars/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]

Log Type: stdout

Log Upload Time: Thu Jun 25 16:40:56 +0800 2020

Log Length: 5141

Showing 4096 bytes of 5141 total. Click here for the full log.

-25 16:40:55 INFO  ApplicationMaster:57 - Waiting for spark context initialization...
arg is not exist and use System currentTimeMillis!
2020-06-25 16:40:55 ERROR ApplicationMaster:94 - User class threw exception: java.lang.IllegalArgumentException: Unable to instantiate SparkSession with Hive support because Hive classes are not found.
java.lang.IllegalArgumentException: Unable to instantiate SparkSession with Hive support because Hive classes are not found.
	at org.apache.spark.sql.SparkSession$Builder.enableHiveSupport(SparkSession.scala:878)
	at com.mskj.dopdev.scfp.dm_scfp_num_type_value$.main(dm_scfp_num_type_value.scala:34)
	at com.mskj.dopdev.scfp.dm_scfp_num_type_value.main(dm_scfp_num_type_value.scala)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:673)
2020-06-25 16:40:55 INFO  ApplicationMaster:57 - Final app status: FAILED, exitCode: 13, (reason: User class threw exception: java.lang.IllegalArgumentException: Unable to instantiate SparkSession with Hive support because Hive classes are not found.
	at org.apache.spark.sql.SparkSession$Builder.enableHiveSupport(SparkSession.scala:878)
	at com.mskj.dopdev.scfp.dm_scfp_num_type_value$.main(dm_scfp_num_type_value.scala:34)
	at com.mskj.dopdev.scfp.dm_scfp_num_type_value.main(dm_scfp_num_type_value.scala)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:673)
)
2020-06-25 16:40:55 ERROR ApplicationMaster:94 - Uncaught exception: 
org.apache.spark.SparkException: Exception thrown in awaitResult: 
	at org.apache.spark.util.ThreadUtils$.awaitResult(ThreadUtils.scala:226)
	at org.apache.spark.deploy.yarn.ApplicationMaster.runDriver(ApplicationMaster.scala:447)
	at org.apache.spark.deploy.yarn.ApplicationMaster.run(ApplicationMaster.scala:275)
	at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$3.run(ApplicationMaster.scala:805)
	at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$3.run(ApplicationMaster.scala:804)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:422)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1875)
	at org.apache.spark.deploy.yarn.ApplicationMaster$.main(ApplicationMaster.scala:804)
	at org.apache.spark.deploy.yarn.ApplicationMaster.main(ApplicationMaster.scala)
Caused by: java.lang.IllegalArgumentException: Unable to instantiate SparkSession with Hive support because Hive classes are not found.
	at org.apache.spark.sql.SparkSession$Builder.enableHiveSupport(SparkSession.scala:878)
	at com.mskj.dopdev.scfp.dm_scfp_num_type_value$.main(dm_scfp_num_type_value.scala:34)
	at com.mskj.dopdev.scfp.dm_scfp_num_type_value.main(dm_scfp_num_type_value.scala)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:673)
2020-06-25 16:40:55 INFO  ApplicationMaster:57 - Deleting staging directory hdfs://master01-prod.cdh:8020/user/root/.sparkStaging/application_1591413191159_0057
2020-06-25 16:40:55 WARN  NativeCodeLoader:60 - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2020-06-25 16:40:55 INFO  ShutdownHookManager:57 - Shutdown hook called

重點關注：reason: User class threw exception: java.lang.IllegalArgumentException: Unable to instantiate SparkSession with Hive support because Hive classes are not found
初步分析是缺少hive相關的jar

2.檢查hive相關jar包
找一臺可以正常submit的服務器，對比分析發現spark所需要的jar包都在對應位置，因此排除缺少jar包問題

3.檢查spark配置
正常服務器的spark配置：

異常服務器的spark配置：

這下真相大白了，spark-conf的配置不同

4.unlink spark-conf ，創建正確的軟鏈接

5.驗證問題是否解決

[root@master02-prod alternatives]# spark-submit --class xxxx  --master yarn --deploy-mode cluster --driver-memory 4g --executor-memory 2g --executor-cores 1 --queue thequeue  /tmp/xxxx.jar
20/06/25 18:06:55 INFO client.RMProxy: Connecting to ResourceManager at master01-prod.cdh/10.0.101.86:8032
20/06/25 18:06:55 INFO yarn.Client: Requesting a new application from cluster with 3 NodeManagers
20/06/25 18:06:55 INFO conf.Configuration: resource-types.xml not found
20/06/25 18:06:55 INFO resource.ResourceUtils: Unable to find 'resource-types.xml'.
20/06/25 18:06:55 INFO yarn.Client: Verifying our application has not requested more than the maximum memory capability of the cluster (13458 MB per container)
20/06/25 18:06:55 INFO yarn.Client: Will allocate AM container, with 4505 MB memory including 409 MB overhead
20/06/25 18:06:55 INFO yarn.Client: Setting up container launch context for our AM
20/06/25 18:06:55 INFO yarn.Client: Setting up the launch environment for our AM container
20/06/25 18:06:55 INFO yarn.Client: Preparing resources for our AM container
20/06/25 18:06:55 INFO yarn.Client: Uploading resource file:/tmp/dopEtlDev.jar -> hdfs://master01-prod.cdh:8020/user/root/.sparkStaging/application_1591413191159_0064/dopEtlDev.jar
20/06/25 18:06:55 INFO yarn.Client: Uploading resource file:/tmp/spark-410edde9-c2d8-4fd4-9a12-1f7473cc89f5/__spark_conf__1869574925881857964.zip -> hdfs://master01-prod.cdh:8020/user/root/.sparkStaging/application_1591413191159_0064/__spark_conf__.zip
20/06/25 18:06:55 INFO spark.SecurityManager: Changing view acls to: root
20/06/25 18:06:55 INFO spark.SecurityManager: Changing modify acls to: root
20/06/25 18:06:55 INFO spark.SecurityManager: Changing view acls groups to:
20/06/25 18:06:55 INFO spark.SecurityManager: Changing modify acls groups to:
20/06/25 18:06:55 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(root); groups with view permissions: Set(); users  with modify permissions: Set(root); groups with modify permissions: Set()
20/06/25 18:06:56 INFO conf.HiveConf: Found configuration file file:/etc/hive/conf.cloudera.hive/hive-site.xml
20/06/25 18:06:56 INFO security.YARNHadoopDelegationTokenManager: Attempting to load user's ticket cache.
20/06/25 18:06:56 INFO yarn.Client: Submitting application application_1591413191159_0064 to ResourceManager
20/06/25 18:06:56 INFO impl.YarnClientImpl: Submitted application application_1591413191159_0064
20/06/25 18:06:57 INFO yarn.Client: Application report for application_1591413191159_0064 (state: ACCEPTED)
20/06/25 18:06:57 INFO yarn.Client:
	 client token: N/A
	 diagnostics: AM container is launched, waiting for AM container to Register with RM
	 ApplicationMaster host: N/A
	 ApplicationMaster RPC port: -1
	 queue: root.thequeue
	 start time: 1593079616105
	 final status: UNDEFINED
	 tracking URL: http://master01-prod.cdh:8088/proxy/application_1591413191159_0064/
	 user: root
20/06/25 18:06:58 INFO yarn.Client: Application report for application_1591413191159_0064 (state: ACCEPTED)
20/06/25 18:06:59 INFO yarn.Client: Application report for application_1591413191159_0064 (state: ACCEPTED)
20/06/25 18:07:00 INFO yarn.Client: Application report for application_1591413191159_0064 (state: RUNNING)
20/06/25 18:07:00 INFO yarn.Client:
	 client token: N/A
	 diagnostics: N/A
	 ApplicationMaster host: slave03-prod.cdh
	 ApplicationMaster RPC port: 40442
	 queue: root.thequeue
	 start time: 1593079616105
	 final status: UNDEFINED
	 tracking URL: http://master01-prod.cdh:8088/proxy/application_1591413191159_0064/
	 user: root
20/06/25 18:07:01 INFO yarn.Client: Application report for application_1591413191159_0064 (state: RUNNING)
20/06/25 18:07:02 INFO yarn.Client: Application report for application_1591413191159_0064 (state: RUNNING)
20/06/25 18:07:03 INFO yarn.Client: Application report for application_1591413191159_0064 (state: RUNNING)
20/06/25 18:07:04 INFO yarn.Client: Application report for application_1591413191159_0064 (state: RUNNING)
20/06/25 18:07:05 INFO yarn.Client: Application report for application_1591413191159_0064 (state: RUNNING)
20/06/25 18:07:06 INFO yarn.Client: Application report for application_1591413191159_0064 (state: RUNNING)
20/06/25 18:07:07 INFO yarn.Client: Application report for application_1591413191159_0064 (state: RUNNING)
20/06/25 18:07:08 INFO yarn.Client: Application report for application_1591413191159_0064 (state: RUNNING)
20/06/25 18:07:09 INFO yarn.Client: Application report for application_1591413191159_0064 (state: RUNNING)
20/06/25 18:07:10 INFO yarn.Client: Application report for application_1591413191159_0064 (state: RUNNING)
20/06/25 18:07:11 INFO yarn.Client: Application report for application_1591413191159_0064 (state: RUNNING)
20/06/25 18:07:12 INFO yarn.Client: Application report for application_1591413191159_0064 (state: RUNNING)
20/06/25 18:07:13 INFO yarn.Client: Application report for application_1591413191159_0064 (state: RUNNING)
20/06/25 18:07:14 INFO yarn.Client: Application report for application_1591413191159_0064 (state: RUNNING)
20/06/25 18:07:15 INFO yarn.Client: Application report for application_1591413191159_0064 (state: RUNNING)
20/06/25 18:07:16 INFO yarn.Client: Application report for application_1591413191159_0064 (state: RUNNING)
20/06/25 18:07:17 INFO yarn.Client: Application report for application_1591413191159_0064 (state: RUNNING)
20/06/25 18:07:18 INFO yarn.Client: Application report for application_1591413191159_0064 (state: RUNNING)
20/06/25 18:07:19 INFO yarn.Client: Application report for application_1591413191159_0064 (state: RUNNING)
20/06/25 18:07:20 INFO yarn.Client: Application report for application_1591413191159_0064 (state: RUNNING)
20/06/25 18:07:21 INFO yarn.Client: Application report for application_1591413191159_0064 (state: RUNNING)
20/06/25 18:07:22 INFO yarn.Client: Application report for application_1591413191159_0064 (state: FINISHED)
20/06/25 18:07:22 INFO yarn.Client:
	 client token: N/A
	 diagnostics: N/A
	 ApplicationMaster host: slave03-prod.cdh
	 ApplicationMaster RPC port: 40442
	 queue: root.thequeue
	 start time: 1593079616105
	 final status: SUCCEEDED
	 tracking URL: http://master01-prod.cdh:8088/proxy/application_1591413191159_0064/
	 user: root
20/06/25 18:07:22 INFO util.ShutdownHookManager: Shutdown hook called
20/06/25 18:07:22 INFO util.ShutdownHookManager: Deleting directory /tmp/spark-410edde9-c2d8-4fd4-9a12-1f7473cc89f5
20/06/25 18:07:22 INFO util.ShutdownHookManager: Deleting directory /tmp/spark-e847fe28-3bd5-4c74-bd1b-2b8fc7e5c4e6

參考資料

1. https://java.wekeepcoding.com/article/11916051/Unable+to+instantiate+SparkSession+with+Hive+support+because+Hive+classes+are+not+found
2. https://blog.csdn.net/weixin_43552807/article/details/105209467?ops_request_misc=%257B%2522request%255Fid%2522%253A%2522159307823319725250158297%2522%252C%2522scm%2522%253A%252220140713.130102334…%2522%257D&request_id=159307823319725250158297&biz_id=0&utm_medium=distribute.pc_search_result.none-task-blog-2_allsobaiduend~default-1-105209467.ecpm_v1_rank_business_v1&utm_term=Unable+to+instantiate+SparkSes