student.txt文件內容如下
001,Rajiv,Hyderabad
002,siddarth,Kolkata
003,Rajesh,Delhi
test.pig腳本內容如下：
student = LOAD 'hdfs://dn1:9000/student.txt' USING PigStorage(',')as(id:int,name:chararray,city:chararray);
Dump student;
grunt> exec test.pig
2020-05-21 21:29:42,781 [main] INFO org.apache.hadoop.conf.Configuration.deprecation - yarn.resourcemanager.system-metrics-publisher.enabled is deprecated. Instead, use yarn.system-metrics-publisher.enabled
2020-05-21 21:29:42,802 [main] INFO org.apache.hadoop.conf.Configuration.deprecation - yarn.resourcemanager.system-metrics-publisher.enabled is deprecated. Instead, use yarn.system-metrics-publisher.enabled
2020-05-21 21:29:42,874 [main] INFO org.apache.hadoop.conf.Configuration.deprecation - yarn.resourcemanager.system-metrics-publisher.enabled is deprecated. Instead, use yarn.system-metrics-publisher.enabled
2020-05-21 21:29:43,048 [main] INFO org.apache.pig.tools.pigstats.ScriptState - Pig features used in the script: UNKNOWN
2020-05-21 21:29:43,076 [main] INFO org.apache.hadoop.conf.Configuration.deprecation - yarn.resourcemanager.system-metrics-publisher.enabled is deprecated. Instead, use yarn.system-metrics-publisher.enabled
2020-05-21 21:29:43,078 [main] INFO org.apache.pig.data.SchemaTupleBackend - Key [pig.schematuple] was not set... will not generate code.
2020-05-21 21:29:43,078 [main] INFO org.apache.pig.newplan.logical.optimizer.LogicalPlanOptimizer - {RULES_ENABLED=[AddForEach, ColumnMapKeyPrune, ConstantCalculator, GroupByConstParallelSetter, LimitOptimizer, LoadTypeCastInserter, MergeFilter, MergeForEach, NestedLimitOptimizer, PartitionFilterOptimizer, PredicatePushdownOptimizer, PushDownForEachFlatten, PushUpFilter, SplitFilter, StreamTypeCastInserter]}
2020-05-21 21:29:43,096 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler - File concatenation threshold: 100 optimistic? false
2020-05-21 21:29:43,099 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer - MR plan size before optimization: 1
2020-05-21 21:29:43,099 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer - MR plan size after optimization: 1
2020-05-21 21:29:43,129 [main] INFO org.apache.hadoop.conf.Configuration.deprecation - yarn.resourcemanager.system-metrics-publisher.enabled is deprecated. Instead, use yarn.system-metrics-publisher.enabled
2020-05-21 21:29:43,133 [main] INFO org.apache.hadoop.yarn.client.RMProxy - Connecting to ResourceManager at dn2/192.168.100.204:8032
2020-05-21 21:29:43,150 [main] INFO org.apache.pig.tools.pigstats.mapreduce.MRScriptState - Pig script settings are added to the job
2020-05-21 21:29:43,151 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - mapred.job.reduce.markreset.buffer.percent is not set, set to default 0.3
2020-05-21 21:29:43,151 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - This job cannot be converted run in-process
2020-05-21 21:29:43,238 [Thread-26] INFO org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferClient - SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false
2020-05-21 21:29:43,466 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - Added jar file:/usr/local/pig/pig-0.17.0-core-h2.jar to DistributedCache through /tmp/temp1262446127/tmp-1930475542/pig-0.17.0-core-h2.jar
2020-05-21 21:29:43,487 [Thread-29] INFO org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferClient - SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false
2020-05-21 21:29:43,529 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - Added jar file:/usr/local/pig/lib/automaton-1.11-8.jar to DistributedCache through /tmp/temp1262446127/tmp244094970/automaton-1.11-8.jar
2020-05-21 21:29:43,579 [Thread-31] INFO org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferClient - SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false
2020-05-21 21:29:43,615 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - Added jar file:/usr/local/pig/lib/antlr-runtime-3.4.jar to DistributedCache through /tmp/temp1262446127/tmp700929672/antlr-runtime-3.4.jar
2020-05-21 21:29:43,631 [Thread-33] INFO org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferClient - SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false
2020-05-21 21:29:43,702 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - Added jar file:/usr/local/pig/lib/joda-time-2.9.3.jar to DistributedCache through /tmp/temp1262446127/tmp432040203/joda-time-2.9.3.jar
2020-05-21 21:29:43,704 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - Setting up single store job
2020-05-21 21:29:43,704 [main] INFO org.apache.pig.data.SchemaTupleFrontend - Key [pig.schematuple] is false, will not generate code.
2020-05-21 21:29:43,704 [main] INFO org.apache.pig.data.SchemaTupleFrontend - Starting process to move generated code to distributed cacche
2020-05-21 21:29:43,705 [main] INFO org.apache.pig.data.SchemaTupleFrontend - Setting key [pig.schematuple.classes] with classes to deserialize []
2020-05-21 21:29:43,755 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 1 map-reduce job(s) waiting for submission.
2020-05-21 21:29:43,773 [JobControl] INFO org.apache.hadoop.yarn.client.RMProxy - Connecting to ResourceManager at dn2/192.168.100.204:8032
2020-05-21 21:29:43,828 [JobControl] INFO org.apache.hadoop.mapreduce.JobResourceUploader - Disabling Erasure Coding for path: /tmp/hadoop-yarn/staging/hadoop/.staging/job_1590063597719_0005
2020-05-21 21:29:43,832 [JobControl] WARN org.apache.hadoop.mapreduce.JobResourceUploader - No job jar file set. User classes may not be found. See Job or Job#setJar(String).
2020-05-21 21:29:43,849 [JobControl] INFO org.apache.pig.builtin.PigStorage - Using PigTextInputFormat
2020-05-21 21:29:43,857 [JobControl] INFO org.apache.hadoop.mapreduce.lib.input.FileInputFormat - Total input files to process : 1
2020-05-21 21:29:43,857 [JobControl] INFO org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input paths to process : 1
2020-05-21 21:29:43,860 [JobControl] INFO org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input paths (combined) to process : 1
2020-05-21 21:29:43,878 [Thread-36] INFO org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferClient - SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false
2020-05-21 21:29:43,945 [Thread-38] INFO org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferClient - SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false
2020-05-21 21:29:44,017 [JobControl] INFO org.apache.hadoop.mapreduce.JobSubmitter - number of splits:1
2020-05-21 21:29:44,163 [Thread-40] INFO org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferClient - SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false
2020-05-21 21:29:44,288 [JobControl] INFO org.apache.hadoop.mapreduce.JobSubmitter - Submitting tokens for job: job_1590063597719_0005
2020-05-21 21:29:44,288 [JobControl] INFO org.apache.hadoop.mapreduce.JobSubmitter - Executing with tokens: []
2020-05-21 21:29:44,293 [JobControl] INFO org.apache.hadoop.mapred.YARNRunner - Job jar is not present. Not adding any jar to the list of resources.
2020-05-21 21:29:44,377 [JobControl] INFO org.apache.hadoop.yarn.client.api.impl.YarnClientImpl - Submitted application application_1590063597719_0005
2020-05-21 21:29:44,402 [JobControl] INFO org.apache.hadoop.mapreduce.Job - The url to track the job: http://dn2:8088/proxy/application_1590063597719_0005/
2020-05-21 21:29:44,402 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - HadoopJobId: job_1590063597719_0005
2020-05-21 21:29:44,402 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Processing aliases student
2020-05-21 21:29:44,402 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - detailed locations: M: student[1,10],student[-1,-1] C: R:
2020-05-21 21:29:44,425 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 0% complete
2020-05-21 21:29:44,425 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Running jobs are [job_1590063597719_0005]
2020-05-21 21:29:49,448 [main] WARN org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Ooops! Some job has failed! Specify -stop_on_failure if you want Pig to stop immediately on failure.
2020-05-21 21:29:49,449 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - job job_1590063597719_0005 has failed! Stop running all dependent jobs
2020-05-21 21:29:49,449 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 100% complete
2020-05-21 21:29:49,451 [main] INFO org.apache.hadoop.yarn.client.RMProxy - Connecting to ResourceManager at dn2/192.168.100.204:8032
2020-05-21 21:29:49,500 [main] INFO org.apache.hadoop.yarn.client.RMProxy - Connecting to ResourceManager at dn2/192.168.100.204:8032
2020-05-21 21:29:49,561 [main] ERROR org.apache.pig.tools.pigstats.mapreduce.MRPigStatsUtil - 1 map reduce job(s) failed!
2020-05-21 21:29:49,561 [main] INFO org.apache.pig.tools.pigstats.mapreduce.SimplePigStats - Script Statistics:
HadoopVersion PigVersion UserId StartedAt FinishedAt Features
3.2.1 0.17.0 hadoop 2020-05-21 21:29:43 2020-05-21 21:29:49 UNKNOWN
Failed!
Failed Jobs:
JobId Alias Feature Message Outputs
job_1590063597719_0005 student MAP_ONLY Message: Job failed! hdfs://dn1:9000/tmp/temp1262446127/tmp1730881622,
Input(s):
Failed to read data from "hdfs://dn1:9000/student.txt"
Output(s):
Failed to produce result in "hdfs://dn1:9000/tmp/temp1262446127/tmp1730881622"
Counters:
Total records written : 0
Total bytes written : 0
Spillable Memory Manager spill count : 0
Total bags proactively spilled: 0
Total records proactively spilled: 0
Job DAG:
job_1590063597719_0005
2020-05-21 21:29:49,561 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Failed!
2020-05-21 21:29:49,566 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR 1066: Unable to open iterator for alias student
Details at logfile: /home/hadoop/pig_1590067556707.log
grunt>
經過覈查確認此錯誤爲Pig版本與Hadoop版本不兼容引起