Pig Grunt之簡單命令及實例說明

原創

常笑笑

2020-06-23 19:51

Pig大的運行方法：

1、腳本

2、Grunt

3、嵌入式

Grunt

1、自動補全機制（命令補全、不支持文件名補全）

2、autocomplete文件

3、Eclipse插件PigPen

進入Grunt shell命令

[hadoop@master pig]$ ./bin/pig
2013-04-13 23:00:19,909 [main] INFO org.apache.pig.Main - Apache Pig version 0.10.0 (r1328203) compiled Apr 19 2012, 22:54:12
2013-04-13 23:00:19,909 [main] INFO org.apache.pig.Main - Logging error messages to: /opt/pig/pig_1365865219902.log
2013-04-13 23:00:20,237 [main] INFO org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting to hadoop file system at: hdfs://192.168.154.100:9000
2013-04-13 23:00:20,536 [main] INFO org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting to map-reduce job tracker at: 192.168.154.100:9001

幫助（help）

grunt> help
Commands:
<pig latin statement>; - See the PigLatin manual for details: http://hadoop.apache.org/pig
File system commands:
    fs <fs arguments> - Equivalent to Hadoop dfs command: http://hadoop.apache.org/common/docs/current/hdfs_shell.html
Diagnostic commands:
    describe <alias>[::<alias] - Show the schema for the alias. Inner aliases can be described as A::B.
    explain [-script <pigscript>] [-out <path>] [-brief] [-dot] [-param <param_name>=<param_value>]
        [-param_file <file_name>] [<alias>] - Show the execution plan to compute the alias or for entire script.
        -script - Explain the entire script.
        -out - Store the output into directory rather than print to stdout.
        -brief - Don't expand nested plans (presenting a smaller graph for overview).
        -dot - Generate the output in .dot format. Default is text format.
        -param <param_name - See parameter substitution for details.
        -param_file <file_name> - See parameter substitution for details.
        alias - Alias to explain.
    dump <alias> - Compute the alias and writes the results to stdout.
Utility Commands:
    exec [-param <param_name>=param_value] [-param_file <file_name>] <script> -
        Execute the script with access to grunt environment including aliases.
        -param <param_name - See parameter substitution for details.
        -param_file <file_name> - See parameter substitution for details.
        script - Script to be executed.
    run [-param <param_name>=param_value] [-param_file <file_name>] <script> -
        Execute the script with access to grunt environment.
        -param <param_name - See parameter substitution for details.
        -param_file <file_name> - See parameter substitution for details.
        script - Script to be executed.
    sh <shell command> - Invoke a shell command.
    kill <job_id> - Kill the hadoop job specified by the hadoop job id.
    set <key> <value> - Provide execution parameters to Pig. Keys and values are case sensitive.
        The following keys are supported:
        default_parallel - Script-level reduce parallelism. Basic input size heuristics used by default.
        debug - Set debug on or off. Default is off.
        job.name - Single-quoted name for jobs. Default is PigLatin:<script name>
        job.priority - Priority for jobs. Values: very_low, low, normal, high, very_high. Default is normal
        stream.skippath - String that contains the path. This is used by streaming.
        any hadoop property.
    help - Display this message.
    quit - Quit the grunt shell.

查看（ls、cd 、cat）

grunt> ls
hdfs://192.168.154.100:9000/user/hadoop/in <dir>
hdfs://192.168.154.100:9000/user/hadoop/out <dir>

grunt> cd in
grunt> ls
hdfs://192.168.154.100:9000/user/hadoop/in/test1.txt<r 1>   12
hdfs://192.168.154.100:9000/user/hadoop/in/test2.txt<r 1>   13
hdfs://192.168.154.100:9000/user/hadoop/in/test_1.txt<r 1>   328
hdfs://192.168.154.100:9000/user/hadoop/in/test_2.txt<r 1>   139
grunt> cat test1.txt
hello world

複製到本地（copyToLocal）

grunt> ls
hdfs://192.168.154.100:9000/user/hadoop/in/test1.txt<r 1>   12
hdfs://192.168.154.100:9000/user/hadoop/in/test2.txt<r 1>   13
hdfs://192.168.154.100:9000/user/hadoop/in/test_1.txt<r 1>   328
hdfs://192.168.154.100:9000/user/hadoop/in/test_2.txt<r 1>   139
grunt> copyToLocal test1.txt ttt

[root@master pig]# ls -l ttt
-rwxrwxrwx. 1 hadoop hadoop 12 4月 13 23:06 ttt
[root@master pig]#

執行操作系統命令：sh

grunt> sh jps
2098 DataNode
1986 NameNode
2700 Jps
2539 RunJar
2297 JobTracker
2211 SecondaryNameNode
2411 TaskTracker
grunt>

Pig數據模型

Bag：表

Tuple：行，記錄

Field：屬性

Pig不要求同一個bag裏面的各個tuple有相同數量或相同類型的field

Pig latin常用語句

LOAD：指出載入數據的方法

FOREACH：逐行掃描進行某種處理

FILTER：過濾行