hive表信息查詢、查看錶結構、表操作等

轉載:http://blog.csdn.net/lskyne/article/details/38427895

問題導讀：
1.如何查看hive表結構？
2.如何查看錶結構信息？
3.如何查看分區信息？
4.哪個命令可以模糊搜索表

1.hive模糊搜索表

show tables like '*name*';

2.查看錶結構信息

desc formatted table_name;
desc table_name;

3.查看分區信息

show partitions table_name;

4.根據分區查詢數據

select table_coulm from table_name where partition_name = '2014-02-25';

5.查看hdfs文件信息

dfs -ls /user/hive/warehouse/table02;

6.從文件加載數據進表(OVERWRITE覆蓋,追加不需要OVERWRITE關鍵字)

LOAD DATA LOCAL INPATH 'dim_csl_rule_config.txt' OVERWRITE into table dim.dim_csl_rule_config;
 --從查詢語句給table插入數據
INSERT OVERWRITE TABLE test_h02_click_log PARTITION(dt) select *
from stage.s_h02_click_log where dt='2014-01-22' limit 100;

7.導出數據到文件

hive> insert overwrite local directory '/opt/xuzhiguo' select * from pdca_factory_t where factory_id<4;
Query ID = root_20171228114351_ab465584-9575-4ec2-a6ad-6b7e6ecf227d
Total jobs = 1
Launching Job 1 out of 1
In order to change the average load for a reducer (in bytes):
  set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
  set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
  set mapreduce.job.reduces=<number>
Starting Spark Job = d4aa5361-e1d3-4b0a-802a-62a3c8124a13

Query Hive on Spark job[0] stages:
0

Status: Running (Hive on Spark job[0])
Job Progress Format
CurrentTime StageId_StageAttemptId: SucceededTasksCount(+RunningTasksCount-FailedTasksCount)/TotalTasksCount [StageCost]
2017-12-28 11:44:13,438 Stage-0_0: 0(+1)/1
2017-12-28 11:44:16,547 Stage-0_0: 0(+1)/1
2017-12-28 11:44:19,602 Stage-0_0: 0(+1)/1
2017-12-28 11:44:22,660 Stage-0_0: 0(+1)/1
2017-12-28 11:44:25,692 Stage-0_0: 0(+1)/1
2017-12-28 11:44:28,724 Stage-0_0: 0(+1)/1
2017-12-28 11:44:29,741 Stage-0_0: 1/1 Finished
Status: Finished successfully in 30.46 seconds
Moving data to local directory /opt/xuzhiguo
OK
Time taken: 39.248 seconds
hive> 

insert overwrite directory '/tmp/csl_rule_cfg' select a.* from dim.dim_csl_rule_config a;
 hive -e "select day_id,pv,uv,ip_count,click_next_count,second_bounce_rate,return_visit,pg_type

from tmp.tmp_h02_click_log_baitiao_ag_sum where day_id in ('2014-03-06','2014-03-07','2014-03-08','2014-03-09','2014-03-10');">

/home/jrjt/testan/baitiao.dat;

8.自定義udf函數

  1.繼承UDF類
  2.重寫evaluate方法
  3.把項目打成jar包
  4.hive中執行命令add jar /home/jrjt/dwetl/PUB/UDF/udf/GetProperty.jar;
  5.創建函數create temporary function get_pro as 'jd.Get_Property'//jd.jd.Get_Property爲類路徑;

9.查詢顯示列名 及 行轉列顯示 

set hive.cli.print.header=true; // 打印列名
set hive.cli.print.row.to.vertical=true; // 開啓行轉列功能, 前提必須開啓打印列名功能
set hive.cli.print.row.to.vertical.num=1; // 設置每行顯示的列數

10.查看錶文件大小,下載文件到某個目錄,顯示多少行到某個文件

dfs -du hdfs://BJYZH3-HD-JRJT-4137.jd.com:54310/user/jrjt/warehouse/stage.db/s_h02_click_log;
dfs -get /user/jrjt/warehouse/ods.db/o_h02_click_log_i_new/dt=2014-01-21/000212_0 /home/jrjt/testan/;
head -n 1000 文件名 > 文件名

11.殺死某個任務  不在hive shell中執行

hadoop job -kill job_201403041453_58315

12.hive-wui路徑

   http://172.17.41.38/jobtracker.jsp

13.刪除分區

alter table tmp_h02_click_log_baitiao drop partition(dt='2014-03-01');
alter table d_h02_click_log_basic_d_fact drop partition(dt='2014-01-17');

14.hive命令行操作15.hive上操作hadoop文件基本命令

    查看文件大小

dfs -du /user/jrjt/warehouse/tmp.db/tmp_h02_click_log/dt=2014-02-15;

    刪除文件

dfs -rm /user/jrjt/warehouse/tmp.db/tmp_h02_click_log/dt=2014-02-15;

16.插入數據sql、導出數據sql
    1.insert 語法格式爲：

    基本的插入語法：

INSERT OVERWRITE TABLE tablename [PARTITON(partcol1=val1,partclo2=val2)]select_statement FROM from_statement
insert overwrite table test_insert select * from test_table;

    對多個表進行插入操作：

FROM fromstatte
INSERT OVERWRITE TABLE tablename1 [PARTITON(partcol1=val1,partclo2=val2)]select_statement1
INSERT OVERWRITE TABLE tablename2 [PARTITON(partcol1=val1,partclo2=val2)]select_statement2

from test_table
insert overwrite table test_insert1
select key
insert overwrite table test_insert2
select value;

    insert的時候，from子句即可以放在select 子句後面，也可以放在 insert子句前面。

    hive不支持用insert語句一條一條的進行插入操作，也不支持update操作。數據是以load的方式加載到建立好的表中。數據一旦導入就不可以修改。

    2.通過查詢將數據保存到filesystem

INSERT OVERWRITE [LOCAL] DIRECTORY directory SELECT.... FROM .....

    導入數據到本地目錄：

insert overwrite local directory '/home/zhangxin/hive' select * from test_insert1;

    產生的文件會覆蓋指定目錄中的其他文件，即將目錄中已經存在的文件進行刪除。

    導出數據到HDFS中：

insert overwrite directory '/user/zhangxin/export_test' select value from test_table;

    同一個查詢結果可以同時插入到多個表或者多個目錄中：

from test_insert1
insert overwrite local directory '/home/zhangxin/hive' select *
insert overwrite directory '/user/zhangxin/export_test' select value;

17.mapjoin的使用 應用場景：1.關聯操作中有一張表非常小 2.不等值的鏈接操作

select /*+ mapjoin(A)*/ f.a,f.b from A t join B f  on ( f.a=t.a and f.ftime=20110802)

18.perl啓動任務

perl /home/jrjt/dwetl/APP/APP/A_H02_CLICK_LOG_CREDIT_USER/bin/a_h02_click_log_credit_user.pl
APP_A_H02_CLICK_LOG_CREDIT_USER_20140215.dir >& /home/jrjt/dwetl/LOG/APP/20140306/

<span style="white-space:pre;">                     </span> a_h02_click_log_credit_user.pl.4.log

19.查看perl進程

ps -ef|grep perl

20.hive命令移動表數據到另外一張表目錄下並添加分區

dfs -cp /user/jrjt/warehouse/tmp.db/tmp_h02_click_log/dt=2014-02-18 /user/jrjt/warehouse/ods.db/o_h02_click_log/;
dfs -cp /user/jrjt/warehouse/tmp.db/tmp_h02_click_log_baitiao/* /user/jrjt/warehouse/dw.db/

<span style="white-space:pre;">                         </span>d_h02_click_log_baitiao_basic_d_fact/;--複製所有分區數據
 alter table d_h02_click_log_baitiao_basic_d_fact add partition(dt='2014-03-11')

<span style="white-space:pre;"> </span>location '/user/jrjt/warehouse/dw.db/d_h02_click_log_baitiao_basic_d_fact/dt=2014-03-11';

21.導出白條數據

hive -e "select day_id,pv,uv,ip_count,click_next_count,second_bounce_rate,return_visit,pg_type

from tmp.tmp_h02_click_log_baitiao_ag_sum where day_id like '2014-03%';"> /home/jrjt/testan/baitiao.xlsx;

22.hive修改表名

ALTER TABLE o_h02_click_log_i RENAME TO o_h02_click_log_i_bk;

23.hive複製表結構

CREATE TABLE d_h02_click_log_baitiao_ag_sum LIKE tmp.tmp_h02_click_log_baitiao_ag_sum;

24.hive官網網址

   https://cwiki.apache.org/conflue ... ionandConfiguration

   http://www.360doc.com/content/12/0111/11/7362_178698714.shtml

25.hive添加字段

alter table tmp_h02_click_log_baitiao_ag_sum

<span style="white-space:pre;"> </span>add columns(current_session_timelenth_count bigint comment '頁面停留總時長');
 ALTER TABLE tmp_h02_click_log_baitiao CHANGE current_session_timelenth

<span style="white-space:pre;"> </span>current_session_timelenth bigint comment '當前會話停留時間';

26.hive開啓簡單模式不啓用mr

set hive.fetch.task.conversion=more;

27.以json格式輸出執行語句會讀取的input table和input partition信息

Explain dependency query