一.需求描述
今天在hdfs上看到，一個從sqoop導入的表，只有幾個大的文件，而不像其它的表，都是一些小文件。
備註:
測試環境只有4個節點，然後HDFS上剛好4個文件，不確定是否會影響查詢的性能。
sqoop導入命令:
sqoop import \
--connect jdbc:mysql://10.31.1.122:3306/test \
--username root \
--password abc123 \
--table fact_sale \
--fields-terminated-by '\0001' \
--delete-target-dir \
--num-mappers 4 \
--hive-import \
--hive-database test \
--hive-table ods_fact_sale \
--hive-overwrite
二.做實驗求證
2.1 複製一張表
複製一張表，查看數據在hdfs上的分佈
hive>
> create table ods_fact_sale_new as select * from ods_fact_sale;
Query ID = root_20211130170738_57662781-2673-4615-bd9f-c46716eb2c8a
Total jobs = 3
Launching Job 1 out of 3
Number of reduce tasks is set to 0 since there's no reduce operator
21/11/30 17:07:39 INFO client.ConfiguredRMFailoverProxyProvider: Failing over to rm69
Starting Job = job_1638236643110_0020, Tracking URL = http://hp3:8088/proxy/application_1638236643110_0020/
Kill Command = /opt/cloudera/parcels/CDH-6.3.1-1.cdh6.3.1.p0.1470567/lib/hadoop/bin/hadoop job -kill job_1638236643110_0020
Hadoop job information for Stage-1: number of mappers: 117; number of reducers: 0
2021-11-30 17:07:47,001 Stage-1 map = 0%, reduce = 0%
2021-11-30 17:08:05,365 Stage-1 map = 1%, reduce = 0%, Cumulative CPU 16.41 sec
2021-11-30 17:08:06,464 Stage-1 map = 3%, reduce = 0%, Cumulative CPU 59.92 sec
2021-11-30 17:08:09,680 Stage-1 map = 4%, reduce = 0%, Cumulative CPU 70.22 sec
2021-11-30 17:08:20,423 Stage-1 map = 5%, reduce = 0%, Cumulative CPU 86.55 sec
2021-11-30 17:08:25,784 Stage-1 map = 6%, reduce = 0%, Cumulative CPU 120.01 sec
2021-11-30 17:08:27,925 Stage-1 map = 7%, reduce = 0%, Cumulative CPU 130.45 sec
2021-11-30 17:08:28,953 Stage-1 map = 8%, reduce = 0%, Cumulative CPU 137.82 sec
2021-11-30 17:08:31,098 Stage-1 map = 9%, reduce = 0%, Cumulative CPU 140.74 sec
2021-11-30 17:08:44,957 Stage-1 map = 11%, reduce = 0%, Cumulative CPU 188.62 sec
2021-11-30 17:08:46,012 Stage-1 map = 12%, reduce = 0%, Cumulative CPU 191.78 sec
2021-11-30 17:08:48,115 Stage-1 map = 13%, reduce = 0%, Cumulative CPU 206.06 sec
2021-11-30 17:08:51,288 Stage-1 map = 14%, reduce = 0%, Cumulative CPU 218.2 sec
2021-11-30 17:09:03,029 Stage-1 map = 15%, reduce = 0%, Cumulative CPU 244.56 sec
2021-11-30 17:09:04,071 Stage-1 map = 16%, reduce = 0%, Cumulative CPU 267.54 sec
2021-11-30 17:09:05,095 Stage-1 map = 17%, reduce = 0%, Cumulative CPU 272.11 sec
2021-11-30 17:09:10,375 Stage-1 map = 18%, reduce = 0%, Cumulative CPU 284.04 sec
2021-11-30 17:09:16,804 Stage-1 map = 19%, reduce = 0%, Cumulative CPU 299.07 sec
2021-11-30 17:09:23,242 Stage-1 map = 21%, reduce = 0%, Cumulative CPU 335.36 sec
2021-11-30 17:09:29,602 Stage-1 map = 22%, reduce = 0%, Cumulative CPU 350.85 sec
2021-11-30 17:09:30,647 Stage-1 map = 23%, reduce = 0%, Cumulative CPU 366.74 sec
2021-11-30 17:09:40,210 Stage-1 map = 24%, reduce = 0%, Cumulative CPU 378.36 sec
2021-11-30 17:09:42,302 Stage-1 map = 25%, reduce = 0%, Cumulative CPU 404.13 sec
2021-11-30 17:09:43,366 Stage-1 map = 26%, reduce = 0%, Cumulative CPU 405.49 sec
2021-11-30 17:09:48,711 Stage-1 map = 27%, reduce = 0%, Cumulative CPU 429.61 sec
2021-11-30 17:09:58,316 Stage-1 map = 28%, reduce = 0%, Cumulative CPU 442.71 sec
2021-11-30 17:09:59,394 Stage-1 map = 29%, reduce = 0%, Cumulative CPU 469.52 sec
2021-11-30 17:10:00,423 Stage-1 map = 30%, reduce = 0%, Cumulative CPU 482.96 sec
2021-11-30 17:10:01,477 Stage-1 map = 31%, reduce = 0%, Cumulative CPU 484.26 sec
2021-11-30 17:10:09,882 Stage-1 map = 32%, reduce = 0%, Cumulative CPU 496.46 sec
2021-11-30 17:10:15,159 Stage-1 map = 33%, reduce = 0%, Cumulative CPU 521.86 sec
2021-11-30 17:10:17,285 Stage-1 map = 34%, reduce = 0%, Cumulative CPU 533.32 sec
2021-11-30 17:10:18,382 Stage-1 map = 35%, reduce = 0%, Cumulative CPU 545.64 sec
2021-11-30 17:10:26,819 Stage-1 map = 36%, reduce = 0%, Cumulative CPU 563.88 sec
2021-11-30 17:10:27,909 Stage-1 map = 37%, reduce = 0%, Cumulative CPU 576.51 sec
2021-11-30 17:10:33,238 Stage-1 map = 38%, reduce = 0%, Cumulative CPU 590.45 sec
2021-11-30 17:10:40,630 Stage-1 map = 39%, reduce = 0%, Cumulative CPU 616.39 sec
2021-11-30 17:10:41,669 Stage-1 map = 40%, reduce = 0%, Cumulative CPU 631.7 sec
2021-11-30 17:10:45,964 Stage-1 map = 41%, reduce = 0%, Cumulative CPU 644.92 sec
2021-11-30 17:10:52,369 Stage-1 map = 42%, reduce = 0%, Cumulative CPU 658.5 sec
2021-11-30 17:10:56,648 Stage-1 map = 44%, reduce = 0%, Cumulative CPU 687.38 sec
2021-11-30 17:11:04,034 Stage-1 map = 45%, reduce = 0%, Cumulative CPU 712.14 sec
2021-11-30 17:11:09,339 Stage-1 map = 46%, reduce = 0%, Cumulative CPU 724.9 sec
2021-11-30 17:11:10,392 Stage-1 map = 47%, reduce = 0%, Cumulative CPU 739.8 sec
2021-11-30 17:11:15,735 Stage-1 map = 48%, reduce = 0%, Cumulative CPU 752.7 sec
2021-11-30 17:11:19,899 Stage-1 map = 49%, reduce = 0%, Cumulative CPU 766.04 sec
2021-11-30 17:11:24,089 Stage-1 map = 50%, reduce = 0%, Cumulative CPU 778.33 sec
2021-11-30 17:11:28,301 Stage-1 map = 51%, reduce = 0%, Cumulative CPU 804.63 sec
2021-11-30 17:11:33,656 Stage-1 map = 52%, reduce = 0%, Cumulative CPU 817.47 sec
2021-11-30 17:11:37,916 Stage-1 map = 53%, reduce = 0%, Cumulative CPU 830.2 sec
2021-11-30 17:11:38,942 Stage-1 map = 54%, reduce = 0%, Cumulative CPU 845.97 sec
2021-11-30 17:11:43,149 Stage-1 map = 55%, reduce = 0%, Cumulative CPU 858.89 sec
2021-11-30 17:11:48,365 Stage-1 map = 56%, reduce = 0%, Cumulative CPU 871.54 sec
2021-11-30 17:11:53,694 Stage-1 map = 57%, reduce = 0%, Cumulative CPU 898.79 sec
2021-11-30 17:11:55,823 Stage-1 map = 58%, reduce = 0%, Cumulative CPU 911.2 sec
2021-11-30 17:12:01,113 Stage-1 map = 59%, reduce = 0%, Cumulative CPU 923.85 sec
2021-11-30 17:12:06,403 Stage-1 map = 60%, reduce = 0%, Cumulative CPU 937.85 sec
2021-11-30 17:12:07,471 Stage-1 map = 61%, reduce = 0%, Cumulative CPU 952.42 sec
2021-11-30 17:12:11,716 Stage-1 map = 62%, reduce = 0%, Cumulative CPU 965.43 sec
2021-11-30 17:12:21,272 Stage-1 map = 64%, reduce = 0%, Cumulative CPU 1006.65 sec
2021-11-30 17:12:23,411 Stage-1 map = 65%, reduce = 0%, Cumulative CPU 1019.55 sec
2021-11-30 17:12:31,940 Stage-1 map = 66%, reduce = 0%, Cumulative CPU 1033.41 sec
2021-11-30 17:12:35,119 Stage-1 map = 67%, reduce = 0%, Cumulative CPU 1059.51 sec
2021-11-30 17:12:36,194 Stage-1 map = 68%, reduce = 0%, Cumulative CPU 1060.69 sec
2021-11-30 17:12:43,564 Stage-1 map = 69%, reduce = 0%, Cumulative CPU 1086.14 sec
2021-11-30 17:12:48,796 Stage-1 map = 70%, reduce = 0%, Cumulative CPU 1097.74 sec
2021-11-30 17:12:49,853 Stage-1 map = 71%, reduce = 0%, Cumulative CPU 1114.22 sec
2021-11-30 17:12:54,113 Stage-1 map = 72%, reduce = 0%, Cumulative CPU 1128.75 sec
2021-11-30 17:13:00,422 Stage-1 map = 73%, reduce = 0%, Cumulative CPU 1142.09 sec
2021-11-30 17:13:02,562 Stage-1 map = 74%, reduce = 0%, Cumulative CPU 1155.07 sec
2021-11-30 17:13:07,790 Stage-1 map = 75%, reduce = 0%, Cumulative CPU 1181.78 sec
2021-11-30 17:13:12,028 Stage-1 map = 76%, reduce = 0%, Cumulative CPU 1195.25 sec
2021-11-30 17:13:17,350 Stage-1 map = 77%, reduce = 0%, Cumulative CPU 1223.03 sec
2021-11-30 17:13:18,391 Stage-1 map = 78%, reduce = 0%, Cumulative CPU 1224.25 sec
2021-11-30 17:13:21,566 Stage-1 map = 79%, reduce = 0%, Cumulative CPU 1236.93 sec
2021-11-30 17:13:30,101 Stage-1 map = 80%, reduce = 0%, Cumulative CPU 1261.89 sec
2021-11-30 17:13:32,244 Stage-1 map = 81%, reduce = 0%, Cumulative CPU 1277.29 sec
2021-11-30 17:13:35,445 Stage-1 map = 82%, reduce = 0%, Cumulative CPU 1289.5 sec
2021-11-30 17:13:42,881 Stage-1 map = 83%, reduce = 0%, Cumulative CPU 1303.41 sec
2021-11-30 17:13:47,163 Stage-1 map = 85%, reduce = 0%, Cumulative CPU 1331.02 sec
2021-11-30 17:13:54,603 Stage-1 map = 86%, reduce = 0%, Cumulative CPU 1356.7 sec
2021-11-30 17:14:00,955 Stage-1 map = 88%, reduce = 0%, Cumulative CPU 1384.56 sec
2021-11-30 17:14:07,221 Stage-1 map = 89%, reduce = 0%, Cumulative CPU 1399.0 sec
2021-11-30 17:14:09,324 Stage-1 map = 90%, reduce = 0%, Cumulative CPU 1412.39 sec
2021-11-30 17:14:12,457 Stage-1 map = 91%, reduce = 0%, Cumulative CPU 1424.71 sec
2021-11-30 17:14:19,858 Stage-1 map = 92%, reduce = 0%, Cumulative CPU 1453.16 sec
2021-11-30 17:14:24,116 Stage-1 map = 93%, reduce = 0%, Cumulative CPU 1465.13 sec
2021-11-30 17:14:27,319 Stage-1 map = 94%, reduce = 0%, Cumulative CPU 1479.25 sec
2021-11-30 17:14:29,451 Stage-1 map = 95%, reduce = 0%, Cumulative CPU 1495.19 sec
2021-11-30 17:14:31,556 Stage-1 map = 96%, reduce = 0%, Cumulative CPU 1508.42 sec
2021-11-30 17:14:39,025 Stage-1 map = 97%, reduce = 0%, Cumulative CPU 1521.05 sec
2021-11-30 17:14:43,231 Stage-1 map = 98%, reduce = 0%, Cumulative CPU 1549.65 sec
2021-11-30 17:14:46,323 Stage-1 map = 99%, reduce = 0%, Cumulative CPU 1562.56 sec
2021-11-30 17:14:50,410 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 1577.67 sec
MapReduce Total cumulative CPU time: 26 minutes 17 seconds 670 msec
Ended Job = job_1638236643110_0020
Stage-4 is selected by condition resolver.
Stage-3 is filtered out by condition resolver.
Stage-5 is filtered out by condition resolver.
Moving data to directory hdfs://nameservice1/user/hive/warehouse/test.db/.hive-staging_hive_2021-11-30_17-07-38_496_3990528869308235621-1/-ext-10001
Moving data to directory hdfs://nameservice1/user/hive/warehouse/test.db/ods_fact_sale_new
MapReduce Jobs Launched:
Stage-Stage-1: Map: 117 Cumulative CPU: 1577.67 sec HDFS Read: 31436862126 HDFS Write: 31421104192 HDFS EC Read: 0 SUCCESS
Total MapReduce CPU Time Spent: 26 minutes 17 seconds 670 msec
OK
Time taken: 434.49 seconds
hive>
可以看到很多的小文件， 都是256M左右一個
2.2 查詢看那個錶快一些
實驗可以看到，小文件更多的這個會慢一些，大概慢了20%左右
hive>
>
>
> select count(*) from ods_fact_sale;
Query ID = root_20211130171557_f5f80c10-c199-4140-95c2-4abf5b77a809
Total jobs = 1
Launching Job 1 out of 1
Number of reduce tasks determined at compile time: 1
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapreduce.job.reduces=<number>
21/11/30 17:15:57 INFO client.ConfiguredRMFailoverProxyProvider: Failing over to rm69
Starting Job = job_1638236643110_0021, Tracking URL = http://hp3:8088/proxy/application_1638236643110_0021/
Kill Command = /opt/cloudera/parcels/CDH-6.3.1-1.cdh6.3.1.p0.1470567/lib/hadoop/bin/hadoop job -kill job_1638236643110_0021
Hadoop job information for Stage-1: number of mappers: 117; number of reducers: 1
2021-11-30 17:16:04,618 Stage-1 map = 0%, reduce = 0%
2021-11-30 17:16:13,088 Stage-1 map = 1%, reduce = 0%, Cumulative CPU 5.68 sec
2021-11-30 17:16:17,293 Stage-1 map = 4%, reduce = 0%, Cumulative CPU 27.19 sec
2021-11-30 17:16:20,450 Stage-1 map = 5%, reduce = 0%, Cumulative CPU 33.03 sec
2021-11-30 17:16:26,849 Stage-1 map = 6%, reduce = 0%, Cumulative CPU 38.79 sec
2021-11-30 17:16:27,877 Stage-1 map = 9%, reduce = 0%, Cumulative CPU 60.57 sec
2021-11-30 17:16:33,151 Stage-1 map = 10%, reduce = 0%, Cumulative CPU 66.33 sec
2021-11-30 17:16:38,499 Stage-1 map = 14%, reduce = 0%, Cumulative CPU 87.98 sec
2021-11-30 17:16:39,563 Stage-1 map = 15%, reduce = 0%, Cumulative CPU 93.0 sec
2021-11-30 17:16:49,110 Stage-1 map = 18%, reduce = 0%, Cumulative CPU 114.77 sec
2021-11-30 17:16:50,160 Stage-1 map = 19%, reduce = 0%, Cumulative CPU 119.78 sec
2021-11-30 17:16:52,274 Stage-1 map = 20%, reduce = 0%, Cumulative CPU 125.68 sec
2021-11-30 17:16:58,605 Stage-1 map = 21%, reduce = 0%, Cumulative CPU 136.39 sec
2021-11-30 17:16:59,631 Stage-1 map = 24%, reduce = 0%, Cumulative CPU 152.96 sec
2021-11-30 17:17:06,037 Stage-1 map = 25%, reduce = 0%, Cumulative CPU 158.79 sec
2021-11-30 17:17:08,166 Stage-1 map = 26%, reduce = 0%, Cumulative CPU 169.92 sec
2021-11-30 17:17:09,207 Stage-1 map = 27%, reduce = 0%, Cumulative CPU 174.52 sec
2021-11-30 17:17:11,374 Stage-1 map = 28%, reduce = 0%, Cumulative CPU 179.8 sec
2021-11-30 17:17:13,501 Stage-1 map = 29%, reduce = 0%, Cumulative CPU 185.61 sec
2021-11-30 17:17:17,737 Stage-1 map = 30%, reduce = 0%, Cumulative CPU 191.06 sec
2021-11-30 17:17:18,763 Stage-1 map = 32%, reduce = 0%, Cumulative CPU 200.7 sec
2021-11-30 17:17:20,864 Stage-1 map = 33%, reduce = 0%, Cumulative CPU 211.88 sec
2021-11-30 17:17:26,228 Stage-1 map = 35%, reduce = 0%, Cumulative CPU 223.32 sec
2021-11-30 17:17:28,279 Stage-1 map = 36%, reduce = 0%, Cumulative CPU 227.79 sec
2021-11-30 17:17:29,327 Stage-1 map = 37%, reduce = 0%, Cumulative CPU 233.25 sec
2021-11-30 17:17:30,360 Stage-1 map = 38%, reduce = 0%, Cumulative CPU 238.17 sec
2021-11-30 17:17:36,716 Stage-1 map = 40%, reduce = 0%, Cumulative CPU 255.0 sec
2021-11-30 17:17:38,807 Stage-1 map = 41%, reduce = 0%, Cumulative CPU 259.56 sec
2021-11-30 17:17:39,878 Stage-1 map = 42%, reduce = 0%, Cumulative CPU 265.37 sec
2021-11-30 17:17:40,937 Stage-1 map = 43%, reduce = 0%, Cumulative CPU 270.65 sec
2021-11-30 17:17:45,199 Stage-1 map = 44%, reduce = 0%, Cumulative CPU 282.03 sec
2021-11-30 17:17:47,306 Stage-1 map = 45%, reduce = 0%, Cumulative CPU 287.64 sec
2021-11-30 17:17:48,338 Stage-1 map = 46%, reduce = 0%, Cumulative CPU 293.17 sec
2021-11-30 17:17:51,494 Stage-1 map = 48%, reduce = 0%, Cumulative CPU 304.12 sec
2021-11-30 17:17:55,725 Stage-1 map = 49%, reduce = 0%, Cumulative CPU 309.81 sec
2021-11-30 17:17:56,745 Stage-1 map = 50%, reduce = 0%, Cumulative CPU 315.43 sec
2021-11-30 17:17:58,882 Stage-1 map = 51%, reduce = 0%, Cumulative CPU 326.09 sec
2021-11-30 17:18:01,018 Stage-1 map = 52%, reduce = 0%, Cumulative CPU 331.7 sec
2021-11-30 17:18:04,171 Stage-1 map = 54%, reduce = 0%, Cumulative CPU 342.8 sec
2021-11-30 17:18:07,359 Stage-1 map = 55%, reduce = 0%, Cumulative CPU 348.55 sec
2021-11-30 17:18:09,483 Stage-1 map = 56%, reduce = 0%, Cumulative CPU 360.16 sec
2021-11-30 17:18:10,519 Stage-1 map = 57%, reduce = 0%, Cumulative CPU 365.93 sec
2021-11-30 17:18:13,696 Stage-1 map = 58%, reduce = 0%, Cumulative CPU 371.6 sec
2021-11-30 17:18:15,797 Stage-1 map = 59%, reduce = 0%, Cumulative CPU 377.41 sec
2021-11-30 17:18:16,863 Stage-1 map = 60%, reduce = 0%, Cumulative CPU 383.7 sec
2021-11-30 17:18:19,010 Stage-1 map = 61%, reduce = 0%, Cumulative CPU 389.66 sec
2021-11-30 17:18:21,110 Stage-1 map = 62%, reduce = 0%, Cumulative CPU 395.19 sec
2021-11-30 17:18:24,275 Stage-1 map = 63%, reduce = 0%, Cumulative CPU 406.42 sec
2021-11-30 17:18:26,422 Stage-1 map = 64%, reduce = 0%, Cumulative CPU 411.91 sec
2021-11-30 17:18:27,484 Stage-1 map = 65%, reduce = 0%, Cumulative CPU 417.81 sec
2021-11-30 17:18:28,538 Stage-1 map = 66%, reduce = 0%, Cumulative CPU 423.1 sec
2021-11-30 17:18:30,662 Stage-1 map = 67%, reduce = 0%, Cumulative CPU 428.52 sec
2021-11-30 17:18:33,820 Stage-1 map = 68%, reduce = 0%, Cumulative CPU 433.48 sec
2021-11-30 17:18:35,916 Stage-1 map = 69%, reduce = 0%, Cumulative CPU 444.98 sec
2021-11-30 17:18:38,034 Stage-1 map = 70%, reduce = 0%, Cumulative CPU 449.98 sec
2021-11-30 17:18:41,220 Stage-1 map = 71%, reduce = 0%, Cumulative CPU 454.77 sec
2021-11-30 17:18:42,279 Stage-1 map = 72%, reduce = 0%, Cumulative CPU 460.59 sec
2021-11-30 17:18:43,335 Stage-1 map = 73%, reduce = 0%, Cumulative CPU 466.26 sec
2021-11-30 17:18:45,444 Stage-1 map = 74%, reduce = 0%, Cumulative CPU 471.77 sec
2021-11-30 17:18:48,625 Stage-1 map = 75%, reduce = 0%, Cumulative CPU 483.1 sec
2021-11-30 17:18:50,718 Stage-1 map = 76%, reduce = 0%, Cumulative CPU 488.49 sec
2021-11-30 17:18:52,856 Stage-1 map = 77%, reduce = 0%, Cumulative CPU 494.37 sec
2021-11-30 17:18:54,948 Stage-1 map = 79%, reduce = 0%, Cumulative CPU 505.83 sec
2021-11-30 17:19:01,237 Stage-1 map = 81%, reduce = 0%, Cumulative CPU 522.77 sec
2021-11-30 17:19:03,370 Stage-1 map = 82%, reduce = 0%, Cumulative CPU 528.25 sec
2021-11-30 17:19:05,537 Stage-1 map = 83%, reduce = 0%, Cumulative CPU 533.65 sec
2021-11-30 17:19:06,604 Stage-1 map = 84%, reduce = 0%, Cumulative CPU 539.31 sec
2021-11-30 17:19:07,672 Stage-1 map = 85%, reduce = 0%, Cumulative CPU 545.17 sec
2021-11-30 17:19:12,929 Stage-1 map = 86%, reduce = 0%, Cumulative CPU 556.72 sec
2021-11-30 17:19:13,967 Stage-1 map = 87%, reduce = 0%, Cumulative CPU 562.17 sec
2021-11-30 17:19:15,002 Stage-1 map = 88%, reduce = 0%, Cumulative CPU 567.55 sec
2021-11-30 17:19:18,154 Stage-1 map = 89%, reduce = 0%, Cumulative CPU 573.05 sec
2021-11-30 17:19:20,258 Stage-1 map = 90%, reduce = 30%, Cumulative CPU 579.56 sec
2021-11-30 17:19:22,373 Stage-1 map = 91%, reduce = 30%, Cumulative CPU 585.18 sec
2021-11-30 17:19:25,495 Stage-1 map = 92%, reduce = 30%, Cumulative CPU 596.09 sec
2021-11-30 17:19:27,588 Stage-1 map = 93%, reduce = 30%, Cumulative CPU 602.05 sec
2021-11-30 17:19:29,687 Stage-1 map = 94%, reduce = 30%, Cumulative CPU 607.61 sec
2021-11-30 17:19:31,752 Stage-1 map = 95%, reduce = 30%, Cumulative CPU 613.03 sec
2021-11-30 17:19:32,786 Stage-1 map = 95%, reduce = 31%, Cumulative CPU 613.08 sec
2021-11-30 17:19:33,816 Stage-1 map = 96%, reduce = 31%, Cumulative CPU 618.77 sec
2021-11-30 17:19:34,875 Stage-1 map = 97%, reduce = 31%, Cumulative CPU 624.77 sec
2021-11-30 17:19:37,995 Stage-1 map = 97%, reduce = 32%, Cumulative CPU 630.36 sec
2021-11-30 17:19:40,040 Stage-1 map = 98%, reduce = 32%, Cumulative CPU 636.1 sec
2021-11-30 17:19:41,065 Stage-1 map = 100%, reduce = 32%, Cumulative CPU 647.59 sec
2021-11-30 17:19:43,109 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 649.37 sec
MapReduce Total cumulative CPU time: 10 minutes 49 seconds 370 msec
Ended Job = job_1638236643110_0021
MapReduce Jobs Launched:
Stage-Stage-1: Map: 117 Reduce: 1 Cumulative CPU: 649.37 sec HDFS Read: 31436910990 HDFS Write: 109 HDFS EC Read: 0 SUCCESS
Total MapReduce CPU Time Spent: 10 minutes 49 seconds 370 msec
OK
767830000
Time taken: 226.531 seconds, Fetched: 1 row(s)
hive>
> select count(*) from ods_fact_sale_new;
Query ID = root_20211130171959_562c8885-810e-4eb7-95f0-1b93806fc5e9
Total jobs = 1
Launching Job 1 out of 1
Number of reduce tasks determined at compile time: 1
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapreduce.job.reduces=<number>
21/11/30 17:20:00 INFO client.ConfiguredRMFailoverProxyProvider: Failing over to rm69
Starting Job = job_1638236643110_0022, Tracking URL = http://hp3:8088/proxy/application_1638236643110_0022/
Kill Command = /opt/cloudera/parcels/CDH-6.3.1-1.cdh6.3.1.p0.1470567/lib/hadoop/bin/hadoop job -kill job_1638236643110_0022
Hadoop job information for Stage-1: number of mappers: 117; number of reducers: 1
2021-11-30 17:20:07,057 Stage-1 map = 0%, reduce = 0%
2021-11-30 17:20:16,317 Stage-1 map = 2%, reduce = 0%, Cumulative CPU 12.12 sec
2021-11-30 17:20:17,346 Stage-1 map = 4%, reduce = 0%, Cumulative CPU 31.0 sec
2021-11-30 17:20:23,542 Stage-1 map = 6%, reduce = 0%, Cumulative CPU 42.9 sec
2021-11-30 17:20:24,580 Stage-1 map = 9%, reduce = 0%, Cumulative CPU 60.83 sec
2021-11-30 17:20:30,767 Stage-1 map = 10%, reduce = 0%, Cumulative CPU 72.7 sec
2021-11-30 17:20:31,787 Stage-1 map = 13%, reduce = 0%, Cumulative CPU 90.64 sec
2021-11-30 17:20:36,938 Stage-1 map = 15%, reduce = 0%, Cumulative CPU 102.56 sec
2021-11-30 17:20:38,974 Stage-1 map = 16%, reduce = 0%, Cumulative CPU 114.11 sec
2021-11-30 17:20:40,025 Stage-1 map = 17%, reduce = 0%, Cumulative CPU 120.16 sec
2021-11-30 17:20:44,160 Stage-1 map = 19%, reduce = 0%, Cumulative CPU 132.24 sec
2021-11-30 17:20:45,188 Stage-1 map = 21%, reduce = 0%, Cumulative CPU 144.21 sec
2021-11-30 17:20:51,353 Stage-1 map = 23%, reduce = 0%, Cumulative CPU 161.71 sec
2021-11-30 17:20:52,390 Stage-1 map = 24%, reduce = 0%, Cumulative CPU 167.54 sec
2021-11-30 17:20:53,428 Stage-1 map = 26%, reduce = 0%, Cumulative CPU 179.5 sec
2021-11-30 17:20:58,556 Stage-1 map = 27%, reduce = 0%, Cumulative CPU 191.23 sec
2021-11-30 17:20:59,574 Stage-1 map = 29%, reduce = 0%, Cumulative CPU 203.33 sec
2021-11-30 17:21:00,591 Stage-1 map = 30%, reduce = 0%, Cumulative CPU 209.26 sec
2021-11-30 17:21:03,692 Stage-1 map = 31%, reduce = 0%, Cumulative CPU 214.77 sec
2021-11-30 17:21:05,736 Stage-1 map = 32%, reduce = 0%, Cumulative CPU 220.48 sec
2021-11-30 17:21:06,757 Stage-1 map = 33%, reduce = 0%, Cumulative CPU 231.27 sec
2021-11-30 17:21:07,781 Stage-1 map = 34%, reduce = 0%, Cumulative CPU 237.48 sec
2021-11-30 17:21:10,844 Stage-1 map = 35%, reduce = 0%, Cumulative CPU 243.45 sec
2021-11-30 17:21:12,886 Stage-1 map = 36%, reduce = 0%, Cumulative CPU 249.22 sec
2021-11-30 17:21:13,902 Stage-1 map = 38%, reduce = 0%, Cumulative CPU 260.89 sec
2021-11-30 17:21:16,991 Stage-1 map = 39%, reduce = 0%, Cumulative CPU 272.39 sec
2021-11-30 17:21:20,085 Stage-1 map = 41%, reduce = 0%, Cumulative CPU 283.48 sec
2021-11-30 17:21:21,102 Stage-1 map = 42%, reduce = 0%, Cumulative CPU 289.43 sec
2021-11-30 17:21:22,119 Stage-1 map = 43%, reduce = 0%, Cumulative CPU 294.72 sec
2021-11-30 17:21:23,180 Stage-1 map = 44%, reduce = 0%, Cumulative CPU 300.83 sec
2021-11-30 17:21:27,310 Stage-1 map = 45%, reduce = 0%, Cumulative CPU 312.29 sec
2021-11-30 17:21:28,338 Stage-1 map = 46%, reduce = 0%, Cumulative CPU 318.05 sec
2021-11-30 17:21:29,354 Stage-1 map = 47%, reduce = 0%, Cumulative CPU 324.44 sec
2021-11-30 17:21:30,377 Stage-1 map = 48%, reduce = 0%, Cumulative CPU 330.22 sec
2021-11-30 17:21:33,463 Stage-1 map = 49%, reduce = 0%, Cumulative CPU 335.68 sec
2021-11-30 17:21:34,490 Stage-1 map = 50%, reduce = 0%, Cumulative CPU 341.08 sec
2021-11-30 17:21:36,521 Stage-1 map = 52%, reduce = 0%, Cumulative CPU 357.65 sec
2021-11-30 17:21:40,635 Stage-1 map = 53%, reduce = 0%, Cumulative CPU 363.49 sec
2021-11-30 17:21:41,651 Stage-1 map = 55%, reduce = 0%, Cumulative CPU 375.38 sec
2021-11-30 17:21:43,700 Stage-1 map = 56%, reduce = 0%, Cumulative CPU 387.34 sec
2021-11-30 17:21:47,844 Stage-1 map = 57%, reduce = 0%, Cumulative CPU 393.01 sec
2021-11-30 17:21:48,861 Stage-1 map = 59%, reduce = 0%, Cumulative CPU 403.26 sec
2021-11-30 17:21:50,900 Stage-1 map = 61%, reduce = 0%, Cumulative CPU 415.11 sec
2021-11-30 17:21:55,003 Stage-1 map = 62%, reduce = 0%, Cumulative CPU 425.91 sec
2021-11-30 17:21:56,018 Stage-1 map = 63%, reduce = 0%, Cumulative CPU 431.68 sec
2021-11-30 17:21:58,088 Stage-1 map = 65%, reduce = 0%, Cumulative CPU 444.05 sec
2021-11-30 17:22:02,221 Stage-1 map = 67%, reduce = 0%, Cumulative CPU 455.75 sec
2021-11-30 17:22:03,237 Stage-1 map = 68%, reduce = 0%, Cumulative CPU 461.62 sec
2021-11-30 17:22:05,279 Stage-1 map = 69%, reduce = 0%, Cumulative CPU 473.68 sec
2021-11-30 17:22:09,410 Stage-1 map = 71%, reduce = 0%, Cumulative CPU 485.65 sec
2021-11-30 17:22:10,443 Stage-1 map = 72%, reduce = 0%, Cumulative CPU 491.52 sec
2021-11-30 17:22:11,467 Stage-1 map = 73%, reduce = 0%, Cumulative CPU 496.87 sec
2021-11-30 17:22:12,484 Stage-1 map = 74%, reduce = 0%, Cumulative CPU 502.88 sec
2021-11-30 17:22:16,669 Stage-1 map = 76%, reduce = 0%, Cumulative CPU 519.71 sec
2021-11-30 17:22:18,722 Stage-1 map = 77%, reduce = 0%, Cumulative CPU 525.59 sec
2021-11-30 17:22:19,739 Stage-1 map = 78%, reduce = 0%, Cumulative CPU 531.27 sec
2021-11-30 17:22:23,809 Stage-1 map = 80%, reduce = 0%, Cumulative CPU 548.52 sec
2021-11-30 17:22:25,847 Stage-1 map = 81%, reduce = 0%, Cumulative CPU 554.38 sec
2021-11-30 17:22:26,884 Stage-1 map = 82%, reduce = 0%, Cumulative CPU 559.93 sec
2021-11-30 17:22:29,986 Stage-1 map = 83%, reduce = 0%, Cumulative CPU 566.03 sec
2021-11-30 17:22:31,009 Stage-1 map = 85%, reduce = 0%, Cumulative CPU 577.94 sec
2021-11-30 17:22:37,253 Stage-1 map = 87%, reduce = 0%, Cumulative CPU 595.02 sec
2021-11-30 17:22:38,271 Stage-1 map = 88%, reduce = 0%, Cumulative CPU 600.47 sec
2021-11-30 17:22:40,439 Stage-1 map = 89%, reduce = 29%, Cumulative CPU 607.17 sec
2021-11-30 17:22:44,547 Stage-1 map = 91%, reduce = 29%, Cumulative CPU 619.18 sec
2021-11-30 17:22:46,586 Stage-1 map = 91%, reduce = 30%, Cumulative CPU 625.24 sec
2021-11-30 17:22:47,605 Stage-1 map = 92%, reduce = 30%, Cumulative CPU 630.67 sec
2021-11-30 17:22:51,707 Stage-1 map = 93%, reduce = 30%, Cumulative CPU 636.5 sec
2021-11-30 17:22:52,757 Stage-1 map = 95%, reduce = 31%, Cumulative CPU 647.99 sec
2021-11-30 17:22:54,808 Stage-1 map = 96%, reduce = 31%, Cumulative CPU 653.5 sec
2021-11-30 17:22:57,912 Stage-1 map = 97%, reduce = 31%, Cumulative CPU 659.35 sec
2021-11-30 17:22:58,932 Stage-1 map = 97%, reduce = 32%, Cumulative CPU 665.3 sec
2021-11-30 17:22:59,950 Stage-1 map = 98%, reduce = 32%, Cumulative CPU 671.18 sec
2021-11-30 17:23:01,996 Stage-1 map = 99%, reduce = 32%, Cumulative CPU 677.33 sec
2021-11-30 17:23:05,054 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 683.3 sec
2021-11-30 17:23:06,076 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 685.07 sec
MapReduce Total cumulative CPU time: 11 minutes 25 seconds 70 msec
Ended Job = job_1638236643110_0022
MapReduce Jobs Launched:
Stage-Stage-1: Map: 117 Reduce: 1 Cumulative CPU: 685.07 sec HDFS Read: 31429476658 HDFS Write: 109 HDFS EC Read: 0 SUCCESS
Total MapReduce CPU Time Spent: 11 minutes 25 seconds 70 msec
OK
767830000
Time taken: 188.179 seconds, Fetched: 1 row(s)
hive>