對於sqoop增量同步Oracle數據到hive的命令參數以及如何定製自動增量job的測試已經再前面幾篇文章詳細測試過了，連接：
1、sqoop避免輸入密碼自動增量job腳本介紹
這篇文章是基於上面連接的文章繼續做的拓展，上篇文章結尾說了如果一個表很大。我第一次初始化一部分最新的數據到hive表，如果沒初始化進來的歷史數據今天發生了變更，那merge-key的增量方式會不會報錯呢？之所以會提出這個問題，是因爲筆者真的有這個測試需求，接下來先對oracle端的庫表數據做下修改，來模擬這種場景。
第一步：先插入一條數據
當前時間爲：
SQL> select sysdate from dual;
SYSDATE
-------------------
2019-03-25 18:20:26
爲了模擬我是有一部分歷史數據沒有導入到hive表，我這裏先給oracle表插入一條歷史數據：
SQL> select * from inr_job;
EMPNO ENAME JOB SAL ETLTIME
---------- ---------- --------- ---------- -------------------
1 er CLERK 800 2019-03-22 17:24:42
2 ALLEN SALESMAN 1600 2019-03-22 17:24:42
3 WARD SALESMAN 1250 2019-03-22 17:24:42
4 JONES MANAGER 2975 2019-03-22 17:24:42
5 MARTIN SALESMAN 1250 2019-03-22 17:24:42
6 zhao DBA 1000 2019-03-22 17:24:42
7 yan BI 100 2019-03-22 17:24:42
8 dong JAVA 400 2019-03-22 17:24:42
8 rows selected.
SQL> insert into inr_job values(9,'test','test',200,sysdate-20);
1 row created.
SQL> commit;
Commit complete.
SQL> select * from inr_job;
EMPNO ENAME JOB SAL ETLTIME
---------- ---------- --------- ---------- -------------------
1 er CLERK 800 2019-03-22 17:24:42
2 ALLEN SALESMAN 1600 2019-03-22 17:24:42
3 WARD SALESMAN 1250 2019-03-22 17:24:42
4 JONES MANAGER 2975 2019-03-22 17:24:42
5 MARTIN SALESMAN 1250 2019-03-22 17:24:42
6 zhao DBA 1000 2019-03-22 17:24:42
7 yan BI 100 2019-03-22 17:24:42
8 dong JAVA 400 2019-03-22 17:24:42
9 test test 200 2019-03-05 18:53:23--模仿沒初始化到hive表的his數據
9 rows selected.
接下來手動更新一下這個歷史數據：
SQL> update inr_job set sal=999,etltime=sysdate where empno=9;
1 row updated.
SQL> commit;
Commit complete.
SQL> select * from inr_job;
EMPNO ENAME JOB SAL ETLTIME
---------- ---------- --------- ---------- -------------------
1 er CLERK 800 2019-03-22 17:24:42
2 ALLEN SALESMAN 1600 2019-03-22 17:24:42
3 WARD SALESMAN 1250 2019-03-22 17:24:42
4 JONES MANAGER 2975 2019-03-22 17:24:42
5 MARTIN SALESMAN 1250 2019-03-22 17:24:42
6 zhao DBA 1000 2019-03-22 17:24:42
7 yan BI 100 2019-03-22 17:24:42
8 dong JAVA 400 2019-03-22 17:24:42
9 test test 999 2019-03-25 18:54:39
9 rows selected.
現在數據發生了變動，然後去執行一下增量腳本：
[root@hadoop hadoop]# sqoop job --exec auto_job
Warning: /hadoop/sqoop/../accumulo does not exist! Accumulo imports will fail.
Please set $ACCUMULO_HOME to the root of your Accumulo installation.
19/03/25 18:55:49 INFO sqoop.Sqoop: Running Sqoop version: 1.4.7
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/hadoop/hbase/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/hadoop/hive/lib/log4j-slf4j-impl-2.6.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
19/03/25 18:55:51 INFO oracle.OraOopManagerFactory: Data Connector for Oracle and Hadoop is disabled.
19/03/25 18:55:51 INFO manager.SqlManager: Using default fetchSize of 1000
19/03/25 18:55:51 INFO tool.CodeGenTool: Beginning code generation
19/03/25 18:55:52 INFO manager.OracleManager: Time zone has been set to GMT
19/03/25 18:55:52 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM INR_JOB t WHERE 1=0
19/03/25 18:55:52 INFO orm.CompilationManager: HADOOP_MAPRED_HOME is /hadoop
Note: /tmp/sqoop-root/compile/f64e34273a58459369885b96fe46a1ad/INR_JOB.java uses or overrides a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
19/03/25 18:55:56 INFO orm.CompilationManager: Writing jar file: /tmp/sqoop-root/compile/f64e34273a58459369885b96fe46a1ad/INR_JOB.jar
19/03/25 18:55:56 INFO manager.OracleManager: Time zone has been set to GMT
19/03/25 18:55:56 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM INR_JOB t WHERE 1=0
19/03/25 18:55:56 INFO tool.ImportTool: Incremental import based on column ETLTIME
19/03/25 18:55:56 INFO tool.ImportTool: Lower bound value: TO_TIMESTAMP('2019-03-25 18:50:07.0', 'YYYY-MM-DD HH24:MI:SS.FF')
19/03/25 18:55:56 INFO tool.ImportTool: Upper bound value: TO_TIMESTAMP('2019-03-25 18:55:56.0', 'YYYY-MM-DD HH24:MI:SS.FF')
19/03/25 18:55:56 INFO manager.OracleManager: Time zone has been set to GMT
19/03/25 18:55:56 INFO mapreduce.ImportJobBase: Beginning import of INR_JOB
19/03/25 18:55:56 INFO Configuration.deprecation: mapred.jar is deprecated. Instead, use mapreduce.job.jar
19/03/25 18:55:56 INFO manager.OracleManager: Time zone has been set to GMT
19/03/25 18:55:56 INFO Configuration.deprecation: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps
19/03/25 18:55:56 INFO client.RMProxy: Connecting to ResourceManager at /192.168.1.66:8032
19/03/25 18:55:59 INFO db.DBInputFormat: Using read commited transaction isolation
19/03/25 18:55:59 INFO mapreduce.JobSubmitter: number of splits:1
19/03/25 18:56:00 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1553503985304_0013
19/03/25 18:56:00 INFO impl.YarnClientImpl: Submitted application application_1553503985304_0013
19/03/25 18:56:00 INFO mapreduce.Job: The url to track the job: http://hadoop:8088/proxy/application_1553503985304_0013/
19/03/25 18:56:00 INFO mapreduce.Job: Running job: job_1553503985304_0013
19/03/25 18:56:10 INFO mapreduce.Job: Job job_1553503985304_0013 running in uber mode : false
19/03/25 18:56:10 INFO mapreduce.Job: map 0% reduce 0%
19/03/25 18:56:19 INFO mapreduce.Job: map 100% reduce 0%
19/03/25 18:56:20 INFO mapreduce.Job: Job job_1553503985304_0013 completed successfully
19/03/25 18:56:20 INFO mapreduce.Job: Counters: 30
File System Counters
FILE: Number of bytes read=0
FILE: Number of bytes written=144777
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=87
HDFS: Number of bytes written=38
HDFS: Number of read operations=4
HDFS: Number of large read operations=0
HDFS: Number of write operations=2
Job Counters
Launched map tasks=1
Other local map tasks=1
Total time spent by all maps in occupied slots (ms)=5870
Total time spent by all reduces in occupied slots (ms)=0
Total time spent by all map tasks (ms)=5870
Total vcore-milliseconds taken by all map tasks=5870
Total megabyte-milliseconds taken by all map tasks=6010880
Map-Reduce Framework
Map input records=1
Map output records=1
Input split bytes=87
Spilled Records=0
Failed Shuffles=0
Merged Map outputs=0
GC time elapsed (ms)=100
CPU time spent (ms)=3220
Physical memory (bytes) snapshot=189059072
Virtual memory (bytes) snapshot=2147303424
Total committed heap usage (bytes)=102236160
File Input Format Counters
Bytes Read=0
File Output Format Counters
Bytes Written=38
19/03/25 18:56:20 INFO mapreduce.ImportJobBase: Transferred 38 bytes in 23.7426 seconds (1.6005 bytes/sec)
19/03/25 18:56:20 INFO mapreduce.ImportJobBase: Retrieved 1 records.
19/03/25 18:56:20 INFO tool.ImportTool: Final destination exists, will run merge job.
19/03/25 18:56:20 INFO Configuration.deprecation: mapred.output.key.class is deprecated. Instead, use mapreduce.job.output.key.class
19/03/25 18:56:20 INFO client.RMProxy: Connecting to ResourceManager at /192.168.1.66:8032
19/03/25 18:56:22 INFO input.FileInputFormat: Total input paths to process : 2
19/03/25 18:56:23 INFO mapreduce.JobSubmitter: number of splits:2
19/03/25 18:56:23 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1553503985304_0014
19/03/25 18:56:23 INFO impl.YarnClientImpl: Submitted application application_1553503985304_0014
19/03/25 18:56:23 INFO mapreduce.Job: The url to track the job: http://hadoop:8088/proxy/application_1553503985304_0014/
19/03/25 18:56:23 INFO mapreduce.Job: Running job: job_1553503985304_0014
19/03/25 18:56:37 INFO mapreduce.Job: Job job_1553503985304_0014 running in uber mode : false
19/03/25 18:56:37 INFO mapreduce.Job: map 0% reduce 0%
19/03/25 18:56:46 INFO mapreduce.Job: map 100% reduce 0%
19/03/25 18:56:56 INFO mapreduce.Job: map 100% reduce 100%
19/03/25 18:56:57 INFO mapreduce.Job: Job job_1553503985304_0014 completed successfully
19/03/25 18:56:57 INFO mapreduce.Job: Counters: 49
File System Counters
FILE: Number of bytes read=614
FILE: Number of bytes written=435819
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=657
HDFS: Number of bytes written=361
HDFS: Number of read operations=9
HDFS: Number of large read operations=0
HDFS: Number of write operations=2
Job Counters
Launched map tasks=2
Launched reduce tasks=1
Data-local map tasks=2
Total time spent by all maps in occupied slots (ms)=11103
Total time spent by all reduces in occupied slots (ms)=7376
Total time spent by all map tasks (ms)=11103
Total time spent by all reduce tasks (ms)=7376
Total vcore-milliseconds taken by all map tasks=11103
Total vcore-milliseconds taken by all reduce tasks=7376
Total megabyte-milliseconds taken by all map tasks=11369472
Total megabyte-milliseconds taken by all reduce tasks=7553024
Map-Reduce Framework
Map input records=9
Map output records=9
Map output bytes=590
Map output materialized bytes=620
Input split bytes=296
Combine input records=0
Combine output records=0
Reduce input groups=9
Reduce shuffle bytes=620
Reduce input records=9
Reduce output records=9
Spilled Records=18
Shuffled Maps =2
Failed Shuffles=0
Merged Map outputs=2
GC time elapsed (ms)=263
CPU time spent (ms)=3980
Physical memory (bytes) snapshot=670138368
Virtual memory (bytes) snapshot=6394978304
Total committed heap usage (bytes)=508559360
Shuffle Errors
BAD_ID=0
CONNECTION=0
IO_ERROR=0
WRONG_LENGTH=0
WRONG_MAP=0
WRONG_REDUCE=0
File Input Format Counters
Bytes Read=361
File Output Format Counters
Bytes Written=361
19/03/25 18:56:57 INFO tool.ImportTool: Saving incremental import state to the metastore
19/03/25 18:56:57 INFO tool.ImportTool: Updated data for job: auto_job
發現沒有報錯唉，然後去看看hive表:
hive> select * from inr_job;
OK
1 er CLERK 800.0 2019-03-22 17:24:42.0
2 ALLEN SALESMAN 1600.0 2019-03-22 17:24:42.0
3 WARD SALESMAN 1250.0 2019-03-22 17:24:42.0
4 JONES MANAGER 2975.0 2019-03-22 17:24:42.0
5 MARTIN SALESMAN 1250.0 2019-03-22 17:24:42.0
6 zhao DBA 1000.0 2019-03-22 17:24:42.0
7 yan BI 100.0 2019-03-22 17:24:42.0
8 dong JAVA 400.0 2019-03-22 17:24:42.0
9 test test 999.0 2019-03-25 18:54:39.0
Time taken: 0.336 seconds, Fetched: 9 row(s)
沒初始化進來的歷史數據在近期變動之後，如果符合增量條件的話，也會append進來並不會報錯，完全符合筆者需求，其實看看merge-key參數大致原理，也是知道這樣是可行的，畢竟是通過主鍵和最後修改時間去做增量合併。
