Fastload (二) 轉

FASTLOAD的特性:
1)FASTLOAD腳本包含三個主要模塊:
建表模塊、Define模塊、Insert模塊。

2)FASTLOAD分爲兩個階段:
第1階段 - 從數據文本中讀取記錄,只需放到目標AMP上即可
第2階段 - 在AMP內部按Rowhash進行排序
3)FASTLOAD的目標表初始的時候只能是個空表。
4)FASTLOAD的目標表不能定義次索引,但可以定義分區字段。
5)FASTLOAD一次只能對一張表進行加載,不支持對多張表進行加載。
6)FASTLOAD支持多個數據文件一次性加入一張表中,但這些數據文件是串行加載並非並行加載。
7)FASTLOAD在加載的時候會自動剔除文本中的重複記錄,即使目標表定義爲multiset table。

典型的fastload腳本示例如下:

/* 限制錯誤記錄條數*/
ERRLIMIT 1;
/* 數據庫登錄語句*/
LOGON ETL_TD_STAGE,ETL_TD_STAGE;
/* 建表模塊*/
DATABASE TD_STAGE;
DROP TABLE AUCTION_PROPERTY;
DROP TABLE PLOG.AUCTION_PROPERTY_E1;
DROP TABLE PLOG.AUCTION_PROPERTY_E2;
CREATE MULTISET TABLE AUCTION_PROPERTY ,NO FALLBACK ,
NO BEFORE JOURNAL,
NO AFTER JOURNAL,
CHECKSUM = DEFAULT
(
AUCTION_ID CHAR(32) CHARACTER SET LATIN CASESPECIFIC,
PROPERTY_ID DECIMAL(11,0),
PROP_VID INTEGER,
STATUS INTEGER,
GMT_CREATE TIMESTAMP(0),
GMT_MODIFIED TIMESTAMP(0),
INS_DATE TIMESTAMP(0))
PRIMARY INDEX ( AUCTION_ID );
/* 方式一:設置加載方式爲變長,分割符爲Tab鍵*/
/* 同時,進入加載第1階段*/
.SET RECORD VARTEXT " " NOSTOP
BEGIN LOADING AUCTION_PROPERTY ERRORFILES PLOG.AUCTION_PROPERTY_E1,PLOG.AUCTION_PROPERTY_E2;
/* DEFINE模塊,定義數據文本格式和存放路徑*/
DEFINE
AUCTION_ID (VARCHAR(42),nullif='')
,PROPERTY_ID (VARCHAR(21),nullif='')
,PROP_VID (VARCHAR(20),nullif='')
,STATUS (VARCHAR(20),nullif='')
,GMT_CREATE (VARCHAR(25),nullif='')
,GMT_MODIFIED (VARCHAR(25),nullif='')
,INS_DATE (VARCHAR(25),nullif='')
FILE=/bak/Teradata/auction_property.txt;
/* 方式二:設置加載方式爲定長,分割符爲Tab鍵*/
DEFINE
AUCTION_ID (VARCHAR(42),nullif='')
,TAB1 (CHAR(1))
,PROPERTY_ID (VARCHAR(21),nullif='')
,TAB2 (CHAR(1))
,PROP_VID (VARCHAR(20),nullif='')
,TAB3 (CHAR(1))
,STATUS (VARCHAR(20),nullif='')
,TAB4 (CHAR(1))
,GMT_CREATE (VARCHAR(25),nullif='')
,TAB5 (CHAR(1))
,GMT_MODIFIED (VARCHAR(25),nullif='')
,TAB6 (CHAR(1))
,INS_DATE (VARCHAR(25),nullif='') 
,NEWLINE (CHAR(1))
FILE=/bak/Teradata/auction_property.txt;
/* 設置加載起點,也可以設置加載終點(THRU)*/
RECORD 1;
/* INSERT模塊,定義物理表對應的格式*/
INSERT INTO TD_STAGE.AUCTION_PROPERTY(
AUCTION_ID 
,PROPERTY_ID 
,PROP_VID 
,STATUS 
,GMT_CREATE 
,GMT_MODIFIED 
,INS_DATE 
)
VALUES (
:AUCTION_ID 
,:PROPERTY_ID 
,:PROP_VID 
,:STATUS 
,:GMT_CREATE (format 'yyyymmddhhmiss') 
,:GMT_MODIFIED (format 'yyyymmddhhmiss') 
,:INS_DATE (format 'yyyymmddhhmiss') 
);
/* 第1階段結束,進入加載第2階段*/
END LOADING;
/* 數據庫退出語句*/
LOGOFF;

典型的fastload日誌示例如下:
/* 第1階段,文本記錄無序錄入到AMP上*/
**** 19:44:59 Number of recs/msg: 3
**** 19:44:59 Starting to send to RDBMS with record 1
**** 19:45:08 Starting row 100000
**** 19:45:18 Starting row 200000
**** 19:45:28 Starting row 300000
**** 19:45:38 Starting row 400000
**** 19:45:48 Starting row 500000
**** 19:45:58 Starting row 600000
**** 19:46:08 Starting row 700000
**** 19:46:18 Starting row 800000
**** 19:46:28 Starting row 900000
**** 19:46:38 Starting row 1000000
**** 19:46:39 Sending row 1015760
**** 19:46:39 Finished sending rows to the RDBMS
/* 第2階段,AMP內部進行排序*/
===================================================================
= =
= End Loading Phase =
= =
===================================================================
0013 END LOADING;
**** 19:47:58 END LOADING COMPLETE
Total Records Read = 1015760
- skipped by RECORD command = 0
- sent to the RDBMS = 1015760
Total Error Table 1 = 0 ---- Table has been dropped
Total Error Table 2 = 0 ---- Table has been dropped
Total Inserts Applied = 1015760
Total Duplicate Rows = 0
Start: Thu Mar 01 19:46:40 2007
End : Thu Mar 01 19:47:58 2007

2.Using fastload
batch mode運行參數:
3.兩個錯誤日誌表區別:
errortname1 
• Constraint violations
• Conversion errors
• Unavailable AMP conditions
These types of errors always occur during the loading phase of your FastLoad job—after executing the BEGIN LOADING command, but
before the END LOADING command.
errortname2 
Unique primary index violations
This type of error always occurs during the end-loading phase of your FastLoad job—after executing the END LOADING command.


發表評論
所有評論
還沒有人評論,想成為第一個評論的人麼? 請在上方評論欄輸入並且點擊發布.
相關文章