sql loader

轉自：http://wp0140502.javaeye.com/blog/580816

一：sql loader 的特點

　　oracle自己帶了很多的工具可以用來進行數據的遷移、備份和恢復等工作。但是每個工具都有自己的特點。比如說exp和imp可以對數據庫中的數據進行導出和導出的工作，是一種很好的數據庫備份和恢復的工具，因此主要用在數據庫的熱備份和恢復方面。有着速度快，使用簡單，快捷的優點；同時也有一些缺點，比如在不同版本數據庫之間的導出、導入的過程之中，總會出現這樣或者那樣的問題，這個也許是oracle公司自己產品的兼容性的問題吧。

　　sql loader 工具卻沒有這方面的問題，它可以把一些以文本格式存放的數據順利的導入到oracle數據庫中，是一種在不同數據庫之間進行數據遷移的非常方便而且通用的工具。缺點就速度比較慢，另外對blob等類型的數據就有點麻煩了。

二：sql loader 的幫助

命令行敲入“sqlldr”，可查看參數

有效的關鍵字:

　　userid -- ORACLE username/password
　　control -- Control file name
　　log -- Log file name
　　bad -- Bad file name
　　data -- Data file name
　　discard -- Discard file name
　　discardmax -- Number of discards to allow        (全部默認)
　　skip -- Number of logical records to skip (默認0)
　　load -- Number of logical records to load (全部默認)
　　errors -- Number of errors to allow          (默認50)
　　rows -- Number of rows in conventional path bind array or between direct path data saves
　　（默認: 常規路徑 64, 所有直接路徑）
　　bindsize -- Size of conventional path bind array in bytes(默認256000)
　　silent -- Suppress messages during run (header,feedback,errors,discards,partitions)
　　direct -- use direct path                    (默認FALSE)
　　parfile -- parameter file: name of file that contains parameter specifications
　　parallel -- do parallel load                   (默認FALSE)
　　file -- File to allocate extents from
　　skip_unusable_indexes -- disallow/allow unusable indexes or index partitions(默認FALSE)
　　skip_index_maintenance -- do not maintain indexes, mark affected indexes as unusable(默認FALSE)
　　readsize -- Size of Read buffer                (默認1048576)
　　external_table -- use external table for load; NOT_USED, GENERATE_ONLY, EXECUTE(默認NOT_USED)
　　columnarrayrows -- Number of rows for direct path column array(默認5000)
　　streamsize -- Size of direct path stream buffer in bytes(默認256000)
　　multithreading -- use multithreading in direct path
　　resumable -- enable or disable resumable for current session(默認FALSE)
　　resumable_name -- text string to help identify resumable statement
　　resumable_timeout -- wait time (in seconds) for RESUMABLE(默認7200)
　　date_cache -- size (in entries) of date conversion cache(默認1000)

　　PLEASE NOTE: 命令行參數可以由位置或關鍵字指定。前者的例子是 'sqlloadscott/tiger foo'; 後一種情況的一個示例是 'sqlldr

　　control=foouserid=scott/tiger'.位置指定參數的時間必須早於但不可遲於由關鍵字指定的參數。例如,允許 'sqlldr scott/tiger

　　control=foo logfile=log', 但是不允許 'sqlldr scott/tiger control=foo log', 即使參數 'log' 的位置正確。

    因爲業務需要，每天都要把1G左右的文本文件（大約450萬條數據）導入數據庫，而且最要命的是，現在纔開始做，要把過去好幾個月的文件數據導入，兩個月的就有60多個G，傳統的讀取文件然後拆分字段，再一條一條插入，完全不可行了，那簡直是噩夢。剛開始試過，200多萬的數據導入花了3個小時左右，這已經算是很不錯了，要是加了索引，加了一些其他限制，那將更慢。

三、ctl配置文件:ZBroadDetail.ctl

Sql代碼

OPTIONS (errors=100000000,skip=3566018)
LOAD DATA
INFILE "/home/ftp/ZBroadDetail.20091103"
append
INTO TABLE RADIUSDETAIL
Fields terminated by "::"
trailing nullcols
(
NET_ACCT "LTRIM(:NET_ACCT)",
START_TIME DATE "YYYYMMDD HH24MISS",
STOP_TIME DATE "YYYYMMDD HH24MISS",
FRAME_IP,
MAC_ADDRESS,
NAS_IP,
NAS_PORT,
NODE_ID,
INPUT_OCTETS,
OUTPUT_OCTETS,
INPUT_PACKETS,
OUTPUT_PACKETS
)

OPTIONS (errors=100000000,skip=3566018)

LOAD DATA

INFILE "/home/ftp/ZBroadDetail.20091103"

append

INTO TABLE RADIUSDETAIL

Fields terminated by "::"

trailing nullcols

(

NET_ACCT "LTRIM(:NET_ACCT)",

START_TIME DATE "YYYYMMDD HH24MISS",

STOP_TIME DATE "YYYYMMDD HH24MISS",

FRAME_IP,

MAC_ADDRESS,

NAS_IP,  

NAS_PORT,

NODE_ID,

INPUT_OCTETS,

OUTPUT_OCTETS,

INPUT_PACKETS,

OUTPUT_PACKETS

)

四、建表語句：

Sql代碼

-- Create table
create table RADIUSDETAIL
(
NET_ACCT VARCHAR2(22) not null,
START_TIME DATE not null,
STOP_TIME DATE not null,
FRAME_IP VARCHAR2(16),
MAC_ADDRESS VARCHAR2(40),
NAS_IP VARCHAR2(16) not null,
NAS_PORT VARCHAR2(16) not null,
NODE_ID VARCHAR2(10) not null,
INPUT_OCTETS NUMBER(10) not null,
OUTPUT_OCTETS NUMBER(10) not null,
INPUT_PACKETS NUMBER(10) not null,
OUTPUT_PACKETS NUMBER(10) not null
)
partition by range (START_TIME)
(
partition RADIUSDETAIL_200909 values less than (TO_DATE(' 2009-10-01 00:00:00', 'SYYYY-MM-DD HH24:MI:SS', 'NLS_CALENDAR=GREGORIAN'))
tablespace TS_RADIUS_200909
pctfree 10
initrans 1
maxtrans 255
storage
(
initial 64K
minextents 1
maxextents unlimited
),
partition RADIUSDETAIL_200910 values less than (TO_DATE(' 2009-11-01 00:00:00', 'SYYYY-MM-DD HH24:MI:SS', 'NLS_CALENDAR=GREGORIAN'))
tablespace TS_RADIUS_200910
pctfree 10
initrans 1
maxtrans 255
storage
(
initial 64K
minextents 1
maxextents unlimited
),
partition RADIUSDETAIL_200911 values less than (TO_DATE(' 2009-12-01 00:00:00', 'SYYYY-MM-DD HH24:MI:SS', 'NLS_CALENDAR=GREGORIAN'))
tablespace TS_RADIUS_200911
pctfree 10
initrans 1
maxtrans 255
storage
(
initial 64K
minextents 1
maxextents unlimited
),
partition RADIUSDETAIL_200912 values less than (TO_DATE(' 2010-01-01 00:00:00', 'SYYYY-MM-DD HH24:MI:SS', 'NLS_CALENDAR=GREGORIAN'))
tablespace TS_RADIUS_200912
pctfree 10
initrans 1
maxtrans 255
storage
(
initial 64K
minextents 1
maxextents unlimited
)
);
-- Create/Recreate indexes
create bitmap index BIT_NET_ACCT on RADIUSDETAIL (NET_ACCT);

-- Create table
create table RADIUSDETAIL
(
  NET_ACCT       VARCHAR2(22) not null,
  START_TIME     DATE not null,
  STOP_TIME      DATE not null,
  FRAME_IP       VARCHAR2(16),
  MAC_ADDRESS    VARCHAR2(40),
  NAS_IP         VARCHAR2(16) not null,
  NAS_PORT       VARCHAR2(16) not null,
  NODE_ID        VARCHAR2(10) not null,
  INPUT_OCTETS   NUMBER(10) not null,
  OUTPUT_OCTETS  NUMBER(10) not null,
  INPUT_PACKETS  NUMBER(10) not null,
  OUTPUT_PACKETS NUMBER(10) not null
)
partition by range (START_TIME)
(
  partition RADIUSDETAIL_200909 values less than (TO_DATE(' 2009-10-01 00:00:00', 'SYYYY-MM-DD HH24:MI:SS', 'NLS_CALENDAR=GREGORIAN'))
    tablespace TS_RADIUS_200909
    pctfree 10
    initrans 1
    maxtrans 255
    storage
    (
      initial 64K
      minextents 1
      maxextents unlimited
    ),
  partition RADIUSDETAIL_200910 values less than (TO_DATE(' 2009-11-01 00:00:00', 'SYYYY-MM-DD HH24:MI:SS', 'NLS_CALENDAR=GREGORIAN'))
    tablespace TS_RADIUS_200910
    pctfree 10
    initrans 1
    maxtrans 255
    storage
    (
      initial 64K
      minextents 1
      maxextents unlimited
    ),
  partition RADIUSDETAIL_200911 values less than (TO_DATE(' 2009-12-01 00:00:00', 'SYYYY-MM-DD HH24:MI:SS', 'NLS_CALENDAR=GREGORIAN'))
    tablespace TS_RADIUS_200911
    pctfree 10
    initrans 1
    maxtrans 255
    storage
    (
      initial 64K
      minextents 1
      maxextents unlimited
    ),
  partition RADIUSDETAIL_200912 values less than (TO_DATE(' 2010-01-01 00:00:00', 'SYYYY-MM-DD HH24:MI:SS', 'NLS_CALENDAR=GREGORIAN'))
    tablespace TS_RADIUS_200912
    pctfree 10
    initrans 1
    maxtrans 255
    storage
    (
      initial 64K
      minextents 1
      maxextents unlimited
    )
);
-- Create/Recreate indexes 
create bitmap index BIT_NET_ACCT on RADIUSDETAIL (NET_ACCT);

五、數據文件：

Html代碼

554chenhy1::20091102 215228::20091103 000000::114.101.219.208::00:e0:b0:eb:58:3a::61.190.207.16::318767313::0009::89658839::712520776::483771::647601
0558886954::20091102 205212::20091103 000000::117.67.74.67::00:1b:fc:e7:12:20::61.190.200.115::85985994::0014::892854637::127751439::1219981::521422
5552776874::20091102 201133::20091103 000000::114.102.37.7::00:24:e8:09:f9:74::61.190.209.34::18879689::0011::594629766::111886810::614080::295082
561shuang::20091102 204030::20091103 000000::117.57.101.140::00:e0:a0:0f:93:be::61.190.222.23::68158771::0013::61740866::142548707::267982::297281
ads2529670::20091102 225827::20091103 000000::61.190.178.49::00:14:78:49:8e:95::61.190.210.22::35677785::0007::54688807::22344225::71865::77718
05614084872::20091102 225628::20091103 000000::117.69.193.248::00:25:86:8d:a9:7d::61.190.222.3::1090522556::0013::64602789::225039189::331101::346673
561niu77::20091102 211033::20091103 000000::61.191.115.72::00:e0:4c:08:f2:19::61.190.222.17::119551483::0013::32124693::44826012::79114::84360
552g3185402::20091102 204703::20091103 000000::220.178.137.31::00:1d:60:75:55:05::61.190.197.14::587204619::0003::193906940::28375706::261873::261391
564la3386690::20091102 193042::20091103 000000::117.68.80.122::00:25:86:95:ed:7f::61.190.218.14::35661047::0006::16133795::18722492::49295::48160
la561nm8780::20091102 185300::20091103 000000::117.57.94.66::00:25:86:38:07:df::61.190.222.24::102770417::0013::267256413::441666832::667088::692132
5514491492::20091102 192433::20091103 000000::124.73.1.135::00:30:18:c0:da:f9::61.133.137.97::385877693::0001::1640961::13927897::23358::24829
tl5852400::20091102 143341::20091103 000000::220.179.210.208::00:25:56:13:0d:f7::61.190.214.13::68163238::0012::1402826332::146399442::1216008::1013590
05596740939::20091102 234445::20091103 000000::220.179.106.230::00:24:8c:ea:06:6e::61.190.210.8::18904236::000706::19357649::27115887::35378::40429
55475l0vh::20091102 231541::20091103 000000::60.175.72.130::00:1d:60:29:15:79::61.190.207.14::1358957163::0009::9155440::72286818::57366::65709
18956423361::20091102 212834::20091103 000000::114.107.193.11::00:21:85:1f:61:85::61.190.218.13::33572914::0006::229545990::2247512519::1785262::2111780
bl5227271::20091102 231433::20091103 000000::60.174.121.67::00:1d:60:af:64:f5::61.190.226.6::307737662::0017::11418629::45599106::38284::52599
xc2829984::20091102 094159::20091103 000000::124.112.199.24::00:26:18:50:5f:ac::61.190.212.27::69211251::0008::9466980::28853080::93675::91051
05613918815::20091102 203458::20091103 000000::117.57.132.99::00:17:31:19:0b:77::61.190.222.2::1090522559::0013::106679965::347836935::565336::455583
564sc8345115::20091102 184324::20091103 000000::114.104.121.17::00:14:78:e0:3d:9d::61.190.218.142::35656861::000605::19638379::378257763::199378::296845
565aeb662234::20091101 071019::20091103 000000::114.105.20.20::00:21:27:8e:ce:37::61.190.220.37::35657321::001603::3815128047::3083888650::8618842::10227795
05613052533::20091102 185506::20091103 000000::220.179.181.14::00:21:97:44:01:5d::61.190.222.17::119551343::0013::22912037::28588344::120217::124821
dza7013360::20091102 182120::20091103 000000::60.174.10.2::00:1d:92:d6:d9:35::61.190.216.17::52434445::001503::430876369::373789670::834249::569508

554chenhy1::20091102 215228::20091103 000000::114.101.219.208::00:e0:b0:eb:58:3a::61.190.207.16::318767313::0009::89658839::712520776::483771::647601                                                                                    
 0558886954::20091102 205212::20091103 000000::117.67.74.67::00:1b:fc:e7:12:20::61.190.200.115::85985994::0014::892854637::127751439::1219981::521422                                                                                     
 5552776874::20091102 201133::20091103 000000::114.102.37.7::00:24:e8:09:f9:74::61.190.209.34::18879689::0011::594629766::111886810::614080::295082                                                                                       
 561shuang::20091102 204030::20091103 000000::117.57.101.140::00:e0:a0:0f:93:be::61.190.222.23::68158771::0013::61740866::142548707::267982::297281                                                                                       
 ads2529670::20091102 225827::20091103 000000::61.190.178.49::00:14:78:49:8e:95::61.190.210.22::35677785::0007::54688807::22344225::71865::77718                                                                                          
 05614084872::20091102 225628::20091103 000000::117.69.193.248::00:25:86:8d:a9:7d::61.190.222.3::1090522556::0013::64602789::225039189::331101::346673                                                                                    
 561niu77::20091102 211033::20091103 000000::61.191.115.72::00:e0:4c:08:f2:19::61.190.222.17::119551483::0013::32124693::44826012::79114::84360                                                                                           
 552g3185402::20091102 204703::20091103 000000::220.178.137.31::00:1d:60:75:55:05::61.190.197.14::587204619::0003::193906940::28375706::261873::261391                                                                                    
 564la3386690::20091102 193042::20091103 000000::117.68.80.122::00:25:86:95:ed:7f::61.190.218.14::35661047::0006::16133795::18722492::49295::48160                                                                                        
 la561nm8780::20091102 185300::20091103 000000::117.57.94.66::00:25:86:38:07:df::61.190.222.24::102770417::0013::267256413::441666832::667088::692132                                                                                     
 5514491492::20091102 192433::20091103 000000::124.73.1.135::00:30:18:c0:da:f9::61.133.137.97::385877693::0001::1640961::13927897::23358::24829                                                                                           
 tl5852400::20091102 143341::20091103 000000::220.179.210.208::00:25:56:13:0d:f7::61.190.214.13::68163238::0012::1402826332::146399442::1216008::1013590                                                                                  
 05596740939::20091102 234445::20091103 000000::220.179.106.230::00:24:8c:ea:06:6e::61.190.210.8::18904236::000706::19357649::27115887::35378::40429                                                                                      
 55475l0vh::20091102 231541::20091103 000000::60.175.72.130::00:1d:60:29:15:79::61.190.207.14::1358957163::0009::9155440::72286818::57366::65709                                                                                          
 18956423361::20091102 212834::20091103 000000::114.107.193.11::00:21:85:1f:61:85::61.190.218.13::33572914::0006::229545990::2247512519::1785262::2111780                                                                                 
 bl5227271::20091102 231433::20091103 000000::60.174.121.67::00:1d:60:af:64:f5::61.190.226.6::307737662::0017::11418629::45599106::38284::52599                                                                                           
 xc2829984::20091102 094159::20091103 000000::124.112.199.24::00:26:18:50:5f:ac::61.190.212.27::69211251::0008::9466980::28853080::93675::91051                                                                                           
 05613918815::20091102 203458::20091103 000000::117.57.132.99::00:17:31:19:0b:77::61.190.222.2::1090522559::0013::106679965::347836935::565336::455583                                                                                    
 564sc8345115::20091102 184324::20091103 000000::114.104.121.17::00:14:78:e0:3d:9d::61.190.218.142::35656861::000605::19638379::378257763::199378::296845                                                                                         
 565aeb662234::20091101 071019::20091103 000000::114.105.20.20::00:21:27:8e:ce:37::61.190.220.37::35657321::001603::3815128047::3083888650::8618842::10227795                                                                             
 05613052533::20091102 185506::20091103 000000::220.179.181.14::00:21:97:44:01:5d::61.190.222.17::119551343::0013::22912037::28588344::120217::124821                                                                                     
 dza7013360::20091102 182120::20091103 000000::60.174.10.2::00:1d:92:d6:d9:35::61.190.216.17::52434445::001503::430876369::373789670::834249::569508

六、導入命令：

非並行加載：

Sql代碼

sqlldr scott/scott@XE control==/home/ftp/ZBroadDetail.ctl
log==/home/ftp/ZBroadDetail.log

sqlldr scott/scott@XE control==/home/ftp/ZBroadDetail.ctl
log==/home/ftp/ZBroadDetail.log

這種模式下配置文件裏可以設置rows數目，即一次提交量

並行加載（其實也就是不鎖表，可以同時打開多個命令行加載多個文件，向同一個表導入）

Sql代碼

sqlldr kdck/kdck@orcl control=/home/ftp/ZBroadDetail.ctl log=/home/ftp/ZBroadDetail.log parallel=true

sqlldr kdck/kdck@orcl control=/home/ftp/ZBroadDetail.ctl log=/home/ftp/ZBroadDetail.log parallel=true

當該模式下導入的時候，配置文件裏就不能指定rows
（如果沒有索引，可以加上 direct=y 這樣更快）
現在的話，在無索引的前提下，10分鐘可以導入2000萬條，算是非常高效了吧。

10分鐘搞定Mysql主從部署配置

如何使用 JS 判斷用戶是否處於活躍狀態

「Pygors跨平臺GUI」2：安裝MinGW-w64、MSYS2還是WSL2

[轉帖]

python列出centos7內存使用前50的進程信息

「Pygors跨平臺GUI」1：Pygors跨平臺GUI應用研究

一鍵自動化博客發佈工具,用過的人都說好(掘金篇)

lightdb數據庫超時相關控制參數

lightdb秒級增加列和刪除列（not null帶默認值）

Java ThreadPoolShutdown

ORACLE數據庫中創建、刪除--同義詞、視圖、索引、序列

判斷頁面文本框中輸入的類型（數字，符號，字母，漢字）！！

oracle IMP 命令詳解

Oracle查看死鎖

oracle 存儲過程的基本語法及注意事項

https://yachay.unat.edu.pe/blog/index.php?comment_area=format_blog&comment_component=blog&comment_co

linux以太網驅動總結