mysql 批量插入問題

場景：需要實時對mysql一張表中插入大量數據

優化方案：

1.數據庫連接池

2.針對http請求，進行多線程訪問，和寫入

3.批量插入

要點：

a.對於mysql 數據庫配置數據庫url:

jdbc:mysql://ip:3306/table_name?useUnicode=true&characterEncoding=UTF8&connectTimeout=1000&rewriteBatchedStatements=true&socketTimeout=2000

b.preparedStatement批量插入

import com.mchange.v2.c3p0.ComboPooledDataSource;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.beans.factory.annotation.Qualifier;
import org.springframework.stereotype.Repository;
import org.springframework.stereotype.Service;

import javax.sql.DataSource;
import java.sql.*;
import java.util.ArrayList;

@Repository("name")
public class Impl implements IDAO {

    @Autowired
    private DataSource dataSource;


    @Override
    public int batchInsert(ArrayList<RecordDetailDO> list) {

        String sql = "insert into table (record_name, amount, consume_id, created_time) values (?, ?, ?, sysDate())";
        Connection connection = null;
        PreparedStatement ps = null;
        try {
            connection = dataSource.getConnection();
            connection.setAutoCommit(false);
            ps = connection.prepareStatement(sql);

            for (RecordDetailDO recordDetailDO: list) {
                ps.setString(1, recordDetailDO.getRecordNameEnum().toString());
                ps.setInt(2, recordDetailDO.getRefId());
                ps.setBigDecimal(3, recordDetailDO.getAmount());
                ps.addBatch();
            }
            int[] rtl = ps.executeBatch();
            int ok = 0;
            for(int i = 0; i < rtl.length; i++){
                if(rtl[i] != 1){
                    logger.info("consumeRecordDetailBatchInsert有遺漏:" + list.get(i));
                } else {
                    ok++;
                }
            }
            connection.commit();
            connection.setAutoCommit(true);
            return ok;
        } catch (SQLException e) {
            e.printStackTrace();
        }
        return 0;
    }
}

批量插入上千次以上，效果更佳

要點：

http://brian.pontarelli.com/2011/06/21/jdbc-batch-vs-multi-row-inserts/

http://stackoverflow.com/questions/3784197/efficient-way-to-do-batch-inserts-with-jdbc