SQL mapper文件的審計

SSM+MySQL已是Java開發的主流架構。在SSM+MySQL架構中,SQL一般寫在mapper映射文件中,對於mapper文件的審計一直是個頭疼的問題,測試人員很難覆蓋mapper文件中動態SQL的所有情況,開發人員更是無法羅列每一個select id對應的所有SQL。在和開發同事溝通後,覺得自己可以寫一個小工具來解決這一痛點!筆者作爲一名DBA,自學了golang,經過七個月的努力初步完成了myaudit這一款實用性極強的SQLAudit小工具的開發,歡迎使用和吐槽,項目地址:https://github.com/gaozhongzheng/myaudit
下面簡單介紹下myaudit這個小工具:
SQL mapper文件的審計

myaudit是一個審計SSM + MySQL開發架構中SQL映射文件(xml文件)的SQLAudit工具。

設計思路如下:

  • 1.格式化xml文件
  • 2.重寫xml文件
  • 3.構造Mybatis工程
  • 4.調起Mybatis工程打印所有可能的SQL到日誌
  • 5.解析日誌獲取SQL,並根據動態字段類型將SQL中的'?'替換爲具體值
  • 6.調用SOAR獲取審計報告
    爲了上首頁,上一段代碼:

    func ReplaceQuestionMark(query string, db *sql.DB, dbName string) string {
    stmt, err := sqlparser.Parse(query)
    if err != nil {
        common.LogIfError(err, "parse failed: "+query)
        return query
    }
    
    columns, tables, _, err := GetInfo(stmt)
    if err != nil {
        common.LogIfError(err, "get info failed: "+query)
        return query
    }
    
    var column_name string
    var data_type string
    
    column_type := make(map[string]string)
    
    for _, tab := range tables {
        rows, _ := db.Query("select column_name, data_type from columns where table_name = ? and TABLE_SCHEMA = '"+dbName+"'", tab)
        for rows.Next() {
            if err := rows.Scan(&column_name, &data_type); err == nil {
                if _, ok := column_type[column_name]; !ok {
                    column_type[column_name] = data_type
                }
            } else {
                fmt.Println(err)
            }
        }
        rows.Close()
    }
    
    i := 0
    for _, col := range columns {
    
        if column, ok := column_type[col[0]]; ok {
            i++
            delete(column_type, col[0])
            // fmt.Println("here is", query)
            question_index := GetIndex(query, "?")
            if question_index != nil {
                // fmt.Println("this is :", question_index)
                idx, _ := strconv.Atoi(col[1])
                // fmt.Println("idx is :", idx)
                index := question_index[idx-i]
                // fmt.Println("index is :", index)
                if strings.Contains(column, "char") || strings.Contains(column, "text") || strings.Contains(column, "lob") {
                    query = query[:index] + "'SQLAudit'" + query[index+1:]
                } else if strings.Contains(column, "int") {
                    query = query[:index] + "1" + query[index+1:]
                } else if strings.Contains(column, "decimal") || strings.Contains(column, "float") || strings.Contains(column, "double") || strings.Contains(column, "numeric") {
                    query = query[:index] + "1.1" + query[index+1:]
                } else if strings.Contains(column, "timestamp") || strings.Contains(column, "datatime") {
                    query = query[:index] + "'2019-01-01 00:00:00'" + query[index+1:]
                } else if strings.Contains(column, "date") {
                    query = query[:index] + "'2019-01-01'" + query[index+1:]
                } else if strings.Contains(column, "time") {
                    query = query[:index] + "'00:00:00'" + query[index+1:]
                } else if strings.Contains(column, "year") {
                    query = query[:index] + "'2019'" + query[index+1:]
                } else if strings.Contains(column, "enum") {
                    query = query[:index] + "'SQLAudit'" + query[index+1:]
                } else {
                    query = query[:index] + "none" + query[index+1:]
                }
    
            }
        }
    }
    
    stmt, err = sqlparser.Parse(query)
    if err != nil {
        common.LogIfError(err, "parse failed: "+query)
        return query
    }
    
    _, _, limits, err := GetInfo(stmt)
    if err != nil {
        common.LogIfError(err, "get info failed: "+query)
        return query
    }
    
    // 替換limit中的?
    for _, ques := range limits {
        question_index := GetIndex(query, "?")
        if ques[0] != -1 {
            index := question_index[ques[0]-1]
            query = query[:index] + "1" + query[index+1:]
        }
        if ques[1] != -1 {
            index := question_index[ques[1]-1]
            query = query[:index] + "1" + query[index+1:]
        }
    }
    
    // TODO group by和order by中的?替換。
    // 放棄,因爲group by和order by的對象是列而不是值,而列我們是不知道的,所以沒法替換
    // 打印信息
    // fmt.Println("columns:", columns)
    // fmt.Println("tables:", tables)
    // fmt.Println("limits:", limits)
    // fmt.Println("new query:", query)
    return query
    }

    特點:

  • 1.與業務代碼深度解耦
  • 2.解析並審計動態SQL所有可能的情況
  • 3.根據字段類型替換綁定變量值,解決了SQL指紋無法在MySQL中獲取執行計劃的問題

    安裝/部署(Linux)

    方式一

  • step1.克隆源代碼至本地
  • step2.編譯命令文件:cd cmd;go build myaudit.go
  • step3.執行createDir.sh創建相應目錄,目錄含義見目錄說明
  • step4.上傳log4j-1.2.17.jar、mybatis-3.2.8.jar、mysql-connector-java-5.1.47.jar至/usr/local/myaudit/lib,可從此處獲取jar包
  • step5.將step2中編譯好的命令文件拷貝至/usr/local/myaudit/bin並將此目錄添加至環境變量
    注:需先安裝SOAR

    方式二(免編譯快速部署)

  • step1.克隆deployment至本地
  • step2.將/usr/local/myaudit/bin添加至環境變量

    使用myaudit(如有疑問或建議,請郵件至[email protected]

    Usage of myaudit:
    -ftype       mapper|slowlog|normal, default mapper          --待審計文件種類,默認爲mapper(暫時只支持Mybatis項目中的mapper文件)
    -fname       file waitting for audit                      --待審計文件名,不需加擴展名,如MapperSakila.xml傳參時爲MapperSakila
    -conn        $IP:$PORT/$DB, like 127.0.0.1:3306/sakila    --測試環境連接串,待審計文件對應的測試環境數據庫
    -u           username    --MySQL用戶名
    -p           password    --MySQL用戶密碼
    -report-type html|json, default html    --報告類型,默認爲html
  • step1.上傳待審計的SQL映射文件至/usr/local/myaudit/file
  • step2.執行審計(示例):
    myaudit -ftype mapper -fname MapperSakila -conn 127.0.0.1:3306/sakila -u root -p 123456 -report-type html
    Dir '/usr/local/myaudit/tmp/MapperSakila' was created/rebuilt successfully!
    /usr/local/myaudit/file/MapperSakila.xml read ok!
    /usr/local/myaudit/tmp/MapperSakila/tmp.xml read ok!
    /usr/local/myaudit/tmp/MapperSakila/oldMapper.xml read ok!
    /usr/local/myaudit/tmp/MapperSakila/oldMapperFormated.xml read ok!
    Compile and execute java files successfully!
    /usr/local/myaudit/log/mapperLog/MapperSakila.log read ok!
    Database connect success!
    Dir '/usr/local/myaudit/sql/MapperSakila' was created/rebuilt successfully!
    Dir '/usr/local/myaudit/audit/MapperSakila' was created/rebuilt successfully!
    xml file was successfully audited, you can find the audit file 'getFilmInfoTest1_audit.html' in directory /usr/local/myaudit/audit/MapperSakila.
    xml file was successfully audited, you can find the audit file 'getFilmInfoTest2_audit.html' in directory /usr/local/myaudit/audit/MapperSakila.

    MapperSakila.xml解析出的SQL文件在此

    審計報告截圖:

    SQL mapper文件的審計

發表評論
所有評論
還沒有人評論,想成為第一個評論的人麼? 請在上方評論欄輸入並且點擊發布.
相關文章