起因
最近在閱讀數據庫連接池相關的書籍,書中有一小節提到了Statement
與PreparedStatement
的區別,並指出使用PreparedStatement會對SQL進行預編譯,並將預編譯的SQL存儲下來,下次直接使用,提高效率。與之相對的是Statement
,它需要頻繁進行編譯,從這個角度而言,PreparedStatement會比Statement更快,但事實真的是這樣的嗎?書中並沒有對此二者進行深入闡述,只是以餵食般給出了上述結論。筆者一直對PreparedStatement的工作原理比較模糊,它是怎麼進行的預編譯,印象中有說是在客戶端進行的編譯,也有說在數據庫端進行的編譯。因此,爲了搞清楚PreparedStatement的工作原理,查閱相關資料,並將學習結果記錄下來並做分享
本文將圍繞下述兩個問題進行展開:
- PreparedStatement是真的更快嗎?
- 預編譯真的發生了嗎?
- 如果真的發生了預編譯,是在客戶端還是在數據庫端發生?
案例探索
爲了下文更好地表述,事先約定一下環境與術語
- 數據庫指的是關係型數據庫,在本文中特指MySQL
- 客戶端是相對於數據庫而言的說法,因爲對於數據庫端而言,任何連接它的應用程序都可以稱爲數據庫客戶端,包括Java程序,在本文中指的是Java代碼,或者是JDBC-MySQL驅動程序(mysql-connector-java)
- MySQL數據庫版本: 5.6.39 社區版
- mysql-connector-java 版本: 5.1.47
- JDK版本: Oracle 1.8.0_192
如果要做性能測試,出於JVM的工作機制,有經驗的選手一般會考慮到"代碼預熱",一般做法是在main
方法中for循環(如10萬次)調用一個方法使一段代碼"熱"起來,達到JIT編譯門檻,使這段"熱"代碼編譯成爲機器碼,以達到最高的執行效率。但這種預熱法對於測量的結果沒有太大指導意義,正確的姿勢應該是採用JMH(Java Microbenchmark Harness)。
具體可參考R大(RednaxelaFX)在知乎的回答,傳送門
因此,本文的案例也會採用JMH進行測試
- 測試Statement的查詢性能,JMH測試代碼如下:
@State(Scope.Thread)
public class StatementTest {
Connection connection;
Statement statement;
@Setup
public void init() throws Exception {
String url = "jdbc:mysql:///test";
connection = DriverManager.getConnection(url, "root", "xxx");
statement = connection.createStatement();
}
@TearDown
public void close() throws Exception {
if (statement != null) {
statement.close();
}
if (connection != null) {
connection.close();
}
}
@Benchmark
// 預熱2次,每次10秒
@Warmup(iterations = 2, time = 10)
// 預熱完成後測量3次,每次10秒
@Measurement(iterations = 3, time = 10)
public ResultSet m() throws SQLException {
return statement.executeQuery("select * from foo where id = 1");
}
public static void main(String[] args) throws Exception {
Options opt = new OptionsBuilder()
.forks(2) // 測兩輪
.build();
new Runner(opt).run();
}
}
上面這段JMH代碼大概的含義是: 一共測兩輪,每輪一開始會預熱2次(每次10秒),接着開始3次正式開始測量(每次10秒),測量的是statement.executeQuery("select * from foo where id = 1");
的性能
。測量結果如下:
# Run progress: 0.00% complete, ETA 00:01:40
# Fork: 1 of 2
# Warmup Iteration 1: 6383.500 ops/s
# Warmup Iteration 2: 9198.667 ops/s
Iteration 1: 9853.260 ops/s
Iteration 2: 9484.938 ops/s
Iteration 3: 7762.880 ops/s
# Run progress: 50.00% complete, ETA 00:00:52
# Fork: 2 of 2
# Warmup Iteration 1: 8876.019 ops/s
# Warmup Iteration 2: 8821.313 ops/s
Iteration 1: 9678.910 ops/s
Iteration 2: 9688.882 ops/s
Iteration 3: 9805.789 ops/s
Result "com.example.demo.StatementTest.m":
9379.110 ±(99.9%) 2248.990 ops/s [Average]
(min, avg, max) = (7762.880, 9379.110, 9853.260), stdev = 802.012
CI (99.9%): [7130.120, 11628.100] (assumes normal distribution)
# Run complete. Total time: 00:01:43
Benchmark Mode Cnt Score Error Units
StatementTest.m thrpt 6 9379.110 ± 2248.990 ops/s
Process finished with exit code 0
- 測試一般情況下,PrepareStatement的查詢性能(大多數日常開發的姿勢)
@State(Scope.Thread)
public class PrepareStatementTest {
Connection connection;
PreparedStatement preparedStatement;
@Setup
public void init() throws Exception {
// 只變更如下鏈接
String url = "jdbc:mysql:///test";
connection = DriverManager.getConnection(url, "root", "xxx");
// 由Statement變成PreparedStatement
preparedStatement = connection.prepareStatement("select * from foo where id = ?");
}
@TearDown
public void close() throws Exception {
if (preparedStatement != null) {
preparedStatement.close();
}
if (connection != null) {
connection.close();
}
}
@Benchmark
@Warmup(iterations = 2, time = 10)
@Measurement(iterations = 3, time = 10)
public ResultSet m() throws SQLException {
preparedStatement.setLong(1, 1);
return preparedStatement.executeQuery();
}
public static void main(String[] args) throws Exception {
Options opt = new OptionsBuilder()
.forks(2)
.build();
new Runner(opt).run();
}
}
這段JMH測試代碼中,改變的地方只是從Statement變成了PreparedStatement,別的不變,JMH測量結果如下:
# Run progress: 0.00% complete, ETA 00:01:40
# Fork: 1 of 2
# Warmup Iteration 1: 8779.791 ops/s
# Warmup Iteration 2: 9724.843 ops/s
Iteration 1: 9320.048 ops/s
Iteration 2: 8324.464 ops/s
Iteration 3: 10007.290 ops/s
# Run progress: 50.00% complete, ETA 00:00:51
# Fork: 2 of 2
# Warmup Iteration 1: 9068.252 ops/s
# Warmup Iteration 2: 9750.551 ops/s
Iteration 1: 9485.021 ops/s
Iteration 2: 9450.123 ops/s
Iteration 3: 9780.915 ops/s
Result "com.example.demo.PrepareStatementTest.m":
9394.643 ±(99.9%) 1628.668 ops/s [Average]
(min, avg, max) = (8324.464, 9394.643, 10007.290), stdev = 580.799
CI (99.9%): [7765.975, 11023.312] (assumes normal distribution)
# Run complete. Total time: 00:01:42
Benchmark Mode Cnt Score Error Units
PrepareStatementTest.m thrpt 6 9394.643 ± 1628.668 ops/s
Process finished with exit code 0
- 在PreparedStatement的JMH測試代碼中,URL參數添加
useServerPrepStmts=true
,即jdbc:mysql:///test?useServerPrepStmts=true,其它不變,JMH測試結果如下:
# Run progress: 0.00% complete, ETA 00:01:40
# Fork: 1 of 2
# Warmup Iteration 1: 9659.413 ops/s
# Warmup Iteration 2: 9296.215 ops/s
Iteration 1: 8734.479 ops/s
Iteration 2: 9609.639 ops/s
Iteration 3: 9683.444 ops/s
# Run progress: 50.00% complete, ETA 00:00:51
# Fork: 2 of 2
# Warmup Iteration 1: 10151.456 ops/s
# Warmup Iteration 2: 10179.692 ops/s
Iteration 1: 9989.025 ops/s
Iteration 2: 10158.410 ops/s
Iteration 3: 10662.958 ops/s
Result "com.example.demo.PrepareStatementTest.m":
9806.326 ±(99.9%) 1814.637 ops/s [Average]
(min, avg, max) = (8734.479, 9806.326, 10662.958), stdev = 647.117
CI (99.9%): [7991.689, 11620.963] (assumes normal distribution)
# Run complete. Total time: 00:01:43
Benchmark Mode Cnt Score Error Units
PrepareStatementTest.m thrpt 6 9806.326 ± 1814.637 ops/s
Process finished with exit code 0
- 在PreparedStatement的JMH測試代碼中,URL參數添加
cachePrepStmts=true
,即jdbc:mysql:///test?cachePrepStmts=true,其它不變,JMH測試結果如下:
# Run progress: 0.00% complete, ETA 00:01:40
# Fork: 1 of 2
# Warmup Iteration 1: 9725.255 ops/s
# Warmup Iteration 2: 10058.296 ops/s
Iteration 1: 10081.576 ops/s
Iteration 2: 10064.490 ops/s
Iteration 3: 10185.449 ops/s
# Run progress: 50.00% complete, ETA 00:00:52
# Fork: 2 of 2
# Warmup Iteration 1: 9757.321 ops/s
# Warmup Iteration 2: 10143.745 ops/s
Iteration 1: 10142.830 ops/s
Iteration 2: 10127.477 ops/s
Iteration 3: 10113.163 ops/s
Result "com.example.demo.PrepareStatementTest.m":
10119.164 ±(99.9%) 121.980 ops/s [Average]
(min, avg, max) = (10064.490, 10119.164, 10185.449), stdev = 43.499
CI (99.9%): [9997.184, 10241.144] (assumes normal distribution)
# Run complete. Total time: 00:01:43
Benchmark Mode Cnt Score Error Units
PrepareStatementTest.m thrpt 6 10119.164 ± 121.980 ops/s
Process finished with exit code 0
- 在PreparedStatement的JMH測試代碼中,URL參數添加
useServerPrepStmts=true&cachePrepStmts=true
,即jdbc:mysql:///test?useServerPrepStmts=true&cachePrepStmts=true,其它不變,JMH測試結果如下:
# Run progress: 0.00% complete, ETA 00:01:40
# Fork: 1 of 2
# Warmup Iteration 1: 10303.083 ops/s
# Warmup Iteration 2: 10785.386 ops/s
Iteration 1: 10780.442 ops/s
Iteration 2: 10755.745 ops/s
Iteration 3: 10794.132 ops/s
# Run progress: 50.00% complete, ETA 00:00:51
# Fork: 2 of 2
# Warmup Iteration 1: 10416.622 ops/s
# Warmup Iteration 2: 10658.629 ops/s
Iteration 1: 10657.114 ops/s
Iteration 2: 10706.986 ops/s
Iteration 3: 10646.870 ops/s
Result "com.example.demo.PrepareStatementTest.m":
10723.548 ±(99.9%) 176.565 ops/s [Average]
(min, avg, max) = (10646.870, 10723.548, 10794.132), stdev = 62.965
CI (99.9%): [10546.983, 10900.114] (assumes normal distribution)
# Run complete. Total time: 00:01:43
Benchmark Mode Cnt Score Error Units
PrepareStatementTest.m thrpt 6 10723.548 ± 176.565 ops/s
Process finished with exit code 0
把五次測試結果歸納一下:
Statement: 9379.110
PreparedStatement: 9394.643
PreparedStatement useServerPrepStmts: 9806.326
PreparedStatement cachePrepStmts: 10119.164
PreparedStatement useServerPrepStmts cachePrepStmts: 10723.548
可以看到,本案例中使用Statement與PreparedStatement(不添加useServerPrepStmts、cachePrepStmts),吞吐量並無太大差別
PreparedStatement是真的更快嗎?
答:不一定,至少在本案例中,二者相差無幾,除去誤差,基本認爲二者是一致的。在大多數項目中,與關係型數據庫打交道都會直接或間接使用到PreparedStatement,但很少會在連接參數中添加useServerPrepStmts與cachePrepStmts,此時,在效率上,與Statement並無二致
useServerPrepStmts、cachePrepStmts這兩個參數非常重要,是解答後兩個問題的關鍵。
先說useServerPrepStmts,這個參數是讓數據庫端支持prepared statements,即預編譯。也就是說,如果連接參數中沒有添加這個屬性,數據庫端壓根就不會進行預編譯。摘抄MySQL官網兩段話:
Changes in MySQL Connector/J 3.1.0: Added useServerPrepStmts property (default false). The driver will use server-side prepared statements when the server version supports them (4.1 and newer) when this property is set to true. It is currently set to false by default until all bind/fetch functionality has been implemented. Currently only DML prepared statements are implemented for 4.1 server-side prepared statements.
Upgrading from MySQL Connector/J 3.0 to 3.1: Server-side Prepared Statements: Connector/J 3.1 will automatically detect and use server-side prepared statements when they are available (MySQL server version 4.1.0 and newer).
這裏有兩個關鍵信息:
- MySQL 4.1+才支持數據庫端的預編譯,之前的版本並不支持
- 客戶端(mysql-connector-java)版本必須>= 3.1.0
如果客戶端版本>= 3.1.0,且數據庫版本>=4.1.0,那麼客戶端與數據庫端連接時會自動開啓數據庫端的預編譯
但是,客戶端版本自5.0.5之後,客戶端與數據庫端連接時不再自動開啓數據庫端的預編譯
Changes in MySQL Connector/J 5.0.5: Important change: Due to a number of issues with the use of server-side prepared statements, Connector/J 5.0.5 has disabled their use by default. The disabling of server-side prepared statements does not affect the operation of the connector in any way.
To enable server-side prepared statements, add the following configuration property to your connector string: useServerPrepStmts=true
敲重點:因爲數據庫端預編譯太多BUG,mysql-connector-java>=5.0.5不再自動開啓數據庫端預編譯,如果非要開啓,可以在連接參數中添加useServerPrepStmts=true
屬性。
我們使用的客戶端版本爲5.1.47,在JMH中添加了useServerPrepStmts=true開啓數據庫端的預編譯,較未開啓之前性能提升4.4%(9394.643->9806.326)。
預編譯真的發生了嗎? 答: 在未添加useServerPrepStmts=true屬性之前,數據庫端的預編譯並沒有發生,添加之後,開啓了數據庫端的預編譯能力
如果真的發生了預編譯,是在客戶端還是在數據庫端發生?答: 未添加useServerPrepStmts=true屬性之前,是在客戶端進行了預編譯,數據庫端沒有;而添加屬性之後,數據庫端也開啓了預編譯
源碼分析
接下來看一下獲取連接時(DriverManager.getConnection(url, user, password);
)對於useServerPrepStmts屬性的處理
// com.mysql.jdbc.ConnectionImpl#initializePropsFromServer
private void initializePropsFromServer() throws SQLException {
// ...(省略)
//
// Users can turn off detection of server-side prepared statements
// getUseServerPreparedStmts() 用於檢測客戶端版本是否>=3.1.0,以及連接是否配置useServerPrepStmts=true
// versionMeetsMinimum(4, 1, 0)要求數據庫端版本 >= 4.1.0
if (getUseServerPreparedStmts() && versionMeetsMinimum(4, 1, 0)) {
// 此屬性爲true,纔會開啓數據庫端預編譯
this.useServerPreparedStmts = true;
if (versionMeetsMinimum(5, 0, 0) && !versionMeetsMinimum(5, 0, 3)) {
// 5.0.0 <= MySQL數據庫端版本 < 5.0.3,也不支持(或許是有BUG)
this.useServerPreparedStmts = false; // 4.1.2+ style prepared
// statements
// don't work on these versions
}
}
// ...(省略)
}
public boolean getUseServerPreparedStmts() {
return this.detectServerPreparedStmts.getValueAsBoolean();
}
// Think really long and hard about changing the default for this many, many applications have come to be acustomed to the latency profile of preparing stuff client-side, rather than prepare (round-trip), execute (round-trip), close (round-trip).
// 如果沒有設置useServerPrepStmts屬性,默認值爲false
// 自3.1.0版本開始
private BooleanConnectionProperty detectServerPreparedStmts = new BooleanConnectionProperty("useServerPrepStmts", false,
Messages.getString("ConnectionProperties.useServerPrepStmts"), "3.1.0", MISC_CATEGORY, Integer.MIN_VALUE);
從源碼可以看出,開啓數據庫端預編譯需要同時滿足以下條件
- MySQL客戶端版本 >=3.1.0
- MySQL服務端版本 >=4.1,且版本號不能是[5.0.0, 5.0.3)
- 設置連接屬性useServerPrepStmts = true
基本上與官網介紹是吻合,至於版本號不能是[5.0.0, 5.0.3),這點並沒有在官網看到
接着通過Connection創建PreparedStatement
PreparedStatement preparedStatement = connection.prepareStatement("select * from foo where id = ?");
// com.mysql.jdbc.ConnectionImpl#prepareStatement(java.lang.String, int, int)
public java.sql.PreparedStatement prepareStatement(String sql, int resultSetType, int resultSetConcurrency) throws SQLException {
synchronized (getConnectionMutex()) {
checkClosed();
//
// FIXME: Create warnings if can't create results of the given type or concurrency
//
PreparedStatement pStmt = null;
boolean canServerPrepare = true;
String nativeSql = getProcessEscapeCodesForPrepStmts() ? nativeSQL(sql) : sql;
// useServerPreparedStmts賦值過程上面已經分析
// getEmulateUnsupportedPstmts(): 如果驅動檢測到服務端不支持預編譯,是否要啓用客戶端的預編譯來代替,默認是true
if (this.useServerPreparedStmts && getEmulateUnsupportedPstmts()) {
// 根據SQL去判斷服務端是否支持預編譯,因爲有的SQL例如調用存儲過程的命令`call`,`create table`是不支持預編譯的,因此需要將canServerPrepare屬性置爲false
canServerPrepare = canHandleAsServerPreparedStatement(nativeSql);
}
if (this.useServerPreparedStmts && canServerPrepare) {
// 進入此分支代表啓用數據庫端的預編譯
// cachePrepStmts參數值如果爲true則代表需要將PrepStmts緩存起來,默認是false
if (this.getCachePreparedStatements()) {
// cachePrepStmts = true
synchronized (this.serverSideStatementCache) {
pStmt = this.serverSideStatementCache.remove(new CompoundCacheKey(this.database, sql));
if (pStmt != null) {
((com.mysql.jdbc.ServerPreparedStatement) pStmt).setClosed(false);
pStmt.clearParameters();
}
if (pStmt == null) {
try {
pStmt = ServerPreparedStatement.getInstance(getMultiHostSafeProxy(), nativeSql, this.database, resultSetType,
resultSetConcurrency);
if (sql.length() < getPreparedStatementCacheSqlLimit()) {
((com.mysql.jdbc.ServerPreparedStatement) pStmt).isCached = true;
}
pStmt.setResultSetType(resultSetType);
pStmt.setResultSetConcurrency(resultSetConcurrency);
} catch (SQLException sqlEx) {
// Punt, if necessary
if (getEmulateUnsupportedPstmts()) {
pStmt = (PreparedStatement) clientPrepareStatement(nativeSql, resultSetType, resultSetConcurrency, false);
if (sql.length() < getPreparedStatementCacheSqlLimit()) {
this.serverSideStatementCheckCache.put(sql, Boolean.FALSE);
}
} else {
throw sqlEx;
}
}
}
}
} else {
// cachePrepStmts = false
try {
pStmt = ServerPreparedStatement.getInstance(getMultiHostSafeProxy(), nativeSql, this.database, resultSetType, resultSetConcurrency);
pStmt.setResultSetType(resultSetType);
pStmt.setResultSetConcurrency(resultSetConcurrency);
} catch (SQLException sqlEx) {
// Punt, if necessary
if (getEmulateUnsupportedPstmts()) {
pStmt = (PreparedStatement) clientPrepareStatement(nativeSql, resultSetType, resultSetConcurrency, false);
} else {
throw sqlEx;
}
}
}
} else {
// 使用客戶端的預編譯
// 客戶端的預編譯也可以開啓緩存功能(cachePrepStmts)
pStmt = (PreparedStatement) clientPrepareStatement(nativeSql, resultSetType, resultSetConcurrency, false);
}
return pStmt;
}
}
客戶端PreparedStatement的實現類: com.mysql.jdbc.JDBC42PreparedStatement
服務端PreparedStatement的實現類: com.mysql.jdbc.JDBC42ServerPreparedStatement
cachePrepStmts=true是開啓PreparedStatement的緩存功能,該緩存由MySQL驅動實現,同時支持客戶端的PreparedStatement與服務端PreparedStatement,但對兩端的緩存實現方式不一樣,一是緩存的時機不同,二是緩存的value不同
- 客戶端的緩存,是在創建客戶端PrepareStatement的時候進行緩存的,緩存以nativeSql爲key,ParseInfo爲value
public java.sql.PreparedStatement clientPrepareStatement(String sql, int resultSetType, int resultSetConcurrency, boolean processEscapeCodesIfNeeded)
throws SQLException {
checkClosed();
String nativeSql = processEscapeCodesIfNeeded && getProcessEscapeCodesForPrepStmts() ? nativeSQL(sql) : sql;
PreparedStatement pStmt = null;
if (getCachePreparedStatements()) {
// 開啓緩存
PreparedStatement.ParseInfo pStmtInfo = this.cachedPreparedStatementParams.get(nativeSql);
if (pStmtInfo == null) {
pStmt = com.mysql.jdbc.PreparedStatement.getInstance(getMultiHostSafeProxy(), nativeSql, this.database);
// nativeSql爲key,ParseInfo爲value
this.cachedPreparedStatementParams.put(nativeSql, pStmt.getParseInfo());
} else {
pStmt = com.mysql.jdbc.PreparedStatement.getInstance(getMultiHostSafeProxy(), nativeSql, this.database, pStmtInfo);
}
} else {
pStmt = com.mysql.jdbc.PreparedStatement.getInstance(getMultiHostSafeProxy(), nativeSql, this.database);
}
pStmt.setResultSetType(resultSetType);
pStmt.setResultSetConcurrency(resultSetConcurrency);
return pStmt;
}
- 服務端的緩存,是在com.mysql.jdbc.ServerPreparedStatement#close時進行緩存的,以CompoundCacheKey(封裝了catalog與originalSql)爲key,pstmt爲value
// com.mysql.jdbc.ServerPreparedStatement#close
public void close() throws SQLException {
MySQLConnection locallyScopedConn = this.connection;
if (locallyScopedConn == null) {
return; // already closed
}
synchronized (locallyScopedConn.getConnectionMutex()) {
if (this.isCached && isPoolable() && !this.isClosed) {
clearParameters();
this.isClosed = true;
// 緩存
this.connection.recachePreparedStatement(this);
return;
}
this.isClosed = false;
realClose(true, true);
}
}
public void recachePreparedStatement(ServerPreparedStatement pstmt) throws SQLException {
synchronized (getConnectionMutex()) {
if (getCachePreparedStatements() && pstmt.isPoolable()) {
synchronized (this.serverSideStatementCache) {
// 以CompoundCacheKey爲key,pstmt爲value
Object oldServerPrepStmt = this.serverSideStatementCache.put(new CompoundCacheKey(pstmt.currentCatalog, pstmt.originalSql), pstmt);
if (oldServerPrepStmt != null && oldServerPrepStmt != pstmt) {
((ServerPreparedStatement) oldServerPrepStmt).isCached = false;
((ServerPreparedStatement) oldServerPrepStmt).setClosed(false);
((ServerPreparedStatement) oldServerPrepStmt).realClose(true, true);
}
}
}
}
}
總結
本文對PreparedStatement在MySQL下的工作機制進行了探究,並使用JMH編寫性能測試代碼,得出了它並不比Statement更快的結論,並且對"預編譯"的概念進行了闡述,概念包含兩個方面:客戶端的預編譯與數據庫端的預編譯
由於大多數據情況下數據庫連接參數中並不會配置useServerPrepStmts = true,此時應用程序工作在客戶端的預編譯模式下,性能與Statement相比未有明顯提高,儘管開啓服務端預編譯能提升吞吐量,但該方式存在過多的BUG,在生產環境中仍然不建議開啓,避免採坑
從MySQL官網以及互聯網上幾乎找不到關於cachePrepStmts的BUG,暫且可以粗略認爲該功能BUG較少,或者影響可以忽略不計,因此可以嘗試配置cachePrepStmts=true,即開啓PreparedStatement的緩存,達到提升吞吐量的目的
在不配置useServerPrepStmts 、cachePrepStmts的情況下,PreparedStatement並不比Statement更快,是否意味着可以使用Statement代替PreparedStatement?實則不然,因爲PreparedStatement還有一個非常重要的特性是Statement所不具備的: 防止SQL注入
PreparedStatement是如何防止SQL注入呢?且聽下回分解