摘要:當數據庫出問題時能降級從本地緩存的數據中查詢數據, CoralCache就是這樣一個提高微服務可用性的中間件。
背景
有些場景下,微服務依賴數據庫中一些配置項或者數量很少的數據,但當數據庫本身有問題時候,即使數據量很少,這個服務是不能正常工作;因此需要考慮一種能支持全量+極少變更的全局數據的場景,當數據庫出問題時能降級從本地緩存的數據中查詢數據,CoralCache就是這樣一個提高微服務可用性的中間件。
架構
CoralCache中間件架構如下圖所示,通過@EnableLocal註解開啓功能,應用啓動後將配置的表數據一次性加載到內存中,內存中的數據邏輯結構和數據庫中的邏輯結構一樣。
圖1. 架構圖
表達式計算引擎
內存查詢引擎的原理是數據庫查詢降級發生後,Intercepter將攔截到的原始SQL傳入查詢引擎中,查詢引擎解析SQL後得到表名、列名、where條件表達式,遍歷InnerDB中對應表的數據行,並通過表達式計算引擎計算結果,計算結果爲真則添加到結果集中最後返回給調用方。
計算引擎結構如下圖所示,將where條件表達式轉爲後綴表達式後依次遍歷後綴表達式,遇到操作數直接入棧,遇到操作符則根據操作符需要的操作數個數彈棧。
圖2. 表達式計算引擎結構
然後根據操作符和彈出的操作數進行計算,不同操作符對應不同的計算方法,並將計算後的結果重新作爲操作數入棧執到遍歷完成,核心計算流程代碼如下所示:
public Object calc(Expression where, InnerTable table, InnerRow row) {
try {
postTraversal(where);
} catch (Exception e) {
log.warn("calc error: {}", e.getMessage());
return false;
}
for (ExprObj obj : exprList) {
switch (obj.exprType()) {
case ITEM:
stack.push(obj);
break;
case BINARY_OP: {
ExprObj result = calcBinaryOperation(((ExprOperation) obj).getOperationType(), table, row);
stack.push(result);
break;
}
case UNARY_OP: {
ExprObj result = calcSingleOperation(((ExprOperation) obj).getOperationType(), table, row);
stack.push(result);
break;
}
case FUNCTION_OP: {
ExprObj result = calcFunctionOperation(((ExprOperation) obj).getOperationType(), table, row);
stack.push(result);
break;
}
default:
break;
}
}
return stack.pop();
}
常見運算符的實現
邏輯運算
邏輯常見運算符爲<、<=、>、>=、=等,它們的共性都是需要2個操作數並且返回值是布爾類型。
public ExprItem logicalCalculus(InnerTable table, InnerRow row, LogicalOperation logicalOperation) {
ExprObj second = stack.pop();
ExprObj first = stack.pop();
ExprItem result = new ExprItem();
result.setItemType(ItemType.T_CONST_OBJ);
Obj firstObj = getObj((ExprItem) first, table, row);
Obj secondObj = getObj((ExprItem) second, table, row);
boolean value = logicalOperation.apply(firstObj, secondObj);
result.setValue(new Obj(value, ObjType.BOOL));
return result;
}
例子,以"="的實現來展示:
private ExprObj calcBinaryOperation(OperationType type, InnerTable table, InnerRow row) {
ExprObj result = null;
switch (type) {
case T_OP_EQ:
result = logicalCalculus(table, row, (a, b) -> ObjUtil.eq(a, b)); // 等於符號的實現
break;
...
default:
break;
}
return result;
}
public class ObjUtil {
private static ObjType resultType(ObjType first, ObjType second) {
return ObjType.RESULT_TYPE[first.ordinal()][second.ordinal()];
}
public static boolean eq(Obj first, Obj second) {
ObjType type = resultType(first.getType(), second.getType());
switch (type) {
case LONG: {
long firstValue = first.getValueAsLong();
long secondValue = second.getValueAsLong();
return firstValue == secondValue;
}
case DOUBLE: {
double firstValue = first.getValueAsDouble();
double secondValue = second.getValueAsDouble();
return Double.compare(firstValue, secondValue) == 0;
}
case TIMESTAMP: {
java.util.Date firstValue = first.getValueAsDate();
java.util.Date secondValue = first.getValueAsDate();
return firstValue.compareTo(secondValue) == 0;
}
...
default:
break;
}
throw new UnsupportedOperationException(first.getType() + " and " + second.getType() + " not support '=' operation.");
}
}
數學運算
數學運算和邏輯運算的流程都一樣,只不過運算後的結果爲數字類型。
LIKE運算符
除了上面說的邏輯運算和數學運算外,還支持進行模糊匹配的特殊操作符LIKE。
LIKE表達式語法
常見用法如下
LIKE "%HUAWEI" 匹配以HUAWEI結尾的字符串
LIKE "HUAWEI%" 匹配以HUAWEI開頭的字符串
LIKE "A_B" 匹配以"A"起頭且以"Z"爲結尾的字串
LIKE "A?B" 同上
LIKE "%[0-9]%" 匹配含有數字的字符串
LIKE "%[a-z]%" 匹配含有小寫字母字符串
LIKE "%[!0-9]%"匹配不含數字的字符串
?和_都表示單個字符
JAVA中實現LIKE的方案:將LIKE的模式轉爲JAVA中的正則表達式。
LIKE詞法定義
expr := wild-card + expr
| wild-char + expr
| escape + expr
| string + expr
| ""
wild-card := %
wild-char := _
escape := [%|_]
string := [^%_]+ (One or > more characters that are not wild-card or wild-char)
定義Token類
public abstract class Token {
private final String value;
public Token(String value) {
this.value = value;
}
public abstract String convert();
public String getValue() {
return value;
}
}
public class ConstantToken extends Token {
public ConstantToken(String value) {
super(value);
}
@Override
public String convert() {
return getValue();
}
}
public class EscapeToken extends Token {
public EscapeToken(String value) {
super(value);
}
@Override
public String convert() {
return getValue();
}
}
public class StringToken extends Token {
public StringToken(String value) {
super(value);
}
@Override
public String convert() {
return Pattern.quote(getValue());
}
}
public class WildcardToken extends Token {
public WildcardToken(String value) {
super(value);
}
@Override
public String convert() {
return ".*";
}
}
public class WildcharToken extends Token {
public WildcharToken(String value) {
super(value);
}
@Override
public String convert() {
return ".";
}
}
創建Lexer(Tokenizer)
public class Tokenizer {
private Collection<Tuple> patterns = new LinkedList<>();
public <T extends Token> Tokenizer add(String regex, Function<String, Token> creator) {
this.patterns.add(new Tuple<Pattern, Function<String, Token>>(Pattern.compile(regex), creator));
return this;
}
public Collection<Token> tokenize(String clause) throws RuntimeException {
Collection<Token> tokens = new ArrayList<>();
String copy = String.copyValueOf(clause.toCharArray());
int position = 0;
while (!copy.equals("")) {
boolean found = false;
for (Tuple tuple : this.patterns) {
Pattern pattern = (Pattern) tuple.getFirst();
Matcher m = pattern.matcher(copy);
if (m.find()) {
found = true;
String token = m.group(1);
Function<String, Token> fn = (Function<String, Token>) tuple.getSecond();
tokens.add(fn.apply(token));
copy = m.replaceFirst("");
position += token.length();
break;
}
}
if (!found) {
throw new RuntimeException("Unexpected sequence found in input string, at " + position);
}
}
return tokens;
}
}
創建LIKE到正則表達式的轉換映射
public class LikeTranspiler {
private static Tokenizer TOKENIZER = new Tokenizer()
.add("^(\\[[^]]*])", ConstantToken::new)
.add("^(%)", WildcardToken::new)
.add("^(_)", WildcharToken::new)
.add("^([^\\[\\]%_]+)", StringToken::new);
public static String toRegEx(String pattern) throws ParseException {
StringBuilder sb = new StringBuilder().append("^");
for (Token token : TOKENIZER.tokenize(pattern)) {
sb.append(token.convert());
}
return sb.append("$").toString();
}
}
直接調用LikeTranspiler的toRegEx方法將LIKE語法轉爲JAVA中的正則表達式。
private ExprObj calcBinaryOperation(OperationType type, InnerTable table, InnerRow row) {
ExprObj result = null;
switch (type) {
. . .
case T_OP_LIKE:
result = logicalCalculus(table, row, (a, b) -> ObjUtil.like(a, b));
break;
. . .
}
return result;
}
public static boolean like(Obj first, Obj second) {
Assert.state(first.getType() == ObjType.STRING, OperationType.T_OP_LIKE + " only support STRING.");
Assert.state(second.getType() == ObjType.STRING, OperationType.T_OP_LIKE + " only support STRING.");
String firstValue = (String) first.getRelValue();
String secondValue = (String) second.getRelValue();
String regEx = LikeTranspiler.toRegEx(secondValue);
return Pattern.compile(regEx).matcher(firstValue).matches();
}
通過創建詞法分析器並使用此方法進行轉換,我們可以防止LIKE像這樣的子句被轉換爲正則表達式%abc[%]%,該子句應將其中的任何子字符串與其中的子字符串匹配,該子句將與子字符串或匹配任何字符串。abc%.abc[.].abc.abc。
類型計算轉換
不同數據類型在進行計算時需要轉型,具體的轉化入下二維數組中。
// 不同類型計算後的類型
ObjType[][] RESULT_TYPE = {
//UNKNOWN BYTE SHORT INT LONG FLOAT DOUBLE DECIMAL BOOL DATE TIME TIMESTAMP STRING NULL
{ UNKNOWN, UNKNOWN, UNKNOWN, UNKNOWN, UNKNOWN, UNKNOWN, UNKNOWN, UNKNOWN, UNKNOWN, UNKNOWN, UNKNOWN, UNKNOWN, UNKNOWN, UNKNOWN },// UNKNOWN
{ UNKNOWN, LONG, LONG, LONG, LONG, DOUBLE, DOUBLE, DECIMAL, BOOL, UNKNOWN, UNKNOWN, UNKNOWN, LONG, UNKNOWN },// BYTE
{ UNKNOWN, LONG, LONG, LONG, LONG, DOUBLE, DOUBLE, DECIMAL, BOOL, UNKNOWN, UNKNOWN, UNKNOWN, LONG, UNKNOWN },// SHORT
{ UNKNOWN, LONG, LONG, LONG, LONG, DOUBLE, DOUBLE, DECIMAL, BOOL, UNKNOWN, UNKNOWN, UNKNOWN, LONG, UNKNOWN },// INT
{ UNKNOWN, LONG, LONG, LONG, LONG, DOUBLE, DOUBLE, DECIMAL, BOOL, UNKNOWN, UNKNOWN, UNKNOWN, LONG, UNKNOWN },// LONG
{ UNKNOWN, DOUBLE, DOUBLE, DOUBLE, DOUBLE, DOUBLE, DOUBLE, DECIMAL, BOOL, UNKNOWN, UNKNOWN, UNKNOWN, DOUBLE, UNKNOWN },// FLOAT
{ UNKNOWN, DOUBLE, DOUBLE, DOUBLE, DOUBLE, DOUBLE, DOUBLE, DECIMAL, BOOL, UNKNOWN, UNKNOWN, UNKNOWN, DOUBLE, UNKNOWN },// DOUBLE
{ UNKNOWN, DECIMAL, DECIMAL, DECIMAL, DECIMAL, DECIMAL, DECIMAL, DECIMAL, UNKNOWN, UNKNOWN, UNKNOWN, UNKNOWN, DECIMAL, UNKNOWN },// DECIMAL
{ UNKNOWN, BOOL, BOOL, BOOL, BOOL, BOOL, BOOL, BOOL, BOOL, UNKNOWN, UNKNOWN, UNKNOWN, BOOL, UNKNOWN },// BOOL
{ UNKNOWN, UNKNOWN, UNKNOWN, UNKNOWN, UNKNOWN, UNKNOWN, UNKNOWN, UNKNOWN, UNKNOWN, TIMESTAMP, TIMESTAMP, TIMESTAMP, TIMESTAMP, UNKNOWN },// DATE
{ UNKNOWN, UNKNOWN, UNKNOWN, UNKNOWN, UNKNOWN, UNKNOWN, UNKNOWN, UNKNOWN, UNKNOWN, TIMESTAMP, TIMESTAMP, TIMESTAMP, TIMESTAMP, UNKNOWN },// TIME
{ UNKNOWN, UNKNOWN, UNKNOWN, UNKNOWN, UNKNOWN, UNKNOWN, UNKNOWN, UNKNOWN, UNKNOWN, TIMESTAMP, TIMESTAMP, TIMESTAMP, TIMESTAMP, UNKNOWN },// TIMESTAMP
{ UNKNOWN, LONG, LONG, LONG, LONG, DOUBLE, DOUBLE, DECIMAL, BOOL, TIMESTAMP, TIMESTAMP, TIMESTAMP, STRING, UNKNOWN },// STRING
{ UNKNOWN, UNKNOWN, UNKNOWN, UNKNOWN, UNKNOWN, UNKNOWN, UNKNOWN, UNKNOWN, UNKNOWN, UNKNOWN, UNKNOWN, UNKNOWN, UNKNOWN, UNKNOWN },// NULL
};
參考資料
[1] https://codereview.stackexchange.com/questions/36861/convert-sql-like-to-regex/207486
本文分享自華爲雲社區《微服務緩存中間件CoralCache表達式計算引擎詳解》,原文作者:超純的小白兔 。