java自定義hive sparksql thriftServer連接池

由於公司需要做hive和sparksql查詢功能，且都通過thrift server進行查詢，那麼如何有效的創建鏈接並緩存鏈接就是比較關鍵的步驟了。

thrift connection類似於jdbc connection只是引入的driver class不同罷了。由於公司集羣開了kerbrose，所以需要使用代理用戶進行通信，每一次登錄的用戶都會有一個clusterName(代理用戶的名稱，集羣名)。 所以每一個connection就可能都不一樣了。  這裏需要一個動態的連接池來保存已經創建和正在使用的鏈接，並需要有效的釋放鏈接的機制。

廢話不多說，直接上關鍵代碼：

1.基礎類  ObjectPool 

public abstract class ObjectPool {

    static Logger logger = org.slf4j.LoggerFactory.getLogger(ObjectPool.class);
    private static long expirationTime;

    private static HashMap<String, ConcurrentHashMap<String, ConnectionBean>> locked;
    private static HashMap<String, ConcurrentLinkedQueue<ConnectionBean>> unlocked;

    public ObjectPool() {
        if (locked == null) {
            locked = new HashMap<>();
        }
        if (unlocked == null) {
            unlocked = new HashMap<>();
        }
        expirationTime = 30 * 60 * 1000; // 30 minute 過期
    }

    protected abstract ConnectionBean create();

    public abstract boolean validate(ConnectionBean o);

    public abstract void expire(ConnectionBean o);

    public ConnectionBean get(String clusterName) {
        synchronized (locked) {
            String key = Thread.currentThread().getName() + clusterName;
            logger.info("【POOL】 lock the LOCKED map, the clusterName is {}", clusterName);
            long now = System.currentTimeMillis();
            ConcurrentLinkedQueue<ConnectionBean> beans;
            if (!unlocked.isEmpty()) {
                beans = unlocked.get(clusterName);
                if (beans != null) {
                    while (!beans.isEmpty()) {
                        // 獲取頭元素，並在資源隊列中刪除頭元素
                        ConnectionBean bean = beans.poll();
                        // 如果頭元素的時間過期了，那麼關閉連接
                        if (now - bean.getUpdateTime() > expirationTime) {
                            logger.info("【POOL】 the connection is out of time ,bean time is {}", bean.getUpdateTime());
                            // 釋放
                            expire(bean);
                            bean = null;
                        } else {
                            if (validate(bean)) {
                                logger.info("【POOL】 get the connection from poll and the clusterName is {}",
                                        clusterName);
                                bean.setUpdateTime(now);
                                // 放入鎖定的隊列中並返回 鎖定隊列需要
                                locked.get(clusterName).put(key, bean);
                                return bean;
                            } else {
                                // 如果鏈接已經關閉
                                unlocked.remove(clusterName);
                                expire(bean);
                                bean = null;
                            }
                        }
                    }
                }
            }
            // 由於unlock可能爲空，所以初始化對應的clusterName
            unlocked.put(clusterName, new ConcurrentLinkedQueue<ConnectionBean>());
            // 如果沒有鏈接則新建一個操作
            ConnectionBean bean = create();
            logger.info("【POOL】 the pool could not provide a connection, so create it,clusterName is {}",
                    clusterName);
            if (locked.get(clusterName) == null) {
                logger.info("【POOL】 the clusterName in pool is null, create a new Map in LOCKED, clusterName is {}",
                        clusterName);
                locked.put(clusterName, new ConcurrentHashMap<String, ConnectionBean>());
            }
            locked.get(clusterName).put(key, bean);
            return bean;

        }
    }


    public void release(String clusterName) {
        synchronized (locked) {
            String key = Thread.currentThread().getName() + clusterName;
            ConcurrentHashMap<String, ConnectionBean> connectionBeans = locked.get(clusterName);
            ConnectionBean bean = connectionBeans.get(key);
            connectionBeans.remove(key);
            bean.setUpdateTime(System.currentTimeMillis());
            unlocked.get(clusterName).add(bean);
            System.out.println("......................................................" + Thread.currentThread().getName());
        }
    }
}

在這個連接池中 包含了locked 和 unlocked 兩個連接隊列和concurrentHashMap（釋放鏈接主要爲了對應key的釋放，不使用隊列出列）。

使用get 獲取指定的connection，首先 如果連接池中有，且存活時間沒有 超過 expirationTime 默認30min 則會爲用戶使用，但是超過30min則會回收。當獲取到鏈接會，會和當前ThreadName以及集羣名稱作爲key值存儲至locked map中。

2.  ConnectionBeanPool 這個對象繼承了抽象類 並實現相應操作

public class ConnectionBeanPool extends ObjectPool {

    private String url;  // 鏈接url
    private String usr;  // 賬戶名
    private String pwd;  // 密碼

    public ConnectionBeanPool(String driver, String url, String usr, String pwd) {
        super();
        try {
            Class.forName(driver).newInstance();
        } catch (Exception e) {
            e.printStackTrace();
        }
        this.url = url;
        this.usr = usr;
        this.pwd = pwd;
    }

    @Override
    protected ConnectionBean create() {
        try {
            ConnectionBean connectionBean = new ConnectionBean();
            Connection connection = DriverManager.getConnection(url, usr, pwd);
            if (connection == null) {
                System.out.print("null connection");
            }
            connectionBean.setConnection(connection);
            connectionBean.setUpdateTime(new Date().getTime());
            return connectionBean;
        } catch (SQLException e) {
            e.printStackTrace();
            return null;
        }
    }

    @Override
    public void expire(ConnectionBean o) {
        try {
            o.getConnection().close();
        } catch (SQLException e) {
            e.printStackTrace();
        }
    }

    @Override
    public boolean validate(ConnectionBean o) {
        try {
            return (!(o.getConnection()).isClosed());
        } catch (SQLException e) {
            e.printStackTrace();
            return false;
        }
    }
}

3. bean

public class ConnectionBean {

    private Connection connection; // 鏈接信息
    private long updateTime;       // 更新時間

    public Connection getConnection() {
        return connection;
    }

    public void setConnection(Connection connection) {
        this.connection = connection;
    }

    public long getUpdateTime() {
        return updateTime;
    }

    public void setUpdateTime(long updateTime) {
        this.updateTime = updateTime;
    }
}

目前還沒對鏈接數量進行限定。未來需要設置一個最大連接量，如果超過則讓請求等待或者返回錯誤信息。