compass指南針搜索框架學習(二)

一、準備

    個人在學習中採用Struts2 + Hibernate3.2 + Spring2.5 + Compass2.2.0， 一下圖片爲本次學習中用到的jar包：

 

圖中圈出的jar包爲本次學習的主要部分，另外用綠色框圈出的jar包爲分詞器，主要用來做實驗看分詞效果的，選用一個即可。

二、什麼是Compass

    Compass是一個Java搜索框架。它封裝了Lucene，增加了一些Lucene不支持的特性（例如實時更新索引），支持各種數據（Java對象、xml、json）到索引的映射，支持各種數據源（JDBC, Hibernate, iBatis）

 

 

圖解:

• Compass - 一般在程序啓動時建立並被整個程序共享，主要用於建立CompassSession並通過其管理索引數據。與Hibernate的SessionFactory類似
• CompassSession - 用於處理數據的session。與Hibernate的Session類似
• CompassTransaction - 手動進行事務管理，如果不使用，Compass會自動管理事務。與Hibenate的事物管理類似
• CompassTemplate - 將session和transaction透明化。類似Spring提供的HibernateTemplate
• 數據到索引的各種映射 - OSEM, XSEM, JSEM, RSEM。支持通過程序、XML、JSON進行配置。
• CompassGps - Gps的核心模塊，管理GpsDevice，有兩種實現：SingleCompassGps和DualCompassGps。
• CompassGpsDevice - 處理各種數據源到索引的操作：JDBC, Hibernate, iBatis等。不能獨立使用而必須融合到CompassGps中。

三、與Spring、Hibernate整合

 

     這裏主要結合代碼進行。

    

     1.數據庫腳本(Oracle)

--創建表Article
create table ARTICLE
(
  ID      NUMBER,--ID,主鍵
  TITLE   VARCHAR2(100),--標題
  CONTENT CLOB,--文章內容
  PUBDATE DATE--發佈日期
)

   

    2.配置Compass的OSEM 以及Hibernate映射

import java.io.Serializable;
import java.util.Date;

import javax.persistence.Column;
import javax.persistence.Entity;
import javax.persistence.GeneratedValue;
import javax.persistence.Id;
import javax.persistence.Lob;
import javax.persistence.Table;
import javax.persistence.Temporal;
import javax.persistence.TemporalType;

import org.compass.annotations.Index;
import org.compass.annotations.Searchable;
import org.compass.annotations.SearchableId;
import org.compass.annotations.SearchableProperty;
import org.compass.annotations.Store;
import org.hibernate.annotations.GenericGenerator;


@Searchable(alias = "article")
@Entity
@Table(name = "ARTICLE", schema = "SCOTT")
public class Article implements Serializable {

    private static final long serialVersionUID = 1L;

    private Long id;
    private String title;
    private Date pubdate = new Date();
    private String content;

    @SearchableId(
            name = "id",
            store = Store.NO,
            index = Index.NOT_ANALYZED)
    @Id
    @GeneratedValue(generator = "paymentableGenerator")
    @GenericGenerator(name = "paymentableGenerator", strategy = "increment")
    public Long getId() {
        return id;
    }

    public void setId(Long id) {
        this.id = id;
    }

    @SearchableProperty(
            name = "title",
            store = Store.YES,
            index = Index.ANALYZED)
    @Column(name = "TITLE")
    public String getTitle() {
        return title;
    }

    public void setTitle(String title) {
        this.title = title;
    }

    @SearchableProperty(
            name = "pubdate",
            store = Store.NO,
            index = Index.UN_TOKENIZED)
    @Temporal(TemporalType.TIMESTAMP)
    @Column(name = "PUBDATE")
    public Date getPubdate() {
        return pubdate;
    }

    public void setPubdate(Date pubdate) {
        this.pubdate = pubdate;
    }

    @SearchableProperty(
            name = "content",
            index = Index.TOKENIZED,
            store = Store.YES,
            converter = "htmlPropertyConverter")
    @Lob
    @Column(name = "CONTENT")
    public String getContent() {
        return content;
    }

    public void setContent(String content) {
        this.content = content;
    }

}

    說明：

    @Searchable(alias="article")表示這個是可以搜索實體，別名爲article. 
    @SearchableId  這個是實體搜索的標識ID,和hibernate裏的概念差不多，用來區分索引文件裏的實體索引。 
    @SearchableProperty(index = Index.NOT_ANALYZED, store = Store.NO) 表示這個屬性存入索引文件,不進行分詞,不存儲要索引中。

    另外在getContent()方法上的@SearchableProperty中還加入了converter = "htmlPropertyConverter",主要是用來將文章中的HTML標籤進行過濾獲取純文本，在建立到索引中。在後面會具體介紹這個轉換器。

 

    3.建立Compass搜索的類

import java.util.ArrayList;
import java.util.List;

import javax.annotation.Resource;

import org.compass.core.Compass;
import org.compass.core.CompassHighlighter;
import org.compass.core.CompassHits;
import org.compass.core.CompassQuery;
import org.compass.core.CompassQueryBuilder;
import org.compass.core.CompassSession;
import org.compass.core.CompassTemplate;
import org.compass.core.CompassHighlighter.TextTokenizer;
import org.compass.core.CompassQuery.SortPropertyType;
import org.springframework.stereotype.Component;

import com.compass.example.dao.SearchArticleDao;
import com.compass.example.model.Article;

@Component("SearchArticleDao")
public class SearchArticleDaoCompass implements SearchArticleDao {

    @Resource
    private CompassTemplate compassTemplate;

    @Override
    public List<Article> searchWithList(final String queryString) {

        Compass compass = compassTemplate.getCompass();
        CompassSession session = compass.openSession();
        CompassQueryBuilder builder = session.queryBuilder();
        CompassQuery compassQuery = builder.queryString(queryString).toQuery().addSort("article.id",SortPropertyType.STRING);
        CompassHits compassHits = compassQuery.hits();


        List<Article> articles = new ArrayList<Article>();
        for(int i=0; i<compassHits.length(); i++) {
            Article article = (Article) compassHits.data(i);
            CompassHighlighter highlighter = compassHits.highlighter(i);
            String title = highlighter.fragment("title");
            if(title != null) {
                article.setTitle(title);
            }
            String content = highlighter.setTextTokenizer(TextTokenizer.AUTO).fragment("content");
            if(content != null) {
                article.setContent(content);
            }
            articles.add(article);
        }
        return articles;
    }

}

索引的查詢主要是根據傳過來的參數，關鍵字keyword,是搜索的關鍵字

String title = hits.highlighter(i).fragment("title");這段是檢索titile這個屬性有沒有出現搜索的關鍵字，有就將它高亮（其實就是在關鍵字前後加個html標記設置顏色，等下可以看到在配置文件裏可以自由設置高亮的顏色）. 
String content = hits.highlighter(i).setTextTokenizer( 
CompassHighlighter.TextTokenizer.AUTO) 
.fragment("content"); 
這段代碼和上面的title具有一樣的一樣的功能，另外還多了個很重要的功能，自動選擇正文中最匹配關鍵字的內容中的一部分輸出。因爲很多時候一篇文章幾千字，我們只想顯示有關鍵字的那部分的摘要，這時候這個功能就很方便.

 

    4.建立索引，將在服務器啓動時或定時重建索引

import org.compass.gps.CompassGps;
import org.springframework.beans.factory.InitializingBean;

/**
 * 通過quertz 定時調度定時重建索引或自動隨Spring ApplicationContext 啓動而重建
 * 索引的Builder。會在啓動後延時數秒新開線程調用compassGps.index()函數。
 * 默認會在Web應用每次啓動時重建索引，可以設置BuildIndex屬性爲false來禁止此功能。
 * 也可以不用本builder，編寫手動調用compasssGps.index()的代碼。
 * @author YinGuojun
 *
 */
public class CompassIndexBuilder implements InitializingBean {

    /*是否需要建立索引，可以被設置爲false使本Builder失效*/
    private boolean buildIndex = false;
    /*索引操作線程延時啓動的時間，單位爲秒*/
    private int lazyTime = 10;
    /*Compass封裝*/
    private CompassGps compassGps;

    private Thread indexThread = new Thread() {

        @Override
        public void run() {
            try {
                System.out.println("lazyTime: " + lazyTime);
                Thread.sleep(lazyTime * 1000);
                System.out.println("begin compass index ...");
                long beginTime = System.currentTimeMillis();

                // 重建索引.
                // 如果compass實體中定義的索引文件已存在，索引過程中會建立臨時索引，
                // 索引完成後再進行覆蓋.
                compassGps.index();

                long costTime = System.currentTimeMillis() - beginTime;
                System.out.println("compss index finished.");
                System.out.println("costed " + costTime + " milliseconds");

            } catch (InterruptedException e) {
                e.printStackTrace();
            }

        }

    };

    /**
     * 實現<code>InitializingBean</code>接口，在完成注入後調用啓動索引線程.
     *
     * @see org.springframework.beans.factory.InitializingBean#afterPropertiesSet()
     */
    @Override
    public void afterPropertiesSet() throws Exception {
        if (buildIndex) {
            indexThread.setDaemon(true);
            indexThread.setName("Compass Indexer");
            indexThread.start();
        }
    }

    public boolean isBuildIndex() {
        return buildIndex;
    }

    public void setBuildIndex(boolean buildIndex) {
        this.buildIndex = buildIndex;
    }

    public int getLazyTime() {
        return lazyTime;
    }

    public void setLazyTime(int lazyTime) {
        this.lazyTime = lazyTime;
    }

    public CompassGps getCompassGps() {
        return compassGps;
    }

    public void setCompassGps(CompassGps compassGps) {
        this.compassGps = compassGps;
    }

}

  

   5.轉換器

  

import org.apache.log4j.Logger;
import org.compass.core.Property;
import org.compass.core.converter.ConversionException;
import org.compass.core.converter.basic.AbstractBasicConverter;
import org.compass.core.mapping.ResourcePropertyMapping;
import org.compass.core.marshall.MarshallingContext;

import com.compass.example.utils.StringUtil;

public class HtmlPropertyConverter extends AbstractBasicConverter<String> {

    private static Logger logger = Logger.getLogger(HtmlPropertyConverter.class);

    public HtmlPropertyConverter() {
        super();
        // called by application server starting
        logger.info("----------HtmlPropertyConverter Initializing ...");
    }

    /**
     *  搜索的時候被調用
     */
    @Override
    protected String doFromString(String str,
            ResourcePropertyMapping resourcePropertyMapping,
            MarshallingContext context) throws ConversionException {
        logger.info("----------calling doFromString...");
        return str;
    }


    /**
     * 創建索引的時候被調用，此時將文本中的HTML標籤過濾
     */
    @Override
    protected Property createProperty(String value,
            ResourcePropertyMapping resourcePropertyMapping,
            MarshallingContext context) {
        logger.info("----------calling createProperty...");
                  //過濾html標籤
        value = StringUtil.removeHTML(value);

        return super.createProperty(value, resourcePropertyMapping, context);
    }


public class StringUtil {
/**
     * Remove occurences of html, defined as any text
     * between the characters "<" and ">". Optionally
     * replace HTML tags with a space.
     * @param str
     * @param addSpace
     * @return
     */
    public static String removeHTML(String str, boolean addSpace) {

        //System.out.println(str);

        if(str == null) return "";
        StringBuffer ret = new StringBuffer(str.length());
        int start = 0;
        int beginTag = str.indexOf("<");
        int endTag = 0;
        if(beginTag == -1) return str;

        while(beginTag >= start) {
            if(beginTag > 0) {
                ret.append(str.substring(start, beginTag));

                // replace each tag with a space (looks better)
                if(addSpace) ret.append(" ");
            }
            endTag = str.indexOf(">", beginTag);

            // if endTag found move "cursor" forward
            if(endTag > -1) {
                start = endTag + 1;
                beginTag = str.indexOf("<", start);
            }

            // if no endTag found, get rest of str and break
            else {
                ret.append(str.substring(beginTag));
                break;
            }
        }
        // append everything after the last endTag
        if(endTag >-1 && endTag + 1 < str.length()) {
            ret.append(str.substring(endTag + 1));
        }

        //System.out.println(ret.toString());

        return ret.toString().trim();
    }

    /**
     * Remove occurences of html, defined as any text
     * between the characters "<" and ">".
     * Replace any HTML tags with a space.
     * @param str
     * @return
     */
    public static String removeHTML(String str) {
        return removeHTML(str, true);
    }
}

 

   5.配置文件

 

<?xml version="1.0" encoding="UTF-8"?>
<beans
    xmlns="http://www.springframework.org/schema/beans"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xmlns:p="http://www.springframework.org/schema/p"
    xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-2.5.xsd">

    <bean id="annotationConfiguration" class="org.compass.annotations.config.CompassAnnotationsConfiguration"/>

    <bean id="compass" class="org.compass.spring.LocalCompassBean">
        <property name="resourceDirectoryLocations">
            <list>
                <value>classpath:com/compass</value>
            </list>
        </property>
        <!--  如果compass有單獨的配置文件,可以從這裏引入
        <property name="configLocation" value="classpath:compass.cfg.xml"/>
         -->
        <!-- 數據索引存儲位置 -->
        <property name="connection" value="/lucene/indexes"/>

        <property name="classMappings">
            <list>
                <value>com.compass.example.model.Product</value>
                <value>com.compass.example.model.Article</value>
            </list>
        </property>
        <property name="compassConfiguration" ref="annotationConfiguration"/>
        <property name="compassSettings">
            <props>
                <!-- 建立索引位置的另一種方式
                <prop key="compass.engine.connection">
                    file://${user.home}/lucene/indexes
                </prop>
                 -->
                <prop key="compass.transaction.factory">
                    org.compass.spring.transaction.SpringSyncTransactionFactory
                </prop>

                <!-- 指定摘要文本的長度 -->
                <prop key="compass.engine.highlighter.default.fragmenter.simple.size">
                    200
                </prop>

                <!-- 搜索內容高亮顯示 -->
                <prop  key="compass.engine.highlighter.default.formatter.simple.pre">
                    <![CDATA[<span style='background-color:yellow;color:red;'>]]>
                </prop>
                <prop  key="compass.engine.highlighter.default.formatter.simple.post">
                  <![CDATA[</span>]]>
                </prop>
                <!--定義分詞器-->
                <!--
                <prop
                    key="compass.engine.analyzer.default.type">
                   org.wltea.analyzer.lucene.IKAnalyzer
                </prop>
                -->
                <!--
                <prop key="compass.engine.analyzer.MMAnalyzer.CustomAnalyzer">
                    org.wltea.analyzer.lucene.IKAnalyzer
                    jeasy.analysis.MMAnalyzer
                    net.paoding.analysis.analyzer.PaodingAnalyzer
                </prop>
                -->

                <prop key="compass.engine.analyzer.default.type">
                    org.wltea.analyzer.lucene.IKAnalyzer
                </prop>

            </props>
        </property>
        <property name="transactionManager" ref="transactionManager"/>
        <property name="convertersByName">
            <map>
                <entry key="htmlPropertyConverter">
                    <bean class="com.compass.converter.HtmlPropertyConverter"/>
                </entry>
            </map>
        </property>
    </bean>

    <bean id="hibernateGpsDevice" class="org.compass.gps.device.hibernate.HibernateGpsDevice">
        <property name="name" value="hibernateDevice"/>
        <property name="sessionFactory" ref="sessionFactory"/>
        <property name="mirrorDataChanges" value="true"/>
    </bean>

    <!-- 數據庫中的記錄變化後同步更新索引 -->
    <bean id="compassGps" class="org.compass.gps.impl.SingleCompassGps" init-method="start" destroy-method="stop">
        <property name="compass" ref="compass"/>
        <property name="gpsDevices">
            <list>
                <!-- compass2.1
                <bean class="org.compass.spring.device.SpringSyncTransactionGpsDeviceWrapper">
                    <property name="gpsDevice" ref="hibernateGpsDevice"/>
                </bean>
                 -->
                 <!-- compass2.2 -->
                 <ref local="hibernateGpsDevice"/>
            </list>
        </property>
    </bean>

    <!-- compass模板 -->
    <bean id="compassTemplate" class="org.compass.core.CompassTemplate">
        <property name="compass" ref="compass"/>
    </bean>

    <!-- 定時重建索引(利用quartz)或隨Spring ApplicationContext啓動而重建索引 -->
    <bean id="compassIndexBuilder" lazy-init="false" class="com.compass.example.utils.CompassIndexBuilder">
        <property name="compassGps" ref="compassGps"/>
        <property name="buildIndex" value="true"/>
        <property name="lazyTime" value="10"/>
    </bean>
</beans>

 

    6.效果（英文）

 

     中文

 

 

四、問題總結

 

    1.異常there are more terms than documents in field "XXX", but it's impossible to sort on tokenized fields.

    在Luncene的API中對Sort的說明中有以下描述：

    The fields used to determine sort order must be carefully chosen. Documents must contain a single term in such a field, and the value of the term should indicate the document's relative position in a given sort order.The field must be indexed, but should not be tokenized, and does not need to be stored

(unless you happen to want it back with the rest of your document data). In other words:

document.add (new Field ("byNumber", Integer.toString(x), Field.Store.NO, Field.Index.NOT_ANALYZED));

    描述中紅色部分需特別注意，排序的字段必須被索引，並且不應該被tokennized,也就是在註解@SearchableProperty中的index=Index.NOT_ANALYZED, store=Store.NO,括號中的說明不是很明白，希望知道的可以給我點提示，再此謝謝了。

   

    2.異常java.lang.RuntimeException: field "XXX" does not appear to be indexed

    對多個表建索引後進行搜索，在添加排序條件時，如果不指定SortPropertyType，那麼在沒有指定converter的字段上排序時會拋以上異常， 但如果只對單個表建索引，不會有這個問題。

 

五、本次學習在網上查找各種資料的彙總，對引用到別處博客內容的博主表示感謝！文章尚有很多不完善之處，望指正，本人不勝感激！

六、其他資料

     Compass入門

     http://www.yeeach.com/2008/03/23/compass-%E5%85%A5%E9%97%A8%E6%8C%87%E5%8D%97/

  

     關於高亮顯示的解決方案

     http://jdkcn.com/entry/the-better-revolution-about-the-compass-lucene-highlight.html，此網站開放源碼，有助於大家學習