nutch-2.0執行過程及hbase下webpage的變化更新

今天把nutch-2.0整個過程執行了一邊，並且分析了hbase下webpage的變化過程。測試環境是本地環境，即hbase只在本地運行。

整個執行過程如下：

1>inject

2>generate

3>fetch

4>parse

5>updatedb

6>solrindex

webpage的表結構可以參考nutch/conf/gora-hbase-mapping.xml，如下所示

<?xml version="1.0" encoding="UTF-8"?>
<!--
  Licensed to the Apache Software Foundation (ASF) under one or more
  contributor license agreements.  See the NOTICE file distributed with
  this work for additional information regarding copyright ownership.
  The ASF licenses this file to You under the Apache License, Version 2.0
  (the "License"); you may not use this file except in compliance with
  the License.  You may obtain a copy of the License at
  
  http://www.apache.org/licenses/LICENSE-2.0
  
  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License.
-->
<gora-orm>
    
    <table name="webpage">
        <family name="p" maxVersions="1"/> <!-- This can also have params like compression, bloom filters -->
        <family name="f" maxVersions="1"/>
        <family name="s" maxVersions="1"/>
        <family name="il" maxVersions="1"/>
        <family name="ol" maxVersions="1"/>
        <family name="h" maxVersions="1"/>
        <family name="mtdt" maxVersions="1"/>
        <family name="mk" maxVersions="1"/>
    </table>
    <class table="webpage" keyClass="java.lang.String" name="org.apache.nutch.storage.WebPage">
        
        <!-- fetch fields     name="f"                                  -->
        <field name="baseUrl" family="f" qualifier="bas"/>
        <field name="status" family="f" qualifier="st"/>
        <field name="prevFetchTime" family="f" qualifier="pts"/>
        <field name="fetchTime" family="f" qualifier="ts"/>
        <field name="fetchInterval" family="f" qualifier="fi"/>
        <field name="retriesSinceFetch" family="f" qualifier="rsf"/>
        <field name="reprUrl" family="f" qualifier="rpr"/>
        <field name="content" family="f" qualifier="cnt"/>
        <field name="contentType" family="f" qualifier="typ"/>
        <field name="protocolStatus" family="f" qualifier="prot"/>
        <field name="modifiedTime" family="f" qualifier="mod"/>
        
        <!-- parse fields     name="p"                                  -->
        <field name="title" family="p" qualifier="t"/>
        <field name="text" family="p" qualifier="c"/>
        <field name="parseStatus" family="p" qualifier="st"/>
        <field name="signature" family="p" qualifier="sig"/>
        <field name="prevSignature" family="p" qualifier="psig"/>
        
        <!-- score fields    name="s"                                   -->
        <field name="score" family="s" qualifier="s"/>
        <field name="headers" family="h"/>
        <field name="inlinks" family="il"/>
        <field name="outlinks" family="ol"/>
        <field name="metadata" family="mtdt"/>
        <field name="markers" family="mk"/>
    </class>
    
    <table name="host">
      <family name="mtdt" maxVersions="1"/>
      <family name="il" maxVersions="1"/>
      <family name="ol" maxVersions="1"/>
    </table>
    
    <class table="host" keyClass="java.lang.String" name="org.apache.nutch.storage.Host">
      <field name="metadata" family="mtdt"/>
      <field name="inlinks" family="il"/>
      <field name="outlinks" family="ol"/>
    </class>
    
</gora-orm>

一、inject

在bin目錄下創建名爲urls的文件，加入一行：http://www.2345.com/

命令：./nutch inject urls

webpage表結構：

< f >
==============================================>

ROW com.2345.www:http/
COL f : fi //fetchInterval
COL f : ts //fetchTime

< h >
==============================================>

< il >
==============================================>

< mk >
==============================================>

ROW com.2345.www:http/
COL mk : _injmrk_ //inject標記

< mtdt >
==============================================>

ROW com.2345.www:http/
COL mtdt : _csh_ //???暫不明含義

< ol >
==============================================>


==============================================>

< s >
==============================================>

ROW com.2345.www:http/
COL s : s //score

生成的結構已經用紅色標出了，含義參見注釋。

二、generate

命令：./nutch generate

執行後webpage表的變化：

< f >
==============================================>

ROW com.2345.www:http/
COL f : fi
COL f : ts

< h >
==============================================>

< il >
==============================================>

< mk >
==============================================>

ROW com.2345.www:http/
COL mk : _gnmrk_
COL mk : _injmrk_

< mtdt >
==============================================>

ROW com.2345.www:http/
COL mtdt : _csh_

< ol >
==============================================>


==============================================>

< s >
==============================================>

ROW com.2345.www:http/
COL s : s

加紅的是多出的一行，可見在generate之後做了一個標記

三、fetch

命令：./nutch fetch 1345101942-1642860074

執行後webpage表的變化：

< f >
==============================================>

ROW com.2345.www:http/
COL f : bas //baseUrl
COL f : cnt //content
COL f : fi //fetchInterval
COL f : prot //protocolStatus
COL f : pts //prevFetchTime
COL f : st //status
COL f : ts //fetchTime
COL f : typ //contentType

< h >
==============================================>

ROW com.2345.www:http/
COL h : Accept-Ranges
COL h : Cache-Control
COL h : Connection
COL h : Content-Encoding
COL h : Content-Length
COL h : Content-Type
COL h : Date
COL h : ETag
COL h : Expires
COL h : Last-Modified
COL h : Server
COL h : Vary
COL h : _ip

< il >
==============================================>

< mk >
==============================================>

ROW com.2345.www:http/
COL mk : _ftcmrk_
COL mk : _gnmrk_
COL mk : _injmrk_

< mtdt >
==============================================>

ROW com.2345.www:http/
COL mtdt : _csh_

< ol >
==============================================>


==============================================>

< s >
==============================================>

ROW com.2345.www:http/
COL s : s

加入了fetch的標記，加入了header信息，加入了fetch的相關信息，具體參見注釋（僞的）

四、parse

< f >
==============================================>

ROW com.2345.www:http/
COL f : bas
COL f : cnt
COL f : fi
COL f : prot
COL f : pts
COL f : st
COL f : ts
COL f : typ

< h >
==============================================>

ROW com.2345.www:http/
COL h : Accept-Ranges
COL h : Cache-Control
COL h : Connection
COL h : Content-Encoding
COL h : Content-Length
COL h : Content-Type
COL h : Date
COL h : ETag
COL h : Expires
COL h : Last-Modified
COL h : Server
COL h : Vary
COL h : _ip

< il >
==============================================>

< mk >
==============================================>

ROW com.2345.www:http/
COL mk : __prsmrk__
COL mk : _ftcmrk_
COL mk : _gnmrk_
COL mk : _injmrk_

< mtdt >
==============================================>

ROW com.2345.www:http/
COL mtdt : _csh_

< ol >
==============================================>

ROW com.2345.www:http/
COL ol : http://182.xdwan.com/gg442a/
COL ol : http://2012.cntv.cn/
COL ol : http://2345.qq.okooo.com/shuangseqiu/
COL ol : http://25.198game.com/
COL ol : http://book.2345.com/yanqing.html
COL ol : http://buy.2345.com/
COL ol : http://cf.qq.com/
COL ol : http://china.nba.com/
COL ol : http://cn.wsj.com/
COL ol : http://danji.2345.com/
COL ol : http://day.2345.com/joke.htm
COL ol : http://dianying.2345.com/
COL ol : http://dianying.2345.com/detail/53555.html
COL ol : http://ditu.google.cn/
COL ol : http://email.163.com/
COL ol : http://esf.soufun.com/
COL ol : http://fcwr.jstv.com/
COL ol : http://finance.sina.com.cn/
COL ol : http://game.2345.com/
COL ol : http://im.qq.com/qq/all.shtml
COL ol : http://image.baidu.com/
COL ol : http://lux.2345.com/
COL ol : http://lvyou.2345.com/
COL ol : http://m.2345.com/pages/
COL ol : http://mail.10086.cn/
COL ol : http://mail.cn.yahoo.com/
COL ol : http://mail.qq.com/
COL ol : http://mail.sina.com.cn/
COL ol : http://military.china.com/
COL ol : http://mobile.pconline.com.cn/
COL ol : http://mobile.pcpop.com/
COL ol : http://mobile.zol.com.cn/
COL ol : http://mp3.baidu.com/
COL ol : http://mp3.sogou.com/
COL ol : http://my.2345.com/member/edit_info.php
COL ol : http://my.ku6.com/
COL ol : http://news.163.com/12/0815/04/88U0D1K60001124J.html
COL ol : http://news.7k7k.com/roco/
COL ol : http://news.baidu.com/
COL ol : http://news.ifeng.com/
COL ol : http://news.ifeng.com/mainland/special/cqqiangjiqiangjie/content-3/detail_2012_08/15/16829686_0.shtml
COL ol : http://news.ifeng.com/mainland/special/diaoyudaozhengduan/content-3/detail_2012_08/15/16832117_0.shtml
COL ol : http://news.ifeng.com/mainland/special/diaoyudaozhengduan/content-3/detail_2012_08/16/16844487_0.shtml
COL ol : http://news.ifeng.com/mil/
COL ol : http://news.qq.com/
COL ol : http://news.sina.com.cn/
COL ol : http://news.sohu.com/
COL ol : http://pic.2345.com/
COL ol : http://r.dangdang.com/2345kuzhan
COL ol : http://sg.198game.com/
COL ol : http://shouji.2345.com/
COL ol : http://sports.163.com/
COL ol : http://sports.sina.com.cn/
COL ol : http://sports.sohu.com/
COL ol : http://t.qq.com/
COL ol : http://tech.sina.com.cn/mobile/
COL ol : http://tianqi.2345.com/
COL ol : http://tianqi.2345.com/indexs.htm
COL ol : http://tieba.baidu.com/
COL ol : http://tieba.baidu.com/index.html
COL ol : http://tools.2345.com/
COL ol : http://tools.2345.com/weizhang.htm
COL ol : http://tools.2345.com/xingzuoyc.htm
COL ol : http://tools.2345.com/zhgjm.htm
COL ol : http://tuan.27.cn/
COL ol : http://tv.2345.com/detail/11406.html
COL ol : http://tv.2345.com/detail/11453.html
COL ol : http://v.2345.com/
COL ol : http://v.2345.com/dv/
COL ol : http://v.2345.com/live/
COL ol : http://v.2345.com/zongyi/
COL ol : http://v.2345.com/zt/40
COL ol : http://v.ifeng.com/
COL ol : http://video.baidu.com/
COL ol : http://wan.2345.com/sxd_a/
COL ol : http://wangyou.2345.com/
COL ol : http://web.7k7k.com/games/dpcq/2345dhz/2345dhz-wzl.html
COL ol : http://web.7k7k.com/games/frxz2/2345dhz/2345dhz-syyxztsan.html
COL ol : http://web.7k7k.com/games/lb/2345dhz/2345dhz-syyxzt6.html
COL ol : http://web.7k7k.com/games/lb/2345dhz/2345dhz-wzl6.html
COL ol : http://web.qq.com/
COL ol : http://ww.xitek.com/
COL ol : http://www.10010.com/
COL ol : http://www.10086.cn/
COL ol : http://www.12306.cn/mormhweb/
COL ol : http://www.126.com/
COL ol : http://www.163.com/
COL ol : http://www.17173.com/
COL ol : http://www.1ting.com/
COL ol : http://www.2345.com/about/about.htm
COL ol : http://www.2345.com/about/gyhd.htm
COL ol : http://www.2345.com/aihao.htm
COL ol : http://www.2345.com/art.htm
COL ol : http://www.2345.com/bank.htm
COL ol : http://www.2345.com/baoxian.htm
COL ol : http://www.2345.com/bbs.htm
COL ol : http://www.2345.com/blog.htm
COL ol : http://www.2345.com/caipiao.htm
COL ol : http://www.2345.com/car.htm
COL ol : http://www.2345.com/chat.htm
COL ol : http://www.2345.com/child.htm
COL ol : http://www.2345.com/city/
COL ol : http://www.2345.com/corp.htm
COL ol : http://www.2345.com/daxue.htm
COL ol : http://www.2345.com/desk.htm
COL ol : http://www.2345.com/edu.htm
COL ol : http://www.2345.com/eng.htm
COL ol : http://www.2345.com/fashion.htm
COL ol : http://www.2345.com/flower.htm
COL ol : http://www.2345.com/football.htm
COL ol : http://www.2345.com/gov.htm
COL ol : http://www.2345.com/gupiao.htm
COL ol : http://www.2345.com/haob.htm
COL ol : http://www.2345.com/hardware.htm
COL ol : http://www.2345.com/health.htm
COL ol : http://www.2345.com/help/szsy.htm
COL ol : http://www.2345.com/homepage.htm
COL ol : http://www.2345.com/house.htm
COL ol : http://www.2345.com/indexold.htm
COL ol : http://www.2345.com/indexz.htm
COL ol : http://www.2345.com/jianfei.htm
COL ol : http://www.2345.com/jijin.htm
COL ol : http://www.2345.com/job.htm
COL ol : http://www.2345.com/junshi.htm
COL ol : http://www.2345.com/kaoshi.htm
COL ol : http://www.2345.com/kv.htm
COL ol : http://www.2345.com/laonian.htm
COL ol : http://www.2345.com/law.htm
COL ol : http://www.2345.com/love.htm
COL ol : http://www.2345.com/mail.htm
COL ol : http://www.2345.com/manwoman.htm
COL ol : http://www.2345.com/map.htm
COL ol : http://www.2345.com/meishi.htm
COL ol : http://www.2345.com/mm.htm
COL ol : http://www.2345.com/navhtm/city.htm
COL ol : http://www.2345.com/nba.htm
COL ol : http://www.2345.com/newsweek.htm
COL ol : http://www.2345.com/pic.htm
COL ol : http://www.2345.com/qtwz.htm
COL ol : http://www.2345.com/quyi.htm
COL ol : http://www.2345.com/shaoer.htm
COL ol : http://www.2345.com/sheji.htm
COL ol : http://www.2345.com/shenghuo.htm
COL ol : http://www.2345.com/sheying.htm
COL ol : http://www.2345.com/shouji.htm
COL ol : http://www.2345.com/shuma.htm
COL ol : http://www.2345.com/shwz.htm
COL ol : http://www.2345.com/soft/
COL ol : http://www.2345.com/software.htm
COL ol : http://www.2345.com/sport.htm
COL ol : http://www.2345.com/tongxun.htm
COL ol : http://www.2345.com/weibo.htm
COL ol : http://www.2345.com/world.htm
COL ol : http://www.2345.com/xszn/xshelp.htm
COL ol : http://www.2345.com/ylwz.htm
COL ol : http://www.2345.com/yule.htm
COL ol : http://www.2345.com/zt/girl.htm
COL ol : http://www.37wan.com/tg/2345dao/
COL ol : http://www.37wan.com/tg/2345dao_jxqy
COL ol : http://www.39.net/
COL ol : http://www.4399.com/
COL ol : http://www.4738.com/
COL ol : http://www.5200tv.com/
COL ol : http://www.6.cn/
COL ol : http://www.7k7k.com/
COL ol : http://www.9ku.com/
COL ol : http://www.abchina.com/
COL ol : http://www.aili.com/
COL ol : http://www.alipay.com/
COL ol : http://www.baixing.com/
COL ol : http://www.bankcomm.com/
COL ol : http://www.boc.cn/
COL ol : http://www.ccb.com/
COL ol : http://www.cheshi.com/
COL ol : http://www.chinabug.net/
COL ol : http://www.chinaiiss.com/
COL ol : http://www.chinanews.com.cn/
COL ol : http://www.cmbchina.com/
COL ol : http://www.cnfol.com/
COL ol : http://www.cntv.cn/index.shtml
COL ol : http://www.digu.com/
COL ol : http://www.douban.com/
COL ol : http://www.duote.com/
COL ol : http://www.duote.com/soft/2054.html
COL ol : http://www.duote.com/soft/21205.html
COL ol : http://www.duote.com/soft/21413.html
COL ol : http://www.duote.com/soft/54347.html
COL ol : http://www.duote.com/soft/5841.html
COL ol : http://www.fortunechina.com/
COL ol : http://www.hexun.com/
COL ol : http://www.homeinns.com/
COL ol : http://www.hongxiu.com/
COL ol : http://www.hotmail.com/
COL ol : http://www.hupu.com/
COL ol : http://www.ifeng.com/
COL ol : http://www.imobile.com.cn/
COL ol : http://www.iqiyi.com/
COL ol : http://www.junshitx.com/
COL ol : http://www.jxdyf.com/
COL ol : http://www.jxedt.com/
COL ol : http://www.kaixin001.com/
COL ol : http://www.kanshu.com/
COL ol : http://www.ku6.cn/
COL ol : http://www.kugou.com/
COL ol : http://www.m1905.com/vod/
COL ol : http://www.miercn.com/
COL ol : http://www.miitbeian.gov.cn/
COL ol : http://www.mop.com/
COL ol : http://www.onlinedown.net/
COL ol : http://www.onlylady.com/
COL ol : http://www.pcpop.com/
COL ol : http://www.people.com.cn/
COL ol : http://www.qianyan001.com/
COL ol : http://www.qidian.com/
COL ol : http://www.qiushibaike.com/
COL ol : http://www.qq.com/
COL ol : http://www.readnovel.com/
COL ol : http://www.renren.com/
COL ol : http://www.ruichuang.net/join_us/
COL ol : http://www.sina.com.cn/
COL ol : http://www.skycn.com/
COL ol : http://www.sohu.com/
COL ol : http://www.stockstar.com/
COL ol : http://www.taobao.com/
COL ol : http://www.tianya.cn/
COL ol : http://www.tudou.com/
COL ol : http://www.vdolady.com/
COL ol : http://www.vogue.com.cn/
COL ol : http://www.xinhuanet.com/
COL ol : http://www.xunlei.com/
COL ol : http://www.xxsy.net/
COL ol : http://www.youku.com/
COL ol : http://www.zaobao.com/
COL ol : http://www.zb8.com/
COL ol : http://www.zhcw.com/
COL ol : http://www.zhenai.com/901002.html
COL ol : http://www.zhiji.com/
COL ol : http://www.zhulang.com/
COL ol : http://www.zol.com.cn/
COL ol : http://xiaoyouxi.2345.com/zhuanti/110.htm
COL ol : http://xiaoyouxi.2345.com/zhuanti/174.htm
COL ol : http://xiaoyouxi.2345.com/zhuanti/61.htm
COL ol : http://yule.iqiyi.com/zghsy.html
COL ol : http://zhidao.baidu.com/


==============================================>

ROW com.2345.www:http/
COL p : c //text
COL p : sig //signature
COL p : st //parseStatus
COL p : t //title

< s >
==============================================>

ROW com.2345.www:http/
COL s : s

加入了parse標記，加入了ol(outlink)外部鏈接，也即從剛纔fetch的頁面http://www.2345.com/內分析之後得到的鏈接，還加入了parse的信息，參見注釋。

五、updatedb

命令：./nutch updatedb

執行後webpage的變化：

< f >
==============================================>

ROW cn.10086.mail:http/
COL f : fi
COL f : st //status
COL f : ts

ROW cn.10086.www:http/
COL f : fi
COL f : st
COL f : ts

ROW cn.12306.www:http/mormhweb/
COL f : fi
COL f : st
COL f : ts

ROW cn.27.tuan:http/
COL f : fi
COL f : st
COL f : ts

ROW cn.6.www:http/
COL f : fi
COL f : st
COL f : ts

ROW cn.boc.www:http/

< h >
==============================================>

ROW com.2345.www:http/
COL h : Accept-Ranges
COL h : Cache-Control
COL h : Connection
COL h : Content-Encoding
COL h : Content-Length
COL h : Content-Type
COL h : Date
COL h : ETag
COL h : Expires
COL h : Last-Modified
COL h : Server
COL h : Vary
COL h : _ip

< il >
==============================================>

ROW cn.10086.mail:http/
COL il : http://www.2345.com/

ROW cn.10086.www:http/
COL il : http://www.2345.com/

ROW cn.12306.www:http/mormhweb/
COL il : http://www.2345.com/

ROW cn.27.tuan:http/
COL il : http://www.2345.com/

ROW cn.6.www:http/
COL il : http://www.2345.com/

ROW cn.boc.www:http/

< mk >
==============================================>

ROW com.2345.www:http/
COL mk : _injmrk_

< mtdt >
==============================================>

ROW cn.10086.mail:http/
COL mtdt : _csh_

ROW cn.10086.www:http/
COL mtdt : _csh_

ROW cn.12306.www:http/mormhweb/
COL mtdt : _csh_

ROW cn.27.tuan:http/
COL mtdt : _csh_

ROW cn.6.www:http/
COL mtdt : _csh_

ROW cn.boc.www:http/

< ol >
==============================================>

ROW com.2345.www:http/
COL ol : http://182.xdwan.com/gg442a/
COL ol : http://2012.cntv.cn/
COL ol : http://2345.qq.okooo.com/shuangseqiu/
COL ol : http://25.198game.com/
COL ol : http://book.2345.com/yanqing.html
COL ol : http://buy.2345.com/
COL ol : http://cf.qq.com/
COL ol : http://china.nba.com/
COL ol : http://cn.wsj.com/
COL ol : http://danji.2345.com/
COL ol : http://day.2345.com/joke.htm
COL ol : http://dianying.2345.com/
COL ol : http://dianying.2345.com/detail/53555.html
COL ol : http://ditu.google.cn/
COL ol : http://email.163.com/
COL ol : http://esf.soufun.com/
COL ol : http://fcwr.jstv.com/
COL ol : http://finance.sina.com.cn/
COL ol : http://game.2345.com/
COL ol : http://im.qq.com/qq/all.shtml
COL ol : http://image.baidu.com/
COL ol : http://lux.2345.com/
COL ol : http://lvyou.2345.com/
COL ol : http://m.2345.com/pages/
COL ol : http://mail.10086.cn/
COL ol : http://mail.cn.yahoo.com/
COL ol : http://mail.qq.com/
COL ol : http://mail.sina.com.cn/
COL ol : http://military.china.com/
COL ol : http://mobile.pconline.com.cn/
COL ol : http://mobile.pcpop.com/
COL ol : http://mobile.zol.com.cn/
COL ol : http://mp3.baidu.com/
COL ol : http://mp3.sogou.com/
COL ol : http://my.2345.com/member/edit_info.php
COL ol : http://my.ku6.com/
COL ol : http://news.163.com/12/0815/04/88U0D1K60001124J.html
COL ol : http://news.7k7k.com/roco/
COL ol : http://news.baidu.com/
COL ol : http://news.ifeng.com/
COL ol : http://news.ifeng.com/mainland/special/cqqiangjiqiangjie/content-3/detail_2012_08/15/16829686_0.shtml
COL ol : http://news.ifeng.com/mainland/special/diaoyudaozhengduan/content-3/detail_2012_08/15/16832117_0.shtml
COL ol : http://news.ifeng.com/mainland/special/diaoyudaozhengduan/content-3/detail_2012_08/16/16844487_0.shtml
COL ol : http://news.ifeng.com/mil/
COL ol : http://news.qq.com/
COL ol : http://news.sina.com.cn/
COL ol : http://news.sohu.com/
COL ol : http://pic.2345.com/
COL ol : http://r.dangdang.com/2345kuzhan
COL ol : http://sg.198game.com/
COL ol : http://shouji.2345.com/
COL ol : http://sports.163.com/
COL ol : http://sports.sina.com.cn/
COL ol : http://sports.sohu.com/
COL ol : http://t.qq.com/
COL ol : http://tech.sina.com.cn/mobile/
COL ol : http://tianqi.2345.com/
COL ol : http://tianqi.2345.com/indexs.htm
COL ol : http://tieba.baidu.com/
COL ol : http://tieba.baidu.com/index.html
COL ol : http://tools.2345.com/
COL ol : http://tools.2345.com/weizhang.htm
COL ol : http://tools.2345.com/xingzuoyc.htm
COL ol : http://tools.2345.com/zhgjm.htm
COL ol : http://tuan.27.cn/
COL ol : http://tv.2345.com/detail/11406.html
COL ol : http://tv.2345.com/detail/11453.html
COL ol : http://v.2345.com/
COL ol : http://v.2345.com/dv/
COL ol : http://v.2345.com/live/
COL ol : http://v.2345.com/zongyi/
COL ol : http://v.2345.com/zt/40
COL ol : http://v.ifeng.com/
COL ol : http://video.baidu.com/
COL ol : http://wan.2345.com/sxd_a/
COL ol : http://wangyou.2345.com/
COL ol : http://web.7k7k.com/games/dpcq/2345dhz/2345dhz-wzl.html
COL ol : http://web.7k7k.com/games/frxz2/2345dhz/2345dhz-syyxztsan.html
COL ol : http://web.7k7k.com/games/lb/2345dhz/2345dhz-syyxzt6.html
COL ol : http://web.7k7k.com/games/lb/2345dhz/2345dhz-wzl6.html
COL ol : http://web.qq.com/
COL ol : http://ww.xitek.com/
COL ol : http://www.10010.com/
COL ol : http://www.10086.cn/
COL ol : http://www.12306.cn/mormhweb/
COL ol : http://www.126.com/
COL ol : http://www.163.com/
COL ol : http://www.17173.com/
COL ol : http://www.1ting.com/
COL ol : http://www.2345.com/about/about.htm
COL ol : http://www.2345.com/about/gyhd.htm
COL ol : http://www.2345.com/aihao.htm
COL ol : http://www.2345.com/art.htm
COL ol : http://www.2345.com/bank.htm
COL ol : http://www.2345.com/baoxian.htm
COL ol : http://www.2345.com/bbs.htm
COL ol : http://www.2345.com/blog.htm
COL ol : http://www.2345.com/caipiao.htm
COL ol : http://www.2345.com/car.htm
COL ol : http://www.2345.com/chat.htm
COL ol : http://www.2345.com/child.htm
COL ol : http://www.2345.com/city/
COL ol : http://www.2345.com/corp.htm
COL ol : http://www.2345.com/daxue.htm
COL ol : http://www.2345.com/desk.htm
COL ol : http://www.2345.com/edu.htm
COL ol : http://www.2345.com/eng.htm
COL ol : http://www.2345.com/fashion.htm
COL ol : http://www.2345.com/flower.htm
COL ol : http://www.2345.com/football.htm
COL ol : http://www.2345.com/gov.htm
COL ol : http://www.2345.com/gupiao.htm
COL ol : http://www.2345.com/haob.htm
COL ol : http://www.2345.com/hardware.htm
COL ol : http://www.2345.com/health.htm
COL ol : http://www.2345.com/help/szsy.htm
COL ol : http://www.2345.com/homepage.htm
COL ol : http://www.2345.com/house.htm
COL ol : http://www.2345.com/indexold.htm
COL ol : http://www.2345.com/indexz.htm
COL ol : http://www.2345.com/jianfei.htm
COL ol : http://www.2345.com/jijin.htm
COL ol : http://www.2345.com/job.htm
COL ol : http://www.2345.com/junshi.htm
COL ol : http://www.2345.com/kaoshi.htm
COL ol : http://www.2345.com/kv.htm
COL ol : http://www.2345.com/laonian.htm
COL ol : http://www.2345.com/law.htm
COL ol : http://www.2345.com/love.htm
COL ol : http://www.2345.com/mail.htm
COL ol : http://www.2345.com/manwoman.htm
COL ol : http://www.2345.com/map.htm
COL ol : http://www.2345.com/meishi.htm
COL ol : http://www.2345.com/mm.htm
COL ol : http://www.2345.com/navhtm/city.htm
COL ol : http://www.2345.com/nba.htm
COL ol : http://www.2345.com/newsweek.htm
COL ol : http://www.2345.com/pic.htm
COL ol : http://www.2345.com/qtwz.htm
COL ol : http://www.2345.com/quyi.htm
COL ol : http://www.2345.com/shaoer.htm
COL ol : http://www.2345.com/sheji.htm
COL ol : http://www.2345.com/shenghuo.htm
COL ol : http://www.2345.com/sheying.htm
COL ol : http://www.2345.com/shouji.htm
COL ol : http://www.2345.com/shuma.htm
COL ol : http://www.2345.com/shwz.htm
COL ol : http://www.2345.com/soft/
COL ol : http://www.2345.com/software.htm
COL ol : http://www.2345.com/sport.htm
COL ol : http://www.2345.com/tongxun.htm
COL ol : http://www.2345.com/weibo.htm
COL ol : http://www.2345.com/world.htm
COL ol : http://www.2345.com/xszn/xshelp.htm
COL ol : http://www.2345.com/ylwz.htm
COL ol : http://www.2345.com/yule.htm
COL ol : http://www.2345.com/zt/girl.htm
COL ol : http://www.37wan.com/tg/2345dao/
COL ol : http://www.37wan.com/tg/2345dao_jxqy
COL ol : http://www.39.net/
COL ol : http://www.4399.com/
COL ol : http://www.4738.com/
COL ol : http://www.5200tv.com/
COL ol : http://www.6.cn/
COL ol : http://www.7k7k.com/
COL ol : http://www.9ku.com/
COL ol : http://www.abchina.com/
COL ol : http://www.aili.com/
COL ol : http://www.alipay.com/
COL ol : http://www.baixing.com/
COL ol : http://www.bankcomm.com/
COL ol : http://www.boc.cn/
COL ol : http://www.ccb.com/
COL ol : http://www.cheshi.com/
COL ol : http://www.chinabug.net/
COL ol : http://www.chinaiiss.com/
COL ol : http://www.chinanews.com.cn/
COL ol : http://www.cmbchina.com/
COL ol : http://www.cnfol.com/
COL ol : http://www.cntv.cn/index.shtml
COL ol : http://www.digu.com/
COL ol : http://www.douban.com/
COL ol : http://www.duote.com/
COL ol : http://www.duote.com/soft/2054.html
COL ol : http://www.duote.com/soft/21205.html
COL ol : http://www.duote.com/soft/21413.html
COL ol : http://www.duote.com/soft/54347.html
COL ol : http://www.duote.com/soft/5841.html
COL ol : http://www.fortunechina.com/
COL ol : http://www.hexun.com/
COL ol : http://www.homeinns.com/
COL ol : http://www.hongxiu.com/
COL ol : http://www.hotmail.com/
COL ol : http://www.hupu.com/
COL ol : http://www.ifeng.com/
COL ol : http://www.imobile.com.cn/
COL ol : http://www.iqiyi.com/
COL ol : http://www.junshitx.com/
COL ol : http://www.jxdyf.com/
COL ol : http://www.jxedt.com/
COL ol : http://www.kaixin001.com/
COL ol : http://www.kanshu.com/
COL ol : http://www.ku6.cn/
COL ol : http://www.kugou.com/
COL ol : http://www.m1905.com/vod/
COL ol : http://www.miercn.com/
COL ol : http://www.miitbeian.gov.cn/
COL ol : http://www.mop.com/
COL ol : http://www.onlinedown.net/
COL ol : http://www.onlylady.com/
COL ol : http://www.pcpop.com/
COL ol : http://www.people.com.cn/
COL ol : http://www.qianyan001.com/
COL ol : http://www.qidian.com/
COL ol : http://www.qiushibaike.com/
COL ol : http://www.qq.com/
COL ol : http://www.readnovel.com/
COL ol : http://www.renren.com/
COL ol : http://www.ruichuang.net/join_us/
COL ol : http://www.sina.com.cn/
COL ol : http://www.skycn.com/
COL ol : http://www.sohu.com/
COL ol : http://www.stockstar.com/
COL ol : http://www.taobao.com/
COL ol : http://www.tianya.cn/
COL ol : http://www.tudou.com/
COL ol : http://www.vdolady.com/
COL ol : http://www.vogue.com.cn/
COL ol : http://www.xinhuanet.com/
COL ol : http://www.xunlei.com/
COL ol : http://www.xxsy.net/
COL ol : http://www.youku.com/
COL ol : http://www.zaobao.com/
COL ol : http://www.zb8.com/
COL ol : http://www.zhcw.com/
COL ol : http://www.zhenai.com/901002.html
COL ol : http://www.zhiji.com/
COL ol : http://www.zhulang.com/
COL ol : http://www.zol.com.cn/
COL ol : http://xiaoyouxi.2345.com/zhuanti/110.htm
COL ol : http://xiaoyouxi.2345.com/zhuanti/174.htm
COL ol : http://xiaoyouxi.2345.com/zhuanti/61.htm
COL ol : http://yule.iqiyi.com/zghsy.html
COL ol : http://zhidao.baidu.com/


==============================================>

ROW com.2345.www:http/
COL p : c
COL p : sig
COL p : st
COL p : t

< s >
==============================================>

ROW cn.10086.mail:http/
COL s : s //score

ROW cn.10086.www:http/
COL s : s

ROW cn.12306.www:http/mormhweb/
COL s : s

ROW cn.27.tuan:http/
COL s : s

ROW cn.6.www:http/
COL s : s

ROW cn.boc.www:http/

更新db之後，把之前parse得出的urls全部採用倒排索引的方式生成行鍵，與剛開始注入時相同，唯有一處是f:st，已經用藍色標出，指fetch的狀態。

整個過程簡單的就是這樣了，當然這裏測試的只是初始的過程，穩定抓取時還會有複雜的情況出現。

nutch-2.0執行過程及hbase下webpage的變化更新

【簡寫Mybatis-02】註冊機的實現以及SqlSession處理

手繪二維碼

.NET藉助虛擬網卡實現一個簡單異地組網工具

hadoop相關－HBase體系結構

nutch-2.0介紹

nutch-2.0執行過程及hbase下webpage的變化更新

nutch-2.0源碼之InjectorJob

eclipse下運行nutch-2.0

Mac下配置sublime實現LaTeX

https://yachay.unat.edu.pe/blog/index.php?comment_area=format_blog&comment_component=blog&comment_co

linux以太網驅動總結