ELK学习笔记-ES-分词

官方文档位置：https://www.elastic.co/guide/en/elasticsearch/reference/7.5/analysis.html

分词

分词是指将文本转化成一系列的单词（term or token）的过程，也可以叫文本分析
es里称之为Analysis

分词器

分词器是es中专门处理分词的组件,英文为Analyzer ,它的组成如下:
Character Filters

针对原始文本进行处理,比如去除html特殊标记符
Tokenizer
将原始文本按照一定规则切分为单词
Token Filters
针对 tokenizer处理的单词就行再加工,比如转小写、删除或新增等处理

分词器-调用顺序

Analyze API

es提供了一个测试分词的api接口,方便验证分词效果, endpoint是_ analyze

可以直接指定analyzer进行测试
可以直接指定索弓|中的字段进行测试
可以自定义分词器进行测试

直接指定Analyzer进行测试，接口如下：

POST _analyze
{
  "analyzer":"standard",   //分词器,standard是es默认分词器，如果字段里没有指定分词器，会默认使用standard
  "text":"hello world!"    //测试文本
}

输出：

{
  "tokens": [
    {
      "token": "hello",    //分词结果
      "start_offset": 0,    //起始偏移
      "end_offset": 5,    //结束偏移
      "type": "<ALPHANUM>",
      "position": 0    //分词位置
    },
    {
      "token": "world",
      "start_offset": 6,
      "end_offset": 11,
      "type": "<ALPHANUM>",
      "position": 1
    }
  ]
}

直接指定索引中的字段进行测试，接口如下：

POST test_index/_analyze    //测试的索引
{
    "field":"username",    //测试字段
    "text":"hello world"    //测试文本
}

输出：

{
  "tokens": [
    {
      "token": "hello",
      "start_offset": 0,
      "end_offset": 5,
      "type": "<ALPHANUM>",
      "position": 0
    },
    {
      "token": "world",
      "start_offset": 6,
      "end_offset": 11,
      "type": "<ALPHANUM>",
      "position": 1
    }
  ]
}

自定义分词器测试，接口如下：

POST _analyze
{
   "tokenizer":"standard",    //指明分词器
   "filter":["lowercase"],    //过滤成小写
   "text":"Hello World!"
 }

输出：

{
  "tokens": [
    {
      "token": "hello",
      "start_offset": 0,
      "end_offset": 5,
      "type": "<ALPHANUM>",
      "position": 0
    },
    {
      "token": "world",
      "start_offset": 6,
      "end_offset": 11,
      "type": "<ALPHANUM>",
      "position": 1
    }
  ]
}

预定义分词器

es自带如下分词器：
Standard
Simple
Whitespace
Stop
Keyword
Pattern
Language

Standard Analyzer

默认分词器
按词切分，支持多语言
小写处理

演示：

POST _analyze
{
  "analyzer": "standard",
  "text": "The 2 QUICK Brown-Foxes jumped over the lazy dog's bone."
}

输出：（可以看到，所以大写字母已经改成小写）

{
  "tokens": [
    {
      "token": "the",
      "start_offset": 0,
      "end_offset": 3,
      "type": "<ALPHANUM>",
      "position": 0
    },
    {
      "token": "2",
      "start_offset": 4,
      "end_offset": 5,
      "type": "<NUM>",
      "position": 1
    },
    {
      "token": "quick",
      "start_offset": 6,
      "end_offset": 11,
      "type": "<ALPHANUM>",
      "position": 2
    },
    {
      "token": "brown",
      "start_offset": 12,
      "end_offset": 17,
      "type": "<ALPHANUM>",
      "position": 3
    },
    {
      "token": "foxes",
      "start_offset": 18,
      "end_offset": 23,
      "type": "<ALPHANUM>",
      "position": 4
    },
    {
      "token": "jumped",
      "start_offset": 24,
      "end_offset": 30,
      "type": "<ALPHANUM>",
      "position": 5
    },
    {
      "token": "over",
      "start_offset": 31,
      "end_offset": 35,
      "type": "<ALPHANUM>",
      "position": 6
    },
    {
      "token": "the",
      "start_offset": 36,
      "end_offset": 39,
      "type": "<ALPHANUM>",
      "position": 7
    },
    {
      "token": "lazy",
      "start_offset": 40,
      "end_offset": 44,
      "type": "<ALPHANUM>",
      "position": 8
    },
    {
      "token": "dog's",
      "start_offset": 45,
      "end_offset": 50,
      "type": "<ALPHANUM>",
      "position": 9
    },
    {
      "token": "bone",
      "start_offset": 51,
      "end_offset": 55,
      "type": "<ALPHANUM>",
      "position": 10
    }
  ]
}

预定义分词器：
Simple Analyzer

按照非字母切分
小写处理

演示：

POST _analyze
{
  "analyzer": "simple",
  "text": "The 2 QUICK Brown-Foxes jumped over the lazy dog's bone."
}

输出：注（因为simple按照非字母切分，所以文中不是字母的已经全部切掉）

{
  "tokens": [
    {
      "token": "the",
      "start_offset": 0,
      "end_offset": 3,
      "type": "word",
      "position": 0
    },
    {
      "token": "quick",
      "start_offset": 6,
      "end_offset": 11,
      "type": "word",
      "position": 1
    },
    {
      "token": "brown",
      "start_offset": 12,
      "end_offset": 17,
      "type": "word",
      "position": 2
    },
    {
      "token": "foxes",
      "start_offset": 18,
      "end_offset": 23,
      "type": "word",
      "position": 3
    },
    {
      "token": "jumped",
      "start_offset": 24,
      "end_offset": 30,
      "type": "word",
      "position": 4
    },
    {
      "token": "over",
      "start_offset": 31,
      "end_offset": 35,
      "type": "word",
      "position": 5
    },
    {
      "token": "the",
      "start_offset": 36,
      "end_offset": 39,
      "type": "word",
      "position": 6
    },
    {
      "token": "lazy",
      "start_offset": 40,
      "end_offset": 44,
      "type": "word",
      "position": 7
    },
    {
      "token": "dog",
      "start_offset": 45,
      "end_offset": 48,
      "type": "word",
      "position": 8
    },
    {
      "token": "s",
      "start_offset": 49,
      "end_offset": 50,
      "type": "word",
      "position": 9
    },
    {
      "token": "bone",
      "start_offset": 51,
      "end_offset": 55,
      "type": "word",
      "position": 10
    }
  ]
}

Whitespace Analyzer

按照空格切分

演示：

POST _analyze
{
  "analyzer": "whitespace",
  "text": "The 2 QUICK Brown-Foxes jumped over the lazy dog's bone."
}

输出：（注：按空格切分，保持大小写）

{
  "tokens": [
    {
      "token": "The",
      "start_offset": 0,
      "end_offset": 3,
      "type": "word",
      "position": 0
    },
    {
      "token": "2",
      "start_offset": 4,
      "end_offset": 5,
      "type": "word",
      "position": 1
    },
    {
      "token": "QUICK",
      "start_offset": 6,
      "end_offset": 11,
      "type": "word",
      "position": 2
    },
    {
      "token": "Brown-Foxes",
      "start_offset": 12,
      "end_offset": 23,
      "type": "word",
      "position": 3
    },
    {
      "token": "jumped",
      "start_offset": 24,
      "end_offset": 30,
      "type": "word",
      "position": 4
    },
    {
      "token": "over",
      "start_offset": 31,
      "end_offset": 35,
      "type": "word",
      "position": 5
    },
    {
      "token": "the",
      "start_offset": 36,
      "end_offset": 39,
      "type": "word",
      "position": 6
    },
    {
      "token": "lazy",
      "start_offset": 40,
      "end_offset": 44,
      "type": "word",
      "position": 7
    },
    {
      "token": "dog's",
      "start_offset": 45,
      "end_offset": 50,
      "type": "word",
      "position": 8
    },
    {
      "token": "bone.",
      "start_offset": 51,
      "end_offset": 56,
      "type": "word",
      "position": 9
    }
  ]
}

Stop Analyzer

Stop Word指语气助词等修饰性的词语,比如the、an、的、这等等。

keyword Analyzer

不分词，直接将输入作为一个单词输出

演示：

POST _analyze
{
  "analyzer": "keyword",
  "text": "The 2 QUICK Brown-Foxes jumped over the lazy dog's bone."
}

输出：（注：不切分，所以直接输出一整行）

{
  "tokens": [
    {
      "token": "The 2 QUICK Brown-Foxes jumped over the lazy dog's bone.",
      "start_offset": 0,
      "end_offset": 56,
      "type": "word",
      "position": 0
    }
  ]
}

Pattern Analyzer

通过正则表达式自定义分隔符
默认\w+，非字词的符号作为分隔符

演示：

POST _analyze
{
  "analyzer": "pattern",
  "text": "The 2 QUICK Brown-Foxes jumped over the lazy dog's bone."
}

输出：

{
  "tokens": [
    {
      "token": "the",
      "start_offset": 0,
      "end_offset": 3,
      "type": "word",
      "position": 0
    },
    {
      "token": "2",
      "start_offset": 4,
      "end_offset": 5,
      "type": "word",
      "position": 1
    },
    {
      "token": "quick",
      "start_offset": 6,
      "end_offset": 11,
      "type": "word",
      "position": 2
    },
    {
      "token": "brown",
      "start_offset": 12,
      "end_offset": 17,
      "type": "word",
      "position": 3
    },
    {
      "token": "foxes",
      "start_offset": 18,
      "end_offset": 23,
      "type": "word",
      "position": 4
    },
    {
      "token": "jumped",
      "start_offset": 24,
      "end_offset": 30,
      "type": "word",
      "position": 5
    },
    {
      "token": "over",
      "start_offset": 31,
      "end_offset": 35,
      "type": "word",
      "position": 6
    },
    {
      "token": "the",
      "start_offset": 36,
      "end_offset": 39,
      "type": "word",
      "position": 7
    },
    {
      "token": "lazy",
      "start_offset": 40,
      "end_offset": 44,
      "type": "word",
      "position": 8
    },
    {
      "token": "dog",
      "start_offset": 45,
      "end_offset": 48,
      "type": "word",
      "position": 9
    },
    {
      "token": "s",
      "start_offset": 49,
      "end_offset": 50,
      "type": "word",
      "position": 10
    },
    {
      "token": "bone",
      "start_offset": 51,
      "end_offset": 55,
      "type": "word",
      "position": 11
    }
  ]
}

Language Analyzer

提供了30+常见语言的分词器（arabic, armenian, basque, bengali, brazilian, bulgarian,catalan, cjk, czech, danish，, dutch, english...）

注：如果工作中需要多语言分词，可以用Language

中文分词

难点：

中文分词指的是将一个汉字序列切分成- 个一个单独的词。在英文中,单词之间是
以空格作为自然分界符,汉语中词没有一个形式上的分界符。
上下文不同,分词结果迥异,比如交叉歧义问题,比如下面两种分词都合理
- 乒乓球拍/卖/完了
- 乒乓球/拍卖/完了

常见分词系统

实现中英文单词的切分,支持ik smart. ik maxword等模式
可自定义词库,支持热更新分词词典

jieba

python中最流行的分词系统,支持分词和词性标注
支持繁体分词、自定义词典、并行分词等

基于自然语言处理的分词系统：

Hanlp

由一系列模型与算法组成的Java工具包，目标是普及自然语言处理在生产环境中的应用

THULAC

THU Lexical Analyzer for Chinese ,由清华大学自然语言处理与社会人文计算实验室研制推出的一套中文词法分析工具包,具有中文分词和词性标注功能

自定义分词：

Tokenizer

将原始文本按照一定规则切分为单词（term or token）
自带功能如下：
Path Hierarchy按照文件路径进行切割
UAX URL Email按照standard分割,但不会分割邮箱和url
standard按照单词进行分割
letter按照非字符类进行分割
NGram和Edge NGram连词分割
whitespace按照空格进行分割

演示：Path Hierarchy （按文件路径切割）

POST _analyze
{
  "tokenizer": "path_hierarchy",
  "text": "/var/log/xxx.log"
}

输出：
{
  "tokens": [
    {
      "token": "/var",
      "start_offset": 0,
      "end_offset": 4,
      "type": "word",
      "position": 0
    },
    {
      "token": "/var/log",
      "start_offset": 0,
      "end_offset": 8,
      "type": "word",
      "position": 0
    },
    {
      "token": "/var/log/xxx.log",
      "start_offset": 0,
      "end_offset": 16,
      "type": "word",
      "position": 0
    }
  ]
}

Token Filters
对于tokenizer输出的单词( term)进行增加、删除、修改等操作
自带如下功能：

lowercase 将所有term转为小写
stop 删除 stop words
NGram和Edge NGram 连词分割
Synonym 添加近义词的 term

演示：

POST _analyze
{
  "text": "a Hello,world!",    //需要切分的语句
  "tokenizer": "standard",    
  "filter": [                  //定义要测试的切分
    "stop",
    "lowercase",
    {
      "type": "ngram",
      "min_gram":2,    //最小2个
      "max_gram":4    //最大4个
    }  
  ]
}

自定义分词需要在索引配置中设定。

创建分词器：

PUT test_index_1
{
  "settings": {
    "analysis": {
      "analyzer": {
        "my_custom_analyzer":{
          "type":      "custom",
          "tokenizer": "standard",
          "char_filter":[
            "html_strip"
            ],
            "filter":[
              "lowercase",
              "asciifolding"
            ]
        }
      }
    }
  }
}  
}

输出：

{
  "acknowledged": true,
  "shards_acknowledged": true,
  "index": "test_index_1"
}

分词：

POST test_index_1/_analyze
{
  "analyzer": "my_custom_analyzer",    //引用刚才定义的分词器
  "text": "this is a<b> box/b"    //需要分词语句
}

输出：

{
  "tokens": [
    {
      "token": "this",
      "start_offset": 0,
      "end_offset": 4,
      "type": "<ALPHANUM>",
      "position": 0
    },
    {
      "token": "is",
      "start_offset": 5,
      "end_offset": 7,
      "type": "<ALPHANUM>",
      "position": 1
    },
    {
      "token": "a",
      "start_offset": 8,
      "end_offset": 12,
      "type": "<ALPHANUM>",
      "position": 2
    },
    {
      "token": "box",
      "start_offset": 13,
      "end_offset": 16,
      "type": "<ALPHANUM>",
      "position": 3
    },
    {
      "token": "b",
      "start_offset": 17,
      "end_offset": 18,
      "type": "<ALPHANUM>",
      "position": 4
    }
  ]
}

创建分词器，并且设置条件切分：

PUT test_index2
{
  "settings": {
    "analysis": {
      "analyzer": {
        "my_custom_analyzer":{    //定义分词器名字
          "type":"custom",
          "char_filter":[    //设置字符过滤
            "emoticons"    //表情过滤
            ],
            "tokenizer":"punctuation",    //按照标点符号
            "filter":[    //设置词语更改操作
              "lowercase",    //小写 
              "english_stop"    //按英语格式，切分停止词
              ]
       }
      },
      "tokenizer": {
        "punctuation":{
          "type":"pattern",
          "pattern":"[.,!?]"    //添加标点符号
        }
      },
      "char_filter": {
        "emoticons":{
          "type":"mapping",
          "mappings":[
            ":) => _happy_",    //添加表情
            ":( => _sad_"
            ]
        }
      },
      "filter": {
        "english_stop":{
          "type":"stop",
          "stopwords":"_english_"    //按照英文格式分词
        }
      }
    }
  }
}

分词：

POST test_index10/_analyze
{
  "analyzer": "my_custom_analyzer",
  "text": "I'm a :) , person and you?"
}

输出：像a，and，这样的修饰词已经被干掉了

{
  "tokens": [
    {
      "token": "i'm ",
      "start_offset": 0,
      "end_offset": 4,
      "type": "word",
      "position": 0
    },
    {
      "token": " happy ",
      "start_offset": 5,
      "end_offset": 9,
      "type": "word",
      "position": 1
    },
    {
      "token": " person",
      "start_offset": 10,
      "end_offset": 17,
      "type": "word",
      "position": 2
    },
    {
      "token": "  you",
      "start_offset": 18,
      "end_offset": 23,
      "type": "word",
      "position": 3
    }
  ]
}

分词使用说明

创建或更新文档时，会对相应的文档进行分词处理
查询时，会对查询语句进行分词

创建索引时分词：

索引时分词是通过配置Index Mapping中每个字段的analyzer属性实现的,如下:
- 不指定分词时,使用默认standard
- 查询时分词：
- 查询时通过analyzer指定分词器
- 通过index mapping 设置 search_analyzer实现
  注：一般不需要特别指定查询时分词器,直接使用索引时分词器即可,否则会出现无法匹配的情况（查询和索引时使用同一个分词器，这样才能使用同一个分词规则来进行语句分词）
  
  分词的使用建议：
  
  ●明确字段是否需要分词,不需要分词的字段就将type设置为keyword ,可以节省空间和提高写性能
  ●善用_analyze API,查看文档的具体分词结果（可以查看你的分词语句是如何被分词的，再改正）选择一个适合公司业务的分词器，需要多次进行修改和测试

欢迎加入

ELK学习笔记-ES-分词

分词

分词器

分词器-调用顺序

Analyze API

中文分词

常见分词系统

基于自然语言处理的分词系统：

自定义分词：

分词的使用建议：

诈骗（杀猪盘）网站进行渗透测试

Python 潮流周刊#50：我最喜欢的 Python 3.13 新特性！

【Python】保存gym截图

【译】使用 GitHub Copilot 作为你的编码 GPS

Linux 服务器配置-安装portainer-ce社区版

外行也能读懂的网络硬件设备功能原理速成

安装Auto-GPT

ELK學習筆記-ES-分詞

elk學習筆記-es-文檔及索引操作

awk 入門教程

【Zabbix4.2學習筆記】4、數據收集-外部監查

centos 7.5 安裝docker，常用命令

https://yachay.unat.edu.pe/blog/index.php?comment_area=format_blog&comment_component=blog&comment_co

linux以太網驅動總結