來自於李宏毅老師的bert課程，http://speech.ee.ntu.edu.tw/~tlkagk/courses_ML19.html
0. 前置基礎
word can have multiple sense.
e.g. bank is word type, can be multiple word tokens
contextalized embedding: 同一個word type，在不同的上下文中，有不同的word embedding
1. ELMO
embedding from Language Model 語言模型
RNN-based language models(trained from lots of sentences)
e.g.對於某個句子，如何計算某個token的embedding？
A:將bi-RNN的某個token的前後兩個hidden states,拼接起來作爲當前token的embedding
[外鏈圖片轉存失敗,源站可能有防盜鏈機制,建議將圖片保存下來直接上傳(img-N5lOkV8M-1583759173930)(./DL_picture/bert1.png)]
2. BERT
bi-directional encoder representation for transformer
bert = encoder of transformer
備註:訓練bert是不需要label的;半監督的==其實還是有label的; learned from a large amount of text without annotaion
[外鏈圖片轉存失敗,源站可能有防盜鏈機制,建議將圖片保存下來直接上傳(img-Q1GsRRD2-1583759173938)(./DL_picture/bert2.png)]
如何訓練bert呢？
2.1 方法: masked LM
predicting the masked word ; linear multi-class classifier
線性分類器預測被masked的詞彙,該masked token輸出是一個embedding ==> 分類問題
[外鏈圖片轉存失敗,源站可能有防盜鏈機制,建議將圖片保存下來直接上傳(img-1dGnRb7X-1583759173943)(./DL_picture/bert3.png)]
2.2 方法: nexted sentence prediction
CLS: the position that outputs classification results
SEP: the boundary of two sentences
預測兩個句子是否是相連的; CLS是句子的首端,其輸出的embedding再過一個線性分類器,判斷是否相連
[外鏈圖片轉存失敗,源站可能有防盜鏈機制,建議將圖片保存下來直接上傳(img-XhcORqU0-1583759173948)(./DL_picture/bert4.png)]
有哪些應用場景呢？
2.1 情感分析,文檔分類
CLS單詞作爲輸出,再過linear classifier
2.2 Slot filling
2.3 自然語言推理
premise,hypothesis。類似應用場景2.1
2.4 問答場景
**各種場景,那些layer更有效？"
[外鏈圖片轉存失敗,源站可能有防盜鏈機制,建議將圖片保存下來直接上傳(img-zGfpqNSu-1583759173950)(./DL_picture/bert5.png)]
3. ERNIE
enhanced representation through knowledge integration
爲了中文所設計的模型
訓練ernie時,採用"masked prase"; 訓練bert時,採用"masked character"
4. GPT
generative pre-training; 生成式預訓練模型
GPT = transformer decoder; 給一個單詞輸入,可以不斷去生成一篇文章 😃
[外鏈圖片轉存失敗,源站可能有防盜鏈機制,建議將圖片保存下來直接上傳(img-9KVE6XIz-1583759173955)(./DL_picture/bert6.png)]
[外鏈圖片轉存失敗,源站可能有防盜鏈機制,建議將圖片保存下來直接上傳(img-rKxOPkBU-1583759173960)(./DL_picture/bert7.png)]
• Unified Language Model Pre-training for Natural Language Understanding and Generation
• https://arxiv.org/abs/1905.03197
• BERT has a Mouth, and It Must Speak: BERT as a Markov Random Field Language Model
• https://arxiv.org/abs/1902.04094
• Insertion Transformer: Flexible Sequence Generation via Insertion Operations
• https://arxiv.org/abs/1902.03249
• Insertion-based Decoding with automatically Inferred Generation Order
• https://arxiv.org/abs/1902.01370