行业知识
深度强化学习（DRL）算法 附录 6 —— NLP 回顾之预训练模型篇
Self-Attention
模型结构
上��架构以 batch_size 为 1，两个时间步的 X 为例子，计算过程如下：
位置编码
根据 self-attention 的模型结构，改变 X 的输入顺序，不影响 attention 的结果，所以还需要引入额外的位置信息，即位置编码。
假设输入表示 X ∈ R n × d 包含一个序列中 n 个词元的 d 维嵌入表示。位置编码使用相同形状的位置 嵌入矩阵 P ∈ R n × d 输出 X + P ，矩阵第 i 行、第 2 j 列和列 2 j + 1 上的元素为： \text { 假设输入表示 } \mathbf{X} \in \mathbb{R}^{n \times d} \text { 包含一个序列中 } n \text { 个词元的 } d \text { 维嵌入表示。位置编码使用相同形状的位置 } 嵌入矩阵 \mathbf{P} \in \mathbb{R}^{n \times d} 输出\mathbf{X}+\mathbf{P}， 矩阵第 i行、第 2j 列和列 2j+1 上的元素为： 假设输入表示 X∈Rn×d 包含一个序列中 n 个词元的 d 维嵌入表示。位置编码使用相同形状的位置 嵌入矩阵P∈Rn×d输出X+P，矩阵第i行、第2j列和列2j+1上的元素为：
p i , 2 j = sin ( i 1000 0 2 j / d ) p i , 2 j + 1 = cos ( i 1000 0 2 j / d ) \begin{aligned} p_{i, 2 j} & =\sin \left(\frac{i}{10000^{2 j / d}}\right) \\ p_{i, 2 j+1} & =\cos \left(\frac{i}{10000^{2 j / d}}\right) \end{aligned} pi,2jpi,2j+1=sin(100002j/di)=cos(100002j/di)
图里计算机二进制编码的低位和位置编码矩阵的前面几列对应。
除了上面捕获绝对位置信息之外，上述的位置编码还允许模型学习得到输入序列中相对位置信息。 这是因为对于任何确定的位置偏移δ，位置 i+δ 处的位置编码可以线性投影位置 i 处的位置编码来表示。
[ cos ( δ ω j ) sin ( δ ω j ) − sin ( δ ω j ) cos ( δ ω j ) ] [ p i , 2 j p i , 2 j + 1 ] = [ cos ( δ ω j ) sin ( i ω j ) + sin ( δ ω j ) cos ( i ω j ) − sin ( δ ω j ) sin ( i ω j ) + cos ( δ ω j ) cos ( i ω j ) ] = [ sin ( ( i + δ ) ω j ) cos ( ( i + δ ) ω j ) ] = [ p i + δ , 2 j p i + δ , 2 j + 1 ] \begin{aligned} & {\left[\begin{array}{cc} \cos \left(\delta \omega_j\right) & \sin \left(\delta \omega_j\right) \\ -\sin \left(\delta \omega_j\right) & \cos \left(\delta \omega_j\right) \end{array}\right]\left[\begin{array}{c} p_{i, 2 j} \\ p_{i, 2 j+1} \end{array}\right] } \\ = & {\left[\begin{array}{c} \cos \left(\delta \omega_j\right) \sin \left(i \omega_j\right)+\sin \left(\delta \omega_j\right) \cos \left(i \omega_j\right) \\ -\sin \left(\delta \omega_j\right) \sin \left(i \omega_j\right)+\cos \left(\delta \omega_j\right) \cos \left(i \omega_j\right) \end{array}\right] } \\ = & {\left[\begin{array}{l} \sin \left((i+\delta) \omega_j\right) \\ \cos \left((i+\delta) \omega_j\right) \end{array}\right] } \\ = & {\left[\begin{array}{c} p_{i+\delta, 2 j} \\ p_{i+\delta, 2 j+1} \end{array}\right] } \end{aligned} ===[cos(δωj)−sin(δωj)sin(δωj)cos(δωj)][pi,2jpi,2j+1][cos(δωj)sin(iωj)+sin(δωj)cos(iωj)−sin(δωj)sin(iωj)+cos(δωj)cos(iωj)][sin((i+δ)ωj)cos((i+δ)ωj)][pi+δ,2jpi+δ,2j+1]
代码
#@saveclass PositionalEncoding(nn.Module): """位置编码""" def __init__(self, num_hiddens, dropout, max_len=1000): super(PositionalEncoding, self).__init__() self.dropout = nn.Dropout(dropout) # 创建一个足够长的P self.P = torch.zeros((1, max_len, num_hiddens)) X = torch.arange(max_len, dtype=torch.float32).reshape( -1, 1) / torch.pow(10000, torch.arange( 0, num_hiddens, 2, dtype=torch.float32) / num_hiddens) self.P[:, :, 0::2] = torch.sin(X) self.P[:, :, 1::2] = torch.cos(X) def forward(self, X): X = X + self.P[:, :X.shape[1], :].to(X.device) return self.dropout(X)
多头注意力
模型结构
两头注意力
七头注意力
七头注意力连接进行信息融合
掩码多头注意力
和多头注意力是一样的，只不过每个头的 self-attention 变成了 masked self-attention
代码
import mathimport torchfrom torch import nnfrom d2l import torch as d2l#@savedef transpose_qkv(X, num_heads): """为了多注意力头的并行计算而变换形状""" # 输入X的形状:(batch_size，查询或者“键－值”对的个数，num_hiddens) # 输出X的形状:(batch_size，查询或者“键－值”对的个数，num_heads， # num_hiddens/num_heads) X = X.reshape(X.shape[0], X.shape[1], num_heads, -1) # 输出X的形状:(batch_size，num_heads，查询或者“键－值”对的个数, # num_hiddens/num_heads) X = X.permute(0, 2, 1, 3) # 最终输出的形状:(batch_size*num_heads,查询或者“键－值”对的个数, # num_hiddens/num_heads) return X.reshape(-1, X.shape[2], X.shape[3])#@savedef transpose_output(X, num_heads): """逆转transpose_qkv函数的操作""" X = X.reshape(-1, num_heads, X.shape[1], X.shape[2]) X = X.permute(0, 2, 1, 3) return X.reshape(X.shape[0], X.shape[1], -1)#@saveclass DotProductAttention(nn.Module): """Scaled dot product attention. Defined in :numref:`subsec_additive-attention`""" def __init__(self, dropout, **kwargs): super(DotProductAttention, self).__init__(**kwargs) self.dropout = nn.Dropout(dropout) # Shape of `queries`: (`batch_size`, no. of queries, `d`) # Shape of `keys`: (`batch_size`, no. of key-value pairs, `d`) # Shape of `values`: (`batch_size`, no. of key-value pairs, value # dimension) # Shape of `valid_lens`: (`batch_size`,) or (`batch_size`, no. of queries) def forward(self, queries, keys, values, valid_lens=None): d = queries.shape[-1] # Set `transpose_b=True` to swap the last two dimensions of `keys` scores = torch.bmm(queries, keys.transpose(1,2)) / math.sqrt(d) self.attention_weights = masked_softmax(scores, valid_lens) return torch.bmm(self.dropout(self.attention_weights), values)#@saveclass MultiHeadAttention(nn.Module): """多头注意力""" def __init__(self, key_size, query_size, value_size, num_hiddens, num_heads, dropout, bias=False, **kwargs): super(MultiHeadAttention, self).__init__(**kwargs) self.num_heads = num_heads self.attention = d2l.DotProductAttention(dropout) self.W_q = nn.Linear(query_size, num_hiddens, bias=bias) self.W_k = nn.Linear(key_size, num_hiddens, bias=bias) self.W_v = nn.Linear(value_size, num_hiddens, bias=bias) self.W_o = nn.Linear(num_hiddens, num_hiddens, bias=bias) def forward(self, queries, keys, values, valid_lens): # queries，keys，values的形状: # (batch_size，查询或者“键－值”对的个数，num_hiddens) # valid_lens 的形状: # (batch_size，)或(batch_size，查询的个数) # 经过变换后，输出的queries，keys，values 的形状: # (batch_size*num_heads，查询或者“键－值”对的个数， # num_hiddens/num_heads) queries = transpose_qkv(self.W_q(queries), self.num_heads) keys = transpose_qkv(self.W_k(keys), self.num_heads) values = transpose_qkv(self.W_v(values), self.num_heads) if valid_lens is not None: # 在轴0，将第一项（标量或者矢量）复制num_heads次， # 然后如此复制第二项，然后诸如此类。 valid_lens = torch.repeat_interleave( valid_lens, repeats=self.num_heads, dim=0) # output的形状:(batch_size*num_heads，查询的个数， # num_hiddens/num_heads) output = self.attention(queries, keys, values, valid_lens) # output_concat的形状:(batch_size，查询的个数，num_hiddens) output_concat = transpose_output(output, self.num_heads) return self.W_o(output_concat)
Transformer
模型结构
具体的一个 encoder 块内部：
decoder:
代码
import mathimport pandas as pdimport torchfrom torch import nnfrom d2l import torch as d2l#@saveclass PositionWiseFFN(nn.Module): """基于位置的前馈网络""" def __init__(self, ffn_num_input, ffn_num_hiddens, ffn_num_outputs, **kwargs): super(PositionWiseFFN, self).__init__(**kwargs) self.dense1 = nn.Linear(ffn_num_input, ffn_num_hiddens) self.relu = nn.ReLU() self.dense2 = nn.Linear(ffn_num_hiddens, ffn_num_outputs) def forward(self, X): return self.dense2(self.relu(self.dense1(X)))#@saveclass AddNorm(nn.Module): """残差连接后进行层规范化""" def __init__(self, normalized_shape, dropout, **kwargs): super(AddNorm, self).__init__(**kwargs) self.dropout = nn.Dropout(dropout) self.ln = nn.LayerNorm(normalized_shape) def forward(self, X, Y): return self.ln(self.dropout(Y) + X)#@saveclass EncoderBlock(nn.Module): """Transformer编码器块""" def __init__(self, key_size, query_size, value_size, num_hiddens, norm_shape, ffn_num_input, ffn_num_hiddens, num_heads, dropout, use_bias=False, **kwargs): super(EncoderBlock, self).__init__(**kwargs) self.attention = d2l.MultiHeadAttention( key_size, query_size, value_size, num_hiddens, num_heads, dropout, use_bias) self.addnorm1 = AddNorm(norm_shape, dropout) self.ffn = PositionWiseFFN( ffn_num_input, ffn_num_hiddens, num_hiddens) self.addnorm2 = AddNorm(norm_shape, dropout) def forward(self, X, valid_lens): Y = self.addnorm1(X, self.attention(X, X, X, valid_lens)) return self.addnorm2(Y, self.ffn(Y))#@saveclass TransformerEncoder(d2l.Encoder): """Transformer编码器""" def __init__(self, vocab_size, key_size, query_size, value_size, num_hiddens, norm_shape, ffn_num_input, ffn_num_hiddens, num_heads, num_layers, dropout, use_bias=False, **kwargs): super(TransformerEncoder, self).__init__(**kwargs) self.num_hiddens = num_hiddens self.embedding = nn.Embedding(vocab_size, num_hiddens) self.pos_encoding = d2l.PositionalEncoding(num_hiddens, dropout) self.blks = nn.Sequential() for i in range(num_layers): self.blks.add_module("block"+str(i), EncoderBlock(key_size, query_size, value_size, num_hiddens, norm_shape, ffn_num_input, ffn_num_hiddens, num_heads, dropout, use_bias)) def forward(self, X, valid_lens, *args): # 因为位置编码值在-1和1之间， # 因此嵌入值乘以嵌入维度的平方根进行缩放， # 然后再与位置编码相加。 X = self.pos_encoding(self.embedding(X) * math.sqrt(self.num_hiddens)) self.attention_weights = [None] * len(self.blks) for i, blk in enumerate(self.blks): X = blk(X, valid_lens) self.attention_weights[ i] = blk.attention.attention.attention_weights return Xclass DecoderBlock(nn.Module): """解码器中第i个块""" def __init__(self, key_size, query_size, value_size, num_hiddens, norm_shape, ffn_num_input, ffn_num_hiddens, num_heads, dropout, i, **kwargs): super(DecoderBlock, self).__init__(**kwargs) self.i = i self.attention1 = d2l.MultiHeadAttention( key_size, query_size, value_size, num_hiddens, num_heads, dropout) self.addnorm1 = AddNorm(norm_shape, dropout) self.attention2 = d2l.MultiHeadAttention( key_size, query_size, value_size, num_hiddens, num_heads, dropout) self.addnorm2 = AddNorm(norm_shape, dropout) self.ffn = PositionWiseFFN(ffn_num_input, ffn_num_hiddens, num_hiddens) self.addnorm3 = AddNorm(norm_shape, dropout) def forward(self, X, state): enc_outputs, enc_valid_lens = state[0], state[1] # 训练阶段，输出序列的所有词元都在同一时间处理， # 因此state[2][self.i]初始化为None。 # 预测阶段，输出序列是通过词元一个接着一个解码的， # 因此state[2][self.i]包含着直到当前时间步第i个块解码的输出表示 if state[2][self.i] is None: key_values = X else: key_values = torch.cat((state[2][self.i], X), axis=1) state[2][self.i] = key_values if self.training: batch_size, num_steps, _ = X.shape # dec_valid_lens的开头:(batch_size,num_steps), # 其中每一行是[1,2,...,num_steps] dec_valid_lens = torch.arange( 1, num_steps + 1, device=X.device).repeat(batch_size, 1) else: dec_valid_lens = None # 自注意力 X2 = self.attention1(X, key_values, key_values, dec_valid_lens) Y = self.addnorm1(X, X2) # 编码器－解码器注意力。 # enc_outputs的开头:(batch_size,num_steps,num_hiddens) Y2 = self.attention2(Y, enc_outputs, enc_outputs, enc_valid_lens) Z = self.addnorm2(Y, Y2) return self.addnorm3(Z, self.ffn(Z)), stateclass TransformerDecoder(d2l.AttentionDecoder): def __init__(self, vocab_size, key_size, query_size, value_size, num_hiddens, norm_shape, ffn_num_input, ffn_num_hiddens, num_heads, num_layers, dropout, **kwargs): super(TransformerDecoder, self).__init__(**kwargs) self.num_hiddens = num_hiddens self.num_layers = num_layers self.embedding = nn.Embedding(vocab_size, num_hiddens) self.pos_encoding = d2l.PositionalEncoding(num_hiddens, dropout) self.blks = nn.Sequential() for i in range(num_layers): self.blks.add_module("block"+str(i), DecoderBlock(key_size, query_size, value_size, num_hiddens, norm_shape, ffn_num_input, ffn_num_hiddens, num_heads, dropout, i)) self.dense = nn.Linear(num_hiddens, vocab_size) def init_state(self, enc_outputs, enc_valid_lens, *args): return [enc_outputs, enc_valid_lens, [None] * self.num_layers] def forward(self, X, state): X = self.pos_encoding(self.embedding(X) * math.sqrt(self.num_hiddens)) self._attention_weights = [[None] * len(self.blks) for _ in range (2)] for i, blk in enumerate(self.blks): X, state = blk(X, state) # 解码器自注意力权重 self._attention_weights[0][ i] = blk.attention1.attention.attention_weights # “编码器－解码器”自注意力权重 self._attention_weights[1][ i] = blk.attention2.attention.attention_weights return self.dense(X), state @property def attention_weights(self): return self._attention_weightsnum_hiddens, num_layers, dropout, batch_size, num_steps = 32, 2, 0.1, 64, 10lr, num_epochs, device = 0.005, 200, d2l.try_gpu()ffn_num_input, ffn_num_hiddens, num_heads = 32, 64, 4key_size, query_size, value_size = 32, 32, 32norm_shape = [32]train_iter, src_vocab, tgt_vocab = d2l.load_data_nmt(batch_size, num_steps)encoder = TransformerEncoder( len(src_vocab), key_size, query_size, value_size, num_hiddens, norm_shape, ffn_num_input, ffn_num_hiddens, num_heads, num_layers, dropout)decoder = TransformerDecoder( len(tgt_vocab), key_size, query_size, value_size, num_hiddens, norm_shape, ffn_num_input, ffn_num_hiddens, num_heads, num_layers, dropout)net = d2l.EncoderDecoder(encoder, decoder)d2l.train_seq2seq(net, train_iter, lr, num_epochs, tgt_vocab, device) Bert
bert 开启了预训练模型的风潮，使用了带掩码的语言模型，具体就是通过大量的数据，模型获取了语言信息抽取的能力，从而可以通过 fine-tune 应用到各种 NLP 任务上。
3w 的词典，使用了 WordPiece。[cls] A [seq] B [seq]
位置嵌入换成了学习的矩阵。
模型结构
截取了 transformer 的 encoder（代码没有改动）
不同点：
输入
训练（类似完形填空，以及下一个句子预测）
尽管掩蔽语言建模能够编码双向上下文来表示单词，但它不能显式地建模文本对之间的逻辑关系。为了帮助理解两个文本序列之间的关系，BERT在预训练中考虑了一个二元分类任务——下一句预测。在为预训练生成句子对时，有一半的时间它们确实是标签为“真”的连续句子；在另一半的时间里，第二个句子是从语料库中随机抽取的，标记为“假”。
模型参数计算
BERT-base（H = 768，L = 12，A = 12）
Transformer encoder block 里面主要参数有：
嵌入层：H x 30000（vocab_size 约等于 30000）
2. 全连接层：H x 4H + 4H x H（一个 block 里面有两个全连接层）
3. 多头注意力机制层：H x H / head_num x 3（一个头的参数，3代表 Q，K，V 用不同矩阵做线性变换），所有头加起来 H x H x 3，再加上多头注意力机制层的线性变换 H x H，这里可以结合下图理解：
1，2，3 加起来就是 BERT-base 的参数数量。
计算公式： L ∗ 12 H 2 + 30000 ∗ H ≈ 110 M ( H = 768 , L = 12 ) L*12H^{2} + 30000*H \approx 110M (H=768, L=12) L∗12H2+30000∗H≈110M(H=768,L=12)
BERT-large 同理可以计算出参数数量约等于 340M。
GPT-3
截取了 transformer 的 decoder（代码没有改动）
参考
- https://www.bilibili.com/list/1567748478?sid=358497&spm_id_from=333.999.0.0&desc=0&oid=974353636&bvid=BV1L44y1m768&p=2
- 8.1. 序列模型 - 动手学深度学习 2.0.0 documentation
- Visualizing A Neural Machine Translation Model (Mechanics of Seq2seq Models With Attention)
- The Illustrated Transformer
- https://www.coursera.org/learn/attention-models-in-nlp/lecture/AMz8y/masked-self-attention
- BERT 论文逐段精读【论文精读】_哔哩哔哩_bilibili
- GPT，GPT-2，GPT-3 论文精读【论文精读】_哔哩哔哩_bilibili
