小工具知识网

当前位置: 首页 > 行业知识 > 矩阵求导笔记

行业知识

矩阵求导笔记

2026-11-17 00:48:48 来源:互联网转载

文章目录

  • 1. ML中为什么需要矩阵求导
  • 2. 向量函数与矩阵求导初印象
  • 3. YX 拉伸术
    • 3.1 f(x)为标量,X为列向量
    • 3.2 f(x)为列向量,X 为标量
    • 3.3 f(x)为列向量,X 为列向量
    • 4. 常见矩阵求导公式
      • 4.1 Y = A T X Y=A^TX Y=ATX
      • 4.2 Y = X T A X Y=X^TAX Y=XTAX

        1. ML中为什么需要矩阵求导

        • 简洁

          ()

          用方程式表示如下:

          y 1 = w 1 X 11 + w 2 X 12 (1) y_1=w_1X_{11}+w_2X_{12}\tag{1} y1​=w1​X11​+w2​X12​(1)

          y 2 = w 1 X 21 + w 2 X 22 (2) y_2=w_1X_{21}+w_2X_{22}\tag{2} y2​=w1​X21​+w2​X22​(2)

          转换成矩阵表示如下:

          ()

          Y = X W (3) Y=XW\tag{3} Y=XW(3)

          Y = [ y 1 y 2 ] , X = [ x 11 x 12 x 21 x 22 ] , W = [ w 1 w 2 ] (4) Y=\begin{bmatrix}y_1\\\\y_2\end{bmatrix},X=\begin{bmatrix}x_{11}&&x_{12}\\\\x_{21}&&x_{22}\end{bmatrix},W=\begin{bmatrix}w_{1}\\\\w_{2}\end{bmatrix}\tag{4} Y= ​y1​y2​​ ​,X= ​x11​x21​​​x12​x22​​ ​,W= ​w1​w2​​ ​(4)

        • 快速

          当使用python 中的numpy 库时候,在相对于 for 循环,Numpy 本身的计算提速相当快

        • 源代码

          import timeimport numpy as npif __name__ == "__main__":    N = 1000000    a = np.random.rand(N)    b = np.random.rand(N)    start = time.time()    c = np.dot(a,b)    stop = time.time()    print(f"c={c}")    print("vectorized version: " + str(1000*(stop-start))+"ms")    c = 0    start1 = time.time()    for i in range(N):        c += a[i]*b[i]    stop1 = time.time()    print(f"c={c}")    print("for loop: " + str(1000*(stop1-start1))+"ms")    times1 = (stop1-start1)/(stop-start)    print(f"times1={times1}")
          • 结果
            c=250071.8870070607vectorized version: 6.549358367919922msc=250071.88700706122for loop: 265.43641090393066mstimes1=40.52861303239898# 向量化居然比单独的for循环快40倍

            2. 向量函数与矩阵求导初印象

            • 标量函数:输出为标量的函数

              f ( x ) = x 2 ⇒ x ∈ R → x 2 ∈ R f(x)=x^2\Rightarrow x\in R\rightarrow x^2 \in R f(x)=x2⇒x∈R→x2∈R

              f ( x ) = x 1 2 + x 2 2 ⇒ [ x 1 x 2 ] ∈ R 2 → x 1 2 + x 2 2 ∈ R f(x)=x_1^2+x_2^2\Rightarrow \begin{bmatrix}x_1\\\\x_2\end{bmatrix}\in R^2\rightarrow x_1^2+x_2^2 \in R f(x)=x12​+x22​⇒ ​x1​x2​​ ​∈R2→x12​+x22​∈R

            • 向量函数:输出为向量或矩阵的函数

              输入标量,输出向量

              f ( x ) = [ f 1 ( x ) = x f 2 ( x ) = x 2 ] ⇒ x ∈ R , [ x x 2 ] ∈ R 2 f(x)=\begin{bmatrix}f_1(x)=x\\\\f_2(x)=x^2\end{bmatrix}\Rightarrow x\in R,\begin{bmatrix}x\\\\x^2\end{bmatrix} \in R^2 f(x)= ​f1​(x)=xf2​(x)=x2​ ​⇒x∈R, ​xx2​ ​∈R2

              输入标量,输出矩阵

              f ( x ) = [ f 11 ( x ) = x f 12 ( x ) = x 2 f 21 ( x ) = x 3 f 22 ( x ) = x 4 ] ⇒ x ∈ R , [ x x 2 x 3 x 4 ] ∈ R 2 × 2 f(x)=\begin{bmatrix}f_{11}(x)=x&&f_{12}(x)=x^2\\\\f_{21}(x)=x^3&&f_{22}(x)=x^4\end{bmatrix}\Rightarrow x\in R,\begin{bmatrix}x&&x^2\\\\x^3&&x^4\end{bmatrix} \in R^{2\times2} f(x)= ​f11​(x)=xf21​(x)=x3​​f12​(x)=x2f22​(x)=x4​ ​⇒x∈R, ​xx3​​x2x4​ ​∈R2×2

              输入向量,输出矩阵

              f ( x ) = [ f 11 ( x ) = x 1 + x 2 f 12 ( x ) = x 1 2 + x 2 2 f 21 ( x ) = x 1 3 + x 2 3 f 22 ( x ) = x 1 4 + x 2 4 ] ⇒ [ x 1 x 2 ] ∈ R 2 , [ x 1 + x 2 x 1 2 + x 2 2 x 1 3 + x 2 3 x 1 4 + x 2 4 ] ∈ R 2 × 2 f(x)=\begin{bmatrix}f_{11}(x)=x_1+x_2&&f_{12}(x)=x_1^2+x_2^2\\\\f_{21}(x)=x_1^3+x_2^3&&f_{22}(x)=x_1^4+x_2^4\end{bmatrix}\Rightarrow \begin{bmatrix}x_1\\\\x_2\end{bmatrix} \in R^2,\begin{bmatrix}x_1+x_2&&x_1^2+x_2^2\\\\x_1^3+x_2^3&&x_1^4+x_2^4\end{bmatrix} \in R^{2\times2} f(x)= ​f11​(x)=x1​+x2​f21​(x)=x13​+x23​​​f12​(x)=x12​+x22​f22​(x)=x14​+x24​​ ​⇒ ​x1​x2​​ ​∈R2, ​x1​+x2​x13​+x23​​​x12​+x22​x14​+x24​​ ​∈R2×2

            • 总结

              矩阵求导的本质

              d A d B = 矩阵 A 中的每个元素对矩阵 B 中的每个元素求导 \frac{\mathrm{d}A}{\mathrm{d}B}=矩阵A中的每个元素对矩阵B中的每个元素求导 dBdA​=矩阵A中的每个元素对矩阵B中的每个元素求导

              3. YX 拉伸术

              3.1 f(x)为标量,X为列向量

              • 标量不变,向量拉伸
              • YX中,Y前面横向拉,X后面纵向拉

                d f ( x ) d x , Y = f ( x ) 为标量, X = [ x 1 x 2 ⋮ x n ] 为列向量 \frac{\mathrm{d}f(x)}{\mathrm{d}x},Y=f(x)为标量,X=\begin{bmatrix}x_1\\\\x_2\\\\\vdots\\\\x_n\end{bmatrix}为列向量 dxdf(x)​,Y=f(x)为标量,X= ​x1​x2​⋮xn​​ ​为列向量

                f ( x ) = f ( x 1 , x 2 , . . . . , x n ) 为标量 f(x)=f(x_1,x_2,....,x_n)为标量 f(x)=f(x1​,x2​,....,xn​)为标量

              • 标量 f ( x ) f(x) f(x)不变,向量X 因为在YX拉伸术中在Y后面,所以向量X纵向拉伸,实际上就是将多元函数的偏导写在一个列向量中

                d f ( x ) d x = [ ∂ f ( x ) ∂ x 1 ∂ f ( x ) ∂ x 2 ⋮ ∂ f ( x ) ∂ x n ] \frac{\mathrm{d}f(x)}{\mathrm{d}x}=\begin{bmatrix}\frac{\partial f(x)}{\partial x_1}\\\\\frac{\partial f(x)}{\partial x_2}\\\\\vdots\\\\\frac{\partial f(x)}{\partial x_n}\end{bmatrix} dxdf(x)​= ​∂x1​∂f(x)​∂x2​∂f(x)​⋮∂xn​∂f(x)​​ ​

                3.2 f(x)为列向量,X 为标量

                f ( x ) = [ f 1 ( x ) f 2 ( x ) ⋮ f n ( x ) ] ; X 为标量 f(x)=\begin{bmatrix}f_1(x)\\\\f_2(x)\\\\\vdots\\\\f_n(x)\end{bmatrix};X 为标量 f(x)= ​f1​(x)f2​(x)⋮fn​(x)​ ​;X为标量

                • 标量不变,向量拉伸
                • YX中,Y前面横向拉,X后面纵向拉

                  d f ( x ) d x = [ ∂ f 1 ( x ) ∂ x ∂ f 2 ( x ) ∂ x … ∂ f n ( x ) ∂ x ] \frac{\mathrm{d}f(x)}{\mathrm{d}x}=\begin{bmatrix}\frac{\partial f_1(x)}{\partial x}&&\frac{\partial f_2(x)}{\partial x}&&\dots&&\frac{\partial f_n(x)}{\partial x}\end{bmatrix} dxdf(x)​=[∂x∂f1​(x)​​​∂x∂f2​(x)​​​…​​∂x∂fn​(x)​​]

                  3.3 f(x)为列向量,X 为列向量

                  f ( x ) = [ f 1 ( x ) f 2 ( x ) ⋮ f n ( x ) ] ; X = [ x 1 x 2 ⋮ x n ] 为列向量 f(x)=\begin{bmatrix}f_1(x)\\\\f_2(x)\\\\\vdots\\\\f_n(x)\end{bmatrix};X=\begin{bmatrix}x_1\\\\x_2\\\\\vdots\\\\x_n\end{bmatrix}为列向量 f(x)= ​f1​(x)f2​(x)⋮fn​(x)​ ​;X= ​x1​x2​⋮xn​​ ​为列向量

                  • 第一步先固定Y ,将 X 纵向拉

                    d f ( x ) d x = [ ∂ f ( x ) ∂ x 1 ∂ f ( x ) ∂ x 2 ⋮ ∂ f ( x ) ∂ x n ] \frac{\mathrm{d}f(x)}{\mathrm{d}x}=\begin{bmatrix}\frac{\partial f(x)}{\partial x_1}\\\\\frac{\partial f(x)}{\partial x_2}\\\\\vdots\\\\\frac{\partial f(x)}{\partial x_n}\end{bmatrix} dxdf(x)​= ​∂x1​∂f(x)​∂x2​∂f(x)​⋮∂xn​∂f(x)​​ ​

                  • 第二步,看每一个项 ∂ f ( x ) ∂ x 1 \frac{\partial f(x)}{\partial x_1} ∂x1​∂f(x)​,其中f(x)为列向量, x 1 x_1 x1​为标量,那么可以看出要进行 Y 横向拉

                    ∂ f ( x ) ∂ x 1 = [ ∂ f 1 ( x ) ∂ x 1 ∂ f 2 ( x ) ∂ x 1 … ∂ f n ( x ) ∂ x 1 ] \frac{\partial f(x)}{\partial x_1}=\begin{bmatrix}\frac{\partial f_1(x)}{\partial x_1}&&\frac{\partial f_2(x)}{\partial x_1}&&\dots&&\frac{\partial f_n(x)}{\partial x_1}\end{bmatrix} ∂x1​∂f(x)​=[∂x1​∂f1​(x)​​​∂x1​∂f2​(x)​​​…​​∂x1​∂fn​(x)​​]

                  • 第三步 ,将每项整合如下

                    d f ( x ) d x = [ ∂ f 1 ( x ) ∂ x 1 ∂ f 2 ( x ) ∂ x 1 … ∂ f n ( x ) ∂ x 1 ∂ f 1 ( x ) ∂ x 2 ∂ f 2 ( x ) ∂ x 2 … ∂ f n ( x ) ∂ x 2 ⋮ ⋮ … ⋮ ∂ f 1 ( x ) ∂ x n ∂ f 2 ( x ) ∂ x n … ∂ f n ( x ) ∂ x n ] \frac{\mathrm{d}f(x)}{\mathrm{d}x}=\begin{bmatrix}\frac{\partial f_1(x)}{\partial x_1}&&\frac{\partial f_2(x)}{\partial x_1}&&\dots&&\frac{\partial f_n(x)}{\partial x_1}\\\\\frac{\partial f_1(x)}{\partial x_2}&&\frac{\partial f_2(x)}{\partial x_2}&&\dots&&\frac{\partial f_n(x)}{\partial x_2}\\\\\vdots&&\vdots&&\dots&&\vdots\\\\\frac{\partial f_1(x)}{\partial x_n}&&\frac{\partial f_2(x)}{\partial x_n}&&\dots&&\frac{\partial f_n(x)}{\partial x_n}\end{bmatrix} dxdf(x)​= ​∂x1​∂f1​(x)​∂x2​∂f1​(x)​⋮∂xn​∂f1​(x)​​​∂x1​∂f2​(x)​∂x2​∂f2​(x)​⋮∂xn​∂f2​(x)​​​…………​​∂x1​∂fn​(x)​∂x2​∂fn​(x)​⋮∂xn​∂fn​(x)​​ ​

                    4. 常见矩阵求导公式

                    4.1 Y = A T X Y=A^TX Y=ATX

                    f ( x ) = A T X ; A = [ a 1 , a 2 , … , a n ] T ; X = [ x 1 , x 2 , … , x n ] T , 求 d f ( x ) d X f(x)=A^TX;\quad A=[a_1,a_2,\dots,a_n]^T;\quad X=[x_1,x_2,\dots,x_n]^T,求\frac{\mathrm{d}f(x)}{\mathrm{d}X} f(x)=ATX;A=[a1​,a2​,…,an​]T;X=[x1​,x2​,…,xn​]T,求dXdf(x)​

                    • 由于 A T = 1 × n , X = n × 1 , 那么 f ( x ) 为标量,即表示数值 A^T=1\times n,X=n\times1,那么f(x)为标量,即表示数值 AT=1×n,X=n×1,那么f(x)为标量,即表示数值,
                    • 标量不变,向量拉伸
                    • YX中,Y前面横向拉,X后面纵向拉

                      f ( x ) = ∑ i = 1 N a i x i f(x)=\sum_{i=1}^Na_ix_i f(x)=i=1∑N​ai​xi​

                      d f ( x ) d X = [ ∂ f ( x ) ∂ x 1 ∂ f ( x ) ∂ x 2 ⋮ ∂ f ( x ) ∂ x n ] \frac{\mathrm{d}f(x)}{\mathrm{d}X}=\begin{bmatrix}\frac{\partial f(x)}{\partial x_1}\\\\\frac{\partial f(x)}{\partial x_2}\\\\\vdots\\\\\frac{\partial f(x)}{\partial x_n}\end{bmatrix} dXdf(x)​= ​∂x1​∂f(x)​∂x2​∂f(x)​⋮∂xn​∂f(x)​​ ​

                    • 可以计算 ∂ f ( x ) ∂ x i \frac{\partial f(x)}{\partial x_i} ∂xi​∂f(x)​

                      ∂ f ( x ) ∂ x i = a i \frac{\partial f(x)}{\partial x_i}=a_i ∂xi​∂f(x)​=ai​

                    • 可得如下:

                      d f ( x ) d X = [ a 1 a 2 ⋮ a n ] = A \frac{\mathrm{d}f(x)}{\mathrm{d}X}=\begin{bmatrix}a_1\\\\a_2\\\\\vdots\\\\a_n\end{bmatrix}=A dXdf(x)​= ​a1​a2​⋮an​​ ​=A

                    • 结论:

                      当 f ( x ) = A T X 当f(x)=A^TX 当f(x)=ATX

                      d f ( x ) d X = A \frac{\mathrm{d}f(x)}{\mathrm{d}X}=A dXdf(x)​=A

                      4.2 Y = X T A X Y=X^TAX Y=XTAX

                      f ( x ) = X T A X ; A = [ a 11 a 12 … a 1 n a 21 a 22 … a 2 n ⋮ ⋮ … ⋮ a n 1 a n 2 … a n n ] ; X = [ x 1 , x 2 , … , x n ] T , 求 d f ( x ) d X f(x)=X^TAX;\quad A=\begin{bmatrix}a_{11}&&a_{12}&&\dots&&a_{1n}\\\\a_{21}&&a_{22}&&\dots&&a_{2n}\\\\\vdots&&\vdots&&\dots&&\vdots\\\\a_{n1}&&a_{n2}&&\dots&&a_{nn}\end{bmatrix};\quad X=[x_1,x_2,\dots,x_n]^T,求\frac{\mathrm{d}f(x)}{\mathrm{d}X} f(x)=XTAX;A= ​a11​a21​⋮an1​​​a12​a22​⋮an2​​​…………​​a1n​a2n​⋮ann​​ ​;X=[x1​,x2​,…,xn​]T,求dXdf(x)​

上一篇:揭秘网址前缀的秘密,如何正确使用网址前缀?

下一篇:家政服务谈判艺术,巧妙讲价秘籍大揭秘!