[Math] 矩陣微分

數學知識：矩陣微分
線代啟示錄
Wiki Matrix calculus

簡介：矩陣微分定義與基本性質

定義：scalar by vector 的導數

假設 $f$ 為 function，且擁有 $p$ 個獨立變數 $x_1,x_2,\cdots ,x_p$
令 $\mathbf{x}=[x_1,x_2,\cdots ,x_p{]}^T$
$\frac{\partial f}{\partial \mathbf{x}}={[\frac{\partial f}{\partial x_1},\frac{\partial f}{\partial x_2},\cdots ,\frac{\partial f}{\partial x_p}]}^T$

定理(1)：$\frac{\partial {\mathbf{x}}^T\mathbf{x}}{\partial \mathbf{x}}=2\mathbf{x}$

$Proof:$ $$\begin{array}{rl}\frac{\partial {\mathbf{x}}^T\mathbf{x}}{\partial \mathbf{x}}& =[\frac{\partial {\mathbf{x}}^T\mathbf{x}}{\partial x_1},\frac{\partial {\mathbf{x}}^T\mathbf{x}}{\partial x_1},\frac{\partial {\mathbf{x}}^T\mathbf{x}}{\partial x_p}{]}^T\\ & =[\frac{\partial \sum _{i=1}^n{x}_i^2}{\partial x_1},\frac{\partial \sum _{i=1}^n{x}_i^2}{\partial x_2},\cdots ,\frac{\partial \sum _{i=1}^n{x}_i^2}{\partial x_p}{]}^T\\ & =[2x_1,2x_2,\cdots ,2x_p{]}^T\\ & =2[x_1,x_2,\cdots ,x_p{]}^T\\ & =2\mathbf{x}\end{array}$$

定理(2)：$\frac{\partial {\mathbf{x}}^T\mathbf{y}}{\partial \mathbf{x}}=\mathbf{y}\iff \frac{\partial xy}{\partial x}=y$

$Proof:$ $$\begin{array}{rl}\frac{\partial {\mathbf{x}}^T\mathbf{y}}{\partial \mathbf{x}}& =[\frac{\partial {\mathbf{x}}^T\mathbf{y}}{\partial x_1},\frac{\partial {\mathbf{x}}^T\mathbf{y}}{\partial x_2},\cdots ,\frac{\partial {\mathbf{x}}^T\mathbf{y}}{\partial x_p}]\\ & =[\frac{\partial \sum _{i=1}^p{x}_i{y}_i}{\partial x_1},\frac{\partial \sum _{i=1}^p{x}_i{y}_i}{\partial x_2},\cdots ,\frac{\partial \sum _{i=1}^p{x}_i{y}_i}{\partial x_p}{]}^T\\ & =[y_1,y_2,\cdots ,y_p{]}^T\\ & =\mathbf{y}\end{array}$$

定理(3)：$\frac{\partial {\mathbf{A}}^T\mathbf{x}}{\partial \mathbf{x}}=\mathbf{A}\iff \frac{\partial ax}{\partial x}=a$

$Proof:$ $$\begin{array}{rl}Let\mathbf{A}& =[a_{ij}{]}_{p\times p}\\ \frac{\partial \mathbf{Ax}}{\partial \mathbf{x}}& =\frac{\partial }{\partial \mathbf{x}}\mathbf{Ax}\\ & =\frac{\partial }{\partial \mathbf{x}}{[\sum _{j=1}^p{a}_{1j}{x}_j,\sum _{j=1}^p{a}_{2j}{x}_j,\cdots ,\sum _{j=1}^p{a}_{pj}{x}_j]}^T\\ & =\left[\begin{array}{c}\frac{\partial }{\partial \mathbf{x}}\sum _{j=1}^p{a}_{1j}{x}_j\\ \frac{\partial }{\partial \mathbf{x}}\sum _{j=1}^p{a}_{2j}{x}_j\\ ⋮\\ \frac{\partial }{\partial \mathbf{x}}\sum _{j=1}^p{a}_{pj}{x}_j\end{array}\right]\\ & =\left[\begin{array}{cccc}\frac{\partial }{\partial x_1}\sum _{j=1}^p{a}_{1j}{x}_j& \frac{\partial }{\partial x_2}\sum _{j=1}^p{a}_{1j}{x}_j& \cdots & \frac{\partial }{\partial x_p}\sum _{j=1}^p{a}_{1j}{x}_j\\ \frac{\partial }{\partial x_1}\sum _{j=1}^p{a}_{2j}{x}_j& \frac{\partial }{\partial x_2}\sum _{j=1}^p{a}_{2j}{x}_j& \cdots & \frac{\partial }{\partial x_p}\sum _{j=1}^p{a}_{2j}{x}_j\\ ⋮& ⋮& \ddots & ⋮\\ \frac{\partial }{\partial x_1}\sum _{j=1}^p{a}_{pj}{x}_j& \frac{\partial }{\partial x_2}\sum _{j=1}^p{a}_{pj}{x}_j& \cdots & \frac{\partial }{\partial x_p}\sum _{j=1}^p{a}_{pj}{x}_j\end{array}\right]\\ & =\mathbf{A}\end{array}$$

定理(4)：$(i)\frac{\partial {\mathbf{x}}^T\mathbf{A}\mathbf{x}}{\partial \mathbf{x}}=(\mathbf{A}+{\mathbf{A}}^T)\mathbf{x}(ii)If\mathbf{A}={\mathbf{A}}^T\Rightarrow \frac{\partial {\mathbf{x}}^T\mathbf{A}\mathbf{x}}{\partial \mathbf{x}}=2\mathbf{A}\mathbf{x}\iff \frac{\partial }{\partial x}a{x}^2=2x$

$Proof:$ $$\begin{array}{rl}Let\mathbf{A}& =[a_{ij}{]}_{p\times p}\\ f(\mathbf{x})& ={\mathbf{x}}^T\mathbf{A}\mathbf{x}=\sum _{i=1}^p\sum _{j=1}^p{x}_i{a}_{ij}{x}_j\\ \frac{\partial f}{\partial x_i}& =\frac{\partial }{\partial x_i}\left[\sum _{i=1}^p\sum _{j=1}^p{x}_i{a}_{ij}{x}_j\right]\\ & =\sum _{j=1}^p{a}_{ij}{x}_j+\sum _{j=1}^p{x}_j{a}_{ji}\\ & =\sum _{j=1}^p{a}_{ij}{x}_j+\sum _{j=1}^p{a}_{ji}{x}_j\\ \Rightarrow & \frac{\partial f}{\partial \mathbf{x}}\\ & =\mathbf{A}\mathbf{x}+{\mathbf{A}}^T\mathbf{x}\\ & =(\mathbf{A}+{\mathbf{A}}^T)\mathbf{x}\end{array}$$ 乘法律：
若 $h(x)=f(x)g(x)$，$f(x)$ 和 $g(x)$ 在 $x$ 都是可微
$h^{\prime }(x)=f^{\prime }(x)g(x)+f(x)g^{\prime }(x)$

定義：scalar by matrix 的導數

假設 $f$ 為 function，且擁有 $m\times n$ matrix $\mathbf{X}$ 變數 $${\mathbf{X}}_{\mathbf{m}\times \mathbf{n}}=\left[\begin{array}{cccc}{x}_{11}& x_{12}& \cdots & x_{1n}\\ x_{21}& x_{22}& \cdots & x_{2n}\\ ⋮& ⋮& \ddots & ⋮\\ x_{m1}& x_{m2}& \cdots & x_{mn}\end{array}\right]$$
假設所有 $\frac{\partial f}{\partial x_{ij}}$ 皆存在
$$\frac{\partial f}{\partial \mathbf{X}}={\left[\begin{array}{cccc}\frac{\partial f}{\partial x_{11}}& \frac{\partial f}{\partial x_{12}}& \cdots & \frac{\partial f}{\partial x_{1n}}\\ \frac{\partial f}{\partial x_{21}}& \frac{\partial f}{\partial x_{22}}& \cdots & \frac{\partial f}{\partial x_{2n}}\\ ⋮& ⋮& \ddots & ⋮\\ \frac{\partial f}{\partial x_{m1}}& \frac{\partial f}{\partial x_{m2}}& \cdots & \frac{\partial f}{\partial x_{mn}}\end{array}\right]}_{m\times n}$$

定義：matrix by scalar 的導數

假設 $\mathbf{F}$ 為 matrix function，且擁有 $x$ 變數
且所有 $\frac{\partial f_{ij}}{\partial x}$ 皆存在
$$\frac{\partial \mathbf{F}}{\partial x}={\left[\begin{array}{cccc}\frac{\partial f_{11}}{\partial x}& \frac{\partial f_{12}}{\partial x}& \cdots & \frac{\partial f_{1n}}{\partial x}\\ \frac{\partial f_{21}}{\partial x}& \frac{\partial f_{22}}{\partial x}& \cdots & \frac{\partial f_{2n}}{\partial x}\\ ⋮& ⋮& \ddots & ⋮\\ \frac{\partial f_{m1}}{\partial x}& \frac{\partial f_{m2}}{\partial x}& \cdots & \frac{\partial f_{mn}}{\partial x}\end{array}\right]}_{m\times n}$$

定理(5)：${\mathbf{X}}_{n\times p}\Rightarrow \frac{\partial tr({\mathbf{X}}^T\mathbf{X})}{\partial \mathbf{X}}=2\mathbf{X}$

$Proof:$ $$\begin{array}{rl}{\mathbf{X}}^T\mathbf{X}& ={\left[\begin{array}{cccc}{x}_{11}& x_{12}& \cdots & x_{1p}\\ x_{21}& x_{22}& \cdots & x_{2p}\\ ⋮& ⋮& \ddots & ⋮\\ x_{n1}& x_{n2}& \cdots & x_{np}\end{array}\right]}^T\left[\begin{array}{cccc}{x}_{11}& x_{12}& \cdots & x_{1p}\\ x_{21}& x_{22}& \cdots & x_{2p}\\ ⋮& ⋮& \ddots & ⋮\\ x_{n1}& x_{n2}& \cdots & x_{np}\end{array}\right]\\ & =\left[\begin{array}{cccc}\sum _{i=1}^n{x}_{i1}{x}_{i1}& \sum _{i=1}^n{x}_{i1}{x}_{i2}& \cdots & \sum _{i=1}^n{x}_{i1}{x}_{ip}\\ \sum _{i=1}^n{x}_{i2}{x}_{i1}& \sum _{i=1}^n{x}_{i2}{x}_{i2}& \cdots & \sum _{i=1}^n{x}_{i2}{x}_{ip}\\ ⋮& ⋮& \ddots & ⋮\\ \sum _{i=1}^n{x}_{ip}{x}_{i1}& \sum _{i=1}^n{x}_{ip}{x}_{i2}& \cdots & \sum _{i=1}^n{x}_{ip}{x}_{ip}\end{array}\right]\\ \\ tr({\mathbf{X}}^T\mathbf{X})& =(\sum _{i=1}^n{x}_{i1}{x}_{i1}+\sum _{i=1}^n{x}_{i2}{x}_{i2}+\cdots +\sum _{i=1}^n{x}_{ip}{x}_{ip})\\ & =\sum _{i=1}^n\sum _{j=1}^p{x}_{ij}^2\\ \\ \frac{\partial tr({\mathbf{X}}^T\mathbf{X})}{\partial x_{ij}}& =\frac{\partial \sum _{i=1}^n\sum _{j=1}^p{x}_{ij}^2}{\partial x_{ij}}\\ & =2x_{ij}\\ \Rightarrow \frac{\partial tr({\mathbf{X}}^T\mathbf{X})}{\partial \mathbf{X}}& =2\mathbf{X}\end{array}$$

定理(6)：${\mathbf{A}}_{n\times p},{\mathbf{X}}_{n\times p}\Rightarrow \frac{\partial tr({\mathbf{A}}^T\mathbf{X})}{\partial \mathbf{X}}=\mathbf{A}$

$Proof:$ $$\begin{array}{rl}{\mathbf{A}}^T\mathbf{X}& =\left[\begin{array}{cccc}{a}_{11}& a_{21}& \cdots & a_{n1}\\ a_{12}& a_{22}& \cdots & a_{n1}\\ ⋮& ⋮& \ddots & ⋮\\ a_{1p}& a_{2p}& \cdots & a_{np}\end{array}\right]\left[\begin{array}{cccc}{x}_{11}& x_{12}& \cdots & x_{1p}\\ x_{21}& x_{22}& \cdots & x_{2p}\\ ⋮& ⋮& \ddots & ⋮\\ x_{n1}& x_{n2}& \cdots & x_{np}\end{array}\right]\\ & =\left[\begin{array}{cccc}\sum _{i=1}^n{a}_{i1}{x}_{i1}& \sum _{i=1}^n{a}_{i1}{x}_{i2}& \cdots & \sum _{i=1}^n{a}_{i1}{x}_{ip}\\ \sum _{i=1}^n{a}_{i2}{x}_{i1}& \sum _{i=1}^n{a}_{i2}{x}_{i2}& \cdots & \sum _{i=1}^n{a}_{i2}{x}_{ip}\\ ⋮& ⋮& \ddots & ⋮\\ \sum _{i=1}^n{a}_{ip}{x}_{i1}& \sum _{i=1}^n{a}_{ip}{x}_{i2}& \cdots & \sum _{i=1}^n{a}_{ip}{x}_{ip}\end{array}\right]\\ \\ & tr({\mathbf{A}}^T\mathbf{X})=\sum _{i=1}^n\sum _{j=1}^p{a}_{ij}{x}_{ij}\\ & \frac{\partial tr({\mathbf{A}}^T\mathbf{X})}{\partial x_{ij}}=a_{ij}\Rightarrow \frac{\partial tr({\mathbf{A}}^T\mathbf{X})}{\partial \mathbf{X}}=\mathbf{A}\end{array}$$

定理(7)：${\mathbf{A}}_{n\times n},{\mathbf{X}}_{n\times p},{\mathbf{B}}_{p\times n}\Rightarrow \frac{\partial tr({\mathbf{A}}^T\mathbf{X}\mathbf{B})}{\partial \mathbf{X}}={\mathbf{AB}}^T$

$Proof:$ $$\begin{array}{rl}{\mathbf{A}}^T\mathbf{XB}& ={\left[\begin{array}{cccc}{a}_{11}& a_{12}& \cdots & a_{1n}\\ a_{21}& a_{22}& \cdots & a_{2n}\\ ⋮& ⋮& \ddots & ⋮\\ a_{n1}& a_{n2}& \cdots & a_{nn}\end{array}\right]}^T\left[\begin{array}{cccc}{x}_{11}& x_{12}& \cdots & x_{1p}\\ x_{21}& x_{22}& \cdots & x_{2p}\\ ⋮& ⋮& \ddots & ⋮\\ x_{n1}& x_{n2}& \cdots & x_{np}\end{array}\right]\left[\begin{array}{cccc}{b}_{11}& b_{12}& \cdots & b_{1n}\\ b_{21}& b_{22}& \cdots & b_{2n}\\ ⋮& ⋮& \ddots & ⋮\\ b_{p1}& b_{p2}& \cdots & b_{pn}\end{array}\right]\\ & =\left[\begin{array}{cccc}{a}_{11}& a_{21}& \cdots & a_{n1}\\ a_{12}& a_{22}& \cdots & a_{n2}\\ ⋮& ⋮& \ddots & ⋮\\ a_{1n}& a_{2n}& \cdots & a_{nn}\end{array}\right]\left[\begin{array}{cccc}{x}_{11}& x_{12}& \cdots & x_{1p}\\ x_{21}& x_{22}& \cdots & x_{2p}\\ ⋮& ⋮& \ddots & ⋮\\ x_{n1}& x_{n2}& \cdots & x_{np}\end{array}\right]\left[\begin{array}{cccc}{b}_{11}& b_{12}& \cdots & b_{1n}\\ b_{21}& b_{22}& \cdots & b_{2n}\\ ⋮& ⋮& \ddots & ⋮\\ b_{p1}& b_{p2}& \cdots & b_{pn}\end{array}\right]\\ & =\left[\begin{array}{cccc}\sum _{i=1}^n{a}_{i1}{x}_{i1}& \sum _{i=1}^n{a}_{i1}{x}_{i2}& \cdots & \sum _{i=1}^n{a}_{i1}{x}_{ip}\\ \sum _{i=1}^n{a}_{i2}{x}_{i1}& \sum _{i=1}^n{a}_{i2}{x}_{i2}& \cdots & \sum _{i=1}^n{a}_{i2}{x}_{ip}\\ ⋮& ⋮& \ddots & ⋮\\ \sum _{i=1}^n{a}_{in}{x}_{i1}& \sum _{i=1}^n{a}_{in}{x}_{i2}& \cdots & \sum _{i=1}^n{a}_{in}{x}_{ip}\end{array}\right]\left[\begin{array}{cccc}{b}_{11}& b_{12}& \cdots & b_{1n}\\ b_{21}& b_{22}& \cdots & b_{2n}\\ ⋮& ⋮& \ddots & ⋮\\ b_{p1}& b_{p2}& \cdots & b_{pn}\end{array}\right]\\ & =\left[\begin{array}{cccc}\sum _{j=1}^p\sum _{i=1}^n{a}_{i1}{x}_{ij}{b}_{j1}& \sum _{j=1}^p\sum _{i=1}^n{a}_{i1}{x}_{ij}{b}_{j2}& \cdots & \sum _{j=1}^p\sum _{i=1}^n{a}_{i1}{x}_{ij}{b}_{jn}\\ \sum _{j=1}^p\sum _{i=1}^n{a}_{i2}{x}_{ij}{b}_{j1}& \sum _{j=1}^p\sum _{i=1}^n{a}_{i2}{x}_{ij}{b}_{j2}& \cdots & \sum _{j=1}^p\sum _{i=1}^n{a}_{i2}{x}_{ij}{b}_{jn}\\ ⋮& ⋮& \ddots & ⋮\\ \sum _{j=1}^p\sum _{i=1}^n{a}_{in}{x}_{ij}{b}_{j1}& \sum _{j=1}^p\sum _{i=1}^n{a}_{in}{x}_{ij}{b}_{j2}& \cdots & \sum _{j=1}^p\sum _{i=1}^n{a}_{in}{x}_{ij}{b}_{jn}\end{array}\right]\\ \\ & tr(\mathbf{AXB})=\sum _{k=1}^n\sum _{j=1}^p\sum _{i=1}^n{a}_{ik}{x}_{ij}{b}_{jk}\\ & \frac{\partial tr(\mathbf{AXB})}{\partial x_{ij}}=\sum _{k=1}^n{a}_{ik}{b}_{jk}\Rightarrow \frac{\partial tr(\mathbf{AXB})}{\partial \mathbf{X}}=\mathbf{A}{\mathbf{B}}^T\\ \\ & \mathbf{A}{\mathbf{B}}^T=\left[\begin{array}{cccc}\sum _{i=1}^n{a}_{1i}{b}_{1i}& \sum _{i=1}^n{a}_{1i}{b}_{2i}& \cdots & \sum _{i=1}^n{a}_{1i}{b}_{pi}\\ \sum _{i=1}^n{a}_{2i}{b}_{1i}& \sum _{i=1}^n{a}_{2i}{b}_{2i}& \cdots & \sum _{i=1}^n{a}_{2i}{b}_{pi}\\ ⋮& ⋮& \ddots & ⋮\\ \sum _{i=1}^n{a}_{ni}{b}_{1i}& \sum _{i=1}^n{a}_{ni}{b}_{2i}& \cdots & \sum _{i=1}^n{a}_{ni}{b}_{pi}\end{array}\right]\end{array}$$

定理(8)：${\mathbf{A}}_{n\times n},{\mathbf{X}}_{n\times p}\Rightarrow \frac{\partial tr({\mathbf{X}}^T\mathbf{A}\mathbf{X})}{\partial \mathbf{X}}=(\mathbf{A}+{\mathbf{A}}^T)\mathbf{X}$

$Proof:$ $$\begin{array}{rl}{\mathbf{X}}^T\mathbf{AX}& ={\left[\begin{array}{cccc}{x}_{11}& x_{12}& \cdots & x_{1p}\\ x_{21}& x_{22}& \cdots & x_{2p}\\ ⋮& ⋮& \ddots & ⋮\\ x_{n1}& x_{n2}& \cdots & x_{np}\end{array}\right]}^T\left[\begin{array}{cccc}{a}_{11}& a_{12}& \cdots & a_{1n}\\ a_{21}& a_{22}& \cdots & a_{2n}\\ ⋮& ⋮& \ddots & ⋮\\ a_{n1}& a_{n2}& \cdots & a_{nn}\end{array}\right]\left[\begin{array}{cccc}{x}_{11}& x_{12}& \cdots & x_{1p}\\ x_{21}& x_{22}& \cdots & x_{2p}\\ ⋮& ⋮& \ddots & ⋮\\ x_{n1}& x_{n2}& \cdots & x_{np}\end{array}\right]\\ & =\left[\begin{array}{cccc}{x}_{11}& x_{21}& \cdots & x_{n1}\\ x_{12}& x_{22}& \cdots & x_{n2}\\ ⋮& ⋮& \ddots & ⋮\\ x_{1p}& x_{2p}& \cdots & x_{np}\end{array}\right]\left[\begin{array}{cccc}{a}_{11}& a_{12}& \cdots & a_{1n}\\ a_{21}& a_{22}& \cdots & a_{2n}\\ ⋮& ⋮& \ddots & ⋮\\ a_{n1}& a_{n2}& \cdots & a_{nn}\end{array}\right]\left[\begin{array}{cccc}{x}_{11}& x_{12}& \cdots & x_{1p}\\ x_{21}& x_{22}& \cdots & x_{2p}\\ ⋮& ⋮& \ddots & ⋮\\ x_{n1}& x_{n2}& \cdots & x_{np}\end{array}\right]\\ & =\left[\begin{array}{cccc}\sum _{i=1}^n{x}_{i1}{a}_{i1}& \sum _{i=1}^n{x}_{i1}{a}_{i2}& \cdots & \sum _{i=1}^n{x}_{i1}{a}_{in}\\ \sum _{i=1}^n{x}_{i2}{a}_{i1}& \sum _{i=1}^n{x}_{i2}{a}_{i2}& \cdots & \sum _{i=1}^n{x}_{i2}{a}_{ip}\\ ⋮& ⋮& \ddots & ⋮\\ \sum _{i=1}^n{x}_{ip}{a}_{i1}& \sum _{i=1}^n{x}_{ip}{a}_{i2}& \cdots & \sum _{i=1}^n{x}_{ip}{a}_{in}\end{array}\right]\left[\begin{array}{cccc}{x}_{11}& x_{12}& \cdots & x_{1p}\\ x_{21}& x_{22}& \cdots & x_{2p}\\ ⋮& ⋮& \ddots & ⋮\\ x_{n1}& x_{n2}& \cdots & x_{np}\end{array}\right]\\ & =\left[\begin{array}{cccc}\sum _{j=1}^n\sum _{i=1}^n{x}_{i1}{a}_{ij}{x}_{j1}& \sum _{j=1}^n\sum _{i=1}^n{x}_{i1}{a}_{ij}{x}_{j2}& \cdots & \sum _{j=1}^n\sum _{i=1}^n{x}_{i1}{a}_{ij}{x}_{jp}\\ \sum _{j=1}^n\sum _{i=1}^n{x}_{i2}{a}_{ij}{x}_{j1}& \sum _{j=1}^n\sum _{i=1}^n{x}_{i2}{a}_{ij}{x}_{j2}& \cdots & \sum _{j=1}^n\sum _{i=1}^n{x}_{i2}{a}_{ij}{x}_{jp}\\ ⋮& ⋮& \ddots & ⋮\\ \sum _{j=1}^n\sum _{i=1}^n{x}_{ip}{a}_{ij}{x}_{j1}& \sum _{j=1}^n\sum _{i=1}^n{x}_{ip}{a}_{ij}{x}_{j2}& \cdots & \sum _{j=1}^n\sum _{i=1}^n{x}_{ip}{a}_{ij}{x}_{jp}\end{array}\right]\\ \end{array}$$ $$\begin{array}{rl}tr({\mathbf{X}}^T\mathbf{A}\mathbf{X})& =\sum _{k=1}^p\sum _{j=1}^n\sum _{i=1}^n{x}_{ip}{a}_{ij}{x}_{jp}\\ \frac{\partial tr({\mathbf{X}}^T\mathbf{A}\mathbf{X})}{\partial x_{ip}}& =\sum _{j=1}^n{a}_{ij}{x}_{jp}+\sum _{j=1}^n{x}_{jp}{a}_{ji}\\ & =\sum _{j=1}^n{a}_{ij}{x}_{jp}+\sum _{j=1}^n{a}_{ji}{x}_{jp}\\ \Rightarrow \frac{\partial tr({\mathbf{X}}^T\mathbf{A}\mathbf{X})}{\partial \mathbf{X}}& =(\mathbf{A}+{\mathbf{A}}^T)\mathbf{X}\end{array}$$乘法律：
若 $h(x)=f(x)g(x)$，$f(x)$ 和 $g(x)$ 在 $x$ 都是可微
$h^{\prime }(x)=f^{\prime }(x)g(x)+f(x)g^{\prime }(x)$

定理(9)：${\mathbf{A}}_{p\times p},{\mathbf{X}}_{n\times p}\Rightarrow \frac{\partial tr(\mathbf{X}\mathbf{A}{\mathbf{X}}^T)}{\partial \mathbf{X}}=\mathbf{X}(\mathbf{A}+{\mathbf{A}}^T)$

$Proof:$ $$\begin{array}{rl}\mathbf{XA}{\mathbf{X}}^T& =\left[\begin{array}{cccc}{x}_{11}& x_{12}& \cdots & x_{1p}\\ x_{21}& x_{22}& \cdots & x_{2p}\\ ⋮& ⋮& \ddots & ⋮\\ x_{n1}& x_{n2}& \cdots & x_{np}\end{array}\right]\left[\begin{array}{cccc}{a}_{11}& a_{12}& \cdots & a_{1p}\\ a_{21}& a_{22}& \cdots & a_{2p}\\ ⋮& ⋮& \ddots & ⋮\\ a_{p1}& a_{p2}& \cdots & a_{pp}\end{array}\right]{\left[\begin{array}{cccc}{x}_{11}& x_{12}& \cdots & x_{1p}\\ x_{21}& x_{22}& \cdots & x_{2p}\\ ⋮& ⋮& \ddots & ⋮\\ x_{n1}& x_{n2}& \cdots & x_{np}\end{array}\right]}^T\\ & =\left[\begin{array}{cccc}{x}_{11}& x_{12}& \cdots & x_{1p}\\ x_{21}& x_{22}& \cdots & x_{2p}\\ ⋮& ⋮& \ddots & ⋮\\ x_{n1}& x_{n2}& \cdots & x_{np}\end{array}\right]\left[\begin{array}{cccc}{a}_{11}& a_{12}& \cdots & a_{1p}\\ a_{21}& a_{22}& \cdots & a_{2p}\\ ⋮& ⋮& \ddots & ⋮\\ a_{p1}& a_{p2}& \cdots & a_{pp}\end{array}\right]\left[\begin{array}{cccc}{x}_{11}& x_{21}& \cdots & x_{n1}\\ x_{12}& x_{22}& \cdots & x_{n2}\\ ⋮& ⋮& \ddots & ⋮\\ x_{1p}& x_{2p}& \cdots & x_{np}\end{array}\right]\\ & =\left[\begin{array}{cccc}\sum _{i=1}^p{x}_{1i}{a}_{i1}& \sum _{i=1}^p{x}_{1i}{a}_{i2}& \cdots & \sum _{i=1}^p{x}_{1i}{a}_{ip}\\ \sum _{i=1}^p{x}_{2i}{a}_{i1}& \sum _{i=1}^p{x}_{2i}{a}_{i2}& \cdots & \sum _{i=1}^p{x}_{2i}{a}_{ip}\\ ⋮& ⋮& \ddots & ⋮\\ \sum _{i=1}^p{x}_{ni}{a}_{i1}& \sum _{i=1}^p{x}_{ni}{a}_{i2}& \cdots & \sum _{i=1}^p{x}_{ni}{a}_{ip}\end{array}\right]\left[\begin{array}{cccc}{x}_{11}& x_{21}& \cdots & x_{n1}\\ x_{12}& x_{22}& \cdots & x_{n2}\\ ⋮& ⋮& \ddots & ⋮\\ x_{1p}& x_{2p}& \cdots & x_{np}\end{array}\right]\\ & =\left[\begin{array}{cccc}\sum _{j=1}^p\sum _{i=1}^p{x}_{1i}{a}_{ij}{x}_{1j}& \sum _{j=1}^p\sum _{i=1}^p{x}_{1i}{a}_{ij}{x}_{2j}& \cdots & \sum _{j=1}^p\sum _{i=1}^p{x}_{1i}{a}_{ij}{x}_{pj}\\ \sum _{j=1}^p\sum _{i=1}^p{x}_{2i}{a}_{ij}{x}_{1j}& \sum _{j=1}^p\sum _{i=1}^p{x}_{2i}{a}_{ij}{x}_{2j}& \cdots & \sum _{j=1}^p\sum _{i=1}^p{x}_{2i}{a}_{ij}{x}_{pj}\\ ⋮& ⋮& \ddots & ⋮\\ \sum _{j=1}^p\sum _{i=1}^p{x}_{pi}{a}_{ij}{x}_{1j}& \sum _{j=1}^p\sum _{i=1}^p{x}_{pi}{a}_{ij}{x}_{2j}& \cdots & \sum _{j=1}^p\sum _{i=1}^p{x}_{ni}{a}_{ij}{x}_{nj}\end{array}\right]\\ \end{array}$$ $$\begin{array}{rl}tr(\mathbf{X}\mathbf{A}{\mathbf{X}}^T)& =\sum _{k=1}^n\sum _{j=1}^p\sum _{i=1}^p{x}_{ki}{a}_{ij}{x}_{kj}\\ \frac{\partial tr(\mathbf{X}\mathbf{A}{\mathbf{X}}^T)}{\partial x_{ki}}& =\sum _{j=1}^p{a}_{ij}{x}_{kj}+\sum _{j=1}^p{x}_{kj}{a}_{ji}\\ & =\sum _{j=1}^p{x}_{kj}{a}_{ji}+\sum _{j=1}^p{x}_{kj}{a}_{ij}\\ \Rightarrow \frac{\partial tr({\mathbf{X}}^T\mathbf{A}\mathbf{X})}{\partial \mathbf{X}}& =\mathbf{X}(\mathbf{A}+{\mathbf{A}}^T)\end{array}$$ 乘法律：
若 $h(x)=f(x)g(x)$，$f(x)$ 和 $g(x)$ 在 $x$ 都是可微
$h^{\prime }(x)=f^{\prime }(x)g(x)+f(x)g^{\prime }(x)$ $$

定理(10)：${\mathbf{X}}_{p\times p}\Rightarrow \frac{\partial |\mathbf{X}|}{\partial \mathbf{X}}=|\mathbf{X}|({\mathbf{X}}^{-1}{)}^T$

$Proof:$ $$\begin{array}{rl}|\mathbf{X}|=det\mathbf{X}& =\sum _{j=1}^n{x}_{ij}(-1{)}^{i+j}det{\mathbf{X}}_{ij}\\ & =\sum _{j=1}^n{x}_{ij}{c}_{ij}\\ \\ \frac{\partial det\mathbf{X}}{\partial x_{ij}}& =\frac{\partial \sum _k{x}_{ik}{c}_{ik}}{\partial x_{ij}}\\ & =c_{ij}\\ & ={((\text{adj}X{)}^T)}_{ij}\\ & ={((detX)(X^{-1}{)}^T)}_{ij}\\ \Rightarrow \frac{\partial |\mathbf{X}|}{\partial \mathbf{X}}& =|\mathbf{X}|({\mathbf{X}}^{-1}{)}^T\\ \end{array}$$
----------------------------------------------------------------------------------------------------------------------------------------
$$\begin{gathered} det\mathbf{X}=\sum _{j=1}^n{x}_{ij}{c}_{ij}=\sum _{j=1}^n(-1{)}^{i+j}{x}_{ij}det{\tilde{\mathbf{X}}}_{ij} \\ \left[\begin{array}{cccc}{x}_{11}& x_{12}& \cdots & x_{1n}\\ x_{21}& x_{22}& \cdots & x_{2n}\\ ⋮& ⋮& \ddots & ⋮\\ x_{n1}& x_{n2}& \cdots & x_{nn}\end{array}\right]\left[\begin{array}{cccc}{c}_{11}& c_{21}& \cdots & c_{n1}\\ c_{12}& c_{22}& \cdots & c_{n2}\\ ⋮& ⋮& \ddots & ⋮\\ c_{1n}& c_{2n}& \cdots & c_{nn}\end{array}\right]=\left[\begin{array}{cccc}det\mathbf{X}& 0& \cdots & 0\\ 0& det\mathbf{X}& \cdots & 0\\ ⋮& ⋮& \ddots & ⋮\\ 0& 0& \cdots & det\mathbf{X}\end{array}\right] \\ \mathbf{X}{\mathbf{C}}^T=(det\mathbf{X})\mathbf{I} \end{gathered}$$ ${\mathbf{C}}^T$ 為 $\mathbf{X}$ 的伴隨矩陣 (adjugate 或 classical adjoint)，記作 $\mathrm{adj}\mathbf{X}$
若 $\mathbf{X}$ 可逆，則 ${\mathbf{C}}^T=(det\mathbf{X}){\mathbf{X}}^{-1}$

為何只有對角線為 $det\mathbf{X}$，其餘為 0？
如果 $i\ne j$，那麼 $\mathbf{X}{\mathbf{C}}^T$ 的第 $i$ 行第 $j$ 列的係數是 $\sum _{k=1}^n{x}_{ik}{c}_{jk}$
拉普拉斯公式說明這個和等於 0
（等同把 $\mathbf{X}$ 的第 $j$ 行元素換成第 $i$ 行元素後求行列式。由於有兩行相同，行列式為 0）。
以 $\mathbf{X}$ 的第 2 列和 ${\mathbf{C}}^T$ 的第 1 行相乘為例：
$x_{21}$ 和 $c_{11}$ 思考如圖，左為目前的做法，右為正常的做法

$$\sum _{k=1}^n{x}_{2k}{c}_{1k}\Rightarrow det\left[\begin{array}{cccc}{x}_{21}& x_{22}& \cdots & x_{2n}\\ x_{21}& x_{22}& \cdots & x_{2n}\\ ⋮& ⋮& \ddots & ⋮\\ x_{n1}& x_{n2}& \cdots & x_{nn}\end{array}\right]=0$$

定理(11)：${\mathbf{X}}_{p\times p}\Rightarrow \frac{\partial \ln |\mathbf{X}|}{\partial \mathbf{X}}=({\mathbf{X}}^{-1}{)}^T$

$Proof:$ 根據 定理(10) $$\begin{array}{rl}\frac{\partial \ln \mathbf{|}\mathbf{X}\mathbf{|}}{\partial x}& =\frac{1}{\mathbf{|}\mathbf{X}\mathbf{|}}\frac{\partial }{\partial \mathbf{X}}\mathbf{|}\mathbf{X}\mathbf{|}\\ & =\frac{1}{\mathbf{|}\mathbf{X}\mathbf{|}}\mathbf{|}\mathbf{X}\mathbf{|}({\mathbf{X}}^{-1}{)}^T\\ & =({\mathbf{X}}^{-1}{)}^T\end{array}$$ Chain Rule 連鎖律：
設 $y=f(u)$ 可以對 $u$ 微分，而函數 $u=g(x)$ 是可以對 $x$ 微分
則 $y=f(g(x))$ 是可以對 $x$ 微分的
同時 $$\frac{\mathrm{d}y}{\mathrm{d}x}=\frac{\mathrm{d}y}{\mathrm{d}u}\frac{\mathrm{d}u}{\mathrm{d}x}$$