[ML] Backpropagation

ML：Backpropagation
李宏毅 老師的課程

簡介：Backpropagation

直觀的思考

簡單推導

全微分由來 $$\begin{array}{rl}dL(u_1,u_2,\cdots ,u_N)& =d{u}_1\cdot \frac{\partial L}{\partial u_1}+d{u}_2\cdot \frac{\partial L}{\partial u_2}+\cdots +d{u}_N\cdot \frac{\partial L}{\partial u_N}\\ & =\sum _{i=1}^{N}d{u}_i\cdot \frac{\partial L}{\partial u_i}\\ \frac{\partial L(u_1,u_2,\cdots ,u_N)}{\partial x}& =\sum _{i=1}^N\frac{\partial u_i}{\partial x}\cdot \frac{\partial L}{\partial u_i}\end{array}$$ 推導過程 $$\begin{array}{rl}\frac{\partial C}{\partial W_1}& =\frac{\partial z}{\partial W_1}\frac{\partial C}{\partial z}\\ \frac{\partial z}{\partial W_1}& =x_1\\ \frac{\partial C}{\partial z}& =\frac{\partial a}{\partial z}\frac{\partial C}{\partial a}\\ & =f^{\prime }(z)(\frac{\partial z^{\prime }}{\partial a}\frac{\partial C}{\partial z^{\prime }}+\frac{\partial z^{″}}{\partial a}\frac{\partial C}{\partial z^{″}})\\ & =f^{\prime }(z)(W_3\frac{\partial C}{\partial z^{\prime }}+W_4\frac{\partial C}{\partial z^{″}})\end{array}$$

矩陣形式 $$\begin{array}{rl}\frac{\partial C}{\partial {\mathbf{w}}_{\mathbf{1}}}& =\frac{\partial z}{\partial {\mathbf{w}}_{\mathbf{1}}}\frac{\partial C}{\partial z}\\ & ={\mathbf{x}}^T{f}^{\prime }(z){\mathbf{w}}_{\mathbf{2}}\left[\begin{array}{c}\frac{\partial C}{\partial z^{\prime }}\\ \frac{\partial C}{\partial z^{″}}\end{array}\right]\end{array}$$ $$\begin{array}{rl}\mathbf{x}& =\left[\begin{array}{c}{x}_1\\ x_2\end{array}\right]\\ {\mathbf{w}}_{\mathbf{1}}& =\left[\begin{array}{cc}{W}_1& W_2\end{array}\right]\\ {\mathbf{w}}_{\mathbf{2}}& =\left[\begin{array}{cc}{W}_3& W_4\end{array}\right]\end{array}$$

$$\begin{array}{rl}\mathbf{x}& =\left[\begin{array}{c}{x}_1\\ x_2\end{array}\right]\\ {\mathbf{w}}_{\mathbf{1}}& =\left[\begin{array}{cc}{W}_1& W_2\end{array}\right]\\ {\mathbf{w}}_{\mathbf{2}}& =\left[\begin{array}{cc}{W}_3& W_4\end{array}\right]\\ {\mathbf{z}}_{\mathbf{2}}& =\left[\begin{array}{c}{z}^{\prime }\\ z^{″}\end{array}\right]\end{array}$$ $$\begin{array}{rl}\frac{\partial C}{\partial {\mathbf{w}}_{\mathbf{1}}}& =\frac{\partial z}{\partial {\mathbf{w}}_{\mathbf{1}}}\frac{\partial C}{\partial z}\\ & =\left[\begin{array}{cc}\frac{\partial z}{\partial W_1}& \frac{\partial z}{\partial W_2}\end{array}\right]\frac{\partial a}{\partial z}\frac{\partial C}{\partial a}\\ & ={\mathbf{x}}^T{f}^{\prime }(z)\frac{\partial {\mathbf{z}}_{\mathbf{2}}}{\partial a}\frac{\partial C}{\partial {\mathbf{z}}_{\mathbf{2}}}\\ & ={\mathbf{x}}^T{f}^{\prime }(z)\left[\begin{array}{cc}\frac{\partial z^{\prime }}{\partial a}& \frac{\partial z^{″}}{\partial a}\end{array}\right]\frac{\partial C}{\partial {\mathbf{z}}_{\mathbf{2}}}\\ & ={\mathbf{x}}^T{f}^{\prime }(z){\mathbf{w}}_{\mathbf{2}}\left[\begin{array}{c}\frac{\partial C}{\partial z^{\prime }}\\ \frac{\partial C}{\partial z^{″}}\end{array}\right]\end{array}$$

參考

深度學習的數學地圖：用 Python 實作神經網路的數學模型
機器學習的數學基礎 : AI、深度學習打底必讀