矩阵求导

矩阵求导结果的布局

functon \input	标量变元	向量变元	矩阵变元
实值标量函数	$f (x)$	$f (x)$	$f (X)$
实向量函数	$f (x)$	$f (x)$	$f (X)$
实矩阵函数	$F (x)$	$F (x)$	$F (X)$

一般向量默认为列向量

Example

$\begin{array}{r} f (X_{3 \times 2}) = a_{1} x_{11}^{2} + a_{2} x_{12}^{2} + \dots + a_{6} x_{32}^{2} \\ F_{3 \times 2} (x_{3 \times 1}) = (\begin{array}{c} x_{1} + x_{2} + x_{3} & x_{2} - x_{3} \\ x_{1} & x_{3} \\ 2 x_{1} + x_{2} & x_{1} - x_{2} - x_{3} \end{array}) \end{array}$

矩阵求导即每个 $f$ 对变元中的每个元素逐个求偏导。其中有两种形式

Jacobian 阵形式：对 $x^{T}, X^{T}$ 求偏导
梯度向量形式：对 $x, X$ 求导
求导后结果的布局一般即一下两种情况：
分子布局，就是分子是列向量形式，分母是行向量形式 $\frac{\partial f_{2 \times 1} (x)}{\partial x_{3 \times 1}^{T}} = {[\begin{matrix} \frac{\partial f_{1}}{\partial x_{1}} & \frac{\partial f_{1}}{\partial x_{2}} & \frac{\partial f_{1}}{\partial x_{3}} \\ \frac{\partial f_{2}}{\partial x_{1}} & \frac{\partial f_{2}}{\partial x_{2}} & \frac{\partial f_{2}}{\partial x_{3}} \end{matrix}]}_{2 \times 3}$
分母布局，就是分母是列向量形式，分子是行向量形式

\frac{\partial f_{2 \times 1}^{T} (x)}{\partial x_{3 \times 1}} = {[\begin{matrix} \frac{\partial f_{1}}{\partial x_{1}} & \frac{\partial f_{2}}{\partial x_{1}} \\ \frac{\partial f_{1}}{\partial x_{2}} & \frac{\partial f_{2}}{\partial x_{2}} \\ \frac{\partial f_{1}}{\partial x_{3}} & \frac{\partial f_{2}}{\partial x_{3}} \end{matrix}]}_{3 \times 2}

向量变元，标量函数

行向量偏导

D_{x} f (x) = \frac{\partial f (x)}{\partial x^{T}} = [\frac{\partial f}{\partial x_{1}}, \frac{\partial f}{\partial x_{2}}, \dots, \frac{\partial f}{\partial x_{n}}]

列向量偏导（梯度向量）

\begin{matrix} (4) & \nabla_{x} f (x) = \frac{\partial f (x)}{\partial x} = {[\frac{\partial f}{\partial x_{1}}, \frac{\partial f}{\partial x_{2}}, \dots, \frac{\partial f}{\partial x_{n}}]}^{T} \end{matrix}

两种形式互为转置

矩阵变元，标量函数

$vec (X)$ ：按列堆栈来向量化

\begin{matrix} (5) & vec (X) = {[x_{11}, x_{21}, \dots, x_{m 1}, x_{12}, x_{22}, \dots, x_{m 2}, \dots, x_{1 n}, x_{2 n}, \dots, x_{m n}]}^{T} \end{matrix}

行向量偏导

\begin{matrix} (6) & \begin{aligned} D_{vec X} f (X) & = \frac{\partial f (X)}{\partial {vec}^{T} (X)} \\ = [\frac{\partial f}{\partial x_{11}}, \frac{\partial f}{\partial x_{21}}, \dots, \frac{\partial f}{\partial x_{m 1}}, \frac{\partial f}{\partial x_{12}}, \frac{\partial f}{\partial x_{22}}, \dots, \frac{\partial f}{\partial x_{m 2}}, \dots, \frac{\partial f}{\partial x_{1 n}}, \frac{\partial f}{\partial x_{2 n}} \dots, \frac{\partial f}{\partial x_{m n}}] \end{aligned} \end{matrix}

列向量偏导（梯度向量）

\begin{aligned} \nabla_{vec X} f (X) & = \frac{\partial f (X)}{\partial vec X} \\ = {[\frac{\partial f}{\partial x_{11}}, \frac{\partial f}{\partial x_{21}}, \dots, \frac{\partial f}{\partial x_{m 1}}, \frac{\partial f}{\partial x_{12}}, \frac{\partial f}{\partial x_{22}}, \dots, \frac{\partial f}{\partial x_{m 2}}, \dots, \frac{\partial f}{\partial x_{1 n}}, \frac{\partial f}{\partial x_{2 n}}, \dots, \frac{\partial f}{\partial x_{m n}}]}^{T} \end{aligned}

Jacobian 矩阵形式

\begin{aligned} D_{X} f (X) & = \frac{\partial f (X)}{\partial X_{m \times n}^{T}} = {[\begin{array}{c} \frac{\partial f}{\partial x_{11}} & \frac{\partial f}{\partial x_{21}} & \dots & \frac{\partial f}{\partial x_{m 1}} \\ \frac{\partial f}{\partial x_{12}} & \frac{\partial f}{\partial x_{22}} & \dots & \frac{\partial f}{\partial x_{m 2}} \\ ⋮ & ⋮ & ⋮ & ⋮ \\ \frac{\partial f}{\partial x_{1 n}} & \frac{\partial f}{\partial x_{2 n}} & \dots & \frac{\partial f}{\partial x_{m n}} \end{array}]}_{n \times m} \end{aligned}

梯度矩阵形式（与 $X$ 形状一致）

\begin{aligned} \nabla_{X} f (X) & = \frac{\partial f (X)}{\partial X_{m \times n}} \\ = {[\begin{array}{c} \frac{\partial f}{\partial x_{11}} & \frac{\partial f}{\partial x_{12}} & \dots & \frac{\partial f}{\partial x_{1 n}} \\ \frac{\partial f}{\partial x_{21}} & \frac{\partial f}{\partial x_{22}} & \dots & \frac{\partial f}{\partial x_{2 n}} \\ ⋮ & ⋮ & ⋮ & ⋮ \\ \frac{\partial f}{\partial x_{m 1}} & \frac{\partial f}{\partial x_{m 2}} & \dots & \frac{\partial f}{\partial x_{m n}} \end{array}]}_{m \times n} \end{aligned}

矩阵变元，矩阵函数

设 $X_{m \times n} = (x_{i j})_{i = 1, j = 1}^{m, n}, F_{p \times q} = (f_{i j})_{i = 1, j = 1}^{p, q}$

Jacobian 矩阵形式

D_{X} F (X) = \frac{\partial {vec}_{p q \times 1} (F (X))}{\partial {vec}_{m n \times 1}^{T} X} = {[\begin{matrix} \frac{\partial f_{11}}{\partial x_{11}} & \dots & \frac{\partial f_{11}}{\partial x_{m n}} \\ ⋮ & ⋱ & ⋮ \\ \frac{\partial f_{p q}}{\partial x_{11}} & \dots & \frac{\partial f_{p q}}{\partial x_{m n}} \end{matrix}]}_{p q \times m n}

梯度矩阵形式（与 $X$ 形状一致）

\nabla_{X} F (X) = \frac{\partial {vec}_{p q \times 1}^{T} (F (X))}{\partial {vec}_{m n \times 1} X} = {[\begin{matrix} \frac{\partial f_{11}}{\partial x_{11}} & \dots & \frac{\partial f_{p q}}{\partial x_{11}} \\ ⋮ & ⋱ & ⋮ \\ \frac{\partial f_{11}}{\partial x_{p q}} & \dots & \frac{\partial f_{p q}}{\partial x_{m n}} \end{matrix}]}_{m n \times p q}

矩阵求导公式

矩阵微分

基本形式

\begin{matrix} d f (x) = \frac{\partial f}{\partial x_{1}} d x_{1} + \frac{\partial f}{\partial x_{2}} d x_{2} + \dots + \frac{\partial f}{\partial x_{n}} d x_{n} \\ = (\frac{\partial f}{\partial x_{1}}, \frac{\partial f}{\partial x_{2}}, \dots, \frac{\partial f}{\partial x_{n}}) [\begin{matrix} d x_{1} \\ d x_{2} \\ ⋮ \\ d x_{n} \end{matrix}] \\ = tr ((\frac{\partial f}{\partial x_{1}}, \frac{\partial f}{\partial x_{2}}, \dots, \frac{\partial f}{\partial x_{n}}) [\begin{matrix} d x_{1} \\ d x_{2} \\ ⋮ \\ d x_{n} \end{matrix}]) \end{matrix}

\begin{aligned} d (f (X)) & = \frac{\partial f}{\partial x_{11}} d x_{11} + \frac{\partial f}{\partial x_{12}} d x_{12} + \dots + \frac{\partial f}{\partial x_{1 n}} d x_{1 n} \\ + \frac{\partial f}{\partial x_{21}} d x_{21} + \frac{\partial f}{\partial x_{22}} d x_{22} + \dots + \frac{\partial f}{\partial x_{2 n}} d x_{2 n} \\ + \dots \\ + \frac{\partial f}{\partial x_{m 1}} d x_{m 1} + \frac{\partial f}{\partial x_{m 2}} d x_{m 2} + \dots + \frac{\partial f}{\partial x_{m n}} d x_{m n} \\ = tr ({[\begin{array}{c} \frac{\partial f}{\partial x_{11}} & \frac{\partial f}{\partial x_{21}} & \dots & \frac{\partial f}{\partial x_{m 1}} \\ \frac{\partial f}{\partial x_{12}} & \frac{\partial f}{\partial x_{22}} & \dots & \frac{\partial f}{\partial x_{m 2}} \\ ⋮ & ⋮ & ⋮ & ⋮ \\ \frac{\partial f}{\partial x_{1 n}} & \frac{\partial f}{\partial x_{2 n}} & \dots & \frac{\partial f}{\partial x_{m n}} \end{array}]}_{n \times m} {[\begin{array}{c} d x_{11} & d x_{12} & \dots & d x_{1 n} \\ d x_{21} & d x_{22} & \dots & d x_{2 n} \\ ⋮ & ⋮ & ⋮ & ⋮ \\ d x_{m 1} & d x_{m 2} & \dots & d x_{m n} \end{array}]}_{m \times n}) \end{aligned}

可以看出 $d (f (X)) = tr (\frac{\partial f (X)}{\partial X^{T}} d X)$ ，因此如果能全微分成 $tr (\dots d X)$ ，则可直接求出导数。

且该结果已证明唯一。即若 $d f (X) = tr (A_{1} d X) = tr (A_{2} d X)$ ，则 $A_{1} = A_{2}$
当矩阵“退化”为向量时仍成立

d F_{p \times q} (X) = {[\begin{matrix} d f_{11} (X) & d f_{12} (X) & \dots & d f_{1 q} (X) \\ d f_{21} (X) & d f_{22} (X) & \dots & d f_{2 q} (X) \\ ⋮ & ⋮ & ⋮ & ⋮ \\ d f_{p 1} (X) & d f_{p 2} (X) & \dots & d f_{p q} (X) \end{matrix}]}_{p \times q}

微分基本公式

$d A_{m \times n} = 0_{m \times n}$
$d (c_{1} F (X) + c_{2} G (X)) = c_{1} d F (X) + c_{2} d G (X)$
对于 $F_{p \times q} (X), G_{q \times s} (X)$ ， $d (F (X) G (X)) = d (F (X)) G (X) + F (X) d G (X)$

此时的微分是矩阵，不能交换乘积的左右顺序。

$d F_{p \times q}^{T} (X) = (d F_{p \times q} (X))^{T}$
$tr (AB) = tr (BA)$
夹层饼

d (A X B) = A d (X) B

行列式（ $X$ 为方阵）

d | X | = | X | tr (X^{- 1} d X) = tr (| X | X^{- 1} d X)

Proof

$| X | = x_{i 1} A_{i 1} + x_{i 2} A_{i 2} + \dots + x_{i n} A_{i n}$

因此，行列式对元素 $x_{i j}$ 的偏导，即为该元素对应的代数余子式，行列式对矩阵求导结果即伴随矩阵。由于伴随矩阵与逆矩阵满足

X^{- 1} = \frac{X^{*}}{| X |}

带入得

\begin{aligned} d | X | & = tr (X^{*} d X) \\ = tr (| X | X^{- 1} d X) \end{aligned}

逆矩阵

d (X^{- 1}) = - X^{- 1} d (X) X^{- 1}

证明思路即 $X X^{- 1} = E$ 两边同时微分得到

常用例

均可对应[[#矩阵求导公式]]
对于实值标量函数求导：

d f (X) = d (tr f (X)) = tr (d f (X))

如果是迹形式的实值标量函数同理：

d (tr F_{p \times p} (X)) = d (\sum_{i = 1}^{p} f_{i i} (X)) = \sum_{i = 1}^{p} d (f_{i i} (X)) = tr (d F_{p \times p} (X))

如果是矢量或矩阵函数，则需对每个函数元素微分求导
【例 1】

\frac{\partial (a^{T} X X^{T} b)}{\partial X} = a b^{T} X + b a^{T} X

【例 2】

\frac{\partial tr (X^{T} X)}{\partial X} = 2 X

【例 3】

\frac{\partial \log | X |}{\partial X} = (X^{- 1})^{T}

其中 $X$ 为方阵
【例 4】

\frac{\partial | X^{- 1} |}{\partial X} = - | X^{- 1} | (X^{- 1})^{T}

其中 $X$ 为方阵
【例 5】

\frac{\partial tr (X + A)^{- 1}}{\partial X} = - ((X + A)^{- 2})^{T}

其中 $X, A$ 均为方阵， $A$ 为常数阵
【例 6】

\frac{\partial | X^{3} |}{\partial X} = \frac{\partial | X |^{3}}{\partial X} = 3 | X |^{3} (X^{- 1})^{T} = 3 | X^{3} | (X^{- 1})^{T}

其中 $X$ 为方阵

Note

手写资料(14)|600
手写资料(15)|600

矩阵求导结果的布局

向量变元，标量函数

矩阵变元，标量函数

矩阵变元，矩阵函数

矩阵求导公式

矩阵微分

基本形式

微分基本公式

常用例

参考