[Math] Normal Equation : Vector derivative(Numerator Layout)를 이용한 유도

Orinary Least Square는 다음과 같은 최소화 문제임.

$$\underset{\textbf{x}}{\text{argmin}}\|\textbf{b}-A\textbf{x}\|_2^2$$

$\|\textbf{b}-A\textbf{x}\|_2^2$를 전개하면 다음과 같음.

$$\begin{aligned}\|\textbf{b}-A\textbf{x}\|_2^2&=(\textbf{b}-A\textbf{x})^T(\textbf{b}-A\textbf{x})\\ &=(\textbf{b}^T-\textbf{x}^TA^T)(\textbf{b}-A\textbf{x})\\ &=\textbf{b}^T\textbf{b}-\textbf{x}^TA^T\textbf{b}-\textbf{b}^TA\textbf{x}+\textbf{x}^TA^TA\textbf{x}\\ \end{aligned}$$

OLS는 다음을 참고

2022.04.28 - [.../Math] - Ordinary Least Squares : OLS, 최소자승법

Ordinary Least Squares : OLS, 최소자승법

denominator layout으로 전개한 글은 다음을 참고

2022.04.28 - [.../Math] - Normal Equation : Vector derivative를 이용한 유도

Normal Equation : Vector derivative를 이용한 유도

 

 

 

필요한 Vector Calculus 공식(?)은 다음과 같음. (Numerator Layout임)

• $\frac{\partial A\textbf{x}}{\partial \textbf{x}} = A$
• $\frac{\partial \textbf{x}^TA}{\partial \textbf{x}} = A^T$
• $\frac{\partial \textbf{x}^TS\textbf{x}}{\partial \textbf{x}} = \textbf{x}^TS$

2022.05.05 - [.../Math] - Commonly used Vector derivatives.

Commonly used Vector derivatives.

2022.05.08 - [.../Math] - [Math] Matrix Calculus : Numerator Layout

[Math] Matrix Calculus : Numerator Layout

 

기본적인 Transpose 공식(?)은 다음과 같음.

• $(\textbf{x}^T\textbf{b})^T=\textbf{b}^T\textbf{x}$
• $(A^T\textbf{x})^T=\textbf{x}^TA$

 

최소값에서 1st order derivative는 0 인 점을 이용하여 위 식에서 solution $\hat{\textbf{x}}$를 구함.

$$\begin{aligned} \frac{\partial \|\textbf{b}-A\hat{\textbf{x}}\|_2^2}{\partial\hat{\textbf{x}}}&= \frac{\partial (\textbf{b}^T\textbf{b}-\hat{\textbf{x}}^TA^T\textbf{b}-\textbf{b}^TA\hat{\textbf{x}}+\hat{\textbf{x}}^TA^TA\hat{\textbf{x}}) }{\partial \hat{\textbf{x}}} \\ &=-(A^T\textbf{b})^T-\textbf{b}^TA+2\hat{\textbf{x}}^TA^TA\\ &=0 \end{aligned}$$

이를 정리하면 solution $\hat{\textbf{x}}$에 대한 closed-form expression을 얻음.

$$\begin{aligned} 2\hat{\textbf{x}}^TA^TA&=(A^T\textbf{b})^T+\textbf{b}^TA\\ 2(\hat{\textbf{x}}^TA^TA)^T&=\left[ (A^T\textbf{b})^T +\textbf{b}^TA \right]^T \\ 2A^TA\hat{\textbf{x}}&=\left[ \textbf{b}^TA +\textbf{b}^TA \right]^T \\ 2A^TA\hat{\textbf{x}}&=\left[ 2\textbf{b}^TA \right]^T \\ 2A^TA\hat{\textbf{x}}&=2A^T\textbf{b}\\ A^TA\hat{\textbf{x}}&=A^T\textbf{b}\\ \hat{\textbf{x}}&=(A^TA)^{-1}A^T\textbf{b} \\ \end{aligned}$$