Positive Definite Matrices

Say we have a $n\times n$ matrix $A$ then $A$ is Positive Definite Matrix if any of the below condition is satisfies,

info

$1.$ All the eigenvalues are positive.

note

$\quad\lambda_1\gt 0,\lambda_2\gt 0,\cdots,\lambda_n\gt 0$

info

$2.$ All leading determinants are positive.

note

For Example:
Say we have a $3\times 3$ matrix $A$
$A=\begin{bmatrix} a_{11} & a_{12} & a_{13}\\ a_{21} & a_{22} & a_{23}\\ a_{31} & a_{32} & a_{33}\\ \end{bmatrix}$
$A$ is a Positive Definite Matrix if,
$\quad$
$\text{det}\left( \begin{bmatrix} a_{11} \end{bmatrix} \right)\gt 0;\quad$
$\quad$
$\text{det}\left( \begin{bmatrix} a_{11} & a_{12} \\ a_{21} & a_{22} \\ \end{bmatrix} \right)\gt 0;\quad$
$\quad$
$\text{det}\left( \begin{bmatrix} a_{11} & a_{12} & a_{13}\\ a_{21} & a_{22} & a_{23}\\ a_{31} & a_{32} & a_{33}\\ \end{bmatrix} \right)\gt 0$

info

$3.$ All pivots are positive.

info

$4.$ A $n\times n$ matrix say $A$ is Positive Definite Matrix, if

danger

$$\begin{matrix} \vec{x}^TA\vec{x}\gt 0;\quad\forall x\in\mathbb{R}^n \char`\\ \vec{0} \end{matrix}$$

More interesting properties of positive definite matrices

● If a $n\times n$ matrix $A$ is a positive definite matrix then $A^{-1}$ is also positive definite matrix

note

Proof:
Here we can use property number $1$ "A matrix is positive definite matrix if all the eigenvalues are positive."
Say that the eigenvalues of $A$ are $\lambda_1,\lambda_2,\cdots,\lambda_n$
Then the eigenvalues of $A^{-1}$ are $1/\lambda_1,1/\lambda_2,\cdots,1/\lambda_n$
$A$ is a positive definite matrix, it's eigenvalues are positive, so
$\lambda_1\gt 0,\lambda_2\gt 0,\cdots,\lambda_n\gt 0$

$$\Rightarrow \quad1/\lambda_1\gt 0,\quad1/\lambda_2\gt 0,\cdots,1/\lambda_n\gt 0$$

So by property $1$ we can say that "$A^{-1}$ is a Positive Definite Matrix"

● Say we have two $n\times n$ positive definite matrix $A$ and $B$ then $A+B$ is also positive definite matrix

note

Proof:
Here we can use property number $5$
"A matrix (say A) is positive definite matrix if $\vec{x}^TA\vec{x}\gt 0;\quad\forall x\in\mathbb{R}^n \char`\\ \vec{0}$"
We know that $A$ is positive definite matrix so $\vec{x}^TA\vec{x}\gt 0;\quad\forall x\in\mathbb{R}^n \char`\\ \vec{0}$
We know that $B$ is positive definite matrix so $\vec{x}^TB\vec{x}\gt 0;\quad\forall x\in\mathbb{R}^n \char`\\ \vec{0}$
For $A+B$ to be positive definite matrix we need

$$\vec{x}^T(A+B)\vec{x}\gt 0;\quad\forall x\in\mathbb{R}^n \char`\\ \vec{0} \\ \quad \\ \vec{x}^T(A+B)\vec{x} = \underbrace{\vec{x}^TA\vec{x}}_{(\gt 0;\forall x\in\mathbb{R}^n \char`\\ \vec{0})} + \underbrace{\vec{x}^TB\vec{x}}_{(\gt 0;\forall x\in\mathbb{R}^n \char`\\ \vec{0})} \\ \quad \\ \Rightarrow \vec{x}^T(A+B)\vec{x}\gt 0;\quad\forall x\in\mathbb{R}^n \char`\\ \vec{0} \\$$

So by property $5$ we can say that "$A + B$ is a Positive Definite Matrix"

● Say we have a $m\times n$ matrix $A$ and $\text{Rank}(A)=n$ then $A^TA$ is a positive definite matrix

note

Proof:
For $A^TA$ to be a positive definite matrix we need,

$$\vec{x}^T(A^TA)\vec{x}\gt 0;\quad\forall x\in\mathbb{R}^n \char`\\ \vec{0}$$

$\vec{x}^T(A^TA)\vec{x}=(\vec{x}^TA^T)(A\vec{x})$
$\Rightarrow \vec{x}^T(A^TA)\vec{x}=(A\vec{x})^T(A\vec{x})$
Here $A\vec{x}$ is a vector in $\mathbb{R}^n$, so $(A\vec{x})^T(A\vec{x})=\|A\vec{x}\|^2$
$\|A\vec{x}\|^2$ is the sum of squares so it can't be negative $\|A\vec{x}\|^2\geq 0$ and $\|A\vec{x}\|^2$ is $0$ only for $\vec{x}=\vec{0}$
because $\text{Rank}(A)=n$ so columns are independent, so $A\vec{x}=\vec{0}$ if $\vec{x}=\vec{0}$ so it's Null space have only $\vec{0}$
$\Rightarrow \vec{x}^T(A^TA)\vec{x}\gt 0;\quad\forall x\in\mathbb{R}^n \char`\\ \vec{0}$
So we can say that "$A^TA$ is a Positive Definite Matrix"

---

Example $1$

Say $A=\begin{bmatrix} 2&6\\6&19 \end{bmatrix}$ check if $A$ is Positive Definite Matrix or not, using properties described above.
Let's try using the $4^{th}$ property and see if

$$\vec{x}^TA\vec{x}\gt 0;\quad\forall x\in\mathbb{R}^n \char`\\ \vec{0}$$

Say $\vec{x}=\begin{bmatrix} x_1\\x_2 \end{bmatrix}$
$\vec{x}^TA\vec{x}= \begin{bmatrix} x_1&x_2 \end{bmatrix} \begin{bmatrix} 2&6\\6&19 \end{bmatrix} \begin{bmatrix} x_1\\x_2 \end{bmatrix}$

$\vec{x}^TA\vec{x}=2x_1^2+12x_1x_2+19x^2$
Now we want to know that is $2x_1^2+12x_1x_2+19x^2\gt 0$
Now let's use Calculus.

Our function is $f(x_1,x_2)=2x_1^2+12x_1x_2+19x^2$
Let's find the critical points,

• First we take derivative w.r.t. $x_1$.

  $$\begin{matrix} \displaystyle \frac{\partial f(x_1,x_2)}{\partial x_1}=& \displaystyle \frac{\partial (2x_1^2+12x_1x_2+19x^2)}{\partial x_1} \\ \displaystyle \frac{\partial f(x_1,x_2)}{\partial x_1}=&\displaystyle 4x_1+12x_2 \end{matrix}$$

  Now equate $\displaystyle \frac{\partial f(x_1,x_2)}{\partial x_1}=0$

  $$\begin{matrix} 4x_1+12x_2=0 \end{matrix} \quad\tag{\color{red}{1}}$$

• Now we take derivative w.r.t. $x_2$.

  $$\begin{matrix} \displaystyle \frac{\partial f(x_1,x_2)}{\partial x_2}=& \displaystyle \frac{\partial (2x_1^2+12x_1x_2+19x^2)}{\partial x_2} \\ \displaystyle \frac{\partial f(x_1,x_2)}{\partial x_2}=&\displaystyle 12x_1 + 38x_2 \end{matrix}$$

  Now equate $\displaystyle \frac{\partial f(x_1,x_2)}{\partial x_1}=0$

  $$\begin{matrix} 12x_1 + 38x_2=0 \end{matrix} \quad\tag{\color{red}{2}}$$

  $x_1=0$ and $x_2=0$ satisfies this system of equations $(1)\text{ and }(2)$.
  Now we got a point $(x_1,x_2)=(0,0)$ but we don't know that, Is it minimum, maximum or saddle point.
  To check that, Is it minimum, maximum or saddle point. We need $2^{nd}$ derivative test

• $\frac{\partial^2 f(x_1,x_2)}{\partial x_1^2}$

  $$\begin{matrix} \displaystyle \frac{\partial^2 f(x_1,x_2)}{\partial x_1^2}=& \displaystyle \frac{\partial (4x_1+12x_2)}{\partial x_1} \\ \displaystyle \frac{\partial f(x_1,x_2)}{\partial x_1}=&\displaystyle 4 \end{matrix}$$

• $\frac{\partial^2 f(x_1,x_2)}{\partial x_2^2}$

  $$\begin{matrix} \displaystyle \frac{\partial^2 f(x_1,x_2)}{\partial x_2^2}=& \displaystyle \frac{\partial (12x_1 + 38x_2)}{\partial x_2} \\ \displaystyle \frac{\partial^2 f(x_1,x_2)}{\partial x_2^2}=&\displaystyle 38 \end{matrix}$$

• $\frac{\partial^2 f(x_1,x_2)}{\partial x_1x_2}$

  $$\begin{matrix} \displaystyle \frac{\partial^2 f(x_1,x_2)}{\partial x_1x_2}=& \displaystyle \frac{\partial (4x_1+12x_2)}{\partial x_2} \\ \displaystyle \frac{\partial^2 f(x_1,x_2)}{\partial x_1x_2}=&\displaystyle 12 \end{matrix}$$

  (Recommended reading)
  Now we have our Hessian matrix (say $H$)

  $$Hf(x_1,x_2)=\begin{bmatrix} \displaystyle \frac{\partial f}{\partial x_1^2 } & \displaystyle \frac{\partial f}{\partial x_1x_2}\\ \displaystyle \frac{\partial f}{\partial x_2x_1} & \displaystyle \frac{\partial f}{\partial x_2^2} \end{bmatrix}$$

  $Hf(x_1,x_2)=\begin{bmatrix} 4&12\\ 12&38 \end{bmatrix}$
  $\text{det}(Hf(x_1,x_2))\gt 0$ so it rules out the possibility of being a saddle point, so $f(x_1,x_2)$ either have a maximum or a minimum.
  And because $\displaystyle \frac{\partial^2 f(x_1,x_2)}{\partial x_1^2}\gt 0$ so $f(x_1,x_2)$ have a minimum values at $(0,0)$ and that minimum value is $0$

So we had seen that the minimum value of $f(x_1,x_2)=2x_1^2+12x_1x_2+19x^2$ is $0$
$\vec{x}^TA\vec{x}\gt 0$ except at $\vec{x}=\vec{0}$, this means that $A$ is a Positive Definite Matrix

---

Example $2$

Say $A=\begin{bmatrix} 2&6\\6&7 \end{bmatrix}$ check if $A$ is Positive Definite Matrix or not, using properties described above.
Let's try using the $4^{th}$ property and see if

$$\vec{x}^TA\vec{x}\gt 0;\quad\forall x\in\mathbb{R}^n \char`\\ \vec{0}$$

Say $\vec{x}=\begin{bmatrix} x_1\\x_2 \end{bmatrix}$
$\vec{x}^TA\vec{x}= \begin{bmatrix} x_1&x_2 \end{bmatrix} \begin{bmatrix} 2&6\\6&7 \end{bmatrix} \begin{bmatrix} x_1\\x_2 \end{bmatrix}$

$\vec{x}^TA\vec{x}=2x_1^2+12x_1x_2+7x^2$
Now we want to know that is $2x_1^2+12x_1x_2+7x^2\gt 0$
Now let's use Calculus.

Our function is $f(x_1,x_2)=2x_1^2+12x_1x_2+7x^2$
Let's find the critical points,

• First we take derivative w.r.t. $x_1$.

  $$\begin{matrix} \displaystyle \frac{\partial f(x_1,x_2)}{\partial x_1}=& \displaystyle \frac{\partial (2x_1^2+12x_1x_2+7x^2)}{\partial x_1} \\ \displaystyle \frac{\partial f(x_1,x_2)}{\partial x_1}=&\displaystyle 4x_1+12x_2 \end{matrix}$$

  Now equate $\displaystyle \frac{\partial f(x_1,x_2)}{\partial x_1}=0$

  $$\begin{matrix} 4x_1+12x_2=0 \end{matrix} \quad\tag{\color{red}{1}}$$

• Now we take derivative w.r.t. $x_2$.

  $$\begin{matrix} \displaystyle \frac{\partial f(x_1,x_2)}{\partial x_2}=& \displaystyle \frac{\partial (2x_1^2+12x_1x_2+7x^2)}{\partial x_2} \\ \displaystyle \frac{\partial f(x_1,x_2)}{\partial x_2}=&\displaystyle 12x_1 + 14x_2 \end{matrix}$$

  Now equate $\displaystyle \frac{\partial f(x_1,x_2)}{\partial x_1}=0$

  $$\begin{matrix} 12x_1 + 14x_2=0 \end{matrix} \quad\tag{\color{red}{2}}$$

  $x_1=0$ and $x_2=0$ satisfies this system of equations $(1)\text{ and }(2)$.
  Now we got a point $(x_1,x_2)=(0,0)$ but we don't know that, Is it minimum, maximum or saddle point.
  To check that, Is it minimum, maximum or saddle point. We need $2^{nd}$ derivative test

• $\frac{\partial^2 f(x_1,x_2)}{\partial x_1^2}$

  $$\begin{matrix} \displaystyle \frac{\partial^2 f(x_1,x_2)}{\partial x_1^2}=& \displaystyle \frac{\partial (4x_1+12x_2)}{\partial x_1} \\ \displaystyle \frac{\partial f(x_1,x_2)}{\partial x_1}=&\displaystyle 4 \end{matrix}$$

• $\frac{\partial^2 f(x_1,x_2)}{\partial x_2^2}$

  $$\begin{matrix} \displaystyle \frac{\partial^2 f(x_1,x_2)}{\partial x_2^2}=& \displaystyle \frac{\partial (12x_1 + 14x_2)}{\partial x_2} \\ \displaystyle \frac{\partial^2 f(x_1,x_2)}{\partial x_2^2}=&\displaystyle 14 \end{matrix}$$

• $\frac{\partial^2 f(x_1,x_2)}{\partial x_1x_2}$

  $$\begin{matrix} \displaystyle \frac{\partial^2 f(x_1,x_2)}{\partial x_1x_2}=& \displaystyle \frac{\partial (4x_1+12x_2)}{\partial x_2} \\ \displaystyle \frac{\partial^2 f(x_1,x_2)}{\partial x_1x_2}=&\displaystyle 12 \end{matrix}$$

  (Recommended reading)
  Now we have our Hessian matrix (say $H$)

  $$Hf(x_1,x_2)=\begin{bmatrix} \displaystyle \frac{\partial f}{\partial x_1^2 } & \displaystyle \frac{\partial f}{\partial x_1x_2}\\ \displaystyle \frac{\partial f}{\partial x_2x_1} & \displaystyle \frac{\partial f}{\partial x_2^2} \end{bmatrix}$$

  $Hf(x_1,x_2)=\begin{bmatrix} 4&12\\ 12&14 \end{bmatrix}$
  $\text{det}(Hf(x_1,x_2))\lt 0$ so $(0,0)$ is the saddle point and $f(0,0)=0$.

So we had seen that $f(x_1,x_2)=2x_1^2+12x_1x_2+19x^2$ has neither maximum nor minimum point.
$(0,0)$ is a saddle point of $f(x_1,x_2)$ and $f(0,0)=0$
So $\vec{x}^TA\vec{x}\ngtr 0 ;\quad\forall x\in\mathbb{R}^n \char`\\ \vec{0}$ , this means that $A$ is a not a Positive Definite Matrix