【Study Notes】Mathematics for Machine Learning: Linear Algebra (Week 2)

Introduction

This is a study note of the course Mathematics for Machine Learning: Linear Algebra on the coursera. In week 2 section, we will look at vector’s modulus, and look at a way to combine vectors together to get the dot product, which can take us on to find the scalar and vector projections. In addition, basis vector and vector space will be defined in this section, and we will learn how to prove linear independence and linear combination.

Modulus

Modulus is length of a vector, so if a vector is given $r=a\hat i+b\hat j = \begin{bmatrix} a\\b\\ \end{bmatrix}$, the modulus of vector $r$ is equal to $|r| = \sqrt{a^2+b^2}$ . In addition, we also use dot product to get the modulus of a vector, i.e. we can get the modulus of a vector by the vector dotted by itself, $|r|^2=r\cdot r= \begin{bmatrix} a\\b\\ \end{bmatrix}\cdot \begin{bmatrix} a\\b\\ \end{bmatrix}=a^2+b^2$ .

Dot Product

The dot product is a scalar number given by multiplying the components of the vector in turn, and adding those up. e.g. given two vector $r = \begin{bmatrix}r_1\\r_2\\ \vdots \\r_n\\ \end{bmatrix}$ and $s = \begin{bmatrix}s_1\\s_2\\\vdots\\s_n\\ \end{bmatrix}$, we can find the dot product is $r\cdot s =r_1s_1+r_2s_2+\cdots+r_ns_n$ .

• commutative: $r\cdot s$ is equal to $s\cdot r$ .
• distributive: $r\cdot (s+t)$ is equal to $r\cdot s+r\cdot t$ .
• associative: $r\cdot(as)$ is equal to $a(r\cdot s)$, where $a$ is a scalar.

Interestingly, if we take the dot product of a vector with itself, we can get the square of its size of its modulus. i.e. $|r|^2=r\cdot r= \begin{bmatrix} a\\b\\ \end{bmatrix}\cdot \begin{bmatrix} a\\b\\ \end{bmatrix}=a^2+b^2$.

Cosine & Dot Product

Let’s start from a right triangle. For a right triangle, it have 3 egdes, respectively called legs if the side adjacent to the right angle, and called hypotenuse if the side oppsite the right angle.

According consine rule, we get a equation.
$c^2=a^2+b^2-2ab\cos\theta$
Then we define a vector $s$ and a vector $r$ as the following figure.

$|r-s|^2=|r|^2+|s|^2-2|r||s|\cos\theta \tag{1}$
Let’s make dot product of $r-s$ with itself.
$\begin{aligned} (r-s)\cdot(r-s)&=r\cdot r-s\cdot r-s\cdot r+s\cdot s\\ &=|r|^2-2s\cdot r + |s|^2\tag{2}\\ \end{aligned}$
Then, we do that comparison between $(1)$ and $(2)$.
$r\cdot s=|r||s|\cos\theta$

• $r\cdot s=|r||s|\cos\theta$
• $\theta=90^\circ$, $r\cdot s=|r||s|\cos\theta=0$.
• $\theta=0^\circ$, $r\cdot s=|r||s|\cos\theta=|r||s|$
• $\theta=180^\circ$, $r\cdot s=|r||s|\cos\theta=-|r||s|$

Vector Projection

Given two vector as following figure, we can make the projection of the vector $s$ onto the vector $r$.

Obviously, we can get $\cos\theta={\text{adj}\over{\text{hyp}}}={{\text{adj}}\over{|s|}}$ in the right triangle.
In addition, we use dot product to get a equation $r\cdot s = |r||s|\cos\theta$.
So, $|s|\cos\theta={{r\cdot s}\over{|r|}}$ is used to take the scalar projection of vetor $s$ onto the vector $r$.
We also can get the vector projection by using ${{r\cdot s} \over {|r||r|}}\times r={{r\cdot s} \over {r\cdot r}}\times r$.

• scalar projection: $|s|\cos\theta={{r\cdot s}\over{|r|}}$
• vector projection: ${{r\cdot s} \over {|r||r|}}\times r={{r\cdot s} \over {r\cdot r}}\times r$

Linear Dependence & Independence

Linear Dependence

Vectors are linearly dependent if it is possible to write one vector as a linear combination of the others. For example, the vectors $a$, $b$ and $c$ are linearly dependent if $a=q_1b+q_2c$ where $q_1$ and $q_2$ are scalars.

Linear Independence

We say that two vectors are linearly independent if they are not linearly dependent, that is, we cannot write one of the vectors as a linear combination of the others.