# Neural Network Linear Algreba Prior to this topic is the Perceptron and Multilayer Perceptron. This is information about linear algebra specific to Neural Networks. ## Vectors In an 2-dimensional space, a vector is simply a line, it has a direction and a magnitude. We can represent this vector as: `[x y]`. In a 3-dimensional space the vector would be three directions and magnitudes: `[x y z]`. In 4, 5, ... up to N dimensional space, a vector would be a single list of direction and magnitudes, one for each dimension to represent itself. I have worked with vectors often in game development. A 2D game may have a vector of `[1.3 -7.0]` which could be the direction and speed of a bullet. The two values constitute a direction in an x/y coordinate system and their magnitude is like their force. These vector values can be normalized to within a range of -1 to 1 to get only their direction without their magnitude of force. **Note** these are other ways we can notate a vector: ``` text [x y] [x] // one tall square bracket on each side [y] (x) // one tall pair of parenthsis (y) ``` ### Vector Operations #### 1) Scalar Each element is multiplied by the scalar and produces another vector. ``` text [2] * 2 = [4] [3] [6] ``` #### 2) Element Wise Each element is added to another vector and produces another vector. ``` text [2] + [-1] = [1] [3] [ 5] [8] ``` #### 3) Dot Product Two vectors multiplied together and summed to produce a scalar. Each element in the same row multiplied together. The result of each row added together. ``` text [2] . [-1] = ((2 * -1) + (3 * 5)) = 13 [3] [ 5] ``` ## Matrix A matrix, instead of a linear list of values, is a 2-dimensional grid of values. They are referred to by rows x columns. This is an example of a 2x3 matrix: ``` text [a b c] [d e f] ``` Here is an example of a matrix implementation in Javascript ``` javascript function Matrix(rows, cols) { this.rows = rows; this.cols = cols; this.matrix = []; // Initialize our 2d array with 0 for (var i=0; i < rows; i++) { this.matrix[i] = []; for (var j=0; j < cols; j++) { this.matrix[i][j] = 0; } } } var m = new Matrix(2, 3); ``` ### Matrix Operations #### 1) Scalar Each element is multiplied by the scalar and produces a matrix of the same size. ``` text [ 2 3] * 2 = [ 4 6] [-4 9] [-8 18] [ 2 3] + 2 = [ 4 5] [-4 9] [-2 11] ``` Here is an implementation of this method in our Javascript example. ``` javascript // Multiplication Matrix.prototype.scalarMultiply = function (scalar) { for (var i=0; i < rows; i++) { for (var j=0; j < cols; j++) { this.matrix[i][j] *= scalar; } } } // Addition Matrix.prototype.scalarAdd = function (scalar) { for (var i=0; i < rows; i++) { for (var j=0; j < cols; j++) { this.matrix[i][j] += scalar; } } } ``` #### 2) Element Wise Each element is added to the corresponding element in the matrices producting a matrix of the same size. ``` text [a b] + [e f] = [a+e b+f] [c d] [g h] [x+g d+h] ``` We can update our scalarMultiple and scalarAdd to work with other matrices. ``` javascript // Multiplication Matrix.prototype.multiply = function (n) { if (n instanceof Matrix) { for (var i=0; i < rows; i++) { for (var j=0; j < cols; j++) { this.matrix[i][j] *= n.matrix[i][j]; // element wise (Hadamard Product) } } } else { for (var i=0; i < rows; i++) { for (var j=0; j < cols; j++) { this.matrix[i][j] += n // scalar } } } } // Addition Matrix.prototype.add = function (n) { if (n instanceof Matrix) { for (var i=0; i < rows; i++) { for (var j=0; j < cols; j++) { this.matrix[i][j] += n.matrix[i][j]; // element wise } } } else { for (var i=0; i < rows; i++) { for (var j=0; j < cols; j++) { this.matrix[i][j] += n // scalar } } } } ``` #### 4) Dot Product A dot product can only be calculated on two matrices of the same size. The number of columns of one must equal the number of rows of the other. ``` text A = [a b c] 2x3 [d e f] B = [g h] 3x2 [i j] [k l] A . B ``` These two matrices can be dot producted'ed. But they are not communicative, meaning they cannot be multiplied in either order and produce the same result, `A . B != B . A`. The resulting matrix will always be the number of rows of A by the number of colums of B. `A . B = 2x2 Matrix`. To calculate the values of the resultant 2x2 matrix we will perform the Vector Dot Product of each row of A to its corresponding column of B. - Row 1 in A dot Column 1 in B for position 1,1. - Row 1 in A dot Column 2 in B for position 1,2. - Row 2 in A dot Column 1 in B for position 2,1. - Row 2 in A dot Column 2 in B for position 2,2. ``` text A = [a b c] 2x3 [d e f] B = [g h] 3x2 [i j] [k l] A . B = C C = [p11 p12] [p21 p22] C = [ (a*g + b*i + c*k) (a*h + b*j + c*l) ] [ (d*g + e*i + f*k) (d*h + e*j + f*l) ] ``` We can expand our Javascript example Matrix to have a dot product operation. ``` javascript Matrix.prototype.dot = function (n) { if (n instanceof Matrix) { // check size if (this.cols !== n.rows) { return undefined; } let result = new Matrix(this.rows, n.cols); for (let i=0; i < result.rows; i++) { for (let j=0; j < result.cols; j++) { // dot product of values in col let sum = 0; for (let k=0; k < this.cols; k++) { sum += this.matrix[i][k] * n.matrix[k][j]; } result.matrix[i][j] = sum; } } return result; } else { return undefined; } } ``` #### 5) Transpose Another important operation for matrices in neural networks is the transpose operation. It basically makes the rows the columns and columns the rows. ``` text [a b c] [d e f] transposed becomes: [a d] [b e] [c f] ``` And we can implement in our Javacript Matrix as: ``` javascript Matrix.prototype.transpose = function() { let result = new Matrix(this.cols, this.rows); for (var i=0; i < this.rows; i++) { for (var j=0; j < this.cols; j++) { result.matrix[j][i] = this.matrix[i][j]; } } return result; } ```