Introduction to Deep Learning Fundamentals at IIT Kharagpur

Computer Science and Engineering| Indian Institute of Technology Kharagpur cse.iitkgp.ac.in Deep Learning CS60010 Abir Das, Sudeshna Sarkar Computer Science and Engineering Department Indian Institute of Technology Kharagpur http://cse.iitkgp.ac.in/~adas/ http://www.facweb.iitkgp.ac.in/~sudeshna/

Computer Science and Engineering| Indian Institute of Technology Kharagpur cse.iitkgp.ac.in Agenda To brush up basics of Linear Algebra. 04 Jan, 2020

Computer Science and Engineering| Indian Institute of Technology Kharagpur cse.iitkgp.ac.in Resources Deep Learning , I. Goodfellow, Y. Bengio, A. Courville. (Chapter 2) 04 Jan, 2020

Computer Science and Engineering| Indian Institute of Technology Kharagpur cse.iitkgp.ac.in Scalars, Vectors, Matrices and Tensors Scalars: They are single numbers. Denoted mostly as lowercase variable names. ?,?,? Vectors: Vectors are array of numbers. Typically denoted as boldface lowercase variable names. Individual components are treated as scalars. ? = ?1,?2, ,?? Matrices: Matrices are 2-D array of numbers. Typically denoted as boldface uppercase variable names. ?1,1 ?1,? ??,1 ??,? ??,1 ??,? ?,? = ?1,?2, ,?? ?,? = [?1,?2, ,??]? ?1,1 ?1,? ? = ,? = Tensors: Arrays with more than 2 dimensions are generally called Tensors. 04 Jan, 2020

Computer Science and Engineering| Indian Institute of Technology Kharagpur cse.iitkgp.ac.in Matrix Operations Transpose: Transpose of a matrix is the mirror image of the matrix across the diagonal line, called the main diagonal of the matrix. The transpose of a matrix ? is denoted as ??, where ?? Addition: Matrices can be added as long as they have the same shape, by adding their corresponding elements. ? = ? + ?, where ??,?= ??,?+??,? Multiplication: In order for the product of the two matrices ? and ? to be defined, ? must have the same number of columns as that of the rows of ?. If ? is of shape ? ? and ? is of shape ? ? then ? is of shape ? ?, the product operation ? = ?? is defined by, ?,?= ??,? ??,?= ??,???,? ? 04 Jan, 2020

Computer Science and Engineering| Indian Institute of Technology Kharagpur cse.iitkgp.ac.in Matrix Operations Elementwise or Hadamard Product: It s a matrix containing the product of the individual elements. It is denoted as ? ? Dot Product: The dot product between two vectors ? and ? of the same dimensionality is the matrix product ??? Matrix product is not commutative (?? = ??does not always hold). However, the dot product between two vectors is commutative, i.e., ??? = ??? Let us consider a system of linear equations as follows, ?1,1?1+ ?1,2?2+ + ?1,???= ?1 ?2,1?1+ ?2,2?2+ + ?2,???= ?2 ??,1?1+ ??,2?2+ + ??,???= ?? 04 Jan, 2020

Computer Science and Engineering| Indian Institute of Technology Kharagpur cse.iitkgp.ac.in System of Linear Equations ?1,1?1+ ?1,2?2+ + ?1,???= ?1 ?2,1?1+ ?2,2?2+ + ?2,???= ?2 ??,1?1+ ??,2?2+ + ??,???= ?? We can write these as, ?1,1 ?1,2 ?2,1 ?2,2 . ??,1 ??,2 . ?1,? ?2,? . ??,? ?1 ?2 . ?? ?1 ?2 . ?? = . ?? = ? 04 Jan, 2020

Computer Science and Engineering| Indian Institute of Technology Kharagpur cse.iitkgp.ac.in System of Linear Equations ?? = ? ? ? ?,? ?,? ? There can be 3 possibilities ? = ? and det ? 0, the solution is unique, ? = ? 1?. When is det ? = 0? ? < ? - underdetermined problem (No. of equations < No. of variables). Infinitely many solutions. What can be a meaningful solution? ? > ? - overdetermined problem (No. of equations > No. of variables). No solution. What can be a meaningful solution? We need to be familiar with the concept of norms for this. 04 Jan, 2020

Computer Science and Engineering| Indian Institute of Technology Kharagpur cse.iitkgp.ac.in Eigenvalues and Eigenvectors Suppose ? is a matrix. The question is does there exist any vector ? for ? so that the operation ?? gives a vector which is nothing but a stretched (and not rotated) version of the vector ?? i.e., ?? = ??,or ?? ? ? = ?. For non-trivial solution det ?? ? = ?? ? = 0 If ? ? ?, then ?? ? = 0 will be a ?? order equation. That means you can have ? solutions of ? - such ? s are called eigenvalues (real or complex conjugate). The corresponding vector ? s are the eigenvectors. Remember that eigenvectors are not unique. This is because if ? is an eigenvector, then a? is also the same eigenvector (as it satisfies ??? = ???). So, we are satisfied with the direction of the eigenvectors only. 04 Jan, 2020

Computer Science and Engineering| Indian Institute of Technology Kharagpur cse.iitkgp.ac.in Standard Results on Eigenvalues and Eigenvectors If ?1,?2, ,?? are eigenvalues of ?, then for any positive integer ?, ?1 eigenvalues of ?, you don t have to compute the eigenvalues of ??. If ? is a non-singular or invertible matrix with eigenvalues ?1,?2, ,??, then ?1 For triangular matrix (upper, lower or diagonal), the eigenvalues are the diagonal elements itself. If a square matrix ? ? ?, is symmetric then all its eigenvalues are real and it has ? linearly independent eigenvectors. The reverse is also true i.e., if a square matrix ? ? ? has ? real eigenvalues and ? real orthogonal eigenvectors, then the matrix is symmetric. ? are eigenvalues of ??. The significance is that if you have ?,?2 ?, ,?? 1 are eigenvalues of ? ?. 1,?2 1, ,?? 04 Jan, 2020

Computer Science and Engineering| Indian Institute of Technology Kharagpur cse.iitkgp.ac.in Standard Results on Eigenvalues and Eigenvectors A matrix ? ? ? is positive definite if ? ? ?,???? > ?. It is positive semi-definite if ???? ? A matrix ? ? ? is negative definite if ? ? ?,???? < ?. It is negative semi-definite if ???? ? If ? is positive definite, ??> 0 ? If ? is positive semi-definite, ?? 0 ? If ? is negative definite, ??< 0 ? If ? is negative semi-definite, ?? 0 ? 04 Jan, 2020

Computer Science and Engineering| Indian Institute of Technology Kharagpur cse.iitkgp.ac.in Vector Norms Vector norm is a real valued function (i.e., its output is always a real number) with the following properties. ||?|| > 0 and ||?|| = 0 only if ? = 0 ?? = ? ||?|| ? + ? ||?|| + ||?|| Triangle inequality 1 ? ?? norm: ? ? ? ?= ?=1 ?? ?0 norm: ? ? norm: ? 0= Number of non-zero elements in ? = max ? |??| 04 Jan, 2020

Computer Science and Engineering| Indian Institute of Technology Kharagpur cse.iitkgp.ac.in Matrix/Induced Norms ?? ? = max ? ? ? You can think of matrix norm as the multiplying capacity of the matrix. ||?|| > 0 and ||?|| = 0 only if ? = ? ? ? = ? ||?|| ? + ? ||?|| + ||?|| Triangle inequality ?? ||?||||?|| Additional 04 Jan, 2020

Computer Science and Engineering| Indian Institute of Technology Kharagpur cse.iitkgp.ac.in Orthogonality Vectors: Two vectors ? and ? ? are orthogonal if ??? = 0. We write ? ?. In general, we have ? + ? ||?|| + ||?|| However, ? ? ?= 2+ ? 2 ? + ? ? This is the Pythagorean theorem. 04 Jan, 2020

Computer Science and Engineering| Indian Institute of Technology Kharagpur cse.iitkgp.ac.in Orthogonality Matrix: A matrix ? is orthogonal when it has orthogonal columns | | | ?.?? = 1 ? = ?? ?1 | ?2 | ?? | ?.?? = 0,? ? ?? Properties: ???= ??? = ? ? 1= ?? ?? ?= ?????? = ??? = 2 Length preserving. ? ? 2 04 Jan, 2020

Computer Science and Engineering| Indian Institute of Technology Kharagpur cse.iitkgp.ac.in System of Linear Equations ?? = ? ? ? ?,? ?,? ? There can be 3 possibilities ? = ? and det ? 0, the solution is unique, ? = ? 1. When is det ? = 0? ? < ? - underdetermined problem (No. of equations < No. of variables). Infinitely many solutions. What can be a meaningful solution? minimize ? = ? ? = ??(???) 1? ? > ? - overdetermined problem (No. of equations > No. of variables). No solution. What can be a meaningful solution? Minimize ? = ?? ? ? = (???) 1??? 2subject to ?? = ? 2 04 Jan, 2020

Computer Science and Engineering| Indian Institute of Technology Kharagpur cse.iitkgp.ac.in Thank You!!