A well-known theorem is Diagonalization, where one of the factors is a diagonal matrix. In this paper we will be describing a similar way to factor/decompose a non-square matrix. The key to both of these ways to factor is eigenvalues and eigenvectors

Bonaccorso, Krystal

Incognito, Andrew

English

CCU Digital Commons

Coastal Carolina University CCU Digital Commons Honors Theses Honors College and Center for Interdisciplinary Studies Spring 5-6-2020 Singular Value Decomposition Krystal Bonaccorso Coastal Carolina University, krbonacco@coastal.edu Andrew Incognito aincogni@coastal.edu Follow this and additional works at: https://digitalcommons.coastal.edu/honors-theses  Part of the Algebra Commons Recommended Citation Bonaccorso, Krystal and Incognito, Andrew, "Singular Value Decomposition" (2020). Honors Theses. 353. https://digitalcommons.coastal.edu/honors-theses/353 This Thesis is brought to you for free and open access by the Honors College and Center for Interdisciplinary Studies at CCU Digital Commons. It has been accepted for inclusion in Honors Theses by an authorized administrator of CCU Digital Commons. For more information, please contact commons@coastal.edu.  [Singular Value Decomposition]   By  [Krystal Bonaccorso]  [Applied Mathematics]            Submitted in Partial Fulfillment of the Requirements for the Degree of Bachelor of [Science] In the HTC Honors College at Coastal Carolina University  [Fall 2019]       Louis E. Keiner Director of Honors HTC Honors College       [Andrew Incognito] [Singular Value Decomposition] [Mathematics] [College of Science]             Singular Value DecompositionKrystal BonaccorsoOctober 6, 2018AbstractA well-known theorem is Diagonalization, where one of the factors isa diagonal matrix. In this paper we will be describing a similar way tofactor/decompose a non-square matrix. The key to both of these ways tofactor is eigenvalues and eigenvectors.1 IntroductionThe theory of singular value decomposition was established by mathemati-cians named Eugenio Beltrami, Camille Jordan, James Joseph Sylvester, Er-hard Schmidt, and Hermann Weyl. Beltrami, Jordan and Sylvester came to thesingular value decomposition through linear algebra in the late 1800s. Then inthe early 1900s Schmidt and Weyl came to singular value decomposition by inte-gral equations. Singular value decomposition is a generalization of the spectraldecomposition, also known as factor analysis.2 Eigenvectors and EigenvaluesSingular Value Decomposition is related to eigenvectors and eigenvalues.Definition: An eigenvector of an n x n matrix A is a nonzero vector ~x ∈ Rn1such that A~x = λ~x for some scalar λ ∈ R.Definition: A scalar λ is called an eigenvalue of A if there is a nontrivial solu-tion, ~x of A~x = λ~x.Example:Let A = 1 65 2 , ~u = 6−5 , and ~v = 3−2.Are ~u and ~v eigenvectors of A?A~u = 1 65 2 6−5 =−2420 = −4 6−5 = −4~uA~v = 1 65 2 3−2 =−911 6= λ 3−2Therefore, ~u is an eigenvector with the eigenvalue of -4, but ~v is not a eigenvectorof A because A~v is not a multiple of ~v.3 DiagonalizationThe Diagonalization Theorem says that an n × n matrix A is diagonalizable ifand only if A has n linearly independent eigenvectors.Example: Find a diagonal matrix D and a non-singular matrix P such thatA = PDP−1, if possible.A =2 0 01 2 1−1 0 12Step 1: Find the eigenvaluesdet(λI −A) =∣∣∣∣∣∣∣∣∣∣λ− 2 0 0−1 λ− 2 −11 0 λ− 1∣∣∣∣∣∣∣∣∣∣= (λ− 2)2(λ− 1) = 0Eigenvalues: λ = 1 and λ = 2.Step 2: Find linearly independent eigenvectors.λ = 1 : ~v1 =0−11λ = 2 : ~v2 =010 , ~v3 =−101Step 3: Construct P and D using the eigenvectors and their correspondingeigenvalues.P =0 0 −1−1 1 01 0 1 and D =1 0 00 2 00 0 24 The Spectral TheoremThe Spectral Theorem for Symmetric Matrices states that an n x n symmetricmatrix A has the following properties:1. A has n real eigenvalues, counting multiplicity.32. The dimension of the eigenspace for each eigenvalue λ equals the multi-plicity of λ as a root of the characteristic equation.3. The eigenspaces are mutually orthogonal, in the sense that eigenvectorscorresponding to different eigenvalues are orthogonal.4. A is orthogonally diagonalizable.Example: LetA =0 2 22 0 22 2 0The characteristic polynomial of A is determined by det(λIA) = 0) giving us,(λ+ 2)2(λ− 4)so the eigenvalues areλ1 = −2, λ2 = −2, andλ3 = 4.Next, to find the eigenvectors corresponding to λ1 and λ2, we solve the linearsystem (−2I −A)x = 0−2 −2 −2−2 −2 −2−2 −2 −2x1x2x3 =0004The eigenvectors will be:~v1 =−110 and~v2 =−101We can see that ~v1 and ~v2 are not orthogonal, since ~v1 · ~v2 6= 0. We can use theGram-Schmidt process. We will let~y1 = ~v1 =−110and~y2 = ~v2 = ~v2 −~x2 · ~y1~y1 · ~y1− 12− 121Let~y∗2 = 2~y2 =−1−12Now the set ~y1 and ~y∗2 is an orthogonal set of eigenvectors. After normalizingthere vectors, we obtain− 1√21√20 and− 1√6− 1√62√65Now we will find the eigenvector that corresponds to the eigenvalue 4, using theequation (4I −A)x = 04 −2 −2−2 4 −2−2 −2 4x1x2x3 =000The eigenvector will be~v3 =111After normalizing this vector we will get,~v3 =1√31√31√3Now we will create P, which is made up of the columns being the eigenvectorswe found.P =− 1√2− 1√61√31√2− 1√61√30 2√61√3Now we can also find D, by using the eigenvalues that correspond to the eigen-vectors we have in P.D =−2 0 00 −2 00 0 46We will also have to find P−1 which is the same as PT .PT =− 1√21√20− 1√6− 1√62√61√31√31√3Putting them all together in the equation A = PDPT . We will getA =− 1√2− 1√61√31√2− 1√61√30 2√61√3−2 0 00 −2 00 0 4− 1√21√20− 1√6− 1√62√61√31√31√35 The Singular Value DecompositionThe decomposition of an m× n matrix A includes an m× n “diagonal” matrixΣ of the formΣ = D 00 0where D is an r × r diagonal matrix.Definition: The number of linearly independent rows or columns is called therank.Definition: A diagonal matrix is a matrix where the entries aij = 0 if i 6= j.Definition: An orthogonal matrix means Q−1 = QT .Theorem: Let A be an m×n matrix with rank r. Then there exists an m×nmatrix Σ where the diagonal entries in D are the first r singular values of A,σ1 ≥ σ2 ≥ · · · ≥ σr > 0, where σi is the square root of an eigenvalue of ATA,and there exists an m×m orthogonal matrix U and an n×n orthogonal matrixV such thatA = UΣV T .7Proof: To determine V and Σ we consider the matrix ATA.(1) ATA is a positive semi-definite symmetric n× n matrix. Therefore, ATA =QΛQT , where Q is an orthogonal matrix of eigenvectors and Λ is a diagonalmatrix of non-negative eigenvalues of ATA.(2) If A = UΣV T thenATA = V ΣTUTUΣV T = V (ΣTΣ)V TTherefore we take V = Q and Σ to have diagonal component D consisting ofthe square roots of the diagonal entries of Λ.To determine U note that AAT = PΛPT , with the same Λ since if ATA~v = λ~v,then AATA~v = λA~v .AAT = UΣV TV ΣTUT = U(ΣΣT )UTTherefore, we take U = P , the eigenvectors of AAT .More precisely, we can take the columns of U to be ~ui =A~viσi.These are orthogonal since(A~viσi)T (A~vjσj)=~vTi ATA~vjσiσj=~vTi σ2j~vjσiσj=σjσi~vTi ~vj = 0since vi and vj are orthogonal.Thus when A is a non-square matrix, A = UΣV T , where U consists of eigen-vectors from AAT , V consists of eigenvectors from ATA and Σ is an non-squarematrix with a diagonal component. Moreover, we have the eigen-esque equation:A~vi = σi~ui, i = 1, ..., rProposition: The symmetric matrices ATA and AAT have the same eigenval-8ues.SupposeATA~v = λ~vApplying A to both sides gives,AATA~v = Aλ~vwhich is equivalent toAAT (A~v) = λ(A~v)A similar argument shows that if λ is an eigenvalue of AAT with eigenvector~u, then λ is an eigenvalue of ATA with eigenvector AT~u. Note this proof alsoshows a relationship between the eigenvectors of ATA and AAT . In fact, if ~v isan eigenvector of ATA then A~v is an eigenvector of AAT .Definition: A symmetric matrix A is called positive semidefinite if all of itseigenvalues are non-negative.Proposition: The matrix ATA is a positive semidefinite matrix.SupposeATA~v = λ~vTake the magnitude of A~v and square it. This means the vector will be dottedwith itself.||A~v||2 = A~v ·A~vWhen taking a dot product it’s the transpose of the vector times the vector.= (A~v)T (A~v)The transpose of a product is the product of the transposes in the reverse order,9gives= (~vTAT )(A~v)Next, using associativity to regroup and using the fact λ is a eigenvalue of ATAwith eigenvector ~v, i.e. ATA~v = λ~v, gives(~vTAT )(A~v) = ~vT (ATA~v) = ~vT (λ~v)Since scalars commute with matrices, we can rewrite the equation so that λ isin the front:= λ~vT~vThe dot product of a vector is the transpose times itself, which is what we haveabove, so this equals= λ~v · ~vThis is the magnitude of the vector:= λ||~v||2.Thus we have||A~v||2 = λ||~v||2.Since the squares are non-negative, we can conclude thatλ ≥ 0,and hence ATA is positive semidefinite.Example: Construct a singular value decomposition of A = 4 11 148 7 −2Step 1: Compute ATA10ATA =4 811 714 −24 11 148 7 −2 =80 100 40100 170 14040 140 200Step 2: Use the equation det(λI − ATA) = 0 to find the eigenvalues of ATA.Consider the matrix 110 (ATA)det(λI −ATA) = detλ− 8 −10 −4−10 λ− 17 −14−4 −14 λ− 20= λ− 8∣∣∣∣∣∣∣λ− 17 −14−14 λ− 20∣∣∣∣∣∣∣+ 10∣∣∣∣∣∣∣−10 −14−4 λ− 20∣∣∣∣∣∣∣− 4∣∣∣∣∣∣∣−10 λ− 17−4 −14∣∣∣∣∣∣∣= (λ− 8)(λ− 17)(λ− 20)− 196 + 10((−10)(λ− 20)− 56)− 4(140 + 4(λ− 17))= λ3 − 45λ2 + 324λ = λ(λ2 − 45λ+ 324)The eigenvalues would equal 36, 9 and 0, but now multiply them by 10 to geteigenvalues for ATA.λ1 = σ21 = 360, λ2 = σ22 = 90 and λ3 = 0.Step 3: Use the equation (λI−ATA) = 0 to find the eigenvectors of the matrixabove.Consider the the equation 110 (λI − ATA) = 0. When distributing the 110 intothe parentheses we would get the equation (λI10 −ATA10 ).(36I − ATA10) =28 −10 −4−10 19 −14−4 −14 16 = 0When (36I − ATA10 ) is put into row reduced echelon form, it gives us:111 0 − 120 1 −10 0 0 = 0Where x1 =t2 x2 = t and x3 = t. If we let t = 2. Therefore x1 = 1, x2 = 2and x3 = 2. We will have to divide by 3 to normalize (length = 1) the vector.~v1 =132323To find the next eigenvector we use(9I − ATA10) =1 −10 −4−10 −8 −14−4 −14 −11 = 0When (9I − ATA10 ) is put into row reduced echelon form, it gives us:1 0 10 1 120 0 0 = 0Where x1 = -t x2 =-t2 and x3 = t. If we let t = 2. Therefore x1 = -2, x2 = -1and x3 = 2. We will have to divide by 3 to normalize (length = 1) the vector.~v2 =− 23− 132312To find the next eigenvector we use(0I − ATA10) =−8 −10 −4−10 −17 −14−4 −14 −20 = 0When (0I − ATA10 ) is put into row reduced echelon form, it gives us:1 0 −20 1 20 0 0 = 0Where x1 = 2t x2 = -2t and x3 = t. If we let t = 1. Therefore x1 = 2, x2 = -2and x3 = 1. We will have to divide by 3 to normalize (length = 1) the vector.~v3 =23− 2313Step 4: Create the matrix V, whose columns are the eigenvectors of ATA.V =[~v1 ~v2 ~v3]=13 −232323 −13 −23232313Step 5: To find the singular values of A, take the square roots of the eigenvaluesof ATA.σ1 = 6√10, σ2 = 3√10, σ3 = 0Step 6: Make a matrix D with the non-zero singular values in the diagonal and13then create the pseudo-diagonal matrix Σ containing D and block matrices ofzeros.D =6√10 00 3√10 ,Σ = [D 0] =6√10 0 00 3√10 0Step 7: Create the matrix U, whose columns are the eigenvectors of AAT . Find~u1 and ~u2 by putting the eigenvector that corresponds with it over the singularvalue to normalize (length = 1) that vector.~u1 =1σ1A~v1 =16√10186 = 3√101√10~u2 =1σ2A~v2 =13√10 3−9 = 1√10− 3√10U = 3√10 1√101√10− 3√10Step 8: Put all of the pieces into the equation A = UΣV T . PUT the ~u1 and~u2 together to get the matrix U, put Σ next to it and finally transpose V.A = 3√10 1√101√10− 3√10 6√10 0 00 3√10 0132323− 23 −122323 −23136 ApplicationsSingular value decomposition is a very useful decomposition. One of the ap-plications is singular value decomposition is called data compression. This iswhere you consider some matrix A with rank five hundred. If we had to encode14this matrix on a computer it would take a lot of memory. We might want toapproximate this matrix with rank one hundred. We can prove this by takingthe n largest singular values of A, replacing the rest with zero and recomputingUΣV T gives you the best n rank approximation to the matrix. Also the totalof the first n singular values divided by the sum of all the singular values isthe percentage of information that those singular values contain. This yields aweird and quick algorithm for matrices and we can use SVD. We will drop allbyt a few singular values and then recompute the approximated matrix. Sincewe only need to store the columns of U and V that get used, we greatly reducethe memory usage. We can use this by converting a picture of a tiger to blackand white and then treating this tiger as a matrix, where each element is thepixel intensity at the relevant location. It is easier to graph the singular valuesof the tiger picture. When we graph the singular values you can see that justfifty of the singular values already make up over 70 percent of the informationcontained in the tiger picture. The last thing that will have to be done is thatwe will have to take some approximations and plot them to see what is the bestone to use to form the tiger picture clear. By using SVD, we are able to com-press a 500x800 pixel image into a 50x500 matrix (for U), 50 singular values,and a 80x50 matrix (for V ).7 ConclusionThe singular value decomposition is used to factor a rectangular matrix. Thekey to the SVD is to apply the spectral theorem to the symmetric matrices ATAand AAT . SVD are used in statistics, machine learning, and computer science.15ReferencesLay, David C., Steven R. Lay, and Judith McDonald. Linear Algebra and ItsApplications. Boston: Pearson, 2016.Stewart, G. W. “On the Early History of the Singular Value Decomposition.”SIAM Review 35, no. 4 (1993): 551–66. https://doi.org/10.1137/1035134.Singular Value Decomposition (SVD),https://www.youtube.com/watch?v=rYz83XPxiZoGibiansky, A. (n.d.). Andrew Gibiansky :: Math [Code]. Retrieved fromhttp://andrew.gibiansky.com/blog/mathematics/cool-linear-algebra-singular-value-decomposition/.16

Singular Value Decomposition

https://digitalcommons.coastal.edu/cgi/viewcontent.cgi?article=1353&amp;context=honors-theses

Singular Value Decomposition

Abstract

Similar works

Full text

Available Versions

CCU Digital Commons