I have a pretty straightforward question and one which is related.  Main question:  Why are the singular values of a matrix (from SVD) called so ?   Related question:  Are they related to singularity of a matrix?
44165
The SVD stands for Singular Value Decomposition. After decomposing a data matrix $\mathbf X$ using SVD, it results in three matrices, two matrices with the singular vectors $\mathbf U$ and $\mathbf V$, and one singular value matrix whose diagonal elements are the singular values. But I want to know why those values are named as singular values. Is there any connection between a singular matrix and these singular values?
50986
This question regards the problem of . Vectors and matrices will be denoted in bold.  Premises. Let $N,K$ be given integers, with $K \gg N &gt; 1$. The transpose of matrix $\mathbf{A}$ will be denoted with $\mathbf{A}^T$. Suppose the following statistical model holds $$ (*) \quad \mathbf{y} = \mathbf{Hx + n}, \quad \mathbf{n} \sim \mathcal{N}_{K}(\mathbf{0}, \mathbf{C})  $$ where $\mathbf{y} \in \mathbb{R}^{K \times 1}$ are the observables, $\mathbf{H} \in \mathbb{R}^{K \times N}$ is a known full-rank matrix, $\mathbf{x} \in \mathbb{R}^{N \times 1}$ is a deterministic vector of unknown parameters (which we want to estimate) and finally $\mathbf{n} \in \mathbb{R}^{K \times 1}$ is a disturbance vector (noise) with a known (positive definite) covariance matrix $\mathbf{C} \in \mathbb{R}^{K \times K}$. The  (ML) estimate of $\mathbf{x}$, denoted with $\hat{\mathbf{x}}_{ML}$, is given by $$ (1) \quad \hat{\mathbf{x}}_{ML} = (\mathbf{H}^T \mathbf{C^{-1}} \mathbf{H})^{-1} \mathbf{H}^T \mathbf{C}^{-1} \mathbf{y} $$ and this is also the standard formula of Generalized Linear Least Squares (GLLS). Consider the standard formula of Ordinary Least Squares (OLS) for a linear model, i.e. $$ (2) \quad \hat{\mathbf{x}}_{OLS} = (\mathbf{H}^T \mathbf{H})^{-1} \mathbf{H}^T \mathbf{y} $$ As a final note on notation, $\mathbf{I}_K$ is the $K \times K$ identity matrix and $\mathbf{O}$ is a matrix of all zeros (with appropriate dimensions). Now, for the problem at hand, assume that $\mathbf{C}^{-1} = \mathbf{I}_K + \mathbf{X}$, where $\mathbf{X} \in \mathbb{R}^{K \times K}$ is a symmetric, invertible matrix (and, for the formalism to make sense, $\mathbf{X}$ is such that $\mathbf{I}_K + \mathbf{X}$ is invertible and the inverse is positive definite). In this case, it can be proven (using matrix inversion lemma) that $$ (3) \quad (\mathbf{H}^T \mathbf{C^{-1}} \mathbf{H})^{-1} \mathbf{H}^T \mathbf{C}^{-1} = (\mathbf{H}^T \mathbf{H})^{-1} \mathbf{H}^T + \mathbf{Q} $$ where the expression for the matrix $\mathbf{Q} \in \mathbb{R}^{N \times K}$ can be found (I will omit it here). Now, finally,  Proposition 1. If $\mathbf{H}^T\mathbf{X} = \mathbf{O}_{N,K}$, then equation $(1)$ degenerates in equation $(2)$, i.e., there exists no difference between GLLS and OLS.  The proof is straigthforward and is valid even if $\mathbf{X}$ is singular. Now, my question is  Question: Can an equation similar to eq. $(3)$ (which "separates" an OLS-term from a second term) be written when $\mathbf{X}$ is a singular matrix?  I am not interested in a closed-form of $\mathbf{Q}$ when $\mathbf{X}$ is singular. But I do am interested in understanding the concept beyond that expression: what is the actual role of $\mathbf{Q}$? In which space does it operate?  I found this problem during a numerical implementation where both OLS and GLLS performed roughly the same (the actual model is $(*)$), and I cannot understand why OLS is not strictly sub-optimal. I found this slightly counter-intuitive, since you know a lot more in GLLS (you know $\mathbf{C}$ and make full use of it, why OLS does not), but this is somehow "useless" if some conditions are met. What are these conditions?  As a final note, I am rather new to the world of Least Squares, since I generally work within a ML-framework (or MMSE in other cases) and never studied the deep aspects of GLLS vs OLS, since, in my case, they are just intermediate steps during the derivation of MLE for a given problem.  If the question is, in your opinion, a bit too broad, or if there is something I am missing, could you please point me in the right direction by giving me references? Preferably well-known books written in standard notation.
121640
I am trying to understand reduction of an unsymmetric real square matrix to Hessenberg form from Numerical Recipes Vol. 3.  In it, the author states that one does not use Gaussian elimination for reducing to Hessenberg form because the Gaussian elimination does not preserve similarity and hence ends up changing the eigenvalues of the matrix, which is undesirable.  Why does Gaussian elimination not preserve similarity?   Also, what conditions decide whether a particular matrix transformation preserves similarity?  Heres a  to where I found it. In case you cant view that, it's on page 594 of Numerical Recipes in C++ (Volume 3).
41694
Can anyone help me with a practical application of linear transformations? I am very curious about how can be an application in real life that can be solved with a linear transformation.
14146
I know that the symmetric matrix has orthogonal eigenvectors corresponding eigenvalues. Also, the eigenvectors from the same eigenvalue are linearly independent.  I need an example of symmetric matrix such that the eigenvectors from the same eigenvalue are not orthogonal.  Also, I know that the eigenvalues of the symmetric matrix are real. But , this statement true iff the entries are real numbers.Right? Since the matrix 2x2 $$A=\begin{bmatrix} 1 &amp;i \\  i &amp;1  \end{bmatrix}$$  has non real eigenvalues
40215
I have been studying linear algebra for a while now, and I still can't understand the basic concept of linear transformation and the easy ''translation'' of them the matrices.  I understand that every proof we make can be done with both concepts, and that they are connected pretty close, and there is a basis which ''moves'' us from one concept to another.  If anyone can write a brief explanation, or some reading material, It would be a big step for me for further understanding of the subject.  Thanks,  Alan
47697
I've started studying Eigenvector and Eigenvalue.   It says in my book that 0 is excluded from being an eigenvector because it breaks the uniqueness of eigenvalue associated with each eigenvector.   But, there is a proof in my book showing that Eigenspace is a subspace. In order for it to be subspace, does that mean that there must be a zero element? But, Eigenvector can not be zero... Am I misunderstanding something?
21815
I was recently thinking what information do we get from a matrix. So if we say the columns (or rows) of a matrix define the basis of a system, say vectors of 3 dimensional space. Then the determinant will tell about the volume of the space enclosed by those vectors. Does trace give any information about the space?
62122
I was wondering. Clearly, we cannot multiply a (1x1)-matrix with a (4x3)-matrix; However, we can multiply a scalar with a matrix. This suggests a difference.  On the other hand, I was, for example, in an econometrics lecture today, where we had for a (Tx1)-vector $\underline{û}=\left( \begin{array}{c} û_1\\ \vdots\\ û_T\end{array}\right)$:     $S_{ûû}:= \sum_{i=1}^T û_i^2$ shall be minimized. We see see that   $S_{ûû}=\underline{û}^T\underline{û}$.   Well, formally, shouldn't it be $(S_{ûû})=\underline{û}^T\underline{û}$ or $S_{ûû}=\det(\underline{û}^T\underline{û})$, to ensure that we stay in the space of matrices and not suddenly go to the space of scalars? So here, the professor (physicist) not only treats $\underline{û}^T\underline{û}$ like a scalar, but also calls it a scalar. Is this formally legit or a wrong simplification (though it does not seem to have any impact, and surely makes life easier)?
17317
I am trying to solve following problem.      Let $A, B \in \mathbb{R}^{n\times n}$ be an orthogonal matrices and $\det(A) = -\det(B)$. How can it be proven that $A+B$ is singular?   I could start with implication:  $\det(A)=-\det(B) \Rightarrow B$ is created from $A$ by swapping two lines or columns.   But I am not sure if this implication is correct.
113153
I would like to know the mathematical concepts behind singular matrices. Matrices that do not have inverses in R throw one of two errors. I have provided some examples of both errors below:    Error in solve.default(W) :  system is computationally singular: reciprocal condition number = 1.58603e-17   Error in solve.default(Z) :  Lapack routine dgesv: system is exactly singular: U[2,2] = 0    Can anyone explain the difference between the two? Why is the first "nearly singular" whereas the second is "exactly singular"?
14822
I am having some problem in computing SVD and PCA in Matlab. I do not know if I am doing theoretical mistakes or programming mistakes.  Starting with a data matrix $X$, PCA computes the eigenvalues $\lambda_i$ of the covariance matrix $X^TX/(n-1)$.  On the other side SVD of $X$ is given by  $X=U\Sigma V^\top$, and so $$X^T X=V \Sigma^T U^T U\Sigma V^T=V \Sigma^2 V^T,$$ where $\Sigma$ is diagonal matrix of singular values with elements $\sigma_i$.  So we have that $$\lambda_i=\sigma_i^2 /(n-1).$$  Now I try a Matlab example:      load hald;     [u s v] = svd(ingredients);     sigma = cov(ingredients);     [a,b] = eig(sigma);     disp('sigma')     disp(diag(s)')     disp('lambda')     disp(diag(b)')   and here is the output:   sigma   211.3369   77.2356   28.4597   10.2667  lambda     0.2372   12.4054   67.4964  517.7969   The obtained values do not respect the original equation. Where is the mistake?
26372
I was reading over some notes of mine from the Linear Algebra class I took a while ago in order to prepare myself for Computer Graphics course I'll be taking this Fall and I ran across a sentence that I can't quite grasp. It states that     Since we have a non-zero vector $x$ in the nullspace of $A - \lambda I$, then $\det(A - \lambda I) = 0$.   From reading Wikipedia page on determinants I got that     ... the system has a unique solution exactly when the determinant is nonzero; when the determinant is zero there are either no solutions or many solutions.   So if we have vector $x$ in the nullspace of $A-\lambda I$, then we have either 0 solutions or many solutions. But I still don't understand why $\det(A - \lambda I)$ has to be equal zero?
34028
I am reading the book Elements of Statistical Learning and trying to understand singular value decomposition (SVD). In particular, what is an orthogonal matrix as it relates to SVD?  According to Wikipedia, an orthogonal matrix is a square matrix, the transpose of an which is equal to its inverse.   As this CrossValidated question points out however, in the SVD example discussed in Elements of Statistical Learning, this does not hold.   What am I missing?
84111
Why must the determinant of a matrix with integer entries be an integer?   Note: I know what a determinant of a matrix is, not sure how to explain this question.  Is that because if the matrix is made with integers the determinant has to be an integer as well?
61110
I don't claim at all to be an expert on this topic. In many (advanced) linear algebra textbooks for undergraduates, I usually find something about the "Jordan Canonical Form" of a matrix.   What is the purpose of such a form? I have taken a usual first-course in linear algebra (did another semester with Axler, but I don't claim to be an expert) and have taken abstract algebra (most familiar with group and ring theory) and have briefly skimmed through linear algebra books covering this material, but I don't quite understand the "big picture" idea, i.e., why is this useful in application? One person once told me it is the "most straightforward and useful algorithm for solving systems of linear equations, once you get beyond 3 variables or so," but maybe I'm missing something, since I usually don't see anything like what this person described to me in the linear algebra books I have. Most textbooks I've seen tend to have a more theoretical focus on this topic.  Also, any suggested texts which have good coverage on this topic would be very helpful.
102606
Apart from simplifying matrix powers, why do want to diagonalize a matrix? Do they have any appealing application which can be used to motivate to study diagonal matrices.  Thanks for any answers.