This is the last post for keeping my learning notes on key definitions in Learn Linear Algebra a Different Way series. You can find the first and second posts here: Notes on Key Definitions and Notes on Key Definitions (1).

The Key Definitions (Chapter 7~9)

  • Adjoint: Suppose $T \in \mathcal{L}(V, W)$. The adjoint of $T$ is the function $T^* : W \to V$ such that

    \[\langle Tv, w \rangle = \langle v, T^*w \rangle\]

    for every $v \in V$ and every $w \in W$

    • Adjoint of a linear map is a linear map: If $T \in \mathcal{L}(V, W)$, then $T^* \in \mathcal{L}(W, V)$

  • Conjugate transpose: The conjugate transpose of an $m$-by-$n$ matrix $A$ is the $n$-by-$m$ matrix $A^*$ obtained by interchanging the rows and columns and then taking the complex conjugate of each entry. In other words, if $j \in {1, 2, \ldots, m}$ and $k \in {1, 2, \ldots, n}$, then

    \[(A^*)_{k,j} = \overline{A_{j,k}}\]
    • Matrix of the adjoint equals the conjugate transpose of matrix of $T$:
    \[M(T^*) = M(T)^*\]

  • Self-adjoint: An operator $T \in \mathcal{L}(V)$ is called self-adjoint if $T = T^*$

    📝 Note A good analogy to keep in mind is that the adjoint on $\mathcal{L}(V)$ plays a role to that of the complex conjugate on $\mathbb{C}$. A complex number $z$ is real if and only if $z = \overline{z}$, thus a self-adjoint operator is analogous to a real number.

    • every eigenvalue of a self-adjoint operator is real
    • $Tv$ is orthogonal to $v$ for all $v \in V$ $\iff$ $T = 0$, assuming $V$ is complex inner product space
    • $T$ is self-adjoint $\iff$ $\langle Tv, v \rangle$ is real for every $v \in V$
    • $T$ is self-adjoint and $\langle Tv, v \rangle = 0$ for every $v \in V$ $\iff$ $T = 0$

    📝 Note On a real inner product space $V$, a nonzero operator $T$ might satisfy $\langle Tv, v \rangle = 0$ for every $v \in V$. For example, if $V = \mathbb{R}^2$ and $T$ is the operator defined by $T(x,y) = (-y, x)$, then $\langle T(x,y), (x,y) \rangle = 0$ for every $(x,y) \in \mathbb{R}^2$, but $T$ is not the zero operator.

  • Normal Operator: An operator on an inner product space is called normal if it commutes with its adjoint. In other words, an operator $T \in \mathcal{L}(V)$ is normal if $TT^* = T^*T$.

    • Every self-adjoint operator is normal, but not every normal operator is self-adjoint.

  • $T$ is normal if and only if $Tv$ and $T^*v$ have the same norm:

    • $null(T) = null(T^*)$
    • $range(T) = range(T^*)$
    • $V = null(T) \oplus range(T)$
    • $T - \lambda I$ is normal for every $\lambda \in \mathbb{F}$
    • If $v \in V$ and $\lambda \in \mathbb{F}$, then $Tv = \lambda v$ if and only if $T^*v = \overline{\lambda}v$
  • The eigenvectors of a normal operator corresponding to distinct eigenvalues are orthogonal

  • Suppose $F = \mathbb{C}$ and $T \in \mathcal{L}(V)$. Then $T$ is normal if and only if there exists self-adjoint operators A and B such that $T = A + iB$

  • Real Spectral Theorem: suppose $F = R$ and $T \in \mathcal{L}(V)$. Then the following are equivalent:
    • $T$ is self-adjoint
    • $T$ has a diagonal matrix with respect to some orthonormal basis of $V$
    • $V$ has an orthonormal basis consisting of eigenvectors of $T$
  • Complex Spectral Theorem: Suppose $F = \mathbb{C}$ and $T \in \mathcal{L}(V)$. Then the following are equivalent:
    • $T$ is normal
    • $T$ has a diagonal matrix with respect to some orthonormal basis of $V$
    • $V$ has an orthonormal basis consisting of eigenvectors of $T$
  • Positive operator: An operator $T \in \mathcal{L}(V)$ is called positive if $T$ is self-adjoint and $\langle Tv, v \rangle \geq 0$ for every $v \in V$. The following are equivalent:
    • $T$ is a positive operator
    • $T$ is self-adjoint and every eigenvalue of $T$ is nonnegative
    • With respect to some orthonormal basis of $V$, $T$ has a diagonal matrix whose diagonal entries are all nonnegative real numbers
    • $T$ has a positive square root
    • $T = S^*S$ for some $S \in \mathcal{L}(V)$
  • Square Root: An operator $S \in \mathcal{L}(V)$ is called a square root of $T \in \mathcal{L}(V)$ if $S^2 = T$.
    • Every positive operator has a unique positive square root.
    • For $T$ a positive operator, the unique positive square root of $T$ is denoted by $\sqrt{T}$.

  • $T$ positive and $\langle Tv, v \rangle = 0$ for every $v \in V$ $\implies$ $T = 0$

  • Isometry: A linear map $S \in \mathcal{L}(V, W)$ is called an isometry if $|Sv| = |v|$ for every $v \in V$. In other words, a linear map is an isometry if it preserves the norm. The following are equivalent:
    • $S$ is an isometry
    • $\langle Sv, Sw \rangle = \langle v, w \rangle$ for every $v, w \in V$
    • $S^*S = I$
  • Unitary operator: An operator $S \in \mathcal{L}(V)$ is called unitary if $S$ is an invertible isometry. The following are equivalent:
    • $S$ is a unitary operator
    • \[S^*S = SS^* = I\]
    • $S$ is invertible and $S^{-1} = S^*$
    • $S^*$ is unitary operator

  • Eigenvalues of unitary operators have absolute value 1

  • Unitary Matrix: An $n$-by-$n$ matrix is called unitary if its columns form an orthonormal list in $\mathbb{F}^n$.

  • QR Factorization: Suppose $A$ is a square matrix with linearly independent columns. Then there exist unique matrices $Q$ and $R$ such that $Q$ is unitary, $R$ is upper-triangular with only positive numbers on the diagonal, and

    \[A = QR\]

  • Positive Definite: A matrix $B$ is called positive definite if $B$ is self-adjoint and

    \[\langle Bx, x \rangle > 0\]

    for every nonzero $x \in \mathbb{F}^n$.

  • Cholesky Factorization: Suppose $B$ is a positive definite matrix. Then there exists a unique upper-triangular matrix $R$ with only positive numbers on the diagonal such that

    \[B = R^*R\]

  • Singular Values: Suppose $T \in \mathcal{L}(V, W)$. The singular values of $T$ are the nonnegative square roots of the eigenvalues of $T^*T$, listed in decreasing order, each included as many times as the dimension of the corresponding eigenspace.

  • Singular Value Decomposition: Suppose $T \in \mathcal{L}(V, W)$ and the positive singular values of $T$ are $s_1, \ldots, s_m$. Then there exist orthonormal bases $e_1, \ldots, e_n$ of $V$ and $f_1, \ldots, f_m$ of $W$ such that

    \[Tv = s_1 \langle v, e_1 \rangle f_1 + \cdots + s_m \langle v, e_m \rangle f_m\]

  • Matrix version of SVD: Suppose $A$ is an $M$-by-$n$ matrix of rank $m \ge 1$. Then there exist an $M$-by-$m$ matrix $B$ with orthonormal columns, an $m$-by-$m$ diagonal matrix $D$ with positive diagonal entries, and an $n$-by-$m$ matrix $C$ with orthonormal columns such that

    \[A = BDC^*\]

  • Upper bound for \(\|Tv\|\): Suppose $T \in \mathcal{L}(V, W)$. Let $s_1$ be the largest singular value of $T$. Then

    \[\|Tv\| \le s_1 \|v\|\]

  • Norm of a linear map: Suppose $T \in \mathcal{L}(V, W)$. The norm of $T$, denoted by $|T|$, is defined by

    \[\|T\| = \max \{\|Tv\| : v \in V \text{ and } \|v\| \le 1\}\]
    • \(\|T^*\| = \|T\|\)

  • Best approximation by linear map whose range has dimension $\le k$: Suppose $T \in \mathcal{L}(V, W)$ and $s_1, \ldots, s_m$ are the positive singular values of $T$. Suppose $1 \le k < m$. Then

    \[\min \{\|T - S\| : S \in \mathcal{L}(V, W) \text{ and } \dim range(S) \le k\} = s_{k+1}\]

    Furthermore, if

    \[Tv = s_1 \langle v, e_1 \rangle f_1 + \cdots + s_m \langle v, e_m \rangle f_m\]

    is a singular value decomposition of $T$ and $T_k$ is defined by

    \[T_kv = s_1 \langle v, e_1 \rangle f_1 + \cdots + s_k \langle v, e_k \rangle f_k\]

    then $\dim \text{range}(T_k) = k$ and \(\|T - T_k\| = s_{k+1}\).

  • Polar Decomposition: Suppose $T \in \mathcal{L}(V)$. Then there exists a unitary operator $S \in \mathcal{L}(V)$ such that $T = S \sqrt{T^*T}$

  • Equality in the sequence of null spaces: Suppose $T \in \mathcal{L}(V)$ and $m$ is a nonnegative integer such that

    \[null(T^m) = null(T^{m+1})\]

    Then \(null(T^k) = null(T^m)\) for every $k \ge m$

    • Null spaces stop growing after a finite number of steps.
    • $V = null(T^m) \oplus range(T^m)$
  • Generalized eigenvectors: Suppose $T \in \mathcal{L}(V)$ and $\lambda$ is an eigenvalue of $T$. A nonzero vector $v \in V$ is called a generalized eigenvector of $T$ corresponding to $\lambda$ if $(T - \lambda I)^k v = 0$ for some positive integer $k$.
    • There is a basis of $V$ consisting of generalized eigenvectors of $T$, given $\mathbb{F} = \mathbb{C}$.
    • Generalized eigenvectors corresponding to distinct eigenvalues are linearly independent.
  • Nilpotent operator: An operator $T \in \mathcal{L}(V)$ is called nilpotent if $T^k = 0$ for some positive integer $k$.
    • If $T$ is nilpotent, then $0$ is the only eigenvalue of $T$.
    • The minimal polynomial of a nilpotent operator is $z^m$ for some positive integer $m$.

  • Generalized eigenspace: Suppose $T \in \mathcal{L}(V)$ and $\lambda$ is an eigenvalue of $T$. The generalized eigenspace of $T$ corresponding to $\lambda$ is defined by

    \[G(\lambda, T) = \{v \in V : (T - \lambda I)^k v = 0 \text{ for some positive integer } k\}\]

    Thus \(G(\lambda, T)\) is the set of all generalized eigenvectors corresponding to $\lambda$, together with the zero vector.

    • $G(\lambda, T) = null((T - \lambda I)^{\dim V})$
    • \(G(\lambda, T)\) is invariant under $T$
    • \((T - \lambda I)\|_{G(\lambda, T)}\) is nilpotent
    • $V = G(\lambda_1, T) \oplus \cdots \oplus G(\lambda_m, T)$
  • Multiplicity: Suppose $T \in \mathcal{L}(V)$. The multiplicity of an eigenvalue $\lambda$ of $T$ is defined to be the dimension of the generalized eigenspace $G(\lambda, T)$.
    • Algebraic multiplicity of $\lambda = \dim null(T - \lambda I)^{\dim V} = \dim G(\lambda, T)$
    • Geometric multiplicity of $\lambda = \dim null(T - \lambda I) = \dim E(\lambda, T)$

  • Characteristic polynomial: Suppose $T \in \mathcal{L}(V)$. Let $\lambda_1, \ldots, \lambda_m$ denote the distinct eigenvalues of $T$, with multiplicities $d_1, \ldots, d_m$. The polynomial

    \[(z - \lambda_1)^{d_1} \cdots (z - \lambda_m)^{d_m}\]

    is called the characteristic polynomial of $T$.

  • Cayley-Hamilton Theorem: Suppose $T \in \mathcal{L}(V)$ and $q$ is the characteristic polynomial of $T$. Then $q(T) = 0$.

  • Multiplicity of an eigenvalue equals number of times on the diagonal: Suppose $T \in \mathcal{L}(V)$ and $\lambda$ is an eigenvalue of $T$. Then the multiplicity of $\lambda$ equals the number of times $\lambda$ appears on the diagonal of any upper-triangular matrix representing $T$.

  • Square roots of operators
    • Suppose $T \in \mathcal{L}(V)$ is nilpotent. Then $I + T$ has a square root.
    • Suppose $V$ is a complex vector space and $T \in \mathcal{L}(V)$ is invertible. Then $T$ has a square root.

  • Jordan Form: Suppose $V$ is a complex vector space and $T \in \mathcal{L}(V)$. Then there exists a basis of $V$ with respect to which the matrix of $T$ has the form

    \[\begin{pmatrix} J_1 & 0 & \cdots & 0 \\ 0 & J_2 & \cdots & 0 \\ \vdots & \vdots & \ddots & \vdots \\ 0 & 0 & \cdots & J_m \end{pmatrix}\]

    where each $J_k$ is a Jordan block

    \[J_k = \begin{pmatrix} \lambda_k & 1 & 0 & \cdots & 0 \\ 0 & \lambda_k & 1 & \cdots & 0 \\ \vdots & \vdots & \vdots & \ddots & 1 \\ 0 & 0 & 0 & \cdots & \lambda_k \end{pmatrix}\]
  • Trace of a matrix: Suppose $A$ is an $n$-by-$n$ matrix. The trace of $A$, denoted by $\operatorname{tr} A$, is defined to be the sum of the diagonal entries of $A$.
    • $\operatorname{tr} AB = \operatorname{tr} BA$
    • trace of matrix of operator is independent of the choice of basis

  • Trace of an operator: Suppose $T \in \mathcal{L}(V)$. The trace of $T$, denoted by $\operatorname{tr} T$, is defined by

    \[\operatorname{tr} T = \operatorname{tr} M(T, (v_1, \ldots, v_n))\]

    where $(v_1, \ldots, v_n)$ is any basis of $V$.

    • On complex vector space, $\operatorname{tr} T$ equals the sum of the eigenvalues of $T$, counting multiplicities.
    • On complex vector space, $\operatorname{tr} T$ equals the negative of the coefficient of $z^{n-1}$ in the characteristic polynomial of $T$.
    • Suppose $V$ is an inner product space, $T \in \mathcal{L}(V)$, and $e_1, \ldots, e_n$ is an orthonormal basis of $V$. Then \(\operatorname{tr} T = \sum_{i=1}^n \langle T e_i, e_i \rangle.\)
    • $\operatorname{tr} T$ is linear functional and $\operatorname{tr} ST = \operatorname{tr} TS$ for every $S, T \in \mathcal{L}(V)$.
    • There do not exist operator $S, T \in \mathcal{L}(V)$ such that $ST - TS = I$.
  • Bilinear form: A bilinear form on $V$ is a function $\beta : V \times V \to \mathbb{F}$ such that for every $v \in V$, the functions $u \mapsto \beta(u, v)$ and $u \mapsto \beta(v, u)$ are linear.
    • A bilinear form is not a linear map from $V \times V$ to $\mathbb{F}$, unless $\beta = 0$.
    • The set of bilinear forms on $V$ is denoted by $V^{(2)}$.
  • Matrix of a bilinear form: Suppose $\beta$ is a bilinear form on $V$ and $(e_1, \ldots, e_n)$ is a basis of $V$. The matrix of $\beta$ with respect to $(e_1, \ldots, e_n)$ is the $n$-by-$n$ matrix $M(\beta, (e_1, \ldots, e_n))$ whose $(j,k)$ entry is $\beta(e_j, e_k)$.
    • $\dim V^{(2)} = (\dim V)^2 = n^2$

  • Change of basis formula: Suppose $\beta$ is a bilinear form on $V$, $(e_1, \ldots, e_n)$ and $(f_1, \ldots, f_n)$ are bases of $V$. Let

    \[A = M(\beta, (e_1, \ldots, e_n)) \quad \text{and} \quad B = M(\beta, (f_1, \ldots, f_n)).\]

    and \(C = M(I, (e_1, \ldots, e_n), (f_1, \ldots, f_n))\). Then

    \[A = C^T B C\]

    📝 Note The matrix of a bilinear form is not the same as the matrix of a linear map. But both matrices are defined to compute the transformation from domain to codomain algebraically.

    • The matrix of a linear map $T \in \mathcal{L}(V, W)$ is defined to compute the transformation of vectors in $V$ to vectors in $W$: $T(v) = M(T)(v)$.
    • The matrix of a bilinear form $B \in V^{(2)}$ is defined to compute the transformation of a pair of vectors in $V$ to a scalar: $B(u, v) = u^\top M(B) v$.
  • Symmetric bilinear form: A bilinear form $\beta$ on $V$ is called symmetric if $\beta(u, v) = \beta(v, u)$ for every $u, v \in V$. The set of symmetric bilinear forms on $V$ is denoted by $V^{(2)}_{\text{sym}}$.
    • Symmetric bilinear forms are diagonalizable: there exists a basis of $V$ with respect to which the matrix of $\beta$ is diagonal.
  • Alternating bilinear form: A bilinear form $\beta$ on $V$ is called alternating if $\beta(v, v) = 0$ for every $v \in V$. The set of alternating bilinear forms on $V$ is denoted by $V^{(2)}_{\text{alt}}$.
    • A bilinear form $\beta$ is alternating if and only if $\beta(u, v) = -\beta(v, u)$ for every $u, v \in V$.
    • The decomposition of $V^{(2)}$ is \(V^{(2)} = V^{(2)}_{\text{alt}} \oplus V^{(2)}_{\text{sym}}.\)
  • Quadratic form: A quadratic form on $V$ is a function $q : V \to \mathbb{F}$ such that there exists a bilinear form $\beta$ on $V$ with $q(v) = \beta(v, v)$ for every $v \in V$.
    • There exists a unique symmetric bilinear form $\beta$ on $V$ such that $q(v) = \beta(v, v)$ for every $v \in V$.

    • There exists a basis $e_1, \ldots, e_n$ of $V$ and $\lambda_1, \ldots, \lambda_n \in \mathbb{F}$ such that

      \[q(x_1 e_1 + \cdots + x_n e_n) = \lambda_1 x_1^2 + \cdots + \lambda_n x_n^2\]

      for every $x_1, \ldots, x_n \in \mathbb{F}$.

  • Multilinear forms: For m a positive integer, an m-linear form on $V$ is a function $\beta : V^m \to \mathbb{F}$ that is linear in each slot when the other slots are fixed. This means that for each $j \in {1, 2, \ldots, m}$ and for every $v_1, \ldots, v_{j-1}, v_{j+1}, \ldots, v_m \in V$, the function $u \mapsto \beta(v_1, \ldots, v_{j-1}, u, v_{j+1}, \ldots, v_m)$ is linear.
    • The set of m-linear forms on $V$ is denoted by $V^{(m)}$.
    • $\dim V^{(m)} = (\dim V)^m = n^m$.
  • Alternating m-linear form: An m-linear form $\beta$ on $V$ is called alternating if $\beta(v_1, \ldots, v_m) = 0$ whenever $v_j = v_k$ for some $j \neq k$. The set of alternating m-linear forms on $V$ is denoted by $V^{(m)}_{\text{alt}}$.
    • If $\beta$ is an alternating m-linear form on $V$, then $\beta(v_1, \ldots, v_m) = 0$ whenever the vectors $v_1, \ldots, v_m$ are linearly dependent.
    • If $\dim V < m$, then every alternating m-linear form on $V$ is zero.

    • Swapping the vectors in any two slots of an alternating m-linear form changes the value of the form by a factor of $-1$.

    • Permutation: A permutation of ${1, 2, \ldots, m}$ is a list of $(j_1, j_2, \ldots, j_m)$ containing each element of ${1, 2, \ldots, m}$ exactly once. The set of all permutations of ${1, 2, \ldots, m}$ is denoted by $\operatorname{perm} m$.

  • Sign of a permutation: The sign of a permutation $(j_1, j_2, \ldots, j_m)$ is defined by

    \[\operatorname{sgn}(j_1, j_2, \ldots, j_m) = (-1)^N\]

    where $N$ is the number of pairs $(k, l)$ with $1 \le k < l \le m$ such that $j_k > j_l$.

    • The sign of a permutation equals $1$ if the permutation is even and equals $-1$ if the permutation is odd.
    • Swapping two entries in a permutation multiplies the sign of the permutation by $-1$.

    • For $\beta \in V^{(m)}_{\text{alt}}$ and a permutation $(j_1, j_2, \ldots, j_m)$, we have

      \[\beta(v_{j_1}, v_{j_2}, \ldots, v_{j_m}) = \operatorname{sgn}(j_1, j_2, \ldots, j_m) \beta(v_1, v_2, \ldots, v_m).\]

  • Formula for any alternating m-linear form: Let $m = \dim V$. Suppose $e_1, \ldots, e_m$ is a basis of $V$ and $v_1, \ldots, v_m \in V$. For each $k \in {1, 2, \ldots, m}$, let $b_{k,1}, \ldots, b_{k,m} \in \mathbb{F}$ be such that

    \[v_k = b_{k,1} e_1 + \cdots + b_{k,m} e_m.\]

    Then for any $\beta \in V^{(m)}_{\text{alt}}$, we have

    \[\beta(v_1, \ldots, v_m) = \beta(e_1, \ldots, e_m) \sum_{(j_1, \ldots, j_m) \in \operatorname{perm} m} \operatorname{sgn}(j_1, \ldots, j_m) b_{1, j_1} \cdots b_{m, j_m}.\]
    • $\dim V^{(dim V)}_{\text{alt}} = 1$

  • Determinant of an operator: Suppose $T \in \mathcal{L}(V)$. The determinant of $T$, denoted by $\det T$, is defined to be the unique number in $\mathbb{F}$ such that

    \[\beta_T = (\det T) \beta\]

    where $\beta_T(v_1, \ldots, v_m) = \beta(Tv_1, \ldots, Tv_m)$ for each list $(v_1, \ldots, v_m)$ of vectors in $V$ and $\beta$ is any nonzero alternating m-linear form on $V$.

    • Determinant of a matrix: Suppose $A$ is an $n$-by-$n$ matrix. The determinant of $A$, denoted by $\det A$, is defined by $\det A = \det T_A$ where $T$ is the operator on $\mathbb{F}^n$ whose matrix with respect to the standard basis is $A$.
    • $\det(v_1, \ldots, v_n)$ is an alternating n-linear form on $\mathbb{F}^n$.

  • Formula for determinant of a matrix: Suppose $A$ is an $n$-by-$n$ matrix. Then

    \[\det A = \sum_{(j_1, \ldots, j_n) \in \operatorname{perm} n} \operatorname{sgn}(j_1, \ldots, j_n) A_{1, j_1} \cdots A_{n, j_n}.\]

  • Characteristic polynomial: Suppose $T \in \mathcal{L}(V)$. The characteristic polynomial of $T$ is defined by

    \[p_T(z) = \det(zI - T)\]

    for every $z \in \mathbb{F}$.

    • The characteristic polynomial of $T$ can be written as

      \[z^n - (\operatorname{tr} T) z^{n-1} + \cdots + (-1)^n \det T\]

      where $n = \dim V$ and $\operatorname{tr} T$ is the trace of $T$.

  • Hadamard’s inequality: Suppose $A$ is an $n$-by-$n$ matrix. Let $v_1, \ldots, v_n$ denote the columns of $A$. Then

    \[|\det A| \le \prod_{k=1}^n \|v_k\|.\]

  • Determinant of Vandermonde matrix: Suppose $x_1, \ldots, x_n$ are scalars. Then the determinant of the Vandermonde matrix $V$ with entries $V_{i,j} = x_i^{j-1}$ for $1 \le i, j \le n$ is given by

    \[\det \begin{pmatrix} 1 & x_1 & x_1^2 & \cdots & x_1^{n-1} \\ 1 & x_2 & x_2^2 & \cdots & x_2^{n-1} \\ \vdots & \vdots & \vdots & \ddots & \vdots \\ 1 & x_n & x_n^2 & \cdots & x_n^{n-1} \end{pmatrix} = \prod_{1 \le i < j \le n} (x_j - x_i).\]

  • Tensor product: The tensor product $V \otimes W$ is defined to be the vector space of all bilinear forms on $V’ \times W’$, where $V’$ and $W’$ are the dual spaces of $V$ and $W$, respectively. For $v \in V$ and $w \in W$, the tensor product $v \otimes w $ is the element of $V \otimes W$ defined by

    \[(v \otimes w)(\phi, \psi) = \phi(v) \psi(w)\]

    for all $(\phi, \psi) \in V’ \times W’$.

    • If $e_1, \ldots, e_m$ is a basis of $V$ and $f_1, \ldots, f_n$ is a basis of $W$, then ${e_i \otimes f_j \mid 1 \le i \le m, 1 \le j \le n}$ is a basis of $V \otimes W$.
    • ${v \otimes w: (v, w) \in V \times W} \neq V \otimes W$.

📝 Note To understand the tensor product, there are 4 ingredients to keep in mind:

  1. The pair $(v, w)$ from $V \times W$ is the blueprint to construct the tensor product.
  2. The tensor product $v \otimes w$ is a bilinear form (function) on $V’ \times W’$.
  3. The input of the bilinear form is any pair $(\phi, \psi)$ from $V’ \times W’$.
  4. The output of the bilinear form is a scalar in $\mathbb{F}$.

A pure tensor product $v \otimes w$ corresponds mathematically to the outer product of the 2 vectors - $v w^\top$ - which is a matrix with rank 1. This matrix is the “core” of the tensor product, that can accept any linear functionals written as linear combinations of the dual basis of $V’$ and $W’$, and produce a scalar output.

Not all elements in the tensor product space are pure tensor products. For example, if $v_1, v_2 \in V$ and $w_1, w_2 \in W$, then $v_1 \otimes w_1 + v_2 \otimes w_2$ is an element of $V \otimes W$ that is not a pure tensor product.

  • Bilinear map: A bilinear map from $V \times W$ to $U$ is a function $\phi : V \times W \to U$ such that for every $v \in V$, the function $w \mapsto \phi(v, w)$ is linear and for every $w \in W$, the function $v \mapsto \phi(v, w)$ is linear.

  • Universal property of tensor product: Suppose $V, W, U$ are vector spaces and $\phi : V \times W \to U$ is a bilinear map. Then there exists a unique linear map $\Phi : V \otimes W \to U$ such that

    \[\Phi(v \otimes w) = \phi(v, w)\]

    for every $(v, w) \in V \times W$.

    • Conversely, suppose $T: V \otimes W \to U$ is a linear map. There exists a unique bilinear map \(T^\# : V \times W \to U\) such that
    \[T^\#(v, w) = T(v \otimes w)\]

    for every $(v, w) \in V \times W$.

📝 Note The proof of this universal property of tensor product leverages the linear map lemma, for each basis vector $e_i \otimes f_j$ of $V \otimes W$, we can send them to the exact same vector in $U$ that the original bilinear map $\Phi$ would send the pair $(e_i, f_j)$ to.