Trefethen & Bau lecture notes in MATLAB

TB-Lecture-01.ipynb

TB-Lecture-02.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Lecture 2: Orthogonal vectors and matrices"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "With real vectors and matrices, the transpose operation is simple and familiar. It also happens to correspond to what we call the **adjoint** mathematically. In the complex case, one also has to conjugate the entries to keep the mathematical structure intact. We call this operator the **hermitian** of a matrix and use a star superscript for it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "A =\n",
      "  Columns 1 through 2\n",
      "   8.1472e-01 + 9.5751e-01i   1.2699e-01 + 1.5761e-01i\n",
      "   9.0579e-01 + 9.6489e-01i   9.1338e-01 + 9.7059e-01i\n",
      "  Columns 3 through 4\n",
      "   6.3236e-01 + 9.5717e-01i   2.7850e-01 + 8.0028e-01i\n",
      "   9.7540e-02 + 4.8538e-01i   5.4688e-01 + 1.4189e-01i\n"
     ]
    }
   ],
   "source": [
    "format compact\n",
    "A = rand(2,4) + 1i*rand(2,4)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Aadjoint =\n",
      "   8.1472e-01 - 9.5751e-01i   9.0579e-01 - 9.6489e-01i\n",
      "   1.2699e-01 - 1.5761e-01i   9.1338e-01 - 9.7059e-01i\n",
      "   6.3236e-01 - 9.5717e-01i   9.7540e-02 - 4.8538e-01i\n",
      "   2.7850e-01 - 8.0028e-01i   5.4688e-01 - 1.4189e-01i\n"
     ]
    }
   ],
   "source": [
    "Aadjoint = A'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To get plain transpose, use a `.^` operator."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Atrans =\n",
      "   8.1472e-01 + 9.5751e-01i   9.0579e-01 + 9.6489e-01i\n",
      "   1.2699e-01 + 1.5761e-01i   9.1338e-01 + 9.7059e-01i\n",
      "   6.3236e-01 + 9.5717e-01i   9.7540e-02 + 4.8538e-01i\n",
      "   2.7850e-01 + 8.0028e-01i   5.4688e-01 + 1.4189e-01i\n"
     ]
    }
   ],
   "source": [
    "Atrans = A.'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Inner products"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If **u** and **v** are column vectors of the same length, then their **inner product** is $\\mathbf{u}^*\\mathbf{v}$. The result is a scalar."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "u = [ 4; -1; 2+2i ],  v = [ -1; 1i; 1 ], \n",
    "innerprod = u'*v"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The inner product has geometric significance. It is used to define length through the 2-norm,"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "length_u_squared = u'*u"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "sum( abs(u).^2 )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "norm_u = norm(u)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It also defines the angle between two vectors as a generalization of the familiar dot product."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "cos_theta = (u'*v) / ( norm(u)*norm(v) )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The angle may be complex when the vectors are complex! "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "theta = acos(cos_theta)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The operations of inverse and hermitian commute."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "A = rand(4,4)+1i*rand(4,4);  (inv(A))'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "inv(A')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So we just write $\\mathbf{A}^{-*}$ for either case. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Orthogonality"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Orthogonality, which is the multidimensional extension of perpendicularity, means that $\\cos \\theta=0$, i.e., that the inner product between vectors is zero. A collection of vectors is orthogonal if they are all pairwise orthogonal. \n",
    "\n",
    "Don't worry about how we are creating the vectors here for now."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Q =\n",
      "\n",
      "  -4.9269e-01  -4.8065e-01   1.7797e-01\n",
      "  -5.4776e-01  -3.5835e-01  -5.7775e-01\n",
      "  -7.6793e-02   4.7542e-01  -6.3431e-01\n",
      "  -5.5235e-01   3.3908e-01   4.8084e-01\n",
      "  -3.8241e-01   5.4733e-01   3.1118e-02\n"
     ]
    }
   ],
   "source": [
    "[Q,~] = qr(rand(5,3),0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Since $\\mathbf{Q}^*\\mathbf{Q}$ is a matrix of all inner products between columns of $\\mathbf{Q}$, those columns are orthogonal if and only if that matrix is diagonal."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "QhQ =\n",
      "\n",
      "   1.0000e+00   1.6133e-16   5.3630e-18\n",
      "   1.6133e-16   1.0000e+00  -1.2726e-16\n",
      "   5.3630e-18  -1.2726e-16   1.0000e+00\n"
     ]
    }
   ],
   "source": [
    "QhQ = Q'*Q"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In fact we have a stronger condition here: the columns are **orthonormal**, meaning that they are orthogonal and each has 2-norm equal to 1. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Given any other vector of length 5, we can compute its inner product with each of the columns of $\\mathbf{Q}$. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "c =\n",
      "\n",
      "  -1.1757e+00\n",
      "   1.0098e+00\n",
      "  -3.8850e-01\n"
     ]
    }
   ],
   "source": [
    "u = rand(5,1);  c = Q'*u"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can then use these coefficients to find a vector in the column space of $\\mathbf{Q}$."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "v =\n",
      "\n",
      "   2.4774e-02\n",
      "   5.0662e-01\n",
      "   8.1680e-01\n",
      "   8.0501e-01\n",
      "   9.9021e-01\n"
     ]
    }
   ],
   "source": [
    "v = Q*c"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As explained in the text, $\\mathbf{r} = \\mathbf{u}-\\mathbf{v}$ is orthogonal to all of the columns of $\\mathbf{Q}$."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ans =\n",
      "\n",
      "  -3.9899e-17\n",
      "   2.9837e-16\n",
      "   1.3878e-16\n"
     ]
    }
   ],
   "source": [
    "r = u-v;  Q'*r"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Consequently, we have decomposed $\\mathbf{u}=\\mathbf{v}+\\mathbf{r}$ into the sum of two orthogonal parts, one lying in the range of $\\mathbf{Q}$. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ans =\n",
      "\n",
      "   2.9143e-16\n"
     ]
    }
   ],
   "source": [
    "v'*r"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Unitary matrices"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We just saw that a matrix whose columns are orthonormal is pretty special. It becomes even more special if the matrix is also square, in which case we call it **unitary**. (In the real case, such matrices are confusingly called _orthogonal_. Ugh.) Say $\\mathbf{Q}$ is unitary and $m\\times m$. Then $\\mathbf{Q}^*\\mathbf{Q}$ is an $m\\times m$ identity matrix---that is, $\\mathbf{Q}^*=\\mathbf{Q}^{-1}$! It can't get much easier in terms of finding the inverse of a matrix. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "[Q,~] = qr(rand(5,5)+1i*rand(5,5));\n",
    "abs( inv(Q) - Q' )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The rank of $\\mathbf{Q}$ is $m$, so continuing the discussion above, the original vector $\\mathbf{u}$ lies in its column space. Hence the remainder $\\mathbf{r}=\\boldsymbol{0}$. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "c = Q'*u; \n",
    "v = Q*c;\n",
    "r = u - v"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is another way to arrive at a fact we already knew: Multiplication by $\\mathbf{Q}^*=\\mathbf{Q}^{-1}$ changes the basis to the columns of $\\mathbf{Q}$."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Matlab",
   "language": "matlab",
   "name": "matlab"
  },
  "language_info": {
   "codemirror_mode": "octave",
   "file_extension": ".m",
   "help_links": [
    {
     "text": "MetaKernel Magics",
     "url": "https://github.com/calysto/metakernel/blob/master/metakernel/magics/README.md"
    }
   ],
   "mimetype": "text/x-octave",
   "name": "matlab",
   "version": "0.11.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}

TB-Lecture-03.ipynb

TB-Lecture-04.ipynb

TB-Lecture-05.ipynb

TB-Lecture-06.ipynb

TB-Lecture-07.ipynb

TB-Lecture-08.ipynb

TB-Lecture-09.ipynb

TB-Lecture-10.ipynb

TB-Lecture-11.ipynb

TB-Lecture-12.ipynb