Skip to content

Instantly share code, notes, and snippets.

@tobydriscoll

tobydriscoll/TB-Lecture-02.ipynb

Last active September 2, 2016 18:25
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save tobydriscoll/7f404b36fd47d2878f90dd76a1d7a9b9 to your computer and use it in GitHub Desktop.
Save tobydriscoll/7f404b36fd47d2878f90dd76a1d7a9b9 to your computer and use it in GitHub Desktop.
Download ZIP
T&B Lecture 2 (julia)
Display the source blob
Display the rendered blob
Raw
TB-Lecture-02.ipynb
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Lecture 2: Orthogonal vectors and matrices"
]
},
{
"cell_type": "markdown",
"metadata": {
"collapsed": true
},
"source": [
"With real vectors and matrices, the transpose operation is simple and familiar. It also happens to correspond to what we call the **adjoint** mathematically. In the complex case, one also has to conjugate the entries to keep the mathematical structure intact. We call this operator the **hermitian** of a matrix and use a star superscript for it."
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/plain": [
"2x4 Array{Complex{Float64},2}:\n",
" 0.725337+0.164674im 0.288889+0.794957im … 0.825082+0.589749im\n",
" 0.188618+0.51475im 0.878958+0.840362im 0.25438+0.839209im"
]
},
"execution_count": 1,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"A = rand(2,4) + 1im*rand(2,4)"
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/plain": [
"4x2 Array{Complex{Float64},2}:\n",
" 0.725337-0.164674im 0.188618-0.51475im \n",
" 0.288889-0.794957im 0.878958-0.840362im\n",
" 0.384954-0.0683305im 0.777636-0.230011im\n",
" 0.825082-0.589749im 0.25438-0.839209im"
]
},
"execution_count": 2,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"Aadjoint = A'"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"To get plain transpose, use a `.^` operator."
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/plain": [
"4x2 Array{Complex{Float64},2}:\n",
" 0.725337+0.164674im 0.188618+0.51475im \n",
" 0.288889+0.794957im 0.878958+0.840362im\n",
" 0.384954+0.0683305im 0.777636+0.230011im\n",
" 0.825082+0.589749im 0.25438+0.839209im"
]
},
"execution_count": 3,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"Atrans = A.'"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Inner products"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"If **u** and **v** are column vectors of the same length, then their **inner product** is $\\mathbf{u}^*\\mathbf{v}$. The result is a scalar. (In Julia, though, the scalar inner product is `dot(u,v)`, whereas the matrix multiplication result is a 1-by-1 matrix that is not equivalent for all purposes.)"
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {
"collapsed": false
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"dot(u,v) gives -2 - 3im\n",
"u'*v gives Complex{Int64}[-2 - 3im]\n"
]
}
],
"source": [
"u = [ 4; -1; 2+2im ]\n",
"v = [ -1; 1im; 1 ]\n",
"println(\"dot(u,v) gives \", dot(u,v))\n",
"println(\"u'*v gives \",u'*v)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The inner product has geometric significance. It is used to define length through the 2-norm,"
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/plain": [
"1-element Array{Complex{Int64},1}:\n",
" 25+0im"
]
},
"execution_count": 5,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"length_u_squared = u'*u"
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/plain": [
"25.0"
]
},
"execution_count": 6,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"sum( abs(u).^2 )"
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/plain": [
"5.0"
]
},
"execution_count": 7,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"norm_u = norm(u)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"It also defines the angle between two vectors as a generalization of the familiar dot product."
]
},
{
"cell_type": "code",
"execution_count": 8,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/plain": [
"1-element Array{Complex{Float64},1}:\n",
" -0.23094-0.34641im"
]
},
"execution_count": 8,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"cos_theta = (u'*v) / ( norm(u)*norm(v) )"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The angle may be complex when the vectors are complex! "
]
},
{
"cell_type": "code",
"execution_count": 9,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/plain": [
"1-element Array{Complex{Float64},1}:\n",
" 1.79019+0.34786im"
]
},
"execution_count": 9,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"theta = acos(cos_theta)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The operations of inverse and hermitian commute."
]
},
{
"cell_type": "code",
"execution_count": 10,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/plain": [
"4x4 Array{Complex{Float64},2}:\n",
" 1.82916+2.59705im -0.816702-1.96154im … -0.530187+0.312639im\n",
" -0.303269-0.356645im 1.02432+0.929259im -0.350343-1.09856im \n",
" -1.69821-0.467787im 0.649194+0.784488im 1.01278-1.67234im \n",
" 0.684181-1.39733im -0.779362+0.402974im -0.378864+2.93732im "
]
},
"execution_count": 10,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"A = rand(4,4)+1im*rand(4,4); (inv(A))'"
]
},
{
"cell_type": "code",
"execution_count": 11,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/plain": [
"4x4 Array{Complex{Float64},2}:\n",
" 1.82916+2.59705im -0.816702-1.96154im … -0.530187+0.312639im\n",
" -0.303269-0.356645im 1.02432+0.929259im -0.350343-1.09856im \n",
" -1.69821-0.467787im 0.649194+0.784488im 1.01278-1.67234im \n",
" 0.684181-1.39733im -0.779362+0.402974im -0.378864+2.93732im "
]
},
"execution_count": 11,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"inv(A')"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"So we just write $\\mathbf{A}^{-*}$ for either case. "
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Orthogonality"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Orthogonality, which is the multidimensional extension of perpendicularity, means that $\\cos \\theta=0$, i.e., that the inner product between vectors is zero. A collection of vectors is orthogonal if they are all pairwise orthogonal. \n",
"\n",
"Don't worry about how we are creating the vectors here for now."
]
},
{
"cell_type": "code",
"execution_count": 12,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/plain": [
"5x3 Array{Float64,2}:\n",
" -0.520352 0.113586 0.187403\n",
" -0.169743 -0.61493 0.140415\n",
" -0.0291219 -0.763508 -0.268636\n",
" -0.746073 0.135121 -0.53895 \n",
" -0.378085 -0.0880756 0.763238"
]
},
"execution_count": 12,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"Q = qr(rand(5,3))[1]"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Since $\\mathbf{Q}^*\\mathbf{Q}$ is a matrix of all inner products between columns of $\\mathbf{Q}$, those columns are orthogonal if and only if that matrix is diagonal."
]
},
{
"cell_type": "code",
"execution_count": 13,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/plain": [
"3x3 Array{Float64,2}:\n",
" 1.0 5.09384e-17 -1.42308e-17\n",
" 5.09384e-17 1.0 6.8765e-17 \n",
" -1.42308e-17 6.8765e-17 1.0 "
]
},
"execution_count": 13,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"QhQ = Q'*Q"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"In fact we have a stronger condition here: the columns are **orthonormal**, meaning that they are orthogonal and each has 2-norm equal to 1. "
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Given any other vector of length 5, we can compute its inner product with each of the columns of $\\mathbf{Q}$. "
]
},
{
"cell_type": "code",
"execution_count": 14,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/plain": [
"3x1 Array{Float64,2}:\n",
" -0.862349\n",
" -0.715461\n",
" 0.514809"
]
},
"execution_count": 14,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"u = rand(5,1); c = Q'*u"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We can then use these coefficients to find a vector in the column space of $\\mathbf{Q}$."
]
},
{
"cell_type": "code",
"execution_count": 15,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/plain": [
"5x1 Array{Float64,2}:\n",
" 0.463935\n",
" 0.658623\n",
" 0.433078\n",
" 0.269245\n",
" 0.781978"
]
},
"execution_count": 15,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"v = Q*c"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"As explained in the text, $\\mathbf{r} = \\mathbf{u}-\\mathbf{v}$ is orthogonal to all of the columns of $\\mathbf{Q}$."
]
},
{
"cell_type": "code",
"execution_count": 16,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/plain": [
"3x1 Array{Float64,2}:\n",
" -3.85016e-18\n",
" -8.15e-18 \n",
" -1.01795e-16"
]
},
"execution_count": 16,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"r = u-v; Q'*r"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Consequently, we have decomposed $\\mathbf{u}=\\mathbf{v}+\\mathbf{r}$ into the sum of two orthogonal parts, one lying in the range of $\\mathbf{Q}$. "
]
},
{
"cell_type": "code",
"execution_count": 17,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/plain": [
"1x1 Array{Float64,2}:\n",
" -1.55597e-17"
]
},
"execution_count": 17,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"v'*r"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Unitary matrices"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We just saw that a matrix whose columns are orthonormal is pretty special. It becomes even more special if the matrix is also square, in which case we call it **unitary**. (In the real case, such matrices are confusingly called _orthogonal_. Ugh.) Say $\\mathbf{Q}$ is unitary and $m\\times m$. Then $\\mathbf{Q}^*\\mathbf{Q}$ is an $m\\times m$ identity matrix---that is, $\\mathbf{Q}^*=\\mathbf{Q}^{-1}$! It can't get much easier in terms of finding the inverse of a matrix. "
]
},
{
"cell_type": "code",
"execution_count": 18,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/plain": [
"5x5 Array{Float64,2}:\n",
" 2.28878e-16 1.46376e-16 1.11886e-16 2.48253e-16 2.83052e-16\n",
" 1.66533e-16 1.24127e-16 2.77556e-17 1.31656e-16 1.57009e-16\n",
" 5.55112e-17 3.51083e-16 8.32667e-17 1.14439e-16 1.14439e-16\n",
" 5.55112e-17 2.08167e-16 2.00148e-16 1.24127e-16 1.66533e-16\n",
" 2.16778e-16 1.8619e-16 1.11022e-16 3.92523e-16 1.57009e-16"
]
},
"execution_count": 18,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"Q = qr(rand(5,5)+1im*rand(5,5))[1]\n",
"abs( inv(Q) - Q' )"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The rank of $\\mathbf{Q}$ is $m$, so continuing the discussion above, the original vector $\\mathbf{u}$ lies in its column space. Hence the remainder $\\mathbf{r}=\\boldsymbol{0}$. "
]
},
{
"cell_type": "code",
"execution_count": 19,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/plain": [
"5x1 Array{Complex{Float64},2}:\n",
" 3.33067e-16-1.66533e-16im \n",
" 0.0-0.0im\n",
" 2.77556e-17-1.249e-16im \n",
" 5.55112e-16-1.11022e-16im \n",
" 1.11022e-16+2.22045e-16im "
]
},
"execution_count": 19,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"c = Q'*u; \n",
"v = Q*c;\n",
"r = u - v"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"This is another way to arrive at a fact we already knew: Multiplication by $\\mathbf{Q}^*=\\mathbf{Q}^{-1}$ changes the basis to the columns of $\\mathbf{Q}$."
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Julia 0.4.3",
"language": "julia",
"name": "julia-0.4"
},
"language_info": {
"file_extension": ".jl",
"mimetype": "application/julia",
"name": "julia",
"version": "0.4.3"
}
},
"nbformat": 4,
"nbformat_minor": 0
}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment