Let's start by pointing out that the method usually referred to as "SVD" that is used in the context of recommendations is not strictly speaking the mathematical Singular Value Decomposition of a matrix but rather an approximate way to compute the low-rank approximation of the matrix by minimizing the squared error loss. A more accurate, albeit more generic, way to call this would be Matrix Factorization.
The basic idea is that we want to decompose our original and very sparse matrix into two low-rank matrices that represent user factors and item factors. This is done by using an iterative approach to minimize the loss function. The most common way to do this is by using Stochastic Gradient Descent, but others such as ALS are also possible. The actual loss function to minimize includes a general bias term and two bias for both the user and the item.